Archiving Youtube Channels with YT-DLP and Plex

Sep 24, 2021 · 8 minute read
#Technology #Plex #Youtube #Guerrilla Archiving

Old Dusty PC after 50 years or so (simulated image)

I’m going to try my hardest not to ramble.

Is it possible to archive youtube channels and use Plex to organize and watch them?

Yes, here is how.

YT-DLP

Located here https://github.com/yt-dlp/yt-dlp

My Plex server is already installed and running on Ubuntu, so thats the POV that this guide is written from. Yt-dlp does have a windows binary though, but there may be some slight differences. If you’re running ubuntu, you can install it via snap https://snapcraft.io/install/yt-dlp/ubuntu

After installing and trying out some commands, you will quickly see that plex is very particular with how videos and folders are named. Initially without any sort of numbering in the filename, plex wouldn’t even find content at all. Using playlist_autonumber helped, but it wasn’t until adding S01E as a prefix, that plex found all the content much quicker. However, this naturally leads to the question of, how do multiple seasons work? Some channels may properly separate their content into different playlists, where others might put videos in multiple playlists which could lead to downloading a video multiple times with the right or wrong flags.

Putting playlists within channel folders didn’t work in my testing. It became simpler to just treat channels and playlists as 1:1 items and putting everything in their own folder at the top level. Technically they are both just lists of videos, but if you do want to have a plex page of a channel with different playlists, you will then need to utilize collections. The type of script would have to be determined by the specific content needed.

In short, most channels fall into a number of different categories based on what type of script is needed.

A.) To download an entire channel

sudo yt-dlp -f "(bv*[vcodec~='^((he|a)vc|h26[45])']+ba) / (bv*+ba/b)" https://www.youtube.com/c/CHANNELURL --cookies /home/ubuntu/Downloads/cookies.txt --compat-options no-live-chat --playlist-reverse --embed-subs --embed-thumbnail --embed-metadata -o '/home/ubuntu/Downloads/Youtube/%(uploader)s [%(channel_id)s]/S01E%(playlist_autonumber)s - %(title)s - %(upload_date)s - [%(id)s].%(ext)s'

This script is good for old, dead channels that havent seen updates. This will download the highest video in h264 or h265 format or just the highest quality if it’s not available. It will start from oldest to newest and will create a single folder and will itemize each video as an episode in a single season (This is to play nice with plex). It places it into a folder with the name and channel ID and treats it as a single season. It references a cookie file, this will be useful if you ever bump into a video that requires signing in.

B.) To download an entire channel as a single feed and then download only new episodes on reruns

sudo yt-dlp -f "(bv*[vcodec~='^((he|a)vc|h26[45])']+ba) / (bv*+ba/b)" https://www.youtube.com/c/CHANNELURL --force-write-archive --download-archive /home/ubuntu/Downloads/Youtube/channelname-archive.txt --cookies /home/ubuntu/Downloads/Youtube/cookies.txt --compat-options no-live-chat --playlist-reverse --embed-subs --embed-thumbnail --embed-metadata --no-overwrites -o '/home/ubuntu/Downloads/Youtube/%(uploader)s [%(channel_id)s]/S01E0%(playlist_autonumber)s - %(title)s - %(upload_date)s - [%(id)s].%(ext)s'

This is the same as above, but it uses --force-write-archive --download-archive which will create a txt file of all the youtube IDs that have been downloaded. On subsequent runs, it downloads all the video IDs, and then cross references them with the archive and downloads any missing.

C.) To download an entire playlist

sudo yt-dlp -f "(bv*[vcodec~='^((he|a)vc|h26[45])']+ba) / (bv*+ba/b)" https://www.youtube.com/playlist?list=PLAYLISTID --cookies /home/ubuntu/Downloads/cookies.txt --compat-options no-live-chat --embed-subs --embed-thumbnail --embed-metadata --no-overwrites -o '/home/ubuntu/Downloads/Youtube/%(playlist)s [%(playlist_id)s]/S01E%(playlist_autonumber)s - %(title)s - %(upload_date)s - [%(id)s].%(ext)s'

This is good for educational lecture series, ended series, playlists that haven’t been updated in a long time. It keeps the all the settings as above, but places it in a folder along side channels. Grouping can then be made with Collections inside Plex.

D.) To download an entire playlist and then download only new episodes on reruns

sudo yt-dlp -f "(bv*[vcodec~='^((he|a)vc|h26[45])']+ba) / (bv*+ba/b)" https://www.youtube.com/playlist?list=PLghL9V9QTN0hnyUH3k_pHNHaYPeupqXKG --force-write-archive --download-archive /home/ubuntu/Downloads/Youtube/archives/wisecrackchristophernolan-archive.txt --cookies /home/ubuntu/Downloads/Youtube/cookies.txt --compat-options no-live-chat --embed-subs --embed-thumbnail --embed-metadata --no-overwrites -o '/home/ubuntu/Downloads/Youtube/%(playlist)s [%(playlist_id)s]/S01E%(playlist_autonumber)s - %(title)s - %(upload_date)s - [%(id)s].%(ext)s'

This is the exact same type of script as B, but for playlists.

E.) To download ALL of the playlists of a channel and then download only new episodes on reruns.

sudo yt-dlp -f "(bv*[vcodec~='^((he|a)vc|h26[45])']+ba) / (bv*+ba/b)" https://www.youtube.com/channel/UC2I6Et1JkidnnbWgJFiMeHA/playlists --force-write-archive --download-archive /home/ubuntu/Downloads/Youtube/archives/channelname-archive.txt --cookies /home/ubuntu/Downloads/Youtube/cookies.txt --compat-options no-live-chat --embed-subs --embed-thumbnail --embed-metadata --no-overwrites -o '/home/ubuntu/Downloads/Youtube/%(playlist)s [%(playlist_id)s]/S01E%(playlist_autonumber)s - %(title)s - %(upload_date)s - [%(id)s].%(ext)s'

For channels that have clean organization of playlists, this one will drop each playlist into a folder that you can easily group up in plex. It just might be the best of the bunch, but beware of doing it on channels with shitty organization on their playlists, it might be a headache to workthrough or some missing videos of someone uploads a video and doesnt put it on any one. Also, if running yt-dlp with the --force-write-archive --download-archiveflag, a video downloaded in one playlist will not be downloaded again if it also exists in another playlist. Running this script without it will cause all files to be downloaded in every playlist regardless if it has already been downloaded before.

But what if the channel I want has thousands upon thousands of videos and I only want recent ones?

F.) To download video IDs of channel,playlist, or all playlists first

Some channels are so big, even with 1G internet, it would take weeks to download them all running 24/7. So for channels with regular updates, some may choose to just archive going foward with the following script to pull the video IDs of all the videos in either a channel or playlist and put them into a text file.

sudo yt-dlp --flat-playlist --skip-download --force-write-archive --download-archive /home/ubuntu/Downloads/Youtube/channelname-archive.txt <URL>

Next, go into the txt and remove any files that you do want to download.

A helpful method would be to find a video you want to start from, search for the ID and then delete all the IDs between that one and the most recent one. Save the file and then run script B or D with the same channel and archive file and it will simply download those missing videos and re-add their video ids to the txt file.

What if one of my favorite youtubers from 2010 lost his mind and makes wierd crappy content now?

E.) To download an entire channel, except for content after a specific date.

sudo yt-dlp -f "(bv*[vcodec~='^((he|a)vc|h26[45])']+ba) / (bv*+ba/b)" https://www.youtube.com/c/CHANNELURL --datebefore 20120101 --cookies /home/ubuntu/Downloads/cookies.txt --compat-options no-live-chat --playlist-reverse --embed-subs --embed-thumbnail --embed-metadata -o '/home/ubuntu/Downloads/Youtube/%(uploader)s [%(channel_id)s]/S01E%(playlist_autonumber)s - %(title)s - %(upload_date)s - [%(id)s].%(ext)s'

This will only pull stuff from before January 1st, 2012.

G.) To download an entire channel, but only of videos containing a specific string in the title.

What about “Podcast”?

sudo yt-dlp -f "(bv*[vcodec~='^((he|a)vc|h26[45])']+ba) / (bv*+ba/b)" https://www.youtube.com/user/aragusea --match-filter 'webpage_url!*=/shorts/' --match-filter 'duration<=10800' --match-filter 'title!*=#shorts' --match-filter 'title!*=PODCAST' --force-write-archive --download-archive /home/ubuntu/Downloads/Youtube/archives/aragusea-archive.txt --cookies /home/ubuntu/Downloads/Youtube/cookies.txt --compat-options no-live-chat --playlist-reverse --embed-subs --embed-thumbnail --embed-metadata --no-overwrites -o '/home/ubuntu/Downloads/Youtube/%(uploader)s [%(channel_id)s]/S01E0%(playlist_autonumber)s - %(title)s - %(upload_date)s - [%(id)s].%(ext)s';

Here’s a sample script that download’s Adam Ragusea’s podcast with a couple other flags to help educate. It only downloads videos that contain PODCAST in the title, does not contain #shorts in the title, does not contain /shorts/ in the url, as well as be shorter than 10,800 seconds (Helps skip livestreams)

Those 7 types of scrips pretty much handle anything. When in doubt, just pull the whole channel and delete what you dont want.

YouTube-Agent.bundle

From https://github.com/ZeroQI/YouTube-Agent.bundle

Download the Zip file: https://github.com/ZeroQI/YouTube-Agent.bundle/archive/refs/heads/master.zip
Unpack the downloaded Zip and rename the contents as “Youtube-Agent.bundle” (remove -master)
Place it inside the Plex Plug-ins folder.
Restart Plex Media Server to make sure that the new plugin is loaded. Can be seen in Settings → Agents.
Create your own YouTube API token https://github.com/ZeroQI/YouTube-Agent.bundle#youtube-api-key
Add your API token to the Agent. Settings → Agents → Shows → YoutubeSeries → Gear

The scripts above are already written to play nice with the agents so there isn’t much further configuration you need to do here. Time to make some libraries.

Create Plex Library

Create a TV Show Library like normal, change the scanner to Plex TV Series and Agent to YouTubeSeries.

Change collections to Hide items which are in collections and set seasons to Hide.

This will help clean things up if/when you add multiple playlists from a single channel to a libary. For example, here are a couple Bon Appétit playlists after being placed into a collection on Plex. I had to copy the channel description and some poster artwork. How it compares with the playlist page

Navigating into the playlists, you can see the plex agent pulls the metadata pretty well.

Automate it

sudo nano /home/ubuntu/hourly.sh

Create your script anywhere, you can essentially just copy and paste versions of the archive scripts above to run hourly. Here I have 3 youtube channels and a playlist. Tom Scott, Green Beetle, Adam Ragusea and his podcast as an example. Add them together as a collection and they can gracefully update as grouped items in plex.

#!/bin/bash sudo yt-dlp -f "(bv*[vcodec~='^((he|a)vc|h26[45])']+ba) / (bv*+ba/b)" https://www.youtube.com/user/aragusea --match-filter 'webpage_url!*=/shorts/' --match-filter 'duration<=10800' --match-filter 'title!*=#shorts' --match-filter 'title!*=PODCAST' --force-write-archive --download-archive /home/ubuntu/Downloads/Youtube/archives/aragusea-archive.txt --cookies /home/ubuntu/Downloads/Youtube/cookies.txt --compat-options no-live-chat --playlist-reverse --embed-subs --embed-thumbnail --embed-metadata --no-overwrites -o '/home/ubuntu/Downloads/Youtube/%(uploader)s [%(channel_id)s]/S01E0%(playlist_autonumber)s - %(title)s - %(upload_date)s - [%(id)s].%(ext)s'; sudo yt-dlp -f "(bv*[vcodec~='^((he|a)vc|h26[45])']+ba) / (bv*+ba/b)" https://www.youtube.com/playlist?list=PLWDQtIyZRZu2w8oFtWGz9UZ8Af8FcMt5j --force-write-archive --download-archive /home/ubuntu/Downloads/Youtube/archives/araguseapodcast-archive.txt --cookies /home/ubuntu/Downloads/Youtube/cookies.txt --compat-options no-live-chat --embed-subs --embed-thumbnail --embed-metadata --no-overwrites -o '/home/ubuntu/Downloads/Youtube/%(playlist)s [%(playlist_id)s]/%(playlist_autonumber)s - %(title)s - %(upload_date)s - [%(id)s].%(ext)s' --playlist-start 2; sudo yt-dlp -f "(bv*[vcodec~='^((he|a)vc|h26[45])']+ba) / (bv*+ba/b)" https://www.youtube.com/c/GreenBeetle --match-filter 'webpage_url!*=/shorts/' --match-filter 'duration>80' --match-filter 'duration<=10800' --match-filter 'title!*=#shorts' --match-filter 'title!*=PODCAST' --force-write-archive --download-archive /home/ubuntu/Downloads/Youtube/archives/GreenBeetle-archive.txt --cookies /home/ubuntu/Downloads/Youtube/cookies.txt --compat-options no-live-chat --playlist-reverse --embed-subs --embed-thumbnail --embed-metadata --no-overwrites -o '/home/ubuntu/Downloads/Youtube/%(uploader)s [%(channel_id)s]/S01E0%(playlist_autonumber)s - %(title)s - %(upload_date)s - [%(id)s].%(ext)s'; sudo yt-dlp -f "(bv*[vcodec~='^((he|a)vc|h26[45])']+ba) / (bv*+ba/b)" https://www.youtube.com/c/TomScottGo --match-filter 'webpage_url!*=/shorts/' --match-filter 'duration>80' --match-filter 'duration<=10800' --match-filter 'title!*=#shorts' --force-write-archive --download-archive /home/ubuntu/Downloads/Youtube/archives/TomScottGo-archive.txt --cookies /home/ubuntu/Downloads/Youtube/cookies.txt --compat-options no-live-chat --playlist-reverse --embed-subs --embed-thumbnail --embed-metadata --no-overwrites -o '/home/ubuntu/Downloads/Youtube/%(uploader)s [%(channel_id)s]/S01E0%(playlist_autonumber)s - %(title)s - %(upload_date)s - [%(id)s].%(ext)s';

Make it executable chmod u+x /home/ubuntu/hourly.sh

It might take more than an hour to run initially, I typically run them once individually, and then add them to the automated script after getting the initial dump. Subsequent runs only take a few minutes for dozens of channels.

When happy, run sudo crontab -e and add the following line.

0 * * * * sh /home/ubuntu/hourly.sh

To run it once an hour.

Hope that all makes sense. Once you have it running for a while, Go into the Plex Library, change the filter to ALL→Episodes→By Date Added and that is basically your Youtube subcription feed in order.

Best of luck out there. ✌️