ytdl-sub

The motivation behind this package is to download media via youtube-dlp and prepare it for consumption in your favorite media player in the most hassle-free way possible.

Everyone stores their media differently, so it is critical to provide comprehensive customization to this process while maintaining simplicity. In addition, we treat media sources differently from each other. We may want to...

Automatically download any and all audio/video file from a channel, user, playlist, etc.
- Download a soundcloud artist's discography, keep it up to date
- Download any new music videos from a YouTube channel playlist
Manually download a single audio/video file
- Download one particular music video
Only download recent audio/video, and remove stale files
- Download and only keep recent podcasts, news videos

We want to cover each of these use cases.

Configuration

Below is the proposed way to define a preset for how you want to download and add metadata. We have two examples, one for downloading Soundcloud discographies, and the other for downloading YouTube playlists, and treating them as music videos. Both are intended to be consumed by Kodi, Jellyfin, and any mp3 music player that supports id3v2.4 tags. For videos, this should (hopefully) be customizable enough to support other use-cases like downloading a playlist as a TV show or multiple movies.

The {variables} used in the strings are derived from the media itself or defined in the overrides section.

config.yaml

# [Required]
# All presets live under this section. Other configurable things for ytdl-sub will
# reside in a different section.
presets:

  # [Required]
  # Custom name that you define for your preset. This preset is intended to download
  # a soundcloud artist's discography.
  soundcloud_discography:
    
    # [Required]
    # This section defines the source you intend on using, with specific
    # configurations for each source. Supports [soundcloud, youtube]
    soundcloud:
      
      # [Required]
      # How to download media from the source, soundcloud supports [albums_and_singles]
      download_strategy: "albums_and_singles"
      
      # [Optional] 
      # Each source has specific options, this is one for soundcloud that skips
      # premiere tracks
      skip_premiere_tracks: True
      
      # [Required, defined dynamically]
      # The username of the soundcloud artist you want to download from
      # username: alis_on
      
    # [Optional]
    # Defines any arguments you want to directly pass to yt_dlp. Use with caution!
    ytdl_options:
      
      # For my case, I want the best audio converted to mp3
      format: 'bestaudio[ext=mp3]'
    
    # [Required]
    # This section defines where and how you want to save files
    output_options:
      
      # [Required]
      # Directory for where you want all your files to land for this preset
      output_directory: "{music_path}/{sanitized_artist}"
      
      # [Required]
      # How you want to name each file that gets downloaded in this preset
      file_name: "[{album_year}] {sanitized_album}/{track_number_padded} - {sanitized_title}.{ext}"
      
      # [Optional]
      # If a thumbnail is available, convert it to your desired file type (webp, ugh)
      convert_thumbnail: "jpg"
      
      # [Optional]
      # If you want to store a thumbnail, define how it gets named
      thumbnail_name: "[{album_year}] {sanitized_album}/folder.jpg"
    
    # [Optional]
    # Define how you want to add metadata to your downloaded media
    metadata_options:
      
      # [Optional]
      # If your final file is mp3, add id3 tags (TODO: support for more file types)
      id3_tags:
        
        # [Optional]
        # id3 version, defaults to 2.4
        id3_version: 2.4
        
        # [Optional]
        # For 2.4 multi-tags, define which character to use for the null separator
        null_separator_char: ";"
        
        # [Required]
        # Define the tag names and values to add to your audio
        tags:
          artist: "{artist}"
          albumartist: "{artist}"
          title: "{title}"
          album: "{album}"
          tracknumber: "{track_number}"
          year: "{album_year}"
          genre: "{genre}"

    # [Optional]
    # This will lazily add/override any media variable that can be used in
    # output or metadata options
    overrides:
      
      # Notice how we use music_path in the output_options.output_directory field above
      music_path: "/mnt/nas/music"

  ######################################################################################
  # Another preset, intended to download all music videos in a YouTube playlist.
  youtube_music_video_playlist:
    
    # [Required]
    # Our source is YouTube, so define the youtube section.
    youtube:
      
      # [Required]
      # Download strategies are different for each source. For YouTube, we want to
      # download the whole playlist (also the only supported download_strategy for now).
      download_strategy: "playlist"
      
      # [Required, defined dynamically]
      # The YouTube playlist id you want to download from
      # playlist_id: "PLVTLbc6i-h_iuhdwUfuPDLFLXG2QQnz-x"
    
    # [Optional]
    ytdl_options:
      # In my case, I want the best format possible, and ignore any errors like
      # geo/age restricted videos.
      format: 'best'
      ignoreerrors: True
    
    # [Required]
    output_options:
      
      # Notice in my overrides, I defined the both music_video_path and kodi_music_video_name.
      output_directory: "{music_video_path}"
      file_name: "{kodi_music_video_name}.{ext}"
      convert_thumbnail: "jpg"
      thumbnail_name: "{kodi_music_video_name}.jpg"
    
    # [Optional]
    metadata_options:
  
      # [Optional]
      # Kodi works with NFO files for videos. I want to format this to be read in as a music video
      nfo:
        
        # [Required]
        # Name of the nfo file
        nfo_name: "{kodi_music_video_name}.nfo"
        
        # [Required]
        # Inside the nfo XML, define the root section.
        nfo_root: "musicvideo"
        
        # [Required]
        # Within the nfo root, define any tags you want set
        tags:
          artist: "{artist}"
          title: "{title}"
          album: "Music Videos"
          year: "{upload_year}"
    
    # [Optional]
    overrides:
      
      # I personally like my paths defined explicitly as variables.
      music_video_path: "/mnt/nas/music"
      
      # Kodi wants music videos all in one folder, named like 'artist - title'
      # This format is used for the video file, thumbnail, and nfo file. So
      # create a shared variable that each of them will use.
      kodi_music_video_name: "{sanitized_artist} - {sanitized_title}"

Usage

That was a lot! If you have made it this far, thanks for your interest. Now we can see how to use our presets to actually download. There are three methods this project aims to build.

Subscriptions

Defining a subscription is for media that you will continually want to download, whether that be performed manually or in a cron job. We implement this by defining yet another yaml.

subscription.yaml

# [Required]
# Custom name that you define for your subscription. Any fields defined here will override anything defined in the
# preset.
ALISON:
  
  # [Required]
  # Name of the preset to use
  preset: "soundcloud_discography"
  
  # We did not define soundcloud.username in the preset on purpose, so we can define it here
  soundcloud:
    username: alis_on
  
  # For this artist, I want to explicitly define the artist name and genres that will get used in the metadata
  # and output paths. An implicit feature is all overrides also create a sanitized variable as well. So
  # `sanitized_artist` can be used, and will run the sanitize function on my artist override.
  overrides:
    artist: "A.L.I.S.O.N."
    genre: "Synthwave;Electronic;Instrumental"

######################################################################################
# Another subscription that can use a different preset, overrides, etc.
Rammstein:

  # [Required]
  preset: "youtube_music_video_playlist"
  
  # [Required]
  # We did not define youtube.playlist_id in the preset on purpose, to define it here.
  youtube:
    playlist_id: "PLVTLbc6i-h_iuhdwUfuPDLFLXG2QQnz-x"
  
  # [Optional]
  overrides:
    artist: "Rammstein"

You can have multiple subscription files for better organizing. We can invoke the subscriptoin download using the command below. This makes it easy to use ytdl-sub in a cron job.

ytdl-sub sub subscription.yaml

One-time Downloads [not available yet]

There are things I want to download, but not subscribe to. In this case, I can invoke ytdl-sub dl and add any required/optional/override fields. This example is equivalent to using the subscription yaml above, but without the yaml.

ytdl-sub dl                                                \
    --preset "soundcloud_discography"                      \
    --soundcloud.username "alis_on"                        \
    --overrides.artist "A.L.I.S.O.N."                      \
    --overrides.genre "Synthwave;Electronic;Instrumental"

REST API [not available yet]

The dream will be to create mobile apps or browser extensions to invoke downloads from the couch. My short-term plan is to self-host a swaggerhub io instance and invoke the rest api from my browser. It will be a while before this gets implemented, but when it does, it will look like this.

# spin-up the server
ytdl-sub server

POST request to invoke the same one-time download example via REST.

curl -X POST -H "Content-type: application/json"                \
    -d '{                                                       \
        "soundcloud.username": "alis_on",                       \
        "overrides.artist": "A.L.I.S.O.N.",                     \
        "overrides.genre": "Synthwave;Electronic;Instrumental"  \
    }'                                                          \
    'localhost:8080/soundcloud_discography'

Other Nice-To-Have Feature Ideas [not available yet]

Check existing media before downloading

Right now, ytdl-sub will naively download everything in the given playlist/artist url. We should check the destination directory first to see if it has already been downloaded. Might involve including a tag with its ytdl id.

More download strategies

Only full artist (soundcloud) and playlists (youtube) are currently supported. Add new ones for single song/video downloads.

Make adding new sources easy to implement

Developers should not have to work around spaghetti to add new sources to ytdl-subscribe. It should be very modularized and definable in a separate file as a plugin.

Musicbrainz Integration

Integration with musicbrainz to fetch correct artist names, song titles, ids, etc and make those tags usable would be a great feature for OCD music hoarders like myself, and would play nicely with existing collections to ensure files land in the same directory. This can be worked around for now by using overrides.

Configurable Time Windows to Delete Stale Media

There are some news channels and podcasts that I'd like to download but only keep the prior 2 weeks-worth of videos. This would have to introspect files that reside in the host filesystem. Extreme caution would be required since we are perma-deleting files.

Contributing

I would like to get this to a point where I create a stable release before I accept PRs from the outside world. In the meantime, any feedback in the form of a Github issue would be greatly appreciated and taken into consideration.

README.md

YoutubeDL-Subscribe