Continuous backup of your Soundcloud-Playlists with youtube-dl

1. Introduction
2. What we’re building
1. 2.1. Exclaimer
  1. 2.1.1. Legal
  2. 2.1.2. Ethical
3. Convenience of Streaming Sites
4. youtube-dl
5. yt-dlp

Introduction

Listening to your favorite song is as easy as going to YouTube (or even better Invidious), Soundcloud or Spotify and stream everything right from your PC or mobile device. Millions of artists and tracks are just waiting for you to discover them. However, streaming is not necessarily the best choice for you and here’s why. Simply put, you don’t own any of the content that you are streaming, and you have no guarantee that the song you liked today will still be there tomorrow. You have no say in how those platforms handle copyright, DMCA-takedowns or Content-ID. Remember back in the 2000s, when you went to the store and purchased an album on CD? As long as you owned the CD, you also owned the content, and nobody was able to take it away from you again (unless you became yet another victim of disc rot). In this article we’ll be utilizing youtube-dl (and/or yt-dlp) to get back to the good ol’ days when we had full ownership of the media we are consuming, without losing the comforts and convenience of such streaming-sites.

What we’re building

We’ll create a fully automated script that’ll pull all the content from playlists on your streaming site(s) of choice and feed them seamlessly into your offline music library. ‘Why?’ you might ask. Well son, because data-hoarding is fun.

Exclaimer

Legal

What we’re doing here might (or might not) violate against existing copyright/piracy laws in your country. In legal terms, you will be downloading a copy of a piece that could be protected by copyright. Here’s the Austrian take on copyright.

The mere downloading of music or videos (which are copyrighted works) from the internet (i.e., without wanting to re-offer the piece of music or video itself) onto a data carrier (e.g., hard disk, DVD, CD) for private use is generally permitted. However, this presupposes that the downloaded piece of music or video, which was saved on one’s own computer, for example, is not subsequently used for commercial purposes. For example, the sale of copied or downloaded pieces of music or videos is prohibited, as this would no longer constitute private use. It is also not permitted to allow other persons to access the downloaded music or videos on the internet (e.g., via file-sharing networks).
…
No private copies of computer programs may be made. However, backup copies shall be excepted insofar as this is necessary for the use of the computer program.
…
Source: Austrian ministry for Education and Digital Media

To summarize this: if you don’t distribute the copy and only store it for private purposes (without commercial interest) you are not violating against the law! While it seems, that most of European countries share the same interpretation of copyright, please verify this first for the country you are living in (or take pre-cautions like VPNs and Proxies before you continue).

Ethical

Please also take into consideration that artists get paid a small portion per stream on most streaming sites. Especially for less-known artists, this might be the primary source of income. If you really enjoy the track, please always consider supporting the artist first. Making music is lots of work and requires lots of know-how and effort and thus needs to be valued. If you are downloading the music with no intention of supporting the artists then, stop continue reading NOW and leave this page! I mean it.

Convenience of Streaming Sites

Personally, I prefer to organize my tracks in playlists. I usually sort them by genre or by artist. So, I do have quite a lot of playlists on Soundcloud. I really love Soundcloud especially for that. It allows me, to find new artists and songs with ease and categorize them into playlists. That’s really something I wouldn’t like to miss out anymore. Even though I valued the old go-to-shop-and-buy-a-random-cd-approach, the streaming approach appeals just a bit more to me. It is just easier to find music that matches your preferences.

youtube-dl

youtube-dl is a CLI application written in Python, specializing on downloading videos from the Web. youtube-dl is open source and actively developed on ytdl-org Github - however the last official release has been a while. While the name suggests that it’ll download from YouTube only, it really supports pretty much every site existing (including Spotify and Soundcloud. You can find a list of all supported sites here).

yt-dlp

as youtube-dl works and gets the job done until today, there is a drop-in-replacement fork I’d like to mention here. It’s yt-dlp - a community driven fork of the original youtube-dl. It is compatible with youtube-dl but extends it with support for additional streaming services. Additionally, yt-dlp gets more frequent releases.
Originally this article was written for youtube-dl, but NOTHING speaks against replacing youtube-dl with yt-dlp. Both will
get the job done.

Installing youtube-dl

If you are on Linux distribution, Python (and pip) are most likely already installed, thus the only thing you’d need to do is install youtube-dl with pip.

# if your distribution still ships with Python 2.x make sure to use pip3 instead of pip
pip3 install youtube-dl

# if you want to use yt-dlp
pip3 install yt-dlp

Downloading your first Video

Once installed on your machine it is time to test downloading your first video. The command line interface of youtube-dl is quite straight-forward. If you just want to download the video, just invoke youtube-dl with the URL to the resource that you want to download. Here’s an example URL from YouTube.

youtube-dl https://www.youtube.com/watch?v=dQw4w9WgXcQ

This will download the video into the Current Working Directory.

Conversion to Audio

Since we are interested in music primarily we’d need to extract the audio from the video. Luckily youtube-dl ships with support for ffmpeg out of the box. And ffmpeg will take care of extracting the music from the video. Make sure that you have installed ffmpeg on your box as well.

# On Fedora/CentOS and RHEL-based distributions
sudo dnf install ffmpeg -y
# On Ubuntu and Debian-based distributions
sudo apt install ffmpeg -y

Additionally, you’ll need to install lame in case you want to convert your audio to the mp3 format. I’m not going to show you how to install lame as this is very distribution-dependent and mp3 is a non-free format. As soon as ffmpeg is installed we can invoke youtube-dl once again, but this time we’ll add additional triggers to the command that tell youtube-dl to extract the audio.

# Download the video with the highest-quality audio
# After download (of video) succeeded, call ffmpeg to extract audio
# Convert extracted audio to flac as output format
# Delete the downloaded video file
youtube-dl --extract-audio --format 'bestaudio' --audio-format 'flac' https://www.youtube.com/watch?v=dQw4w9WgXcQ

# Download the video with the highest-quality audio
# After download (of video) succeeded, call ffmpeg to extract audio
# Convert extracted audio to flac as output format
# Keep the downloaded video file
youtube-dl --keep-video --extract-audio --format 'bestaudio' --audio-format 'ogg' https://www.youtube.com/watch?v=dQw4w9WgXcQ

–extract-audio tells youtube-dl to invoke ffmpeg to extract the audio from the video it downloaded.
–format ‘bestaudio’ tells youtube-dl to find the video-stream (and output format) with the best audio-quality

Working with playlists

The intention of this post was to show you how to download playlists (or in more general terms: lists of media-resources). Luckily youtube-dl has support for this out-of-the-box. Downloading a playlist works exactly like downloading a single media element from a website. You just have additional controls.

Imagine the following playlist:

1) Abba - Dancing Quuen
2) The Prodigy - Invaders Must Die
3) Lady Gaga - Pokerface
4) Amy Winehouse - Back in Black
5) Scooter - Nessaja

with these (self-explanatory) triggers you can easily select which elements of the playlist you want to download.

--playlist-start (e.g. playlist-start 2 to skip Abba)
--playlist-end (e.g. playlist-end 4 to skip Scooter)
note that those commandline-triggers can be set at the same time.
--playlist-items (e.g. playlist-items 1,2,5 will download Abba, The Prodigyy, Scooter)
note that playlist-items cannot be added if either --playlist-start or --playlist-end is present

And finally, since playlists tend to grow over time, we want youtube-dl to keep track of what it has downloaded already, and what not. With the trigger --download-archive you can specify a file that youtube-dl utilizes as database of what has already
been downloaded and can be skipped for future downloads of this playlist. youtube-dl creates the file if it does not yet exist.

Authentication against Soundcloud

Sometimes playlists are private, or Songs are not publicly available and are behind a paywall that require you to be logged in (e.g., when you own a Soundcloud Go subscription). Then you need to tell youtube-dl to authenticate against the Soundcloud Streaming-Servers before downloading anything.

First you need to log in to Soundcloud, then identify the Authentication-Token that your browser sends to authenticate against the Soundcloud-Servers. On Firefox press F12 to open the Developer Console, find the Network-Tab in the Console and then refresh the site. Don’t be overwhelmed for the huge amount of requests that are made.
Firefox Developer Console, Network Inspector, Soundcloud Authentication Token

One of the responses of the server that contains the authentication-token is the me-request. The first element in the JSON-response is the authentication-token that you need.

with the trigger --add-header you can pass additional HTTP-headers to the requests to the server that youtube-dl is downloading the media from. To authenticate add the following header: --add-header 'Authorization':'OAuth <your Soundcloud Authentication Token>'.

Let’s put it all together

#!/usr/bin/env bash

TARGET_DIRECTORY='./Sync'
TARGET_FORMAT='mp3'

# Update youtube-dl everytime we start the script (recommended)
pip3 install --upgrade youtube-dl &> /dev/null

# Declare a Dictionary of Directories (where the files are downloaded to) and the upstream playlist-source
declare -A playlists=( 
    ["Austropop"]="https://soundcloud.com/djraremusic/sets/austropop"
    ["Hands Up"]="https://soundcloud.com/djraremusic/sets/haends-up"
    ["Acoustic"]="https://soundcloud.com/djraremusic/sets/acoustic"
    ["Oldskool & Classics"]="https://soundcloud.com/djraremusic/sets/nostalgisch-wie-fick"
    ["Neuro Funk"]="https://soundcloud.com/djraremusic/sets/zabrochene-schlaeg"
    ["Hardstyle"]="https://soundcloud.com/djraremusic/sets/reverse-bass-cyka"
    ["Psytrance"]="https://soundcloud.com/djraremusic/sets/psychadelic-mindfuk"
    ["Rock & Metal"]="https://soundcloud.com/djraremusic/sets/rock-metal-sammelbecken"
    ["Hardcore"]="https://soundcloud.com/djraremusic/sets/3x-normalpuls"
    ["House & Electro"]="https://soundcloud.com/djraremusic/sets/future-zeigs"
    ["House & Electro"]="https://soundcloud.com/djraremusic/sets/hoartes-house"
    ["House & Electro"]="https://soundcloud.com/djraremusic/sets/producing-schmankerl"
    ["Drum & Bass"]="https://soundcloud.com/djraremusic/sets/fluessigs-dnb"
    ["Mainstream"]="https://soundcloud.com/djraremusic/sets/mood-chill"
    ["Likes & Favs"]="https://soundcloud.com/djraremusic/likes"
)

for name in "${!playlists[@]}"; 
do 
    url=${playlists[$name]}
    mkdir -p "$TARGET_DIRECTORY/$name"
    echo "$datetime: Synchronizing $name ($url)"

    youtube-dl \
        --ignore-errors \
        --yes-playlist \
        --continue \
        --no-overwrites \
        --extract-audio \
        --audio-format "$TARGET_FORMAT" \
        --prefer-ffmpeg \
        --sleep-interval 10 \
        -f 'bestaudio' \
        --download-archive "$TARGET_DIRECTORY/$name/.downloadarchive.txt" \
        --user-agent 'Mozilla/5.0 (X11; Fedora; Linux x86_64; rv:84.0) Gecko/20100101 Firefox/91.0' \
        --add-header 'Authorization':'OAuth xxxxxxxxxxxxxxxxxxxxxxxxxxxxx' \
        --output "$TARGET_DIRECTORY/$name/%(title)s.%(ext)s" \
        $url
done