What we’re building

We’ll create a fully automated script that’ll pull all the content from the streaming site(s) of your choice and feed them seamlessly into your offline music library.

Exclaimer

Legal

What we’re doing here might (or might not) violate against existing copyright/piracy laws in your country. In legal terms, you will be downloading a copy of a piece that could be protected by copyright. Here’s the Austrian take on copyright.

The mere downloading of music or videos (which are copyrighted works) from the internet (i.e., without wanting to re-offer the piece of music or video itself) onto a data carrier (e.g., hard disk, DVD, CD) for private use is generally permitted. However, this presupposes that the downloaded piece of music or video, which was saved on one’s own computer, for example, is not subsequently used for commercial purposes. For example, the sale of copied or downloaded pieces of music or videos is prohibited, as this would no longer constitute private use. It is also not permitted to allow other persons to access the downloaded music or videos on the internet (e.g., via file-sharing networks).
…
No private copies of computer programs may be made. However, backup copies shall be excepted insofar as this is necessary for the use of the computer program.
…
Source: Austrian ministry for Education and Digital Media

To summarize this: if you don’t distribute the copy and only store it for private purposes (without commercial interest) you are not violating against the law! While it seems that most of European countries share the same interpretation of copyright, please verify this first (or take pre-cautions like VPNs and Proxies before you continue).

Ethical

Please also take into consideration that artists get paid a small portion per stream on most streaming sites. Especially for less-known artists, this might be the primary source of income. If you really enjoy the track, please always consider supporting the artist first. Making music is lots of work and requires lots of know-how and effort and thus needs to be valued. If you are downloading the music with no intention of supporting the artists then, stop continue reading NOW and leave this page! I mean it.

Convenience of Streaming Sites

Personally, I prefer to organize my tracks in playlists. I usually sort them by Genre or by artist. So, I do have quite a lot of playlists on Soundcloud. And I really love Soundcloud especially for that. It allows to find new artists and songs with ease and categorize them into Playlists and likes. That’s really something I wouldn’t like to miss out anymore. Even though I valued the old go-to-shop-and-buy-a-random-cd-approach, I prefer the streaming-site approach much more, as it is just easier to find tracks that match your favorite style.

youtube-dl

youtube-dl is a CLI application written in Python, specializing on downloading videos from the Web. youtube-dl is open source and actively developed on ytdl-org Github. While the name suggests that it’ll download from YouTube only, it really supports pretty much every site existing (including Spotify and Soundcloud. You can find a list of all supported sites here).

Installing youtube-dl

If you are on Linux distribution, Python (and pip) are most likely already installed, thus the only thing you’d need to do is install youtube-dl with pip.

# if your distribution still ships with Python 2.x make sure to use pip3 instead of pip
pip3 install youtube-dl

Downloading your first Video

Once installed on your machine it is time to test downloading your first video. The command line interface of youtube-dl is quite straight-forward. If you just want to download the video, just invoke youtube-dl with the URL to the resource that you want to download. Here’s an example URL from YouTube.

youtube-dl https://www.youtube.com/watch?v=dQw4w9WgXcQ

This will download the video into the Current Working Directory.

Conversion to Audio

Since we are interested in music primarily we’d need to extract the Audio from the video. Luckily youtube-dl ships with support for ffmpeg out of the box. And ffmpeg will take care of extracting the music from the video. Make sure that you have installed ffmpeg on your box as well.

# On Fedora/CentOS and RHEL-based distributions
sudo dnf install ffmpeg -y
# On Ubuntu and Debian-based distributions
sudo apt install ffmpeg -y

Additionally, to ffmpeg you’ll need to install lame in case you want to export your audio to the mp3 format. I’m not going to show you how to install lame as this is very distribution-dependent and mp3 is a non-free format. As soon as ffmpeg is installed we can invoke youtube-dl once again, but this time we’ll add additional triggers to the command that tell youtube-dl to extract the audio.

# Download the video with the highest-quality audio
# After download (of video) succeeded, call ffmpeg to extract audio
# Convert extracted audio to flac as output format
# Delete the downloaded video file
youtube-dl --extract-audio --format 'bestaudio' --audio-format 'flac' https://www.youtube.com/watch?v=dQw4w9WgXcQ

# Download the video with the highest-quality audio
# After download (of video) succeeded, call ffmpeg to extract audio
# Convert extracted audio to flac as output format
# Keep the downloaded video file
youtube-dl --keep-video --extract-audio --format 'bestaudio' --audio-format 'ogg' https://www.youtube.com/watch?v=dQw4w9WgXcQ

• –extract-audio tells youtube-dl to invoke ffmpeg to extract the audio from the video it downloaded.
• –format ‘bestaudio’ tells youtube-dl to find the video-stream (and output format) with the best audio-quality

Working with playlists

The intention of this post was to show you how to download playlists (or in more general terms: lists of media-resources). Luckily youtube-dl has support for this out-of-the-box. Downloading a playlist works exactly like downloading a single media element from a website. You just have additional controls.

Imagine the following playlist:

1) Abba - Dancing Quuen
2) The Prodigy - Invaders Must Die
3) Lady Gaga - Pokerface
4) Amy Winehouse - Back in Black
5) Scooter - Nessaja

with these (self-explanatory) triggers you can easily select which elements of the playlist you want to download.

• --playlist-start (e.g. playlist-start 2 to skip Abba)
• --playlist-end (e.g. playlist-end 4 to skip Scooter)
• note that those commandline-triggers can be set at the same time.
• --playlist-items (e.g. playlist-items 1,2,5 will download Abba, The Prodigyy, Scooter)
• note that playlist-items cannot be added if either --playlist-start or --playlist-end is present

And finally, since playlists tend to grow over time, we want to youtube-dl to keep track of what it has downloaded already, and what not. With the trigger --download-archive you can specify a file that youtube-dl utilizes as database of what has already
been downloaded and can be skipped for future downloads of this playlist.

Authentication against Soundcloud

sometimes, playlists are private, or Songs are not publicly available and are behind a paywall that require you to be logged in (like a payed Soundcloud Go-subscription). Then you need to tell youtube-dl to authenticate against the Soundcloud Streaming-Servers before downloading anything.

First you need to log-in to Soundcloud, then identify the Authentication-Token that your browser sends to authenticate against the Soundcloud-Servers. On Firefox press F12 (Developer Console), find the Network-Tab in the Console and then refresh the site. Don’t be overwhelmed for the huge amount of requests that are made.

One of the responses of the server that contains the authentication-token is the me-request. The first element in the JSON-response is the authentication-token that you need.

with the trigger --add-header you can pass additional HTTP-headers to the requests to the server that youtube-dl is downloading the media from. To authenticate add the following header: --add-header 'Authorization':'OAuth <your Soundcloud Authentication Token>'.

Let’s put it all together

#!/usr/bin/env bash

TARGET_DIRECTORY='./Sync'
TARGET_FORMAT='mp3'

# Update youtube-dl everytime we start the script (recommended)
pip3 install --upgrade youtube-dl &> /dev/null

# Declare a Dictionary of Directories (where the files are downloaded to) and the upstream playlist-source
declare -A playlists=( 
    ["Austropop"]="https://soundcloud.com/djraremusic/sets/austropop"
    ["Hands Up"]="https://soundcloud.com/djraremusic/sets/haends-up"
    ["Acoustic"]="https://soundcloud.com/djraremusic/sets/acoustic"
    ["Oldskool & Classics"]="https://soundcloud.com/djraremusic/sets/nostalgisch-wie-fick"
    ["Neuro Funk"]="https://soundcloud.com/djraremusic/sets/zabrochene-schlaeg"
    ["Hardstyle"]="https://soundcloud.com/djraremusic/sets/reverse-bass-cyka"
    ["Psytrance"]="https://soundcloud.com/djraremusic/sets/psychadelic-mindfuk"
    ["Rock & Metal"]="https://soundcloud.com/djraremusic/sets/rock-metal-sammelbecken"
    ["Hardcore"]="https://soundcloud.com/djraremusic/sets/3x-normalpuls"
    ["House & Electro"]="https://soundcloud.com/djraremusic/sets/future-zeigs"
    ["House & Electro"]="https://soundcloud.com/djraremusic/sets/hoartes-house"
    ["House & Electro"]="https://soundcloud.com/djraremusic/sets/producing-schmankerl"
    ["Drum & Bass"]="https://soundcloud.com/djraremusic/sets/fluessigs-dnb"
    ["Mainstream"]="https://soundcloud.com/djraremusic/sets/mood-chill"
    ["Likes & Favs"]="https://soundcloud.com/djraremusic/likes"
)

for name in "${!playlists[@]}"; 
do 
    url=${playlists[$name]}
    mkdir -p "$TARGET_DIRECTORY/$name"
    echo "$datetime: Synchronizing $name ($url)"

    youtube-dl \
        --ignore-errors \
        --yes-playlist \
        --continue \
        --no-overwrites \
        --extract-audio \
        --audio-format "$TARGET_FORMAT" \
        --prefer-ffmpeg \
        --sleep-interval 10 \
        -f 'bestaudio' \
        --download-archive "$TARGET_DIRECTORY/$name/.downloadarchive.txt" \
        --user-agent 'Mozilla/5.0 (X11; Fedora; Linux x86_64; rv:84.0) Gecko/20100101 Firefox/91.0' \
        --add-header 'Authorization':'OAuth xxxxxxxxxxxxxxxxxxxxxxxxxxxxx' \
        --output "$TARGET_DIRECTORY/$name/%(title)s.%(ext)s" \
        $url
done