Automatic Spotify playlist powered by AWS Free Tier
Written on: 2020-11-22T00:00:00.000Z
I love discovering and listening to new music, but when you swerve off the mainstream path of music, it can become very time-consuming. I wanted something similar to the Release Radar playlist from Spotify that is updated more frequently and even more tailored to my personal tastes, so I built it myself.
TL;DR: here’s the playlist.
The idea
My current process of music curation consists of scanning a list of YouTube channels on a weekly basis, and adding new songs I like to a Spotify playlist.
Boiled down to the core, the process I want to automate looks like this:
Let’s look at these two steps independently, and then we’ll link them together. I’ll be using AWS free tiers to build everything.
Finding music: Youtube-powered discovery engine
The basis of the Youtube music discovery engine is simple: build a list of channels that upload music I generally enjoy (eg: curate the curators), and check their channels daily for new uploads.
To do this, I have two options:
- Use a web scraper to navigate to each channel and extract the new uploads uploaded in the 24 hours
- Use YouTube’s search API
Youtube has pretty strict free quotas on their APIs, so the first option would make the most sense economically, but I’m lazy, so I went with the second.
Every action you do via the API costs you a credit. Simple actions (ex: getting the metadata of a video) cost you 1 credit, expensive actions (ex: uploading a video) costs a lot of credits.
To do what I wanted, I fell in the middle of the cost spectrum: while listing the videos of a specific channel is practically free, I needed the videos uploaded in the last 24 hours (as I wanted a daily stream of new music), and using Youtube’s Search API costs 100 credits per action.
I could achieve the same using the less expensive operations, but I want then need to keep track of past uploads for every channel myself to deduce the new uploads daily, and that was more work than I cared to do. With the daily limit of 10000 credits, I had a hard limit of 100 channels to scrap music from daily, which was good enough for now.
The architecture
- Input: A list of Youtube channels
- Output: A list of song titles
The code
Using the Node client for Youtube’s API, I can easily get the uploads of the past 24 hours for a given channel.
const channelUploads = await yt.search.list({ // Search for videos
"channelId": channel.ChannelId, // for a specific channel
"publishedAfter": yesterday, // after timestamp (ISO)
"part": "snippet", // select only basic info
"fields": 'items(snippet/title)' // filter results to keep only the title
})Once I have all the results, I can filter them to only keep the video titles.
Two
Add to playlist: transforming text to music with Spotify
As popular as Spotify may be, we don’t often take the time to actually appreciate how the damn thing works. One of Spotify’s biggest magic trick is being able to decode human’s most stupid and vague searches into plausible results.
Spotify Search’s permissiveness is a huge advantage: we don’t have to normalize our search terms too much. (Spotify is not perfect, so we do have to help it a bit in reality, but I won’t cover that today.)
The architecture
- Input: A song title
- Output: Song added to Spotify playlist
The script will need to imitate what a human would do: first, search Spotify using a query, and secondly, add a song to a playlist (given the search returns a result)
The code
Searching a song on Spotify is quite easy; if we have search results, we’ll only keep the first one (we trust Spotify to know what we want better then ourselves)
SpotifySearchForFirstSong
If the search gives 0 results, do nothing, otherwise add it to our designated playlist:
SpotifyAddTrackToPlaylist
Great! Let’s put those two functions together inside a Lambda that receives as input our search query:
SpotifySearchForSong
Connecting A to B: decoupling with queues
We have two scripts that do their respective jobs very well, but they do not know of the existence of the other. That’s a feature, and we’ll keep it that way, but we will add something so they can be able to communicate.
The Youtube script is supposed to run once a day for less than a minute, and the output is plaintext. The Spotify script only cares to do its job if there are actual song titles to search for.
The spontaneous nature of the system and the data that needs to be communicated screams for a queue, so that’s what we’ll use. It also has the benefit of allowing us to add new sources for song titles in the future without having to modify our Spotify script!
The code (or configuration)
AWS SQS
We will actually need two queues. SQS good practices recommend having your main queue as well as a dead-letter queue which will receive invalid messages that got sent in your main queue. This will stop your main queue from getting poisoned, and your AWS bills from going haywire.
The main queue is a FIFO queue since we only ever want to process each song title once. Once a message bounces just once, we send it to our dead-letter queue. The most common case of a message bouncing will be if we found nothing on Spotify for it. We could simply delete those messages forever, but I want to keep them for debugging purposes.
Youtube Lambda
To run the script daily, we can just create an event inside AWS to trigger our Lambda at a specific schedule:
AWS Console -> CloudWatch -> Events -> Rules -> New Rule
Change the event source to Schedule and add a Cron expression to run daily at a specific time (just before I wake up):
Three
We’ll also modify our script to send the song titles to a SQS Queue instead of simply returning the results
YoutubeLambdaSQS
Spotify Lambda
For this one, we just add the SQS queue as a trigger and modify our script to get the message properly:
SpotifyLambda
Conclusion
It’s working! Mostly. We still need to handle some boring things, like keeping our Spotify token alive, normalizing Youtube titles, and optimizing our lambdas, but the main processes are there.
It’s been one year since I’ve built this system, and it’s been working flawlessly since then will little maintenance needed (just some quality of life improvements).