More information about recent issues with Libspotify
Posted March 22nd, 2016
As some of you may already know, Spotify’s C library Libspotify recently experienced some major issues. The purpose of this blog post is to give you a brief overview of what went wrong and what we’re doing to prevent it from happening again.
It’s important to add that Libspotify isn’t under active development on any platform and is considered deprecated. If you’re building applications for iOS or Android, we strongly recommend that you use the iOS SDK or Android SDK instead.
To give some background on our infrastructure, Spotify’s backend is built on a microservice architecture with teams developing and maintaining services necessary to deliver on their missions. More often than not a service built by one team is used by a whole bunch of other teams. While this helps different parts of Spotify to work more autonomously, it also requires the teams to make sure that services in a dependency-chain don’t suddenly change their API or be taken down or changed without notice, very similarly to how third-party developers and partners rely on Spotify to keep their externally accessible APIs and SDKs working - Spotify can’t make breaking changes or take down an API without prior notice.
Applications using Libspotify are making requests to several different services within Spotify, and during a period between January 21st and February 22nd two of them were either mistakenly taken out of production or failed to restart properly after patching the libc vulnerability, causing Libspotify based applications to stop working properly.
The incidents happened close in time to the planned end-of-life for the Metadata API, Spotify’s old Web API, causing people both internally at Spotify as well as externally to believe that the Metadata API take-down was the cause for Libspotify’s degradation of service. The Metadata API, which is not used by Libspotify, was turned off on January 18th, a few days prior to the Libspotify issues began.
The services used by Libspotify that had stopped working, those returning toplist and search data, were online and working again as of February 22nd. The service owners have revisited their logging and alerting, making sure that they’ll be notified whenever one of their services isn’t starting up or otherwise working as intended. To prevent and clear up any confusion around this topic we’ve begun building a closer relationship between Spotify Platform team and @SpotifyCares in order to provide clearer response to developers when we run into technical issues that might affect our developer tools. We’ve also revisited who has access to the @SpotifyPlatform account to make sure that we can quickly respond in case incidents occur again in the future.
Lastly, we sincerely apologize for any inconvenience that this outage have caused.
The Spotify Platform Team