Spotify is one of Google Cloud's core customers, signaling a change in Google's business ambitions. It's also a profitable one. Spotify announced in its IPO filing in 2018 that it was paying Google €365 million over three years, which equates to just over €10 million (£8.78 million) every month.
But what's the story behind those figures? Spotify has released a new podcast that discusses how the company chose Google Cloud for its migration and the company's pain points and myths about the method.
According to Gustav Söderström, Spotify's chief R&D officer, one overlooked element of cloud migrations is the increased expense and difficulty. There is no magical switch that causes anything to shift. The old system must coexist with the new one.
The operations director, Emil Fredriksson, said that there was a "massive hill you had to scale," both mentally and in terms of workload. Because of the overlap, Spotify ordered "millions and millions of dollars" of servers during the migration period, according to Fredriksson. "It doesn't feel great doing that, knowing these are going to be in use for a short period, but this is what you needed to do," he added.
Spotify was candid about some elements of the trip. Fredriksson was one of the company's first employees, joining as a senior network engineer four months after it was formed. He clarified that he had attempted to shift the recommendation scheme, which was based on Hadoop clusters, to the cloud "quite early on."
"I remember thinking, 'this is the future, this is where we need to go, it's going to make our lives so much simpler' – and it failed miserably," he said. "It was costly to do the calculations, and it was so slow that we couldn't use it at all, so we had to roll back and create our own massive in-data-center analytics cluster to do that."
Fredriksson agreed that the product was immature and that the use case was not a direct match. Nonetheless, the podcast's title – 'when your winning bet becomes your losing bet' – summed up the decisions taken. "We suspected... it was only a matter of time before this happened, but it wasn't the time then," Fredriksson said of cloud migration.
Spotify also acknowledged that its ethos stifled development. Because of its engineering focus and trust in its teams, any procurement negotiations normally resulted in a build rather than a buy decision. "We didn't have any decision-making mechanisms or guidelines for the buy versus build decision at all," said Tyson Singer, vice president of technology and platform.
This was a difficult lesson for the business to remember. "It's a sign of a seasoned engineering or product development manager, or a more mature business, to know when to step back and ask – how can building this particular tool shift the needle on our core strategy?" Söderström said.
As a result, the decision to move was taken – but not without disagreement, as another candid admission revealed. "DevOps, which were quite strong constituents at Spotify at the time, was very anti-cloud," Spotify CEO Daniel Ek explained. "Some of it was because they were afraid of losing their work, and part of it was because they were hired to create an excellent tech stack.
"It took a lot of convincing to get people to stop saying 'look, you're not going to lose your job over this,'" Ek added.
The subsequent decision to use Google Cloud as Spotify's cloud provider benefited both parties – and makes for insightful research in vendor and consumer approaches. In terms of alignment, Spotify noted a 'perfect match in engineering culture,' while still not wanting to be seen as 'just another customer.' As a result, Spotify made a mark on Google Cloud's roadmap as a big-ticket item.
Google Cloud was undoubtedly in its early stages in 2016. It's worth noting that Spotify was already listed as a customer when Evernote announced its move to Google in September of that year. However, aside from Coca-Cola, Philips, and Snapchat, the cupboard was relatively empty for blue-chip names at the time.
The Singer was aware of the risk. "[Google] did not have the maturity of a lot of enterprise-level skills, but working with Google gave us an edge because they were working higher up in the stack; they were working on managed services," Singer said. "If you can pick a cloud provider where you can influence the direction of those managed services so it fits your business needs more substantially, then it becomes a much better proposition."
Google goods, Singer remembered, were better at scaling in some cases. BigTable, a NoSQL database service, is one example; 'the variability and latency was way too much' at low RPS (request per second), argued Singer, but Spotify desired a 'standardized NoSQL-type solution that fits in. With the BigTable model.'
However, it is a two-way lane. Urs Holzle, SVP technological infrastructure, claimed that Google's collaboration with Spotify addressed a significant billing accuracy problem. Many who have seen Corey Quinn of The Duckbill Group dissect an Amazon Web Services (AWS) bill will appreciate the meaning.
Finally, Singer remarked on his experience of 'understanding' the shared advantages of the two firms. "Because Spotify was a small company, as a large company with a lot of buying power, we could control the roadmap and feature set of other companies, and that worked out in the sense of our cloud provider," Singer explained.