Can you say why Spotify's recommendations are so bad? Something like what OP has made should have been relatively simple to make for Spotify for many, many years already, yet that hasn't happen. Is the whole system just rigged to only recommended a few "sponspored" artists?
Because, as I said above, it's a very complex problem :)
I honestly don't know much about recommendations (and what I know I probably cannot tell). But there's definitely continuous work done on them. But it can also be hampered by extremely conflicting requirements (where "some" both means double-digit procent of users and these "some"s overlap with each other):
- some users want more of the same, some users want a more diverse listening experience. Some of these users are the same user, but on different days
- some users mostly prefer curated suggestions, some users want ranodm stuff. They can also be the same user :)
- some users a heavily weigted to only a few artists, some users listen to evereything and anything. And even this can be the same user :)
- there's probably stuff about licensing, availability, contracts etc. at play as well, because in streaming services it's always there, in very bizarre ways
Basically every single tweak to recommendations will break them. And yeah, Spotify employees will complain about this more than anyone else, all the time :)
I doubt that "why does your product suck" is one of the things a Spotify employee is allowed to talk freely about in public!
But I've been watching them, I will speculate. A few years ago, Spotify had two young interns, Sander Dieleman and Aäron van den Oord. We know a bit of what they worked on, because Dieleman blogged on it, and indeed it was something a lot like what OP has made here - only better, I would say. I asked him, and Dieleman was allowed to say that the thing they built was one of the inputs into the then-new Discover Weekly, which made headlines for how outrageously good it was.
But Dieleman and v.d.Oord did not stay at Spotify. They were headhunted by DeepMind, and have had a VERY impressive track record there over the years.
And I wonder why. Was there a conflict between the old school ML of the Echo Nest people and the new fancy neural net kids? Or was it just, as GP alludes to, that the NN methods were just too computationally expensive and they failed to justify their costs to leadership?
