34: Redesigning our Apple TV App *Release Notes*

0:00I'm Dave Zalathuski, Principal Engineer at Spotify. Today we're doing a release note style episode, zooming in on one thing that we just shipped and breaking down how it actually got built. This time, it's the redesigned Spotify app on Apple TV. We'll talk about why we rebuilt it, the engineering and platform challenges along the way, and how modern AI and coding agents helped us move faster, from bootstrapping the app to polishing a great video for our experience for the big screen. I'm joined by Theodore Osiridis, one of the engineers behind the work, to walk through how you go from a constrained, templated UI to a fully native TV OS app, and how AI played a real role in shipping something that looks and feels great in your living room. So if you're into app architecture, cross-platform development are using AI to ship high-quality experiences faster. Stick around. So yeah, let's get started. I'm really glad to have you on the podcast. We've worked together a bunch, but I haven't worked with you here yet.

1:00So before we get started, can you just introduce yourself a little bit. Tell me what's your role at Spotify? How long have you been here? Where are you located in the world? Yeah, I'm Fedoris. I've been at Spotify since 2012, and I'm in the part of the org that's called PPX. PPX stands for platform and partner experience, and the studio at Spotify builds the apps, all the apps that are not the main mobile apps. So you can take a bit as a studio that builds our TV apps, our wearables apps, the apps in the watch, the desktop app, the web player app, et cetera. I was part of the team that would build the first ever web player Spotify. Then I'm transitioning a bit on building the new home experience from mobile for iOS and Android, and then I went to PPX and rebuild the web player from Strats for a second time. Quite the adventure. So today we're here to talk a little bit about building a new Apple TV app. Yeah, so Spotify and Apple TV is, I would say,

2:00in September 2019, we released the first Apple TV app, and at that point, the mobile app moved or tailored specifically for the form factor. And yeah, you could do most of the things you would be able to do 2019 in our mobile app. You could do it on TV, you could play podcasts, you could watch podcasts also, you could play music, you could search, you could navigate, get your recommendations from home. And so then more recently, it sounds like you spent a bunch of time building a new Apple TV app. So tell me about why you and often did that. There were big seats with regards to how Spotify wanted to prioritize video. And we wanted to make video a first-class citizen, like video first experience, specifically on the TV space. And we used TVML, which was Apple's markup language. It was very limited, it was a templated language. Like, we couldn't build a UI that we wanted to build. And that kind of made us have a more limited experience compared to the mobile. And as years went by, the gap between the mobile experience and what we offer

3:03on mobile, compared what we had on Apple TV, started becoming bigger and bigger. Yeah, and so just to dive into the engineering aside of that a little bit, if I understood correctly, it sounds like TVML was really kind of like an XML sort of like declarative way of just saying, here are the things that I want to draw on the screen. But it wasn't like a rich modern UI framework, like what you expect to have in Swift on an iPhone. So you were super limited in what you could even do in TVML, probably on earlier generations of Apple TV. And so this kind of deprecation was probably also a sign of being able to do much more on Apple TV, and thus we wanted to do that. TVML, like you said, it was more like some weird mix of HTML. And it transpiled down to native code, but you can only use specific UI elements. So we couldn't build more complicated UI components. We wanted to build. So we had to rely whatever Apple gave us. And that was quite a big limitation for us. Yeah. No, that sounds really familiar with what I've heard from people. I know that worked on early editions of Apple Watch.

4:04That just those platforms and frameworks are so much simpler back then that you only had a handful of tools you could build from. And people built really amazing things from very simple frameworks. And now they've matured so much. There's so much more advanced. We're getting really rich frameworks that you can build much more powerful apps on. And Spotify's matured so much adding things like video. So it bakes a ton of sense for us to be rebuilding on the new rich framework instead of trying to use the old one on the new platform. Yeah. So we wanted to base the app on top of all the foundational work that happened on our main mobile clients. On our mobile clients, we have our architectures around reusable components. We call them services. And this kind of encapsulates business logic and it's the exposure business logic in the form of APIs. And then you can have a dependency on these APIs and you can start using them from your surface, from your feature. So we wanted to reuse as much as we could. And a good way to think about what one of these service is, it could be the service that you load things from the playlist, for a specific playlist or for a specific album.

5:06So a lot of work happened on the mobile side in order to be able to have this kind of service and you can be able to reuse them. So we wanted to build our app based on these services. We started by looking on how can we push strap a new app and bring all these services that we needed into that new app. And that was, I would say, one of the most challenging parts because the way we bootstrapped our app, we did it once for our main mobile apps, specifically the iOS mobile app. We had to actually bootstrap the app for TV OS specifically. We had to bring all the dependencies in. We had to bring our service runtime in. And we had to, since we had to bring the service in, we needed. And since already in a mobile app, we're in a transitioning phase where some of the services have not been porting into this new service runtime, we had to do the work ourselves. And then we had to look at the dependencies of these services and also port them. So that was kind of the biggest challenge of the work and it was in the beginning. The moment we managed to bootstrap the app, what was left was the tip of the iceberg. We had all the APIs that we needed.

6:08And it was all about building the UI. And the UI was got something custom. We were used from the main mobile app, our design system, and specifically the foundational aspects of it, the tokens, the colors, the themes, the text views and the image views. And then we built, we composed these things into more complicated UI components, like a card or a list row, for example. And then we went featured by feature, start consuming more and more of the APIs that were coming from the mobile app. And on top of this API is we start building the UI tailored specifically to the form factor. I see, so I want to get into the UI in a minute. But first, tell me more about this bootstrapping thing. You set a couple of times, we have to bootstrap the app. What does that mean? So bootstrapping the app means, first of all, bringing all the dependencies in and then instantiating everything, going from what we call a zero scope, where you have everything up and running, but you're not authenticated yet. You don't have a user. You only have instantiated the most important services that will allow you to authenticate, for example. Then moving from that scope to the authenticated scope.

7:12So then you show a login screen, the user logs in. Then as you move to the authenticated scope, you bring more services in, you instantiate them, you start them, because these are the services that actually need a user. I got you. So you're kind of putting together all of the foundational bits you need before users can really interact with it, where you build out all the pieces to connect the Spotify or to play music, authenticate all the stuff you said. Exactly, being able to do a network request or being able to bring the playback library, so you can set commands to it to start playback, like these kind of things. And that already is not the same as something like iOS. You can't just say, take the whole iOS app, cut off the UI that works on a phone and just run that on the TV, and then put a TV UI on top. No, we had to do pretty much the same work that they did on mobile, maybe for the first time. We had to do it on TV OS also. All right, so let's talk about the TV OS UI. I mean, how truly different is that from iOS UI and tell me about the process we're putting it all together and making it work and all of that?

8:13It's actually very different. Not only the UI components that we have to use programmatically are quite different from the mobile app, but also the UI itself, like it's tailored for the form factor of a TV. And it's very similar or exactly the same as our flagship TV app that we build for other platforms, TV platforms. Based on web technologies. So on mobile, you're relying a lot on like tapping things and swiping up and down and left and right. On a TV, you have remote control. And you have what they call a focus engine, where you go down, it needs to find the focus engine is to find where the focus should go pretty much. And you need to accommodate for this kind of interaction patterns. Then the cards are different. All the UI components are different. The fonts, yeah, the fonts are the same. Size is again are different. You can imagine, like, I would say 90% of the UI is completely different from mobile. We only reduce from mobile the themes, the fonts, how we load images. And these are all the same between iOS and TV OS.

9:15Because we are using the underlying Swift UI components that we already have. Quickly on the side, you mentioned our apps for other TVs that are based on web technologies. So we have similar TV apps for things that aren't Apple TV and that's written in a completely different framework. Exactly, they're all written with web technologies and react. And they were built in a way where we wanted to build it once and sip it to many different platforms. And one common thing that all this platform had was a browser environment. So we could package our app. In a bundle, the bundle had access to a browser and then we could load our web app in there. And then we could scale that across multiple TV platforms. LG is Samsung. So you build one feature once and sip it everywhere. And fortunately, Apple doesn't allow us to run a web page into one of their apps. So we had to build it natively. Yep, that's really interesting. So I guess the app on other TVs is effectively a web page. And I guess it's like web player, but ported with TV or UI.

10:16It's pretty much the same playback library that we use for a web player. So it uses all the web technologies, like the video tag and the audio tag in order to play video and audio. Cool, so then tell me a little bit about the app itself. Once you build all of this stuff, what are some of the awesome things that the new app does? The app does pretty much whatever the mobile app does. So for example, we brought AIDJ. We use the service that allowed us to control it. Obviously, it does video. It's a video first experience. We have lyrics and we use the same services that we use on mobile for lyrics. We have jam, our jam feature also. And then it's home. We have search and we have voice search also. We have our normal and as we call them entity pages, like an album page, plainly spades, podcasts. Yeah, I would say pretty much everything. I've heard from a few people that you guys made us to apply some kind of newer, fun AIDJ tools to this. So I'm curious how much did modern AIDJ tools and coding agents and all the fun things that we talk about these days

11:16help get the SAP out faster. We use the AIDJ tools and they helped a lot. I've been using a lot, Cloud Code. For example, my colleagues have been using Chatsibit also, some extent. But it helped a lot in specifically the beginning, like spinning out a UI. What we call the now playing here. We gave it a Figma design, through an MCP server. You got the Figma, look at the design. And it speeds up some code that at least we could start using the app. The agents are really good at going through the code pacing and describing it to you. What I mean by that is that there were a lot of components, a lot of services that a lot of us we haven't worked. Like the team I think was when we started at five, six people. And the mobile app has been built by hundreds of people. So we had to bring a lot of these services and these APIs and we had to understand how they work. So we threw the agents at all these systems, asking them to document how these things work. A lot of times, specifically, the AIDJ

12:18and will give me a dependency tree. They will tell me which services doing what APIs are exposed. If I wanted to ask, how can I do this thing? It will go into the code and will find out if I can do it or not. If not, it would suggest a plan of how I can expose something that I want to do. That saved a lot of time. It's not something I wouldn't be able to do myself, but it's a very time consuming process. So the agents were the catalyst for us to be very fast and being able to release enough from nothing to 100% of the users in less than nine months. Usually, a lot of the services that we wanted to reuse from the mobile app, they're owned by other teams. And all these teams have a backlog. They have their own priorities. So as coming and say, hey, can you spend like a week trying to enable X, Y, Z? These teams didn't have the capacity to do that. So instead of us being blocked by that, we use the agents again to do the necessary changes that we needed.

13:18And the only thing we ask from them is, hey, can you review these changes? And that enabled us to release fast and not being blocked at the end of the day. So I would say agents really helped us get to where we are as fast as we went. Yeah, we got there, as we got there. Yeah, that's really interesting. I mean, it sounds like the places you really focused on were around things like places where you needed other people's help and didn't really understand on your own. And in here, you could have the agent kind of figure a lot of things out for you. And then some of these kind of more mundane, like digging through things that just take a lot of time but don't require an incredible amount of like your own human creativity and expertise. Exactly, yes. I guess two other things that jump out at me in those categories, like a lot of testing things. I'm curious if you got a lot of help from agents and just knowing that all your stuff works are getting through the UX type things, making sure that things were right. Yeah, we used that a lot. Specifically, one of the improvements we're doing right now and how we're using the agencies is that we had to enable our testing infrastructure specifically

14:20for TVOS because our testing infrastructure assumed that we're testing on iOS devices. So all our N20s or our unit tests, they were targeting iOS. So we use the agents to make sure that the testing infrastructure supports TVOS. We had to go and change a lot of macros. In general, do a lot of foundational changes in our testing infrastructure and we did all these things ourselves. And then we went to the testing infrastructure team with a PR and a document where we explained all the changes and why we did them and we said, hey, can you review this? And they reviewed it. And now we're throwing the agents saying, hey, here's the description of how our products should work. Can you please create the end-to-end test for it? And the agent with a lot of context, obviously, is able to write end-to-end test for us. Yeah, that's really interesting, but that also sounds like a thing that's applicable way beyond this thing. Like I guess going back to even the main mobile apps

15:22that sounds super useful. Yes. And we want to adopt that kind of waves of working in more, more places. That makes sense. Are there any kind of fun pieces I didn't cover, like particularly challenging or exciting things along the way? No one problem you had to solve that I haven't covered yet. Yeah, I think another thing that we did with agents that I think is really interesting is how we kind of pixel-perfected our UI by using the agents, every engineer who builds UI components and composes UI and features. What I have to do at some point is look at the design specs, let's say in our case, they're all in Figma. Then look at the up, compare the two, look where things are different and make sure that the UI kind of matches the design specs. And we're talking about pixel perfection here. So this is not a difficult task, but it's a tedious task. Takes a lot of looking at the code, looking at the design specs, kind of tedious. And for me, a bit boring also. So I wanted to see if I can make it automated

16:27and be faster when it comes to that specific process. So I found out that by using MCP servers, we can actually first control the TVOS simulator. So we can have an agent that can say, compile the app, install it in the TVOS simulator, run it, and then navigate, swap, tap, swipe, and navigate around, take screenshots, look at the screenshots and make sense of them. So I was like, okay, half of the thing is solved. Now how can I give to the agent the design specs, the actual designs? And Figma kind of solved that for us because it also exposes an MCP server. And I'm like, okay, perfect. Let's connect the two MCP servers to Cloud Code and give it some really nice context and see what it can do. So first thing I did was after I connected the two MCP servers, one for controlling the TVOS simulator and the other for reading the design specs from Figma, I gave a nice long prompt to Cloud Code,

17:27pretty much told it, go to the home page, look at that specific surface and compare it to the design specs and tell me what did you find that is different. I wasn't expecting to work the first goal and it didn't work on the first goal. But the reason why it didn't work was really interesting because the MCP server and the tool that I used, an open source tool from Facebook that I used to control the TVOS simulator was built for, again, mobile, not for TVs. So it tried to tap or swipe and the simulator crossed every single time that I tried to do that. Then I started digging into the problem. I figured out what it was. I cloned the project and open source product and I through clouded it and I said, hey, this is what's going on. Guide me, tell me where I can go and fix that problem. Where can I go and start supporting the keyboard arrow navigation because that's how you control the simulator. So cloud code described to me like where all these things are,

18:28did the necessary changes, rebuilt the whole project and then made the MCP server talk to the newly built CLI tool with the changes. And then it exposed these new tools like arrow navigation, for example, to the MCP server. So now when I restarted cloud and I gave it a prompt, cloud knew that, okay, don't use tap or swipe. Use the arrow navigation. And then it started navigating, took screenshots. It's so it looked at that the focus has went from one component to the other and figured out, okay, everything works. I can navigate this thing now. And then it went to the homepage, took a screenshot, went to Figma, downloaded the home specs compared to like a visual diff and then it told me, okay, your border radius are off. You're spacing between your cards are off. And then it also knew where in the code all these things were. So it went in, changed the code, then rebuilt, recompiled the app, started the simulator again, took a screenshot,

19:28navigated home, took a screenshot, compared to figure out some new things, did it again. Third time is a charm and then it worked just fine. It was really interesting to see how all these different things work together and how if we give agents a set of tools specialized to do specific jobs, the LLAMs are really good at orchestrating all that work. So that was a really fun project and saved us maybe two hours per surface of a developer going back and forth, comparing the UI to, it took five minutes for the agent. And in this five minutes, it also included compile times and bootstrapping the simulator again. That's super cool. I'm picturing you, like leading back in your chair with a cup of coffee, just watching the UI, it iterated itself. But yeah, it was exactly that. Awesome, well, this was really cool to hear about. Thanks a lot for joining me. This was a lot of fun. Thanks for having me, yeah. Aside from all of the stuff you do in PPX, what else do you nerd out about?

20:29Oh, well, I picked up the guitar a year ago and I'm doing that every single day. So I'm learning how to play the guitar. I'm doing music theory. So I'm a bit nerding out. I'm just a theory and all these things right now. That's great. That's a very appropriate Spotify answer. Right? Yeah, I think so. Thanks for listening. The new Spotify app is available in the Apple TV app store today. Go check it out. Nerdotted Spotify is produced by Spotify Sarah Tazari and by Cplan Armada, who wrote our pixel-perfect theme song. I'm Dave Zolatowski. Thanks for nerding out with us.