Inside the Android Bench & AI LLM Rankings with Matthew McCullough from Google

0:00This episode is brought to you by Capital One. Capital One's tech team isn't just talking about multi-agentic AI They already deployed one. It's called chat concierge and it's simplifying car shopping using self-reflection and layered reasoning with live API checks It doesn't just help buyers find a car they love. It helps schedule a test drive, get pre-approved for financing and Estimate trade and value, advanced, intuitive and deployed. That's how they stack. That's technology at Capital One Howdy, howdy ho and welcome to fantasy fanfellas. I'm Hayden producer of the fantasy fangirls podcast and your resident lover of all things Sanderson and I'm Steven your bookish internet goofball, but you can call me the smash daddy and we are currently deep diving Brandon Sanderson's fantasy epic Missborn, but here's the catch Steven here has not read Missborn before that's right. Hey, hey, so each week you'll get my unfiltered raw Reactions to every single chip and along the way we'll do character deep dives magic explainers and Steven will even try to guess What's next spoiler alert? He'll be wrong news flash. I'm never wrong episodes come out every Wednesday and you can find fantasy fanfellas wherever you get your podcasts

1:23Well, hello, this is Android faithful here with a their holiday special for you Did you not have enough green this week on St. Patrick's Day? Well, guess what we got some extra green on the inside folks for you I'm wind wet down and I'm Ron Richards. We're back when we didn't get enough this week, didn't we? We did not we did more No, but we're suiting out exactly you're making up for lost time No, but we're super super excited because we're here for a special bonus episode extra episode this week Because Google came to us and said hey You guys want to you guys want to have a conversation with a Googler and we're like of course we do, right? And especially when it's one of our favorites, right? One of our favorites. Yes Very very good friend of the show Matthew McCullough VP of product Android developer experience three words, which of course I appreciate very much Yeah, no, so recently we covered it on the show when they announced that I think about a week and a half ago or so When the Android developer team rolled out the the new Android bench along with those

2:25LLM rankings And we joked when you were on the show, but but Jason poked fun at it because those LLM rankings had Gemini at the top And we're like oh of course Gemini is at the top And so but but actually as we dug into the information a lot of like I just thought this is like another like oh developer docs And how to use AI and you're kind of stuff But as I dug into it, I'm like oh, they really did a lot of work here in terms of like Guiding development in the proper use of AI and selecting the right tools And I was really curious when your perspective as a developer when that announcement roll out what you're take away from it was Yeah, I mean if you've been following me on DTS and here you know that I'm not I'm not the most let's say I am not the biggest In general sense AI stand in the room. I've been pretty critical about seven and I I've definitely railed about stuff every day So of course when I saw I I have to admit like I tried to pay a very hard to pay attention especially since you know we I do I do we do know eat personally and I'm personal other people that that that working it

3:26But it's there's a lot of sermon wrong along with a lot of innovation happening right now So I have to admit when I kind of saw it. I was like okay, what's this um and I have to admit like especially after reading about it And especially you know talking to Matthew this feels You know very androidy. It feels like a lot of things that you know It feels like android is listening. It feels like the team is listening. It feels like it's taking our feet It feels like they're trying to do something to address real problems and real feelings and real concerns about you know Like how do we integrate this new technology in our lives in a in a successful? But also in a in like the best way possible for both products and human beings and so Yeah, I was a little skeptical at first and certainly yeah, it was kind of funny that Gemini at the top I did I did kind of like oh, that's kind of funny But but I will say that I I I'm still I'm still kind of like very I wouldn't say bear I can be pretty bearish about this But I do say I have a lot of respect for this project

4:28So yeah, that's so that's glad to hear that and your perspective and so when they when Google reached out and said hey Do you want to talk to Matthew? Of course the first answer is yes Because we just like But then the second answer was like yeah, this is an opportunity to really understand like why are they doing what they're doing What is the motivation about it? What is the approach by it? And then you know, honestly, I was like you know How did Gemini become first? I want to understand that and so we we definitely you know like as we often do in Android faithful We don't pull any punches. We're gonna ask the tough questions So if you listen to the interview we do ask Matthew how did Gemini become why is that on top and and he gave a great answer As well as giving a ton of context and behind the project and the approach And the dedication that they're doing to it and again, I think that they're you know the the Android development team is doing a great job in terms of you know really Building resources for developers and trying to build a best-in-class kind of approach for the platform And I can listen to him talk about it for hours, you know, it's so much fun. So yeah, so enough of that Let's get right into it We hope you enjoy this special bonus episode of Android faithful

5:29In this conversation about Android bench and the LML LM ranking system and just enjoy hanging out Matthew McCullough for half an hour So yeah, and so we'll be back on Tuesday. Well, when we'll miss you You'll be gone, but but flow and Jason. I will be back on Tuesday and yeah, enjoy this interview with Matthew Well, welcome back to Android faithful Matthew McCullough. It's great to see you again Good to see you when Ron. It's it's always wonderful to talk to android faithful Whether it's the official current or the ones that I meet when I'm traveling around. Oh, I'm and I'm both of those right so Yes, you're two people With Matthew you are returning champions. That's always good to have you back on the show and and this time around We're talking about all the exciting news that recently came out earlier this month around Android bench and the work that you guys are doing with LLMs and AI and all this exciting stuff We covered it on the show when it was announced, but we had the opportunity You wanted to come on and talk more about it and we of course jumped at the opportunity of course just to hang out with you

6:30But we had a whole bunch of questions for you So let's get right into it So the first thing I thought of that I wanted to hear from you was that you know before Android bench kind of existed How were developed Android developers actually making decisions about which AI model to use and for at least from your perspective And what problem did that create that you guys thought that Google really needed to solve like why do this? Well, I think we're all devs here and then part of this is no hand raising today for sure But you know, we like to tell ourselves that we're rigorous and we use metrics and we we've got you know a stack rank of our choices And then sometimes we get busy and the answer becomes whatever's at hand Or whatever we have closest at hand and we really started to see that you know Happen or people just reaching for the thing that was already corporately subscribed or whatever they'd used most recently But that doesn't necessarily mean that that was the very best thing For their Android development work and so what we decided to set on a quest sponsored by our our leads in the organization

7:34Is to give people a way to rigorously and repeatedly quantify which one gives them the most benefit That makes a lot of sense can can can concur engineering tends to be very data driven So it makes a lot of sense to have a tool that allows you to analyze and quantify what might something make better than the other um The so the next thing I want to ask you which is honestly a quest especially as an Android developer myself a question very dear and dear to my heart that perhaps um I might have asked um, but you know that there are already plenty of general coding benchmarks are at there You know, we're like several years deep now into kind of like this new advent of like AI as part of our workflow And as like the broader culture and you know There are already general coding benchmarks out there But what specifically about Android development made those the existing coding benchmarks fall short And can you give us a concrete example of some gap that they just didn't quite cover Can indeed I think they've really helped so to anyone who's contributed or built one of those existing benchmarks

8:34I have nothing but appreciation it is driven in industry that I love like forward and and more helpful But much as we know that there's the difference between coding for back end and front end and full stack You know the phrases from the last decade or so there's definitely a difference in coding for JavaScript type script React type stacks versus that of native Android development and there are so many benefits You know all the innovation that we're bringing to the Android platform even like with Android 17 We want to make sure that all of that is making it into the hands of developers who are using AI tools for their coding Not just some sort of last year's tech or three years old tech but specifically the newest stuff And so you know, there's everything from I could numerate a very long list when but it could be you know Compose it could be Kotlin making sure we're using the latest language constructs It's making sure that we're using the latest libraries or even performance techniques You're seeing posts all the time from the DREs in our organization many of your friends I see when we're at conferences. They're always posting tips and tricks

9:35And we don't want the lowest comment denominator or mid I think that's what the cool kids say like But we literally want best practices best architecture best libraries best frameworks latest versions And if you want all of that It's quite the shopping list. I think most of them are going to go tell you you got to go do this yourself So we built on the best we got inspired from the ones that are already there But this is unabashedly Android native development boosting Full stop I just want to comment real quick that that sounds like very encouraging to me because I think you know And I think this is a bit ago, but I know one tip I heard once was like well, you know One of the best ways to optimize it is to use architectures that you know are older architectures that the that you know LLS might be familiar with which Which understandable but as you say that that's not I mean Not to hate not to judge, but that's not how we roll in Android and and that ability to stay like adaptable and flexible and kind of take advantage of

10:38All of the new ideas coming out from your team as well as like people in the community. That's so important So thank you for saying that And in some ways it's kind of funny because you what we don't want to do But it can happen is essentially to be beholden to the automation like well the automation prefers and so we're gonna do Sorry, we want to be best latest greatest most innovative. It needs to serve us So to some degree maybe that's the subtitled mission here is making sure that the automation is serving what we aspire to do Well, and that's really interesting because and it's great because you know you guys are in the weeds on the development side And I have so much respect for you Matthew and your team and win and the work you do and I you know I always joke that I'm not a developer, but I know enough to get me in trouble and I've found myself in a career path where I'm Product doing product management and working with the developers and needing to walk the walk and you know And it's really interesting to hear you say that But my first thing I think of is like okay, well if you're doing it in that sense How walk us through how you actually do that you know like if you're sourcing real issues from GitHub repositories to do this

11:43How do you ensure the tasks are generally testing android expertise rather than just general coding ability like like how How do you make sure that it is serving your guys need versus just general development? Well, I like to kind of draw the illustration of a sales funnel to some degree if anybody's working in a place where they've Had to deal with that where you you've got a lot of option at the top right? You know what are we going after what are we reaching and then we kind of start making Hopefully very intelligent decisions to come down to the set that that makes a lot of sense for us Let's just say with the amount of Android code and what's on GitHub the the top is very wide and we finished with a hundred curated tasks So the steepness of that I don't know what this logo of that line is, but it's very very steep and a lot of energy and a lot of decision Went into choosing those and and they're coupled that you know rubrics that we used for this work One it needed to represent best practices. We covered that a few minutes ago. So that one's already in there

12:43Second, it needed to be modern approaches. It's sweet. We're we're not looking to drive Classic approaches as much. I mean those are great, but we're looking to drive modern approaches So compose for example compose only for the approaches that we've got here And then third they need to be looked at by an expert to make sure that these are Quality changes because when you think about it ultimately what we're doing is is helping train these systems on what great looks like And I want to emphasis that word not what mediocre what compiles That can already be probably be done. What great looks like I want to set the bar extremely high Let's just say there was a lot of stressed people when you set the bar at at great There are more stressed faces at times because it means reviewing every single pull request every single one of the tests Line by line trajectory by trajectory change by change. So you know what somebody said and I get it They're like a hundred I can crank that out in no time

13:45But I feel like these are like a hundred FaberJ eggs and so maybe if he is that kind of mental model Hundreds a lot in that particular case And this was done by GDE's by sui's on the team by product managers by folks in the DRE team Couple of outside consultants that we also use to just get all these different lenses And all of these tests were examined by that expert set Fascinating Yeah I that I mean especially given and it is really encouraging that you know you source, you know Issues from GitHub repository So if you're not familiar watching this in your somehow not familiar You know, I think in the Andrew community one of the things that we have built an identity on it And our most proud of it is our wide network of open source solutions that This entire community not just as contributed to but has become foundational to a lot of the work We do so it makes a lot of sense because we get our best practices Our best ideas of what is great from the project is it I think it made a lot of sense As someone who for better for worse you probably for better

14:46I should say that has worked that on a lot of large large large scale projects and an enterprise environments where You know Especially given its proprietary so we you know unfortunately can't always Talk about or show the practices that we do because of more enterprise hugely scaled projects or you know Just even like the unique challenges of kind of being on a large scale Massive project with a lot of different kind of gears working and a lot of different kind of more enterprise The no concerns can you can you kind of give us like Kind of a any any thoughts on like how well those android bench still reflect that you know given that you know Sometimes enterprises a bit different than open source It's it's a difficult task. I think we did a really good job on it But it's extremely hard, you know We already talked about the steepness of the funnel, but one of the other elements since you ask When about this is is also looking for at least 30 or so tasks. We got the 29 That that had large quantities of code change and I mean a hundred lines and there and the reason is it comes back again to this example of

15:48Well, there was already sweet bench and other things like that You know was able to emit Kotlin code It sometimes you could coax it to emit compose for some of the leading models But not necessarily in the volume that we're looking for for that enterprise or professional work So exactly on your question we need it to do more for us And I think for professional developers to really feel the benefit where they're saying like oh my goodness This is transforming the work day It's helping me be more productive. It's got to be larger quantities of code So that was a specific lens that we used in some of it and so just to re emphasize there's 29 if I remember correctly that have at least a hundred lines of code change And so you're talking about big sets in bigger code bases Now the difficult part about this is given that this first pass of the benchmark And I think we might have time to get to it to talk about version two We'll see if I get in trouble for sneak peeks But for version one, it's all open source and so there's also this tension that these are not Necessarily commercial projects like you might have been contracted or employed to work on in the past

16:54And we have some Has coming up to even add that to future versions So I'm already proud of where we got for this for the answer But maybe it's usual Matthew style. I'm never fully satisfied. So we're gonna try to do more Okay, oh, okay, sorry. Go ahead Ross. I mean, I'm gonna say in the world of the technology development The work's never really done, right? I mean, there's always the first release is the first release So then you build from there, right? So one point. No is a Friday and probably nothing more like oh I will say after covering this stuff weekend a week out we have been commenting I don't know if you listen to show we have a commenting at the relentless pace you guys have put us on in terms of Covering the the output. So I'm not surprised to hear that On the core core, but our job is to make your job difficult by the relentless pace of what we put out good It has been nonstop Well real quickly one one quick question for you, you know, similarly to that question room and health are you know kind of a pity This episode is brought to you by capital one capital one's tech team isn't just talking about multi-agentic AI

17:59They already deployed one. It's called chat concierge and it's simplifying car shopping using self-reflection and layered reasoning with live API checks It doesn't just help buyers find a car they love it helps schedule a test drive get pre-approved for financing An estimate trade-in value advanced intuitive and deployed. That's how they stack That's technology at capital one Howdy howdy ho and welcome to fantasy fanfellas I'm Hayden producer of the fantasy fangirls podcast and your resident lover of all things sanderson and I'm steven your bookish internet Goofball, but you can call me the smash daddy and we are currently deep diving Brandon sanderson's fantasy epic Missborn, but here's the catch Steven here has not read Missborn before that's right. Hey, hey, so each week you'll get my unfiltered raw reactions to every single chip and along the way We'll do character deep dives magic explainers and Steven will even try to guess what's next spoiler alert He'll be wrong news flash. I'm never wrong episodes come out every Wednesday and you can find fantasy fanfellas wherever you get your podcasts

19:01You Gnated standard proposed by google coming out of this, but many projects actually use alternates You know and and many developers use alternate storage and di frameworks How much a team's factor that in when using android benches a reference? You know, this is a place where we're a little bit more thoughtful and the thinking may evolve here So I'm gonna give you the current mode of thinking but it may evolve over time I think something for the last four years that you know my team is supported me on and I've had as kind of For vision for android is there's a balance between being opinionated and not having options And let me just tease that part just a little bit Android is the land of tons of options. You can usually find two three four five maybe ten different options for a library that suits a particular need Language parsing image parsing animations and the like I think that is part of what makes it vibrant and attractive to developers at times But when they're on their learning journey or they just get it done because they're under pressure There's also a nice desire to be helpful with a we've tried we've used this one looks good

20:07And that is a very difficult balance So openness so that you can make choice But a little bit of opinionation so you can actually like what if y'all used what seems to be working for for the for the industry So we've done that balance in this We have not taken any strong stance in the selection of tasks for library or preference for maker or for author foundation That one is when we've largely kind of put gently to the side and looked more at architecture and top-level choices With the exception if you want to call it a library of compose. I believe in that so Religiously that you can't unsell me on all the you know miraculous benefits of compose But aside from that we did not use library choice as a filtering concern Cool all right, so one of the big things coming out of this announcement was the the the the the kind of the grading of LLM's and these the The benchmarks and all that sort of stuff helping people guide are they AI

21:08One of one criticism of public benchmarks is basically that models can cheat by training on test data How are you guys guarding against that with Android venture and how are you going to stay ahead of that as those models evolve? You can you can take a couple of approaches so we'll peel apart in three parts. I think number one Developers in general and Ron We're gonna for this point for just like you're in the developer community now that you're AI enhanced in a product manager You're in team developers and no more no more team out And so I think for for all of those folks, you know You're thinking about the fact that software development is Definitely you're training on all the the open materials that are properly licensed to be able to do so But you know in general people will with open source. I think when I'll come back to yours You know well when we're learning we'll copy and paste something and we may not have always like followed to the T The license there and so there's just risk in industry that open source code gets

22:08Transmagnified from one code based to another and essentially we've designed for the fact and expected That's this this benchmark will eventually find its way into some of the models that are out there And so I just want to make it labeled completely clear We expect that to happen over time. It's just the nature of industry But the second piece that I wanted to bring to this is that we're already planning it's a little bit that joke of You know version 1.0 is any given Friday Well, we already started planning and we're working on 2.0 before 1.0 came out of of this benchmark And what you'll see is we'll evolve using some of the the wisdom and best practices of sweet bench and sweet bench pro If you're familiar with those two They they have a second set effectively a second wave of More closed evals that supplement the first so a yin and yang if you weld this on degree and we're going to go that direction with some of the additional Elements that we add to 2.0 that also plays to your earlier question when I didn't I didn't know we're going to go this route But it's the end of this is it's the enterprise approach. We may be able to have some licensed code bases that we're able to use in evals

23:12But we may not be able to make open source so they're going to contribute to the wisdom the quality of what the models are able to do But they won't necessarily be code bases that are out in the open and so what you've got is for the first round You can test it yourself evaluate yourself look at every single line of what we've selected So from a trustworthyness we gave it all to you. It's all in the open And then after we booked the ability for v2 so that we can kind of stretch our arms a little bit We're going to incorporate some code that comes from other sources that we made license to make this even stronger Okay, well picking off of back up back off of that idea, you know like any good party That lets you bring your own beverages android studio out of three launch bring your own model back in January You know, which if you're not aware let's developers plug in claw gpt and like other models And and so how does android bench complete that picture is it essentially like you know like a buying guy That comes with the open marketplace Definitely not a buying guide and and on top of it, you know, these are changing every single week

24:13I mean if you look at just the relentless pace of model releases I think that you know who's on top who's in second place who's fifth Could change week to week to week and and probably is there's new model releases and there's improvements to existing ones But I think even more importantly is the philosophy behind this and not that it's just a buying guide We have we have two goals for this one to be able to as we begin the show I think I'm giving developers a way to more empirically make choices about what applies to which scenario I mean models cost different depending on whether you're using pros or ultras or some of the nano light, you know kind of super Efficient kind of bits and there are certain scenarios like maybe just code completion where the light models make sense But if you're refactoring you're most important app in your company and you are re-architecting everything about it and wanted to run I feel like that's the time to choose the thing that is currently on the top of the list That's that's where to pay those dollars for the tokens So this is not just a which one one and done

25:16But you could almost say this helps you make discrete choices for the scenario the use case the automation the volume that you're working on So you make choose you may choose farther down the list for cost efficiency because it gets you what you need to And then very lastly is I had the most delightful times This is a rare privilege of a hat to be able to wear in Google I've been able to meet with lots of the large Model makers and like build really good relationships. They're scientists and their researchers and The great part is that everyone that I spoke to is on team android Even if they don't wear a Google branded, you know, propeller hat to some degree And so I think what's super exciting to me is this means that every major model maker is pulling for android to do better And it just feels so so like the android vibe and culture in that way We love that vibe. We love that culture So I got to address the elephant in the room at you, you know, we love the approach. We love the whole thing But looking at the leaderboard

26:18Gemini's on top and some people might raise an eyebrow at the fact that Google's both running the benchmark and Saying that Google's AI model coming in first How are you guys maintaining credibility and trust in those rankings over time and some skeptics might wonder are you guys tipping the scale and Gemini's favor To push forward Google's, you know, LLM in this case The team that puts this together is on team android to just be super clear The second piece though is, you know, that's what hat we wear, but then you know, what are we doing All open all in the open source if anyone is currently a skeptic We put it out there so that you can run it yourself like, you know, be that person that goes and runs the harness Runs it n equals 10 runs it across the models that you care about including even once that are not on our our leaderboard I mean, there's only so many we can do there are a lot of model choices out there today And then go look at what your result is so I would simply say like we did that for the sake of efficiencies to somebody can pop up in a webpage

27:20And see, you know, what we got we put hundreds of hours of labor into doing that But if you need that confidence of getting your own answer We gave you all the source materials all the harness all the tests and then go produce your own result Pretend that graph doesn't exist and go create the graph you trust based on exactly the same test Evalon harness that we've got But that would require someone actually doing work and it's much easier to be snarky unread it than to do work I said there was no red lines for today's conversation, but maybe you found one Will we have off that proclivities for work kind of Okay, well, um next up let's kind of drift a little bit away from android bench and maybe look more kind of a like maybe more Kind of top level because you know these mentioned like You're for yourself a lot of goals and a lot of people talked about kind of a good future and what the future of app smell My look like so for me as someone with boots in the ground We're having a lot of discussions about quality and human review at scale and you know It really does come up these days, especially with AI and you know on one hand for maybe like the broader public

28:23There's the growing trend of micro apps where apps are written buying individual for their own use And it feels like a lot of what's been talked about especially kind of the more kind of commercial Oh regular person spaces Um, but the working concerns of like production projects from like you know tiny to large can be quite different again As an enterprisey working person these are the kind of things that I often think about and you know There's there's different you know requirements in terms of security quality and scalability and it can be hard Especially for someone like me to kind of bridge that mental gap between the two kind of projects Like the two kind of threads maybe in the conversation. I want to ask you Matthew. What are your thoughts on what Us as an industry need to do to fill in these gaps to make AI trustworthy and productive at scale while maintaining you know high quality and safety Yeah, it's for using this. I think you know there's a lot of emergent techniques I don't claim to have the full list But I have a couple things that I hear from industry leaders are working extremely well One when we talked about multi model in this particular case One of the approaches is actually

29:23LLM as it as a judge and I use that term very loosely not the traditional sense for it where One will write the test. I mean I even have a harness for some of my work at home where I use one model to write the code And the other one to write the test and then they flip and critique each other and you know Yes, I'm subject to even writing the silly prompts Your future use depends on how good of a job you you you do in critiquing the other And so I think one of the interesting things that that you know comes from a venture like this is you could even choose a couple of the models to be able to Cross-check each other because they just have such different training techniques and sometimes different behaviors that are really good to To check one another I especially did that around area any area where I'm looking for a performance or best practices Architectural or security concerns and you know, I am definitely not on team expert of our Android team I mean we've got luminaries who've been in for ten years So I'm you know far far more in need of this double check than maybe some of our experts in this case

30:25But that's been one approach and then the second is we're actually open to recommendations For folks to send in we've got a kind of the usual paths through GDEs and the like but also some air contacts on the DRE team For where you'd like us to see putting energy on the V2 point out if you're like hey is I'm using model X model Y model Z I'm feeling like I'm not getting secure approaches performing approaches etc Please send us those signals because we're going to use that we're going to work for about another couple of months on making sure that we have the right plan for 2.0 Even as we're coding and I'd love to take that feedback into account so that we're essentially making the areas you need stronger Stronger through a result of that at the benchmark Well as someone who has a lot of opinions. I really much appreciate that But no and I mean seriously that's it's very encouraging because I think that's the one thing that feels sometimes Drowned out and everything is the lack of voice and ability to steer the course of you know What is a huge movement right now and a huge way of so and when one thing we already did is you know

31:25We have effectively a steer cove of wide range of companies But who come together with us virtually and in person a couple times a year too Essentially be our our steering voice for Android developer you know representatives You've you've participated in that many a time and I think those also we quantify those results We make sure we double click into them and that was already used for some of the decision making in v1.0 So that's another channel as well that we're getting from you know from those on the ground developers who are Refectively leading Android in industry Excellent well So in some of the and you know, we're coming close to the end of our time together But in some of the materials that came out at the announcement You guys mentioned you know the long-term goal of a developer being able to build any app they imagine an Android Right, which is you know a long-term dream of mine I've got windows. I have an app idea that I've been wanting to make for years and Someday I'm gonna find the time when my kids will leave me alone I'm gonna vibe coded. We're gonna use Gemini. I'm gonna do studio. I'm gonna do the whole thing

32:27Something to do where does AI need to in your opinion? Where does AI need to get to in terms of benchmark scores and capability and all the stuff you guys are working on before that Generally becomes realistic and how far away are we from that moment from your point of view Not very far. I feel like users can already touch some version of this I'm not asking people to trust me very much at all if you look at Panda 2 and download the new project If you use the new project agent and download Panda 2 I feel like you already Get some confidence that we're right on the the cusp of this being possible It's super exciting. I mean there's quite a bit of Twitter traffic I don't know if there's a specific hashtag for it, but for people Jenning up ideas with with Panda 2 and new project agent that are Composed best practices good architecture come with the test suite have really good look and feel use material 3 expressive Like these are a lot of really good practices. This is not you're not having to compromise like well It's sorry. It's not native. You know doesn't look really good. It doesn't use the best like no no compromise by default

33:27So we're on the cusp But I'll tell you that even with just a little bit of extra prompting still a little bit of that developer insight But maybe not hands-on developer People are able to make some really impressive apps and next time that we're together ask me about the progress of my friend Grant who has a teenagers along with mine who are in driving school and the like and the app That's out there could use some help and he's like what let's do this This is the era like this is the era to like write a better version of this app and It's really good. It's cool Really good and so I think this is like my own personal validation loop not industry not Twitter not some blog post or something like that Like my friend I was like I like this And so then my daughter was writing in the car with him and had an idea and as soon as he got back home from like you know Being transported to their high school activities and stuff. He was like hammering out the extra feature And to me it's not just this this app to focus on but it's the excitement that we have new people

34:30Raising the bar in apps that are going to be like, you know every day kind of use not just you know single individual use apps And so to me like investing in Android bench is effectively a Sirtoidist way of helping people like Grant create better software for people like my own daughter Oh, that's awesome. That's what you that's what you want to see right it's it's it's it's Adding to the society effect and change and moving forward the innovation You know, I think about as a kid in the 80s learning basic for the first time and feeling like you've unlocked a whole new world And like I you know my kids are seven. They're not quite you know They're they're just starting out on and I think about the tools. They'll be able to do in there your kid you could say She like that's gonna be it's gonna be a lot of fun and you guys are laying the groundwork for it. That's awesome So well Matthew. Thank you so much for your time Are there any other things you want to leave our audience with any teases anything we can look forward to or I know Google Ios around the corner. We're looking forward to seeing you there I'll I'll be careful about the Io teaser bits, but there is exciting stuff coming We will we will not leave you bored for that piece. I saw we skip you that promise

35:35And leading up to it and the second is you know on something like this It's always important like oh one and done Android bench 1.0. Well, it's great. You know, it's been great shipping We have the same team plus expanded plus more funding to keep going And so I think that part that I want to leave you with is like this This is the start. This is not the finish that we're talking about today So please give that feedback. You know when you were asking like please send it with through any of the channels Whether it's social or otherwise some of the chance and the feedback on what should be in it and Effectively stay tuned for models to really if you already think they're reasonably good I think over the next six months you will see step changes in Android capabilities Due to this and so if you're part of the Android community Enjoy and you know get derived as much benefit as you can from this in in building some amazing apps for our family Our partners our kids Awesome. Well, thank you so much for your time. We really appreciate. We'll see you. We'll see you in May at at Google Ios

36:35Thanks Matthew then by when by run You This episode is brought to you by capital one capital one's tech team isn't just talking about multi-agentic AI They already deployed one It's called chat concierge and it's simplifying car shopping using self-reflection and layered reasoning with live API checks It doesn't just help buyers find a car they love it helps schedule a test drive get pre-approved for financing and estimate trade and value Advanced intuitive and deployed. That's how they stack That's technology at capital one howdy howdy ho and welcome to fantasy fanfellas I'm Hayden producer of the fantasy fangirls podcast and your resident lover of all things sanderson and I'm steven your bookish internet Goofball, but you can call me the smash daddy and we are currently deep diving Brandon Sanderson's fantasy epic Missborn, but here's the catch Steven here has not read Missborn before that's right. Hey, hey, so each week you'll get my unfiltered raw reactions to every single chip and along the way

37:35We'll do character deep dives magic explainers and steven will even try to guess what's next spoiler alert He'll be wrong news flash. I'm never wrong episodes come out every Wednesday and you can find fantasy fanfellas wherever you get your podcasts