Software Abundance for Government With Cognition's Russell Kaplan

0:00software abundance for government why do we need it and how do we get there to discuss we have on Russell Kaplan co-founder of cognition who previously spent time at scale and Tesla thanks to cognition for bringing us this episode in Russell welcome to trying to talk thanks for having me Jordan excited to be here so what is wrong with software and government we have a lot of problems with software in the government despite the government being you know actually a lot of the the source of innovation in software for a long time but you know today there's the state of the world is it's pretty sad as a citizen you know you interact with software for the government and a lot a lot could be better you know I just to put some numbers on it you know there is more than a hundred billion dollars a year spent on IT for the US government a lot of these systems are ancient the the GAO did a study of finding that you know in the in the 2010s there were 10 critical legacy systems we needed to modernize less than three or I think only three of them

1:00have even started the process of that modernization and as a country we're we're spending a lot of money and not getting the same results that we see in the private sector and I think what's happening now with AI and software engineering it's it's changing it's changing the private sector but I'm personally excited about how much it could change for the country as well and I think it's actually really important for you know for the sort of the next generation of the United States to get this right you mentioned the hundred billion dollars a year number like what does one dollar get you of in the private sector and how does that kind of comp over to some federal or state departments spending that money so yeah the private sector the way we we buy software is you know we have a problem and we see okay what's the best tool in the market for that problem and we you know we buy whether it's a SaaS solution for my CRM or it's infrastructure for you know scaling my database but the the market tends to be more efficient for the government it's it's a different

2:05story it's really challenging for the government to purchase software directly there's a much higher kind of compliance and regulatory hurdle for software vendors to even start working with the government you know we we face this at that cognition getting to getting to Fed ramp high was a was a journey for us but even once you're even once you're there there's a lot of interaction which a lot of these systems were designed with good intent to make sure that there's no corruption that people are having RFP processes that let government vendors get the government buyers get the best price for what they want but the the net result of this system is that it's enormously slow to get to get software into the government and in particular to reuse software you know like a SaaS tool has a much easier time being bought by a private sector company versus a government agency which which often needs to have a much higher degree of ownership of the product they're using so I mean the net result of this if you look if you look at some of the data

3:06we're still powering most of the country look the critical systems of the country with ancient code you know tens of millions of lines of cobalt our powering our you know our treasury our social security administration and it's it's not getting better so Russell is cobalt not Lindy I mean what's what's wrong with running a government on ancient software languages well the problem is that nobody knows how to write cobalt anymore I talked about cobalt specifically is it is a problem in the private sector too but what we found is that you know that the people who wrote these systems are often no longer there when when changes need to be made and so the result of this is that there's like a small a small cohort of specialists who who kind of learned cobalt many decades ago still still write cobalt and they need to be brought in for any change but there's just fewer and fewer of them and the changes are bigger and bigger so as a result everyone's sort of scared to touch the big mainframe systems that are kind of powering critical infrastructure

4:08for the country this problem happens in the in the private sector too by the way so you know a lot of a lot of banks that we work with at cognition large health insurers you know airlines they're running they're running these large scale systems and I mean to give a credit cobalt is a very performant language you know it's actually like really efficient and really fast and and it's working you know it's kind of working so people don't want to mess with it but what what happens is that when requirements change it's really hard for for us to move with those requirements to update them and and that's where kind of the slowdown really comes let's stay on this for a second so like maybe for the uninitiated like why are there new programming languages and what can they enable besides just like having more people know what Python is and how to use it then the stuff that was invented in the 60s and 70s yeah so I mean just like really brief history of of programming languages which you know but even before we were writing cobalt you know people were writing assembly like in 1948 assembly I sort of became popularized and this was a big upgrade

5:13even from the previous era of punch cards the you know the 1890 census was the first time that punch cards were used in like a real sort of production setting but it was by the the government because they realized that to sort of count the census manually was going to take more than 10 years for the 1890 census and so they were literally not going to get the job done and so the government put out kind of a call for technology to say all right you know what can we can we do to solve this problem and and in 1890 we use punch cards and that was a big deal the 1880s baby boom amateurs is the straw that broke the camel's back too many people not enough counters yeah it was I think it was going to take like 12 or 13 years I think something doing it that doing it the the old-fashioned way but but punch cards you know it's a very it's a very little representation there's you know a hole or not a whole representing kind of a one or a zero originally as a data storage format assembly cobalt modern languages you mentioned you know Python even Java

6:13they kind of all just walk up the ladder of abstraction of making it easier to tell your computer what you wanted to do and so you have to have increasingly less arcane specialized knowledge and increasingly more intuitive interfaces of how do I get my computer to do what I wanted to do and I think AI is actually like the next logical rung on the ladder it's not some fundamentally structurally different thing when it comes to programming it's it's telling your computer what you wanted to do but in English in a way that's really natural for everyone yeah and I think the other thing is like the the older programming languages are optimizing for the constraints of the particular generations of technology so you know now before you know you had way more severe sort of memory and storage and processing restrictions and in today's languages like you have human be like you know pre 2025 or whatever you needed a person to sit down and write every line of code that you were going to deploy and that is I don't know now not really a thing so much

7:16anymore the hardware teams work so hard to optimize the chips you know to keep pushing more as law and and then the lazy software is like my software engineers like myself you know we just stop worrying about garbage collection and memory management and we we relished in the productivity gains without worrying so much about the efficiencies is kind of how it's gone you know a lot of the time you know we do get more more efficient but the the the sort of typically most of the hardware performance improvement gains capture by actually making software easier to write and that's that's one thing relevant both for the government and the private sector that AI might flip this where AI might be able to say hey I'm going to actually write this in really optimized you know assembly or binary directly because I don't need to have this intermediary interface that a human can understand beyond insanely performant code what else can we expect for in our world of software abundance I think the most important thing is that software is going to start flowing more like water it's just something that's that's easy to move around easy to change easy to get

8:21more of and and in particular that a lot more is going to be created as a result if you look at the structure of kind of the the SaaS industry and software as deployed in in government and private sector a lot of the way things are shaped is because of how hard it is to change things right it's like to migrate off of a system of a database that you've installed you've designed that's a massive massive project if you want to you know if you want to even buy another company one of the most complex parts of that historically has been the the integration of the software and infrastructure and the IT systems the different data storage so there's like this sprawling complexity that's emerged and honestly a lot of vendors who are they kind of use the switching costs as a way to build a mode around their business it's oh I'm going to you know land this land this contract we're going to set our stuff up and you know we're going to discount it the first year and then it's going to be impossible to leave and I think a big structural change that's that's about to

9:21happen in the economy you can already start to see you know some of the reactions to this in the public markets is that that that strategy doesn't work anymore you can't hold your customers hostage with switching costs when you know AI is going to do the switching and it's going to work on 24 hours a day and it's not going to get bored with what is often a really tedious process and I think that that ability to move from whatever you have to what's the best tool for your problem is going to is going to lead to a lot of changes so what's cognition doing to making that future possible you know we so we started and we started in January 24 so we're like you know a month more than than two years old by the time we're recording this right now and we started originally as a research lab focused on reasoning and long term planning for software engineering so already there is at the time great progress on chatbots but what about like making things I could think for a really long period of time and using that using that for for software engineering and then we launched Devon the AI software engineer in March of 2024 and that was sort of the first

10:28real like draft of what should an autonomous agent look like which now is of course extremely popular in software to have these more almost like co-workers these digital co-workers as opposed to co-pilots that you're that you're delegating work to and if you think about the this sort of what is the complexity we talked about of the switching costs the challenges of migrating and modernizing there's a there's an architecture part of deciding what's the problem with our status quote and where do we want to get to and that's still done by humans today that's still done by humans but once you've decided on that the execution the implementation detail a lot of times it's pretty toils some stuff it's actually the stuff that engineers really don't love to do you know paying down tech debt refactoring you know file after file of old code and so a cognition as we provide this AI software engineer Devon that people can deploy against their code to really quickly transform it improve it modernize it upgrade it and and at you know at this point where we're used

11:29by a lot of the fortune 500 by kind of global organizations but really focused on large complex systems that require serious amounts of existing context to to do useful changes well let's do the compare and concrete contrast with cloud code Rossum yeah so by the I think cloud code is awesome I think in general the explosion of developer tools in AI and software engineering has been like kind of crazy to see not not just cloud code but you know I think codex you know other sort of IDEs like CLIs the interface is constantly changing so where cognition sits is we we have a sort of like a platform we we have you know an IDE we acquired windsurf the agentic IDE 2025 we built Devon the autonomous agent biggest difference between Devon cloud code it's really are you running in the cloud are you running remote are you running locally so is this something that you can spin up in parallel in a fleet versus something that's running on your machine

12:33and it's like a pretty fundamental architectural difference it's like do you give the agent its own computer that's like the dev tool difference I think the way we work with companies is also pretty different we're less of a you know here's the tool go figure it out and you know cognition we work with again a lot of the largest most complex organizations in the world and these folks don't just have a developer tools problem they often have a like a transformation problem how do I get this major outcome done in three months instead of two years and so we've built for example a pretty large forward deployed engineering team for our for our size of company where we'll go work with the government we'll work with an enterprise to kind of partner together on driving a meaningful outcome and why is why do they need that and not just not just the tools and let it rip or maybe maybe the other question is like do we wait six months or a year and then the technology is going to be so good that like all we need is a model to just like go and fix everything

13:35for us yeah so this is kind of like the AGI maximalist case of like oh if we just sort of solve you know you just have the best possible model then like shouldn't everything else just like just happen and I kind of think the answer is no you know that I have you seen the chart of inflation by sector over time where you can see you know like plasma screen TVs are massively deflationary but you know health care and tuition is is going way up that's kind of that chart is sort of my mental model for the like the post AGI future which is all of the things that are intelligence soluble you know they get like really really deflated but sort of what you're left with is all the rest of the complexity of the real world which is actually quite substantial you know it's like first of all how are you even allowed how do you get the permissions to deploy in the environments you need to deploy in how do you work with like the people who are ultimately in charge of these systems to drive the outcomes they want to drive and and then also just like re re reframe restructure the process of how technology is is built or procured inside an organization and you know the

14:41models are going to keep getting better and they're going to make software easier and easier to create it's like all the other problems that are sort of left are left behind so we're recording this the afternoon of February Friday February 27th it's 222 pm there is now two and a half hours left before Pete Hegseth drops the anvil on anthropic apparently I'm curious Russell you know given that Devon can pull from all the different models sort of what challenges and opportunities that has you know that gives you guys from a product development perspective yeah so I mean I think if you're if you're the DOW I think you're certainly you're certainly frustrated and worried about you know the the decisions of any one model provider affecting you know the mission of what you want to do and look I think every private company has the right to say these are the use cases we want to serve these are the use cases we don't want to serve and you know kudos kudos to sort of an anthropic for saying hey here's here's what we want to do

15:44and not do but I think if you're looking at it's another kind of point on should model providers even be providing the vertical tools on top and is that the best experience for customers you know if anything we see kind of the opposite where the differentiation models is is like decreasing not increasing over time you know if you look at sort of frontier e-value scores for software engineering benchmarks the gap between the best models right now as we're recording this it's it's like less than half of the gap it was 12 months ago and as as companies are spending billions and tens and hundreds of billions of dollars on bigger and bigger clusters bigger models the models themselves are sort of converging together and so if you're if you're a government buyer you care about the outcome that you're driving more typically than like what model am I going to use and so I think in some ways it gives a structural advantage to to like the agent labs you know cognition being one of them as hey we're like focused on the customer problem no matter what models exist or don't exist we're going to kind of combine them in the best way and of course we'll have our own specialized

16:47stuff for very specific narrow use cases um but to drive the outcome you want we have a running gag on trying to talk of the AI mandate of heaven and even though it's been anthropics for a hot minute listeners will recall the world in which it was Gemini's and open a eyes um I feel you know I hear you wrestle on the um models kind of converging in capabilities but when I play with them they do feel different and you know people talk about being better at this or that thing for software like how do you guys go about playing with them and figuring out who to assign what work when we're talking about Devon yeah so on the on the on the mandate of heaven piece it's I think these things are cyclical you know and they sort of they'll one thing that's interesting in software engineering in particular is that the right form factor for building software is constantly changing based on in part the underlying capabilities of the models and so you know when we for example when

17:49we launched Devon in march of 2024 it was it was just at the edge of possible I would say to have it agent that you could really delegate work and come back and in fact honestly it wasn't even really useful for us for like another three months when between when we when we built this prototype that we shared with the world uh it took about three months for us to then use it enough internally a cognition that Devon became the number one contributor to Devon um so that was like a three months lag and then there was another several months lag before it actually started becoming deployed in production settings useful for customers and what's happening is like as as the models improve the the the form factor for how to use them is constantly changing right so when encoding we went from tab completion think you're you're writing you're writing a word doc and you you picked tab to get there at the next response but in your in your code editor to like a local chat experience where you can sort of chat with your code base and ask questions and and do like local agents to now increasingly okay we've got autonomous agents we can delegate work and by the way the form factor might look completely different again six months from now so I think the mandate of

18:52having is actually uh going to probably keep changing constantly based on who is sort of first or or best at the next form factor in every every new form factors like a new a new front uh to to to battle um but as far as evaluating the models themselves um so we built uh a internal an internal kind of pretty comprehensive evaluation suite um the original draft of it was called junior dev uh evaluating you know could these models act like junior developers uh we internally we have a fork of it now internally that's more like a senior dev because the the models you know keep keep getting better uh and what we every when we work with every lab uh to basically before they release models we run our e-vals and we give them feedback and we say hey we think you guys are strong here your week here here are some ideas for how you can make this better um and we have a great partnership with every lab about this they you know I think uh many of them have told us that we have the best private evaluation suite for agente coding tasks um that's like external to you know it is sort of the independent from a model provider um so we care a lot about e-vals

19:56because uh we you know we find our our our customers they want the best models the the the other interesting data point is that no matter what task you give the e-vals scores are consistently worse if you constrain the agent to use one model versus if you can use multiple to your point there are differences right so for example um whether it's personality or whether it's macro context understanding or details um these little differences that up that's really interesting I mean is there a structural reason for that staying true forever uh the difference forever oh yeah like like like if if you know we have our if we're if we're holding equal the distribution of like AI researcher talent and everyone still has like the same amount of chips across the you know three or four labs like is it you know is there like like what is the reason why things are spiky in this direction versus versus another direction yeah so I think I think the the sort of the structural equilibrium is one

21:00of model convergence you know the the capabilities increasingly converging increasingly similar uh to basically similar level similar levels of performance in in every domain and I think the if you look at so why why would that happen I mean the trend lines are in that direction but why why would that happen in steady state um first you have the scaling loss right so it takes exponentially more cost inputs for linear gains in any benchmark you choose right and so if you're operating at small scale it's you know it's easy for one firm to spend a hundred times more than another firm if you are you know a million dollars versus a hundred million dollars but once you're in the okay we're all spending hundreds of billions of dollars that it's hard to get a multi order of magnitude lead of your competitors so I think there's a there's a kind of a scaling loss reason that these things are converging there's also just the practical reality that you know non competes are unenforceable in California and people are moving from one lab to the other all the time I think the half-life of a proprietary algorithmic insight is probably like three months uh it I guess you know even even within the labs you know you have one person working at OpenAI

22:01and their partner working at an entropic and uh you know who know you know so so I think the the half-life of proprietary IP in Silicon Valley is is short and and so if you get to this state of like the models roughly converge maybe there's some personality differences you know not not more capabilities more personality that could persist um but I think the the last point that's relevant for every task is you know we have this this mantra in Silicon Valley that oh like we always want more intelligence you know more intelligence more intelligence more intelligence we've got to we've got to build you know clusters of compute in the galaxy uh to harvest energy of every star to have the most intelligent and I actually do think there are use cases for ever increasing amounts of intelligence but I think this also sort of ignores the fact that for any given application often you reach a threshold of intelligent saturation where for that use case it's enough you know and I can tell you for example today if you if you said hey let's let's build a simple static front-end site for China talk any model any frontier model would do that well today

23:02and so once you're at the level of for a given task of that task is intelligent saturated um you don't really care which model you're using you you care about uh okay it works so now is it fastened as it cheap and I think increasingly more and more domains are going to actually see this like intelligent saturation at which point what model you're using becomes less relevant and the interface and the experience rounded and how it kind of drives outcomes end to end for your company for your government organization matter more um all right driving outcomes let's talk about it um before we go to the the government stuff what are some uh enterprise case studies that you guys have worked on that you think illustrates what um I don't know 2025 2026 models are capable of powering them yeah the the thing that I've been the most sort of uh I guess surprise or impressed by is um the ability of these large organizations to take Devon taking autonomous agents and do massive multi-year projects in weeks or months um so I'll give you I'll give you an

24:02example um there is a law that changed recently in Brazil that changed the taxpayer ID numbers of Brazil to be uh alphanumeric instead of numbers okay think of this as like the Y2K catastrophe of Brazil it's called the CNPJ migration uh so every system you know every system in the country that tracks taxpayer ID numbers for for companies it's like it has to go alphanumeric and it's and it's a different format it's longer and so um you know think of the the banks the healthcare providers the government agencies this is like a huge problem um we work with the largest financial services organization in in Brazil called itau um and they uh you know they added sort of a two-year plan to become compliant with this change it involved upgrading cobalt cobalt mainframes it involved upgrading processing um conceptually it's not complicated but when you have thousands and thousands of different systems uh that'll interact in complex ways it gets really messy really gnarly um and they were able to use debon to get the bulk of that project done in

25:03three weeks uh inside of two years uh and then you can kind of clean up the edges however they wanted but it's been really impressive to see you know stuff like that uh this sort of multi-year to to multi-week project happen more and more so is is this kind of where we're at today is like like the really not fun painful migration stuff um where it's just like you know some version of transposing a um to be in a way which is sort of more modern and functional is the is the current sweet spot for uh software and models so anything that you can kind of validate automatically is a sweet spot i would say and you know i'll give you an example of why i'm working on cognition and examples before this i was at scale ai uh which you know provides data to the frontier labs uh and what we found is that we were we were doing labeling at scale with with sort of human experts saying hey this model response is better than that model response and trying to provide you know reinforcement learning with human feedback to improve these models and what was happening is that

26:06it just kept getting harder and harder to do to do well because every every human response you know it needs to be smarter than the model's own intuition for it to provide useful signal in some sense to make the model better um and as the models get better that gets actually really hard to scale we were finding experts like PhDs in chemistry and true domain subject matter experts in every niche in the world to be able to try to keep eking out better and better performance from these models um in software you have a big a big difference which is that you can just run the code you can compile the code you can test the code and if it works or it doesn't work that signal you can use for reinforcement learning to make these things better so every application you know whether it's in government whether it's in private sector you can think of where we can build an automatic feedback loop i think that's really the key the key enabler to success so migrations are a good example of this because you can build tests to say hey how should the system behave does the new system behave the same way as the old system can we talk cve mitigation yeah yeah cve so a lot of

27:06people are you know worried about security an a i and i think the the worries are real in the sense that people are you know people are using a i in all sorts of ways that they haven't before and attackers are actually using these like discover vulnerabilities uh in like really you know novel ways that would have been really hard to do manually previously um and what's happening now on the other side is that the defenders are kind of fighting are fighting this with a i we have the great existing tooling for sort of scanning and detecting vulnerabilities via like traditional static analysis you know think like a sonar cube or a vericode or a sneak or anything that you can take a it or a code base and say okay what are what's my risk surface area the what what happened a few years ago as you would do that and then you would get thousands of alerts and sometimes you get tens or hundreds or hundreds of thousands of alerts or even millions of alerts at really large organization and so they have to get really like they're large organizations in the world that today have hundreds of thousands of open of open you know alerts of hey this might be insecure here

28:08which if you think about that it's kind of terrifying uh but it's also it's it's challenging because like we just don't have the capacity to go read all of those and staff you know uh staff the team to go fix them there's just there's just more problems than people um and what we're seeing with with Devon and actually just with a i more generally is that this is like a really good use case because you've got tons of alerts it's pretty toil some they need to be triaged and I can do the triaging actually quite well um and so you know some of the largest financial services firms in the world for example they they apply Devon to every cve every single vulnerability that's caught in their entire code base before even going to a human it goes to Devon it goes to the agent and then we try to auto-remediate and we're right now at a roughly 70% fully automatic remediation um success rate right now so the code change suggested by Devon can be accepted and approved in one click no changes needed and that should only go up obviously as as as the malls keep getting better yeah I mean I think this is an important point as like you you are not going to make critical

29:13infrastructure whether that be a bank or a power plant um resilient like to the degree you want it to especially when you have AI attacking it on the other side and the cost of sort of um uh you know getting into these systems starts to decrease um because you know your power plant or water treatment plant uh has had 30 years to hire this offer engineers to clean up this stuff and just hasn't so the only way we get into that world um where they are where they do have a stronger defenses is like there is something which is way cheaper than um what the alternative has been for the past few decades um so it's it's cool that we're at the point where we're seeing the systems they don't even need to be um like vulnerable to automatic AI infiltration to be at risk you know we actually see on the attacker side um humans working with AI has has made attackers much much stronger um you know I'll give you an example uh there was a vulnerability a few months ago called reactor shell very very uh you know this is a 10 out of 10 critical

30:16vulnerability where uh you could essentially remote control any server uh by sending the right uh the right network requests that use this library which is a very popular library so 10 out of 10 severity um the attacker who found this uh found it uh using using AI tools in fact uh product we we offer called deep wiki it's like our code base intelligence product we give it away for free for every source open source repo was used by luckily a good Samaritan researcher to go find issues in this code base and um and then and then just guys basically unlock novel exploits and part of the challenge one of the hard parts of being a security researcher uh is is just like wrapping your head around all of the code that already exists inside some existing system and so when AI is making it easier to ask questions about that code to summarize a you know the attackers get a lot of leverage yeah well let's talk about the sort of understanding the code base dynamic both in your kind of like legacy corporate clients as well as the government

31:18ones like why is that such a challenge to um upgrading them yes so I mean right now if you think of the the state of the world the these models you know maybe they have like context window in the million in the millionish range so you can throw in say a million tokens and okay we can reason about that actually a lot of the the real world production systems in enterprise and governments are much much larger than this right you can have individual code bases that are hundreds of millions or billions of lines of code you can have thousands of systems that plug together in different ways and you know we talk about oh like we still need to understand what we're doing as human engineers I would argue no human engineer actually understands what we're doing inside a large organization at this point anymore the complexity has already escaped you know the constraints of like one person's brain it's just there's just too much stuff it's too interconnected it's too hard and so that the same reason it's challenging for people it's also challenging for for you know most kind of AI systems especially where the like this is a limitation of the models right now

32:18and I would expect in the coming years you'll model should get better at handling bigger and more complex like more complex code but um we do a lot of our research team of cognition has focused specifically on large scale code based understanding to take okay every disparate system how do you look at it together in the same way um and you reason about it it's actually actually a mixture of deep learning and graph algorithms uh to build this like high level graph relationship of the different parts of code and the different systems in an organization that it tends to scale much much higher yeah so let's stay on that for a second how do we go from a million uh million token context window to something that can like actually understand what's going on in our gnarly Brazilian bank so for right now you need something you need more than the models right now I would say you need more than models then like look we hope that cognitive we always want underlying based models to get better every day and we train we train our models that are specialized for specific tasks but the models alone are insufficient to do very large scale code based understanding well um what we found is that if you if you basically try to index everything so

33:22you can kind of throw the billions of lines of code and the many different systems uh you can into kind of structured machine learned representations of what are the key similarities and differences across these different services and what are their relationships uh you basically can build this graph data structure that interconnects how everything works and in like much higher degree of detail and then you can still use lm's when you're zooming in in some specific area to say okay like how do these pieces fit together um to to basically go solve a problem and I think this is a really important point if you look at if you look at kind of AI and software and by the way this is true another AI domain too um it's much easier to make a new thing from scratch than to make changes to an existing thing uh because to make changes to an existing thing you know first you have to understand why the thing is the way it is and the why something is the way it is you know that might be decades of historical context some of its documented in the code some of it might be written

34:22you know in a in a confluence page somewhere else some of it might be in one guy's head who left the organization five years ago and so you have this enormous history um that I think we have to respect when we're trying to make changes to real world systems yeah so I think uh social security is perhaps the paradigmatic example of that like uh no government uh you are so no administration wants to do anything to stop those checks going out um and that plus the census data being so finicky ended up uh during the pandemic enabling hundreds of billions of dollars of fraud because there wasn't like a more modern system that would allow you more visibility into where those checks were going um yeah thought about the government context no totally and I think I think it's I think you know sunlight is the best disinfectant I think it's great that the government is starting to put out um these datasets these public datasets to say hey you know community go where is the fraud you find it you know we're not even going to find a week we actually

35:24assigned uh assigned to Devon to the to the recent like a large large uh large dataset release um from HHS uh to go find like what are the fraudulent patterns uh in here and like very quickly this is you can tell this is a task really well suited for AI because there are anomalous movements of money patterns here that do not add up relative to the distribution and so I think you're going to see I think you're going to see a lot more of that both government agencies using AI internally to fight fraud but then also sharing data externally to sort of leverage the full the full community uh so what are some of the dream projects like what what um uh what were do you really want to sick Devon in the coming years I think um look state capacity matters a lot to me you know we uh as both the citizen and as someone interested in like the well-being of the United States uh it's it's great to see what our country is capable of at its best um but also frustrating to see what it's hindered by um at the worst you know the the incentive structure of how uh how the private sector helps government and the way contracting happens and the this sort of resulting

36:29lock-in and and stickiness of really like sub optimal systems for long periods of time it's actually really it's really frustrating and you know I think it affects us every time we we go to the DMV um and I would love to see a future where we have high state capacity for software that we there is not this big gap between your experience using you know using software with the government and your experience you know using software in every other aspect of your life where things continue to get better um you know the the bits power the atoms like our our interaction with the the physical world it's increasingly governed by the the software systems and so you know one of the things we're trying to do in cognition for government is empower every agency to sort of get to where they want to go you know it really it starts for us with modernization modernization is like the bottleneck for a lot of this a lot of these problems and we work with we work with a ton of agencies at this point you know we work with the army we work with the navy we work with the treasury we work with NASA JPL um I think we have like I think dozens now of of kind of fed ramped deployments and we're just getting started but I'm really excited to

37:32to kind of help level the playing field between public sector and private sector uh and and how has the experience been putting Devon and government versus uh you know financial system or other enterprise guys more there's more parallels than you might expect you know it turns out actually like the largest health insurers in the world um they are also very sensitive to regulation and they are also very sensitive to security they also have enormously complex systems uh and so I would say there's actually more similarities than differences um and that's one of the reasons we're deciding hey you know relatively early in our company journey we want to go help the government too you know it's not it's not this completely different set of problems the problems are the same you have to work with the you know you have to work with your counterparties in different ways to um to kind of be useful but the problems are actually pretty similar and what about if we're talking about like striper notion or I don't know some uh you know sort of yeah they're like still like in valley tech native startups are I think

38:34really different you know there's they there's like uh on there's sort of a spectrum of bi versus build right and I think a lot of what's what's special about Silicon Valley is you know companies are are building things themselves right they're constantly building things they're they're making their own agents they're they're shipping they're they're shipping new things all the time constantly reinventing themselves um and they have a lot of companies where you know their core focus is not software their core focus as an organization is solving some other set of problems either for their customers their citizens or their stakeholders and software is just a tool software is a tool to get the job done and and what's happened is historically these organizations the ones that you know they're not native software organizations they're to some degree you know they're reliant on the software vendors to bring them the the tools for the job and I think if you really play forward what's going to happen with AI and software engineering every company every organization uh every government agency is going

39:36to be in control of its own destiny in a much bigger way you know if you think about how constrained software creation is right now um engineers we're everyone needs more engineering capacity than they have uh you know there's the road map is really long and things get cut all the time and de-scope so that you can prioritize what's needed that's gonna that's gonna start to flip and I think it might be the the result might be that every company has the capabilities of a software company so what do the what does the software engineering starved um healthcare provider or federal bureaucracy like actually need in order to sort of like taste taste the fruits of of that future Russell um besides besides a good procurement process for well for a little bit you you joke about the procurement actually the procurement process is one of the sort of the first beneficiaries of the fruits of of of software abundance you know uh uh people people are joking about the the SaaS apocalypse right now uh or some aren't joking about you some companies are you know their stock is

40:39down you know 30% on on on on this concept and I think I think a lot of the concept in some ways is is overblown um because we're not gonna all vibe code our own systems of record uh tomorrow but I would say the leverage has flipped and procurement organizations are seeing the benefits of this so you know uh one of our our largely fortune 500 uh clients they actually instituted a new procurement process uh with Devon where before they buy any other software they first prompt Devon and say hey can you go build just go try to build this application and Devon is not gonna one shot build a giant company's application in in in one go but you can get a prototype and you can get sort of a taste of something um and then the procurement team goes to the goes to the software vendor and says hey like we want a discount and that's actually an effective negotiating tactic and then people are getting discounts from that already and I know of at least one case where you know it's an infrastructure provider and and the firm decided actually we are going to build

41:40this internally because it's it's actually not that hard and we got the prototype and the prototype works and so uh I think we are starting to see that happen like in the real world in Q1226 this is already happening um but it's gonna it's gonna put pressure on people to deliver value the software to me and that's what I'm personally really excited about is you know less rent seeking more product quality yeah I wonder also on the like the question of how many really good people do you need to get to like passable right you know we've for the past decade or so have had all these you know code for America or like various like rotate for two years into government and on the one hand they do good work and on the other it's like okay maybe maybe you make like a nice front end or like you fix one problem right but the ability for like that one person to fix 10 problems or 50 problems in that two-year cycle uh I imagine these tools are going to be allowed

42:41those uh those folks who take those you know who who take these jobs and do these rotations to have a lot more leverage um it's yeah I know that we we see that all the time and it's um one of the fun things about software is basically everyone always wants more software and so uh what happens is if you're you know you're the individual engineer uh you can just ship a lot more than used to be able to uh and you're more empowered kind of cross-functionally as well right like you can get some help uh with your designs with your agent you can get some help scoping the product roadmap you get to help with the integrations so each individual person is getting a lot more empowered by the way in every function the the product manager feels the same oh I I can prototype this without the and the designer says I can build this in scope it without either these two so you have every every person it's kind of traditionally involved in the the process of building software is more empowered to to tab more ownership of of the outcomes they're driving and I think the result of this is you know you can of course you can get a lot more done with smaller teams but organizations are also just getting a lot more ambitious and we see I mean I would think that the bulk of the

43:44change that's actually happening right now is people are taking the productivity gains and what what more can we ship what more can we pull in on the roadmap yeah and I think from a policy perspective and this is you know a drum I beat a lot is like you need to use these tools even if you're not software engineer because like the possibility space of what you can do from a policy perspective is just going to be um is just going to expand like the the idea I came up with was some like dynamic pricing of you know for the FAA to do like different like drone canals in space for you know delivering your packages or like taking your kid home from daycare or something change pricing and they're awesome yeah search pricing for my daycare vtools and but like that is a very software like that like that that's a big demand on software right and we in New York City right now we have this incredibly dumb version of uh surge pricing um it wasn't

44:46necessarily because the software is complicated but like like they're they're you can just have more creative dynamic things because it's it will no longer be impossible to um do what the equivalent of you know 10 FTEs building you a thing um in 2024 or 2025 so I'm excited for like people to use their imagination um um when it comes to do the things uh this stuff better yeah both both like I mean for policymakers I I think you know it's it's useful to like implement your own policy ideas with these tools but I think it's also really important to to build the mental model of okay what's possible and what's not possible because that mental model changes I think every month I think actually one of the greatest harms that we did in generative AI is we shipped you know Google auto answers at the same time we had chat GPT pro and a lot of people were you know running a Google query on the cheap model serve for free and then all of this AI answer is not very good meanwhile you know if you have the the $200 a month chat GPT you know pro subscription the answer might be research grade

45:51quality and so people were building very inaccurate mental models of what these systems are capable of and everyone's guilty of it including people who are even working on the tools if you're working on building the tools and you're not constantly testing the frontier you know your mental model goes out of date really quickly oh I mean not to give away your e-vals but like what what are you what are you hoping to see in the next few years I think I think we're we're heading to a world where you know saw building software it's it's it's already no longer really about coding to some degree like the writing the code is not is not really the bottleneck anymore um it's it's everything around that so I think humans still have to understand the code we're putting into production and the emerging bottleneck it's actually in review so we launched a product a month ago called Devin Review which it's it's a very human-centric interface for trying to understand what it is increasingly AI generated code you know you have people making changes that are thousands and thousands of lines the volume of code is growing enormously um but I think for where we are right now you know 2026 Q1 you've still got to understand the code that you put into production I would say by

46:562028 that is no longer true I think we will just have much broader specifications of systems that characterize okay you know this is how we want the system to behave it looks something more like writing a spec in English and AI kind of compiles the English spec down to software um but I think in 2026 and probably most of 2027 we're still going to be looking at code trying to understand code and we're not going to yet be at the level of reliability that you can just fully automate these things it reminds me a lot of the way in self-driving so you know when I was a Tesla I was on the autopilot team I was working on the vision role network um there's you know that when when you get to sort of 99.9 percent reliability a lot of drivers start uh really trusting the system because it works you know uh 999 out of every a thousand times and so that one and a thousand where you actually have to take over um you know people just people are human and you pay you pay less attention and so I think we're kind of right now in that uncanny valley phase of AI software engineering where it's like works so well that uh you almost might be too trusting of it but you've still got to you've

48:00still got to understand um what what you're doing you mentioned this earlier but like the the sort of the self-driving form factor for when you get to level five is like you take a nap and that's kind of like it's it's clearer like what the end state we're going towards is I mean how do you guys think about like what the next um kind of interaction paradigm is going to end up being yeah well so self-driving is interesting because you're right at level five is you take a nap but that's kind of the limit right you you still decided where you want to go right where should the car drive you so versus in software I actually think there might be a level six which is which is you don't even decide where you want to go it's actually you know maybe you have some very high level objective for what you want to accomplish um but the like level six autonomy of software is you know the the the kind of AI agent actually deciding deciding the the details of what what to even build in the first way I think the level of abstraction that people are going to be operating on is going to grow really high really really sort of unexpectedly fast and if you if you kind of specify a business

49:05objective or an outcome you want um increasingly we're going to be able to optimize against that objective directly yeah and it's funny because that's actually where I feel the limits most is like that I sort of I like the like the like the first question idea generation like what direction to take this like vague thing I have um and yeah sort of the execution or the research or you know finding all the random stuff on the internet or like building me the MVP like that it can take care of but um it's still and and I guess this comes back to your like can you execute the code and see if it works or not as like that's a that's a hard thing for um for a model to like come up with the policy idea that's going to like fit into all of the sort of constraints we're living in or the episode your question topic or yeah I think right now I mean it's kind of like in a lot of life it's it's all about asking the right questions you know that's I feel like the the key skill of using models right now is like well what what question are you asking you know what what

50:05task are you are you trying to do I do think that's a distinctly human activity that is going to remain human for for a long time and um you know I mean even the way we've structured our society as the democracy ultimately you know we as people are in charge of what we want to do what we want it like the structure of society we want to set up how we want to push forward um and so I do think these things are tools ultimately they are tools for the betterment of society but they're getting much much more capable much more autonomous all the time um when you're working with the clients and for the for your for a deployed engineers like are they oftentimes just kind of squinting around being like oh you guys should yeah you thought you wanted us to do a but like B&C is also something that these models are capable of like how much how much do you see cognition serving the role of like uh uh you know AI to problem finder I think I think that is an area that we we help with a lot right now because usually that the customers understand

51:10their problems but they also don't necessarily have the best mental model of exactly the full universe of problems that are addressable with with AI today um and the thing that's really interesting about you know Devon and just agents in general is once you're plugged into the code you can see all the problems like you can see uh the the the sort of problem discovery process that used to take you know lots of conversation lots of challenges uh within software you know whether it's the security vulnerabilities we talked about or or something else you're getting increasingly automated so a typical a typical engagement for us might be you know a government organization or uh or or or a large you know a large enterprise would come in and say okay you know we have these three outcomes we want to achieve and we think we can do it better with AI you know it's we're gonna we're gonna modernize this legacy system and we want to do it in in in weeks or months instead of years uh we need to build this new this new product this new capability and we want it as fast as possible and it's gonna grow it's gonna grow our business this much or uh you know we need to structurally improve our our testing coverage our validation our

52:10security posture and you know hear the metrics we so they have some set of outcomes and um what we find is that inside each organization there's actually a really wide distribution of how much people are like leaning into using new generation tools to do their job and in every order it doesn't matter if this is the the most like you you think of as the most like legacy old school organization the world there are people in those organizations that are excited about the future and want to try new things and learn always consists of 100% of the time and and those people I think are more empowered than ever to have extraordinary impact um there's also folks who are like I've been doing it this way for 30 years and like I'm super skeptical of all this stuff and you know I think those folks are the evidence is increasingly growing that it might be worth it might be worth taking taking a peek at um what are your what are your calls to action were you hiring for what kind of conversations you want to have coming out of this yeah I mean so we're we're we're hiring a lot in like cognitive or government right now uh for folks who have you know kind of been on the ground

53:12and seen the problems firsthand right so uh I think our our forward deployed engineering organization it's maybe the the fastest growing um of our you know of all the roles in some sense uh people asking like what is the future of software engineering looked like it might look like you always have to understand the problems of your customer because the writing the code part is getting easier and easier and so you might I actually think if you look at like kind of our core research and engineering product team versus you know engineers who wear multiple hats they interact with customers they help shape the product you know the ladder is growing it's growing much more much faster um and in the limit you know we all might be working directly with with other people in some capacity uh so we're growing that a lot and we're looking for folks who um have kind of experience like building in the the public sector uh and then we're we're also growing sort of how we do we call like engagement management uh because these projects are very rarely just about the software you know it's like what are the the end to end organizational problems that we've got to go

54:14solve uh you know of course we have classified deployments and we we work in um secret networks and and so folks with the right clearances and backgrounds are always always super interesting but um I think we're really yeah we're just kind of scratching the surface of how how much this is going to change also um so I turn the mic back on because we have we have a family we have some family Lorda share Russell give it to us well you you were asking me why yeah why why I was so interested in the in the census uh you know the 80 90 census uh and and how we kind of popularized punch cards um yeah my grandmother was she was one of like the first female programmers in in the in the country um back when it was like in a very arcane you know a very arcane activity of messy with punch cards uh later assembly came out she was like super excited about that um she gave me a lot of crap growing up uh that we had it so easy you know in the 2020s like writing code with uh with the computer that you could edit and like you didn't have to worry about dropping things um yeah her her her master's thesis was on the knapsack problem uh which that that line of research

55:20ended up actually being really useful in the Apollo missions um and so yeah part of like my hope for for cognition for a government is like we can we can go full circle and we can help we can help bring the government back to actually where it once was which was the true leader in in technology what did you think about Devon unfortunately she she passed away a few years ago or before Devon came out but I think I think she would look at it and I think she would be proud I think she would be okay that's nice um it's like the the the the way the genders flipped in software engineering were in the first few decades this was like a very female coded field to having a change I wonder if I wonder if actually like the the um all the AI tools are gonna help it flip back again um because the sort of like the if the if like the type of skills well it's just it's just going to rearrange what the labor market looks like and I think different skills being prioritized in a way that

56:20you don't kind of see the gender split which has dominated for the past few decades totally and at a minimum I think it'd be so accessible so early in your life to to learn and use these these types of tools that um yeah you might you might start building applications with AI before you even know what the concept of a gender norm is uh you know software like water they're just letting everywhere um and it's it's gonna be a really fun time to be built in cool all right well first you gotta learn how to speak but maybe we'll give we'll give my daughter like six more months uh uh awesome Russell well thank you for that