Why AI Agents Can Beat the Incumbents

0:00If you want to build an aircraft, you need to procure thousands of suppliers. Someone sends a confirmation of like, hey sorry, like this part is going to arrive two weeks later. And if they missed this email, hundreds of millions of them. Procurement historically may have been more in a box, and now it's like, okay, it's touching legal, it's touching finance. It's touching about your different software systems and people. The opportunity for the AINATIVE startup is to say, we're going to own that entire antenna arc. No company, no enterprise, starts with fully autonomous negotiation agents from day one. Why? Because they don't trust us and they don't trust the technology from day one. And by having this even in the loop approach, we are feeding our agent with all the feedback and all the learnings. And then they suddenly trust us for like 10k negotiations, 20k negotiations, 100k negotiations. When you think about what a durable, vertically-elegant company looks like, what are the qualities that you look for? It's really, really hard to forecast your mode going for it. If you look back at all the best businesses at the early stages, they were... If the incumbent already owns the customer, the data, and the system of record, where does an AINATIVE startup have an advantage?

1:04In this episode, Alaina Berger sits down with A16Z partner, SEMA Amble, and Leo Co-founder and CEO Vlad Kyle to answer that question through one of the most complex parts of the enterprise. Procurement. They get into why so much of the actual work happens outside the system of record. Across emails, spreadsheets, contracts, engineering data, and conversations with suppliers. And Vlad explains how Leo is building agents that can coordinate that context and increasingly take on the job end to end. They also discuss how you earn enough trust to let an agent negotiate on your behalf, while building an internal AI tool can be deceptively easy until you hit the exceptions. And what changes when agents eventually sit on both sides of a transaction? Welcome back to the A16Z podcast. I'm Alaina Berger, and today I'm joined by SEMA Amble, a partner at A16Z and Vlad Kyle, Co-founder and CEO of Leo, which builds AI agents for enterprise procurement.

2:06SEMA, you recently wrote a piece called The Incompanze, are coming, and an ask a question facing almost every AI application company. If an established software vendor already has the customer and the data in the model can work across its tools, where does the startup have an advantage? And we have Vlad here, and Vlad can really help us understand where a startup has an advantage. So I think a good place to start is the cases for and against the incumbents. So if a company can connect a capable AI agent to the software it already uses, where does another application actually have a use case and an advantage there? Yeah, we back up and frame it up a little bit. So historically, we thought about there is the incumbent, and then there's a startup, and it's a fight between distribution and innovation, which to take my partner Alex Rampelle's phrase. However, now there's like this third piece, which you're pointing at, which is the incumbent can layer on one of the models on top, and then there's a much more formidable competitor in the market.

3:12So why do you need an AI native startup if you've got cloud force, which is taking cloud plus sales force and putting the two together, and then you've already got all your data and your employees are all used to using the product. So why another product? I absolutely still think there's obviously still a case for the AI native startup, and it really centers around the fact that the legacy incumbent is limited to their system of record and that record they have, and they're not completing the end to end job. So let me put that more concretely in an example. So say you're a customer, and the customer calls and says they got charged after they got canceled. Resolving that cancellation history isn't just the customer going into the chat and saying, hey, I got overcharged, and the response there, it has to hit billing, it has to look at all the chat history, it has to look at the contract. That's not one system of record. That's the knowledge around that customer and everything it touched, and that's something that one system of record wouldn't touch.

4:16However, the opportunity for the AI native startup is to say, we're going to own that entire end to end arc. So that could be illegal. So owning everything from brief all the way through trial. Well, I can talk more about procurement, but it's really the concept of owning the end end work. Yeah, Vlad, do you want to talk about where that does show up and procurement, like where existing incumbent plus a model just as insufficient? And what have you seen just with the companies that you work with? Sure. So when we think about procurement, I would assume that most people think about like prices, right? So like what is the end price that we negotiated on? And surprisingly, the record looks always very, very simple and very easy. It's just 8K. That's the result, for example. And I mean, it's the same for sales, right? So even if I come to see my auntella, like, hey, we now partner up with another enterprise. Look at the signature here for like this contract. It looks like very easy, but like, SEMA doesn't see like all the work behind it.

5:17Right? So like, there are probably like 30 stakeholder meetings happened, 500 emails, 20 Excel sheets, and that's the same for the counterpart procurement. Right? So you see in your ERP system 8K for aluminum, but you don't see that maybe the supplier did like a pushback and asked for like 10K. You don't see that like a cost engineer had run three weeks of Excel sheets and 3D modeling to find out the prices of the part and everything else. So that's what we see. So like most of the work in procurement actually happens outside of this ERP or any system of record. And I'm sure in the workflow that you described an agent can do a huge number of things and different kinds of agents can do a large number of things to and I actually think that's a good bridge into the next question, which is, you know, a year ago, I think a lot of the incumbents were releasing chapots. And that was kind of the extent of what you would see, but SEMA in this piece that you wrote, you lay out four different kinds of agents retrieval agents, which are kind of the chat bots that were familiar with process agents policy agents and principal agents.

6:23So can you just walk us through all of them and explain kind of what changes in the kind of judgment that's necessary across all of them. Yeah. So a year ago, I made this meme, which was the slap on a chatbot strategy, which is essentially like all the incumbents effectively had. A chat bot that sat on top of the system record, which you could chat with to retrieve information, maybe do some analytics. And that really was in that first bucket of what the retrieval agent is. And maybe let me walk you through an example of what each of like the retrieval, the process, the policy and the principal back to the customer support example, just because it's very it's easier to understand. So imagine if you're a customer and there's a service outage and you're calling in to say, hey, I want to get compensated for the service outage. The retrieval assistant, which they come at me have is going to be able to pull up, yes, this was what the contract term said. And yes, there was an outage and just verify that information. It's pulling up information about the customer. It's in the database and it's just sharing it back and maybe synthesizing the second step in the Asian sequence is the process agent.

7:29So that process agent may be able to pull through an approval of credit and say, OK, based on our policy handbook, it was out from these dates. Therefore, we say you are entitled to money and it can just apply the bill and just do process. There's no judgment involved. Then as you keep going to the policy agent. So the policy agent isn't just going to apply the process, but it's going to say, OK, in this situation, it was out for 20 minutes. That's enough to be considered a significant outage and they're applying that judgment because there isn't really strict definition around it. And then the last stage, which is a principle agent, you're actually weighing, OK, should we offer more compensation because that was a pretty terrible outage and we want to reserve a relationship and it's worth doing more beyond even what a process or policies. But the importance of these four things is that most incumbents started out in if at all in bucket one now, at least they're marketing that they are moving towards process and policy, meaning they're able to apply more judgment.

8:34And if you look at what they've launched, these are workflow agents that will help you get a document signed or input information from a transcription or things like that. They're very much still limited to I would say retrieval and a little bit of process. They're not gotten into more judgment and I can get into why they're all sorts of incentives that are preventing them from that. But the incumbents are trying and I think what they're not able to do, they're like some of the other thing, OK, let me partner with open AI andthropic one of the labs and try to build out, take that model capability and complement what they have and super power it. Yeah, I'm sure there should be. Yeah, well, do you want to say why sort of some of the incumbents are holding back? Yeah, OK, so I would say they're holding back, but they're held back. Yeah, I'm sure they want to be full force, but there's probably two pieces. One, they have this advantage of distribution, right? They have the customer trust, which enables them to then sell more products with the customer.

9:37So take sales force. When agent force launched, it was very easy for customers to say, yeah, I'm going to sign up for the sales force agent, especially if it was offered at almost no extra cost. And so they have this trust, they have the distribution. It's often pretty seamless to turn on the product. The flip side is there's all these internal incentive issues, right, which is if you start getting into more complicated agents, it's an internal conflict with an existing product that's offering workflow versus you're resolving the work. And those are two products that are here, you're resolving a customer support issue and to end versus providing workflow for a human agent. Those are different buyers. How do you, you know, those two teams are in conflict. And then on top of that, I think from a sales perspective, like, what are you selling to the customer? And then oftentimes in the classic like incumbent issue, right, is like, there's two two VPs, right, and they're different orgs and they're selling different products. And like they will never be able to figure out what the right set of incentives and the right person sells it.

10:41But anyway, so there's all these sort of classic incumbent issues. I think that incumbent also runs into. Makes sense. Vlad, we're across this, this kind of spectrum of retrieval agent, processed agent, policy agent, principal agent, where, where does Leo sit? Yeah, so like we spend across like, I was like, all of those categories and it really depends on the complexity and the risk our agents take, right, so you know, sometimes we can like we already have use cases where we learn fully autonomously. Sometimes you have the human in the loop. It really depends on like the budget approval, how complex and how, how risky does this. But maybe like coming back to what Sima said about trust for for the incumbents, like you talked about like internal trust. I think there's like also an external trust thing, right. So like, how do you convince someone to go through from, hey, just like an agent that retrieves some information and maybe runs processes to do to do like something completely autonomously. It's like, obviously there's like a product component to it, but mainly is a people component. So they need to trust you.

11:45And like we as a startup, scale up, we have to under trust, incumbents already have to trust, but this also means they like they can destroy the trust if they ship something like too early and the product maybe doesn't work or it's bad or it's like works decisions in a bad way. So that's that's what we can do. And then interestingly, what was like really surprising for us, like those process agents actually became kind of like a side quest for us because it like when we talk about when we look at the invoice process, like invoice agents as a process. So you retrieve some information. You match this across like other documents and then you push this back to SAP Oracle, like a very clear, clear process. And then we figured out, okay, there's like 100% of the software market, right. So that's that's invoice software. That's how you build it today. But it's actually only like 20% of the work of the job to be done or the problem because 80% of the problem is like, what's if like what is the invoice is fraudulent?

12:49What if there's like a mismatch? Like what is like if we don't take the happy path. And so we like very fast shifter to the next step of agents, like doing those doing those exception, exception handlings. And we convinced the customers by I think we got like lucky being like always slightly ahead of the curve. So we were able to pitch the next generation of agents. So for example, like when we started out three years ago, it was like just retrieving a document like not impressive at all today or like three years ago, this was in like crazy impressive. So we pitched this to customers. We find out, okay, that's a real problem that's a use, we'll use case they would pay for. And then we were able to ship this some weeks later. And then the same way we are doing this now for the next step for process agents and for like fully autonomous agents and for long running agents. We are pitching this to them, fighting on the problem and then we're able to ship this very fast. I think an interesting point on trust is this internal versus external trust. The other lens on that is, yeah, so you need your customer to buy the procurement software and trust to use it for their internal processes.

13:59But one of the really interesting things when we first met, Vlad was that they're doing, they're also doing the negotiation. So you have to trust that the Leo agent is going to then interface with a third party. And, and there is that piece of trust. And of course, I think a lot of people feel burned by the incumbents and like, you know, they're pretty limited and haven't been able to do what they've marketed they've had in the past. So, but putting that aside, like, I don't know, maybe Vlad, I'd love to hear a little bit how you convinced the customers to trust an AI agent to now take on negotiations. Yeah. And can when you do that, can you also just like paint the picture of like what's involved in procurement and who, who are your customers and and what kind of sort of legacy systems are they used to. Yeah. So like when we, when you think about procurement, you like maybe just think of like purchasing or like you when you look at like B2C world, like just you buy something. But actually is like a very intense process process, which like runs the economy, right? And it includes multiple stakeholders and a lot of stakeholders and a lot of a lot of departments, legal, cost engineering, obviously procurement, finance.

15:16All of them have to work on this on this decisions and essentially like when we, like I think I think the reason is like how are we convinced them is like what I already said said earlier on. So like we were like always a little bit ahead of the curve. Okay. So we knew, take a deep technology is coming. So we like even before chat, we came out. Like we, we just started a few weeks before like this chat, we take breakthrough. So we already hear it all of the problems. Then we hear the hype and the let's say like tech bubble and we were able to pitch this enterprises and we quickly figured out, okay, like this is a. Could be like an interesting use case like just chat about applications or retrieval agents document processing. And then we, we are able to find out the problem and then ship this quickly to them. And then obviously like this is like the people factor of like trusting. So we are telling them something about and we're able to really ship something in production. But then there's also like this product perspective where we have a lot of like evils in place, right? So like we.

16:21The like no company and no enterprise starts with fully autonomous negotiation agents from day one, no one does that. Why because they don't trust us and they don't trust the technology from day one. So you have like a very easy approach of like having a human in the loop. And this is like extremely helpful for us because we see like we have like that say like the perfect negotiation agent, which is like over all the perfect procurement negotiator. But we don't know exactly how. A fortune 10 enterprise like the specific fortune 10 enterprise operates and by having this even in the loop approach, we are feeding our agent with all the feedback and all the learnings. And then they suddenly trust us for like 10 K negotiations, 20 K negotiations, 100 K negotiations. And then you also have like other. Other agents that are like more long running. So when we talk about like multi million dollar negotiations where you analyze complex 3D models and technical drawings. There we on purpose have always experts in the loop right. So there's an agent running for multiple hours. And then we ask for feedback of the cost like cost engineer and then it does the next the next work.

17:31And so on. Maybe just to double click on that. How do you what where do you put the human in the loop on the like negotiations side. You mentioned the cost engineer, but like if I were you were going back and forth on a deal. Is it mostly around the like you know data for you know something like cost engineering or is there anything else where you have humans in the loop there. So again, depends on the level of negotiation right. So like we have to distinguish between like negotiations where you just like a negotiation. We're in like we're enterprises. They never did those because they didn't have the capacity. But by by deploying agents, they can just capture savings that they were in the way of right. So they like before agents all before Leo, they just didn't care about everything which happened below 50 K right. So you can just like this is maybe a heck for like other startups. You can just send an enterprise and invoice for 40 K. They were probably not negotiate because they don't have the capacity to do so. Except they have Leo agents, then we are going to negotiate against you. But other than that, they are just like just just paying that and obviously there the risk of like like what is the risk of like you don't you didn't negotiate at all.

18:42So it's the risk now of having a bad negotiation agent nearly zero right. So maybe we miss out on some negotiations, but like it's better than nothing. But still in like in we talked about business relationships and business relationships are not always about the cost and the money right. So maybe you're not spending a lot on the vendor. But maybe you you're like you need this business relationship right. So good example might be like podcasts or marketing services. Okay, that's like probably like a friction of off of the spend. But you don't want to like you have a clear business relationship with with summer like setting up the studio. And you don't want like like random some random people doing that because they already know how how a six in the operates and how you want to record all the stuff. So there we have human and loop approaches where like procurement people care about the relationships so they care about the voice of tone and how it works.

19:43But mostly autonomously and then we have the other set of agents where we always have a human and a loop approach and it's like they're like multiple steps and like negotiating. It's also like it's not only the price right. It's also like how is the contract design. So we're talking about legal how is like collection design. So we talk about finance. Obviously like cost structure. So we're talking about really like cost engineering. Then we talk about commercials that's procurement. And those are not back office people. Those are like highly trained people where they have very specific knowledge of a very specific process of a very specific company and very specific industry. And they feed those long running agents of Leo with those insights. That's another example of how procurement history. We may have been more in a box and now it's like okay it's touching legal is touching finance and it's touching about different software systems and people and both specialists and more general.

20:46Yeah. Can you can we map this on to a specific cost like not a specific customer but a specific vertical like I'm a drone manufacturer. Humanoid robotics or something like that like how many parts do I have to you know order and procure how many factories am I touching how many suppliers am I touching just all of all of those things if you want to pick maybe Vlad a vertical that you know is is just managing all of this complexity with Leo and just kind of take us through what their experiences. I think that would really help just illustrate exactly just everything that you touch again like when we like when we as Leo when we talk about procurement of purchases like we don't talk about like laptops and pencils like we think like that's solved also by Leo agents but that's easy we solved this like three years ago. We talk about like when you like you want to build an aircraft or robots or drones or even like we now like doing a podcast about like AI high but even AI needs to be built right so you need data centers and like building means procuring like someone needs to like if you build an aircraft you need to procure thousands of suppliers you need to build a factory to build this airplane.

22:05And like really small frictions can have like a crazy impact right so like there's a like if you're running a very large project of like building a data center building an aircraft if there's like one specific part which arrives two weeks later this can have a like a damage of like hundreds of millions of dollars and and postpone and postpone the project. So like all of those that's where like all of those decisions have to be coordinated and one part is like you need to figure out like what you need with what suppliers you work what are like what is like the best supplier to like to get this part but then once you decided all of the stuff there's like all this operational back office stuff behind it which minds are like unnecessary boring but again like operational means someone sends a confirmation of like Hey sorry like this part is going to arrive two weeks later and this is like one of 500 emails in the outlook or Gmail over procurement manager and if they missed this email hundreds of millions of damage done and this like this happens regularly because the only thing they store in their system record is then just a date right so like it will like not like not this Wednesday next Wednesday that's what you see in the system but you don't see like this is like a

23:26but you don't see like like maybe that's okay but maybe that's a one hundred million dollar damage and someone has to decide that and that's that's also what our agents are doing right there you're not only retrieving the information there and making the decisions like does this have an impact what kind of impact and how can we resolve this. Yeah when when you are sort of so deeply embedded in the physical world what what kinds of you know physical world problems can you intervene with like some things I would think are just like unsolvable you know like let's say you have a ship and coming in and a bunch of stuff like falls off the ship or the street is closed or whatever like they're they're kind of there's things that like you can't do and obviously they're things that you can do so so we're where can you intervene and where does that really make a difference. But but it's like actually it's about like probability right so obviously like you can't like you can't change if like some like if there's like a damage on the ship like every example that you manage like you you can't change that but you can if you have like all of the context you can predict that.

24:42Because you can predict like how how reliable is a supplier okay so they're like their ways on like protect the goods that you're shipping and if you have like all the context you have like one supplier where like 20% of the goods are missing and then 1% of the good is missing and maybe like this one with 20% is like 10x cheaper but for this use case it's fine for you to pay 10x the amount because you have like a higher probability that this thing actually arrives. So and this is the powerful thing because you have like context not only the win one enterprise so we like we talked about like multiple stakeholders but there's also context on the outside world right so just the agent should like have context of all the news out there maybe even having like context of like some bad say like on polymarket I was like okay those disruptions are going to happen. Then like information about like on the supplier side on the seller side on the demand side and by combining all of those context I wouldn't say that there is a limitation in the long in the long run obviously like today we have like different sets of like probability but we can we can help throughout the process and this is what we are building building at the right so it's much bigger than just procurement into a company it's more like intra component like that.

26:06Like how how are like businesses like how enterprise are doing business with each other so like buyer and supplier side you describe Leo as a as a multi agent system so can you describe what the different agents are doing one level is that we that Leo agents spend across like all those four categories that that see my mentioned in her in her article and it again like depends on the on the risk and the complexity. So we use like all of them so like multiple agents but the other thing is that like in in order to do a job and to end those agents need to share information with each other they need to do this in like a very specific order and when we talk about like a multi agent system this is essentially what we are doing we are solving like the task and to end and because the also like human level task involves eight people eight stakeholders and maybe like three departments and five different software tools we need to cover like all of those to do like the job and to end and those agents need to then communicate with each other and only with a multi agent system you can do a job and to end.

27:20When we started out we like started off with like and retrieval like more like a copilot obviously like three years ago but then the next step was like a single agent but then we very quickly discovered okay like that's like you can't solve and you can't solve and like negotiation even without having a contract agent without maybe like having having an agent looking at the news and everything that I described before so that's what yeah that's what we define as a multi agent system. So if you have like a bolt like an airline company needs to procure a bolt for say you know Boeing needs to can procure a bolt what exactly is that process for procuring the bolt and like where does Leo step in on that process. So like this is like one of one of the one of the purchase is where we can like run fully autonomously and we can do this because of this multi agent system so like first of all someone has a demand right so they need to.

28:20Somehow communicate it and even like this part is extremely complicated so you need to call someone maybe like you like you open up your laptop because you're like a construction worker you open up your laptop only. Every second week and now you are required to like work with SAP or any other like your p system so you can't even like issue the demand so this is like how we make it very easy so you check you you take a photo like you you upload a quote an excel sheet. And it's actually everything that you that you should know about procurement like no one cares outside of the procurement department about like categories, GL accounts, framework contracts, no one cares. We in the procurement will care about no one outside their cars and then our agents take off and they're like they check the inventory they find out okay they ask another plant okay can we like can we sauce those those bolts internally no okay then I'm calling the I'm talking to the sourcing agent finding out do we have internal suppliers to do. We have internal suppliers to have external suppliers then some like another agent has to draft the RFQ send out the RFQ over over email then bunch of emails arrive some of them are like completely nonsense some of them are like just in the email some of them are PDF some of them are excel sheets we retrieve those information.

29:36Then we do like the next step or maybe like based on our price benchmarking there's an opportunity to negotiate and then we have like agents that essentially decide on the next step so negotiation could mean strategic negotiation with a human loop this could mean autonomous negotiation this could mean options and e options calling then the specific agent doing the negotiation and like doing this like end to end finger like confirming the order shipment tracking invoices. And we are able to run this like fully autonomously capture like all the context and then obviously the next powerful fingers do this for like more complex parts where we talk about like direct procurement where we where we also operate. What's a direct procurement. Yeah so like essentially everything I just described is the main goal is here automation right so you can like run this process fully autonomously and then throughout the process you can like generate even more savings right so it's not like so we look at it's like okay what is like this.

30:45Like and to like drop to be done how does it look like so like what are they doing like thousand times a day but actually they want to do it like zero times a day we've like run like fully autonomous agents but there's like also opportunities of like what are they doing zero times a day but if a business would do this thousand times a day that was that would have a crazy PNL impact autonomous negotiations on spend they never negotiated before. So this is and this is like in the indirect procurement part like thing about mRO parts building a factory the bold example that we did but also laptop some pencils marketing services someone who needs to build up this podcast studio does are like all indirect and then we have like direct parts. This is like when you build an airplane does like all these suppliers that actually with that you actually need to build the airplane or to build the drone or to build the robot and then we don't talk about 50,000 suppliers we talk about hundred suppliers or 2000 suppliers maximum and those are like extremely strategically important and you have maybe on one supplier like one billion of spend so you don't want to run an autonomous negotiation.

31:57You want to run a negotiation with which takes three months and we're like crazy prepared and we have engineers on your team analyzing okay what's the industry for aluminum what's the industry for oil how the price change so you like really take over like all of those drawings you check the quality of this part and this is what like where it gets like really exciting deploying deploying agents. Yeah and for something like that presumably like you'd have the expert engineers and the other procurement people kind of more as the front of house and like the agent is more back of house is that the idea or is the agent like actually it's like you you sit across the table and you're shaking hands and it's like the robot and so that the human who's like negotiate like is it so yeah is it is a more back of house or is it like still front of house. It's obviously more back of house because you like because you like need like these complex multimillion dollar negotiations and that's that's again like a beautiful example 90% of the workers preparation.

33:01The end result that you see in your system record is like oh instead of like one billion I paid nine hundred million dollars. Like there's like three months of preparation and like ten people working full time on that and obviously this is like happening in the back but actually we have some use cases where. It's also like helping in real time so think about let's assume we would now have a negotiation and I have like a perfect preparation same as like with like those those those notes those notes that we're having here. Imagine like while we're negotiating I would have like real time insights on my screen popping up where you tell me the in DC for oil change 10% so like it's increased by 10% so that's why we need to increase the prices by 10% and I would have like an initial like an immediate pop up with like that's true like oil increased by 10% but the product has only 30% of oil contains all like you can you you shouldn't increase the price by 10% so.

34:04Yeah, like exciting use cases also like in the real life. Yeah, we know that you know companies like Harvey and Deckergon are really fine tuning models now. What what kind of underlying models you use and and how do you approach things like fine tuning or or. So like we believe like you can so like we use multiple models from like all all providers and we really see this as a like obviously like as a commodity right so they like have. Really good like general business purpose or like reading creating a PDF and like creating creating actually like all of this stuff but we also believe that like for some use cases you. You can get extremely far with like combining the foundation model with with a harness and you can maybe reach like 100% of like the job to be done. But there are also some use cases where you can have like the best foundation model the best harness whatever it means but like the best harness but you still can get only to 80% and like it would say like good examples for that is like for example what.

35:22When we talk about negotiations what like cost engineers doing right so they're like analyzing drawings and then they like defining okay what should this. This part actually cost like that's what it's called like should cost modeling and there's there is definitely like an opportunity where we like thinking and already started. Fine tuning the model to get them to to get them to to 100% and this part in that example is like price benchmarking where think about like the the like you would have a quote and in the perfect world you were just drag drag and drop the quote somewhere and you would get the perfect price. But it's like and all those like all those information say like not publicly available right so there's all like does like all proprietary data based on like one. One enterprise like across multiple enterprises so like general purpose models can't train the models on that.

36:24So what we are like what we are thinking about is like maybe like not training just an lm but I think like what we see now with models like Jeff also popping up where you have like and. You train it on like text data but the out like the output is actually like an outcome or just like the perfect price and you can't do this with harness because. Like if you would give me like an quote from bcg and a quote from a Kinsey day they could do like the exact same work but this could be like a 10 extra price and I would have like no idea like what is better but if you give this to a procurement manager he would like initially have a gut feeling like okay like this quote make sense. I think I think a good example is again like contact creation like always like I don't know like how much I should pay someone for creating a video but it's like a gut feeling behind it if I asked like another video creator of how to how to do that but if you asked them to write on the rules they can't do this because it is like just like gut feeling and so and that's where we see a lot of opportunity actually.

37:31I think I think I think I'm training an agent but not maybe like a classic LLM but more exact like again like what what companies like like models like like Jeff are now doing on the outcome based so we have like price bench market should cost modeling. Seema we've talked a little bit about how labs are really moving into industry specific work or working with incumbents to do this when you think about what a durable vertical a company looks like what are the qualities that you look for one is around you know owning the end to work that we're talking about building up this data asset and being able to do. Something that has been done for many cases. This is all said I think we talk talk a lot about modes it's really really hard to forecast your mode going forward if you look back at all the best businesses at the at the early stages they were they were just thinking about okay i'm winning customer trust i'm selling more to them and there's a lot of opportunity versus okay i'm going to do these six steps and then get to the seventh step and then we'll have a moat and so.

38:40I think we we talk a lot about defense ability and durability and I think part of that is you're locking in the customer they there's more dependencies they find it valuable and you're doing more of the work and here it's it's truly like okay you know old CRM company was a log for all the deals new sales AI agent is actually owning the lot of the sales prep process and the outbound process fielding inbound and doing a bunch of work. The company overall customer is dependent on that product and that's like a really important signal of getting to the moat and everything we talk about in terms of. In terms of stickiness and network effects at all that is sort of downstream of of that initial like customer use in the value of the product yeah. Yeah glad have you had conversations with customers or potential customers who ask you you know why should I buy your product why can't I just you know plug into a model and do this myself or like use whatever existing system of record I have plus a model like like what what do you tell them and and how do you how do you convince them to to you.

39:52Yeah 100% and and that's a very fair question right and like even if you look like internally at Leo so like the the first use case three years ago which like kind of like by real in the procurement world was like. Just like having a quote and then getting this information into SAP so like very like again like technically like but like like tremendous business value. And so you you have like this retrieval agent getting like all of the information putting it in the city we like this was our like first product and we had an engineering team like obviously small just like the free of us or maybe like for people building it and then selling it but this is nowadays a case study. If you if you're applying to work at Leo so we give this to people like to build this and they have like eight hours to do so so what I want to say by that is like a product that we like one of our first use cases can now be somehow built by engineers within eight hours so because it's very easy to build stuff nowadays so obviously there's a question okay well so someone can build this within eight hours okay cool but then couldn't like just procurement partners also just build everything in.

41:06Two months and the answer is like yes you can build this in eight hours and you can build this but you will only reach. 70% let's say like the of the performance and the problem is 70% of performance or currency or however you measured it depends really on the task doesn't mean 70% automation right so it's this can be mean that you're like have 70% of the performance but you still need to do 100% of the work. Because 70% is not not that much so again like all of the people have to have to check the data so maybe you even created like more work more work than the default. Two things to to layer onto what Vlad just said one overall it's good if there's more adoption of the base models or just like GPT products because it means that people are also going to trust. So I think that increased familiarity come for excitement about AI tools is generally just good for the market.

42:17The second thing is I was chatting with the management team of a fortune 500 company two or three weeks ago and one of the things they mentioned was they had tried to build out their own cash collection product is a big enterprise business and they like after I think that they are going to be a lot more. Like after I don't know three or four months of work at a minimum they had found that there wasn't enough context it was poor quality context they had a lot of recordings a lot of screen grabs and they tried to pull it all into one system but they there was no there wasn't good enough and then there was this giant question around OK like you know we've got now two different ERPs and we were about to acquire another one who's going to update all the mappings test that you know test out of the market. Test out OK does it work and then like we keep talking about exception handling you now need to map that onto a totally different system a different way of doing things and I think they quickly realize that the internal bill didn't make sense and so we keep hearing stories of us where people are like OK I'm going to do the internal build and they're like wait a second it's not different from what DIY's ever been in the past which corporates have always tried but I think enterprise companies generally realize that like there there's their core competency and then there's

43:32building internal tools and they should focus on the first camp yeah yeah make sense on yeah so this exactly what I would have meant with like obviously like they can they can get to 80% but those last 20% really matter and you can only get like the matter to get into production so that's why you need like this harness right so you need all those like integrations memory workflows and sometimes you need vertical data to do that and like like the perator perator principle like those 20% can make like 80% of the of the effort like they are making 80% of the effort so like to all like those fortune 50150 company so you can do this but then let's say like procurement workforce of a I procurement agent should then become like one of your core competencies and you should evaluate whether this is going to be a good one. This makes sense on not for you to have this in a circular your cost go.

44:33Well Adam curious of your seeing suppliers start to use agents or AI at all and kind of like what happens when both the buyers and the suppliers are are fully AI enabled. Yeah so like we 100% believe that like in the future there will be like agents on both sides and this this makes so much sense and but like surprisingly what we what we see is like so like when we look at the supplier side is also kind of like equals the sell outside right so what we see is like the the sale side was like always ahead of the procurement side but what we are now seeing with those suppliers for the future of 100 companies this actually is not true so they are like. Maybe advance in like let's say video recordings and like using tools like run all on all the stuff but like not like really having agents deployed the automating to work and the cool thing is as like procurement is like is unsexy right so sales of sexy procurement is unsexy but it's like one process and procurement is the the counterpart but now the good thing is in those industrial companies for to 5000 companies procurement.

45:48So the procurement has the bigger power to the supplier because you as in like typical use case like automotive supplier you dictate to your suppliers what they should use what the quality has to be how they have to how they have to answer to a specific RFQ so the opportunity is now if we serve the procurement department and then Kate and they can dictate what a supplier should use. So we are not like not like why aren't we like also pushing them to like Leo agents that are also helping them automate the work owning them both sides of of the transaction. And I know we were talking earlier today about you know how can you have two parties on the same platform is how does that work. I think it could even extend until like legal work right and these are two very adversarial parties right but like if you have two law firms with clients with different interests but both benefit from knowing okay here's the latest draft here's where we are with the open issues here things that have been agreed upon and just even tracking that that doesn't really exist right now right that's all being human that's being created by humans and so be that coordination effort an agent could be doing yeah yeah it's a lot of work.

47:05Yeah it's a well it's super cool to think about how like you know both sides are kind of maybe evolving in tandem one side might be going a little bit faster as you're talking about Vlad but over time potentially people are just like on the same platform and actually it's like we better coordinated for every one of you. 100% because like like also like where we mentioned in the beginning right so like the obvious like the obvious question is like okay but like we also talk a lot about like negotiation agents what if like both parties have negotiation agents and prices 100% the point where it's like zero something like they have like different interests. But we also discuss that like price is like the outcome of like 5000 different like other tasks that happened and on those 5000 other tasks they have the same incentive sales wants to have as less like little friction as possible buyers want to have a really fast time to market right again like coming back to building aircraft building data centers you want this data center to be built as fast as possible you don't want to be building like you know

48:15like six months later just because it takes so much time to analyze all of the responses from suppliers and you want to make the sale also fast so like all of those 500 5000 other tasks the incentive is exactly the same and that's why you can deploy or Leo can deploy agents also on both sides doing like automating the work of all those other tasks this is the beautiful thing. I think that there used to be this logic of like you shouldn't customize your software too much to to one end user one and customer but I think something about LLM's and AI in general is like it might increasingly be possible to customize without slowing yourself down as a business too much. So I'm curious if if that is something that you're seeing Vlad or SEMA and and kind of what what does that mean for for and buyers of software.

49:15I think that overall principle there's a lot of forward to blood work happening right now and part of that is because the state of the customer data and understanding customer and is a lot harder than understanding and plus one and so we're certainly early stages of deployment overall and that's why there's still a lot of humans as part of this product. And by the way that is something that is harder for the incumbents to do because they're also like not set up in in a way to have even their the way their product feedback cycle works where they have a they have implementation teams that's very much an afterthought versus something that feeds into the product side. The beauty of AI is a it learns over time so that's what we're talking about learning loops and you have the right evil process you can do more and more complicated jobs over time and part of that is automating the deployment itself and so you can. And I'd be curious to hear how glad is doing it but a lot of our companies even. Are doing that at you know at a rapid clip where more of the customization a is being handled in an automated way and be the customer is able to turn the knobs and levers around customization via software.

50:29Versus okay I needed to bring in a the original I was like you know you brought an extension to do your SAP customization and now it's like okay I've got a forward deployed team that's going to build some you know help build and spec it out and ultimately it's going to be completely software. Yeah I mean that's the reason why like when you look at the or structure of Leo like 85% of the people are engineers and even if you look at like the people where you like they don't have an engineering title they're like mostly like an engine background and reason for that is. We obviously like we don't want to be a consulting company right so we make sure that we have like over all the best. Agents in indirect and direct and finance and those part but then like as you mentioned like there is a lot of like forward deployed work to do if you go to enterprises because they have like different nuances in their processes. But how we work is we as you mentioned like building a product in a way where it's like where we reducing this customization but also where it's a lot of like self service.

51:34So the job of our like FDE's and forward deployed engineers is on the one hand making it like self service but like in internities like automating their own job right so like their KPIs like you are seeing this happening like multiple times like you like you're literally your job is like to automate yourself and then if you automate yourself you go to the next task. I think it's like also like an approach at Google so is doing so like but you're 100% agree with you and that's exactly what we are what we are building and what are we how are we doing it. There was recently a very big system of record event and we're not we're not talking about Dreamforce we're talking about the the bot and buyer summit that that Leo hosted in New York. And and just wanted to kind of hear you know stories from the ground and and what what you're seeing among buyers what are people excited about what are people looking ahead toward what are people you know asking you for can you just kind of tell us some stories from from that day and that event.

52:42So I mean it was the third time that we are doing that we're doing this event now we did it in in New York just a few blocks from our from our office here and over 100 procurement leaders and right and what we like we made sure that we like when we do when we do such events that we only invite like high caliber people like see level CPR by vice president. And they're like two things very like very different of of how we do this all like white white is so amazed so the first thing is when we look at like how procurement used to work in the last 26 27 years. A lot of tools in much right so you can like they're like all those technology landscapes and you can find them on LinkedIn you will see like they're like 500 procurement tools but like if you like this is also the reason like but when you talk to procurement people like you know what I mean. Like very painful and like I challenge someone like to find someone who like loves to work with procurement like no one does like you can really state people hate working with procurement and when talking about the request us and talking about the suppliers and evil people in procurement hate working procurement.

53:50So like what's going on if they are like 1000 tools and the reason for that is like the all of the tools. They just made the process more efficient that's all but they never changed how do how those people actually work. And it's like crazy to see that they work in emails and on Microsoft Teams this and an ex of sheets and power points is like the main channel where they work on and it's like zero like zero AI enabled. Obviously in this part and the other thing is like we we give them like a very cross department perspective on procurement right so we are not talking about like look at this crazy invoice feature that we developed but we more looking like someone needs something. And in the end you have it on your table and this can be like across indirect direct logistics finance and you can see how we are doing it here and we also putting it into like more into like a physical world because like AI agents that's very abstract.

54:57So what we are doing is we building a booths and we even have this in our offices in also in New York where you can walk through the booths and experience like all of those agents like really hands on. And that's what the people are and it's like I mean the next event will be worth around 700 people so you can imagine how crazy this goes. Procurement people gone wild yeah yeah it's gonna be that one is in a unique. Yeah so we're doing them like in in Europe, Munich and in New York all the time. Well if you're listening and in procurement you know where to go maybe I guess one question one question for me. What do you think it takes to get to get people excited about procurement is it is it the agents is it the people is it the time save like what or like something else like you mentioned it like procurement is one of these things that I remember people aren't they don't like it's like a universally kind of dislike low NPS area I remember talking to a guy who is out of procurement like a seven or eight years ago as I was looking at this category and he was like I hate talking about this product I I use.

56:15I use, you know, this legacy system of record, I'm on Cooper, and I don't want to buy anything else. I don't want to talk about it. It's fine. It was like the most disgruntled customer call have ever done out of like millions of them. But I'm curious, yeah, what it is that you think, you know, really gets people excited about this category. Yeah, so I mean, it's like, it's like, and this is also why I like procurement. It's like, it's on one hand, like, so like the reason why we started in procurement, like, it's not essentially like what happened, but like, how people reacted it. Right. So if you talk to the people, they're like, really frustrated. So this means it's in like highly emotional topic. But it's like, like, let's be honest, like, be to be sourced, okay. But it's like highly emotional. So that's a good thing. And then if you combine this with like something which is boring and niche, there's also an advantage because, again, like, it's, it's like also easy or easy for us to amaze those people, right? Because like,

57:15the really last revolution they have seen is like 20 years ago. And then maybe a nicer user interface 10 years ago, but nothing else happened. And so you have like, boring, highly emotional, and then plus crazy business impact. Right. So like, it feels like it's unnecessary, but like, I told you like some examples. So like, it has like, obviously crazy PNL impact, but it has impact on like the whole economy. Right. So we're like, we are talking about like, how data centers are built, how aircrafts are built, how cars are built, how drones are built. So it's extremely important that you have a fixed procurement process, not only to like make it happen and build but then also when you talk about like when we look at like the competitive landscape. So to get like 1% margin increase, you need to make 10% more revenue, 10% more sales. So like, if you just

58:15manage to get like 1% savings, it's like equals like 10% of sales that you have to do to get the same outcome in your PNL. So it's tremendously important. And like you combine all of those three things and then you have like a trillion dollar business opportunity. That's my opinion. Like procurement, but they are probably also like other things that like emotional, boring, and have a crazy business impact. Well, well, thank you so much for joining us. This was a ton of fun. Thanks for listening to this episode of the A60Z podcast. If you like this episode, be sure to like, comment, subscribe, leave us a rating or review, and share it with your friends and family. For more episodes, go to YouTube, Apple Podcasts, and Spotify. Follow us on x at a16z and subscribe to our substack at a16z.substack.com. Thanks again for listening, and I'll see you in the next episode. As a reminder, the content here is for informational purposes only.

59:15Should not be taken as legal business, tax, or investment advice, or be used to evaluate any investment or security and is not directed at any investors or potential investors in any a16z Please note that A16z and its affiliates may also maintain investments in the companies discussed in this podcast. For more details, including a link to our investments, please see a16z.com forward slash disclosures.