Your AI Agent Is Costing You More Than You Think

0:00Hello, this is a three day weekend here in the US, Labor Day weekend. So I decided to air an episode that I recorded late last year with O'Reilly and our guest is Abby Aryan and the focus of the discussion will be inference, which is a topic that has grown in importance this year, even. Right? So because many more of us are using frontier models much more heavily and much more intensively. So I hope you enjoyed the episode. And if you haven't, yes, please subscribe to our newsletter, which you can find at gradientflow.substab.com. So I hope you enjoyed the episode. All right. So today we have Abby Aryan. She is the author of the variety book on LLM Ops, as well as the founder of Abidei. So Abby, welcome to the podcast.

1:03Thank you so much, Betty. All right. Let's start actually with the book, which I just, I confess, I just cracked open. LLM Ops. So people probably listening to this have heard of MLOps. So at a high level, I guess the models have changed. They're bigger, they're generative, and so on and so forth. So since you've written this book, have you seen kind of a wider acceptance of the need for LLM Ops? I think more recently, there are more infrastructure companies. So there was a conference happening recently and there was this sort of perception that was sort of messaging across the conference, which was MLOps is dead, although I don't sort of agree with that, but there's a big difference that I think companies have started to pick up on more recently as the infrastructure around this space has sort of started to improve in. They're starting to realize how different were the pipelines that we were managing earlier, especially for the older companies. Like Snorkel, that were in this space for like years and years before the average models came in.

2:08The way they were handling data pipelines and sort of even the observability platforms that we're seeing today, they have changed tremendously. And so what about Abby, the general, so we don't have to go into specific tools, but we can if you want, but you know, if you look at the old MLOps person, and then fast forward, this person is now an LLM Ops person. So on a day-to-day basis, do their suite of tools have they changed massively? I think for MLOps person, the focus was very much around this is my model, how do I containerize my model and how do I put it in production? That was the entire problem and you know, the most of the workers around, can I containerize it? What are the best practices around how I arrange my repository? Are we using templates? Drollbacks happen but not as much because most of the times the stuff was tested and there was not too much interterministic behavior within the models itself. Now that has changed, which is most of

3:14the LLM Ops engineers, their biggest job right now is doing phenobstery, which is controlling the cost. First models are massive. The second thing which has been a big difference is we have shifted from how can we build systems to how can we build systems that can perform and not just perform technically but perform behaviorally as well. What is the cost of the model but also what are the latency? What's the third what looking like? How are we managing the memory across different tasks? So the problem has really shifted when we talk about it. So a lot of focus for MLOps let's create fantastic dashboards that can do everything. Now right now it's no matter which dashboard you create but the monitoring is a very very dynamic. So yeah yeah and actually as you were talking there you know I started thinking yeah of course obviously now the inference is essentially a distributed computing problem right so it was not the case before. Now you have different faces

4:18even of the computation during inference right so you have the pre-fill face and the decode face and then you might need different setups for those. So anecdotally Abby were the people who were MLOps people did they successfully migrate themselves were they able to upscale themselves to become LLM Ops engineers? I know a couple of friends who were MLOps engineers they were teaching MLOps as well which is data breaks folks and VTs and they are now transitioning to LLM Ops but the way they started about is they started focusing very much on can we do it well for these models they weren't really dealing with the infrastructure side of it yet and that's that was their slow transition and right now they're very much at that point where they're thinking okay can we make it easy to just catch these problems within the model inferencing itself but a lot of other problems still stay installed. Then the other side which is like a lot of software engineers who anchored the field and

5:21became AI engineers they have a much easier transition because software the way I look at large and rich models is not just as another machine learning model but you know literally like software 3.0 in that way which is it's an end to end system that will run independently now model isn't just something you plug in but the model is the product tree. So for those people most of this software was built around these ideas which is you know we need strong cohesion we need glow coupling we need to think about how are we doing microservices how the communication happens between different tools that we're using how are we calling up endpoints how are we securing up our endpoints really those questions come easier so the system designs side of things comes easier to people who work in traditional software engineering so the transition has been a little easier for them as compared to people who work traditionally like MLOx engineers. Yeah yeah yeah and hopefully your book will help some of these MLOx people upskilled themselves into this new world all right so let's pivot quickly to agents right so obviously it's a

6:28buzzword just like anything in the space it means different things to different teams so how do you distinguish agentic systems yourself. So there are two words in the space one is agents one is agentic workflows so basically agents are the components really or you can call them the model itself but they're trying to figure out what you meant even if you forgot to tell them that's the core work of an agent and the work of workflow or an agentic workflow and agentic system if you want to call it is to tell these agents what to actually do so one is responsible for execution the other is responsible for the planning side of things. Yeah yeah so I think sometimes when the tech journal is right about these things I think the general public gets the notion that there's this monolithic model that does everything right so but the reality is most teams are moving away from

7:28that design as you describe right so they have an agent that acts as an orchestrator or planner and then parcels out the different steps or tasks needed and then maybe reassembles in the end right. Coming back to your point it's now less of a problem of machine-hinding it again more like a distributed systems problem because we have multiple agents some of these agents will have more load which is they will be like let's say the front-end agents which is communicating to a lot of people. Obviously on the GPUs these need more distribution and when it comes to the other agents that may not be used as much they can be provision based on this is the need and this is the availability that we have so all of that provisioning again is a problem the communication is a problem setting up test across different tasks itself within any dialogue workflow. Now that becomes a problem which is where alloy people are trying to implement cont-extensionally but it's a very complicated problem to solve. Yeah and then there's happy there's also the the problem of compounding

8:31reliability right so let's say for example you have an agentic workflow where one agent passes off to another agent and yet to another third agent so each agent may have a certain amount of reliability but it compounds over time right so it compounds across this pipeline which makes it more challenging. Yep and that's where there's a lot of research we're going on in the space because it's an idea that I've talked about in the book as well which is I basically mentioned it which is obviously at that point when I was writing the book especially like chapter for in which a lot of these ideas were described I was very initially mentioned it which is most of the companies right now are at a monolithic architecture but it's not going to be able to sustain as we go towards application we have to go towards a microservices architecture and the moment we go towards microservices architecture there are a lot of problems one will be the hardware problem the other is consensus building which is like it you have three different agents spread across three different nodes which

9:33would be running very differently so let's say one is running on an edge 100 one is running on something else so how do we achieve consensus if even one of the node ends up hitting so that's an open research work which people are trying to figure out which is can we achieve consensus in agents based on whatever answer the majority is giving or how do we really think about it it should be set up a threshold at which if it's beyond this threshold thing you know this perfectly works so one of the three works that is trying to work in this space is called masjid so they're working on the research side of solving this problem itself in terms of like the tooling itself yeah by the way even back in the microservices days of in software architecture obviously people went overboard too so I think that as with any of these new things there's a bit of trial and error that you have to go through and the better you basically the the better you can test your

10:36systems and have a setup where you can reproduce and try different things the better off you are because probably many times your first stab at designing your system may not be the right one right yeah and this I'll give you two examples of this one so most of the work at AI companies they try to first to use a lot of agent like frameworks you know people have used crew people have used any and they've used uh I hate I hate those not that I hate but yeah sorry sorry my friends improve yeah we need to write this so we code as well and 90% of the people were working in the space seriously have already made that transition which is we're going to write it ourselves same happened for evaluation which is there were a lot of evaluation tools out there what they were doing on the surfaces literally just tracing tracing isn't really solving the problem which is it was just a beautiful dashboard that doesn't really serve much purpose may be for the business teams

11:36but at least for ML engineers who are supposed to debug these problems and you know optimize these systems essentially it was not giving much other than what is the error response that we are getting through everything so again for that one as well most of the companies have 12 of their own evaluation frameworks in house as with now the people who are just starting out obviously they're not but most of the companies that started working large language models in 2023 they've tried every tool out there to 2003 for and right now it's more and more people are staying away from the frameworks and like chain and everything I think there's there's people have undisturbed that most of the frameworks in this space are not super reliable and also kind of honestly a bit bloated right so they come with too many things that you don't need in many ways we're secure bloated codes as well so for example like I reported one of the security bloated codes with Lanchi as well with Lanch Smith back in 2024 so those things obviously get

12:38reported by people get worked on but the companies aren't really proactively working on closing those security bloated codes two open source projects that I like that are not specifically agentique are DSPY and DAML BAML I wanted to give them a shout out so this point I'm about to make there's no EC there's no clear cut answer but one thing I noticed Abby is that people will do the following right so I'm going to take something we do and I'm gonna build agents to do the same thing but the way we do things is I have a I'm just making this up right I have a project manager and then I have a designer I have role B role C and then there's certain emails being exchanged so then people the first tab is let's replicate not just the roles but kind of the exchange and communication and sometimes that actually increases the complexity of the design of your system because maybe you don't need to do it the way the humans do it right so maybe maybe if you go to automation and agents

13:41you don't have to in other words you don't have to over enter more fights your workflow right so what do you think about this observation a very interesting analogy will give you which is people that are trying to replicate intelligence without understanding what intelligence is same for consciousness everybody wants to replicate and create consciousness without understanding consciousness so the same is happening with this as well which is we are trying to replicate a human workflow without really understanding how humans work yeah yeah and sometimes sometimes the humans may not be the most efficient thing like they exchange five emails to do to arrive at something right so and humans are never context defined and in very limiting sense which is even if somebody's job is to do editing they're not just doing editing they're looking at the flow they're looking for a lot of things which you can't really define obviously you can over a period of time but it needs a lot of observation to understand and that skill also depends on who the person is different people

14:45have different skills as well so if we say most of the agentic systems right now they are just glorified say peer if it routines that's the way I look at them right now the if it recipes if this then that yeah yeah robotic process automation I guess is what people hold right so the other kind of thing that people I don't think understand just reading popular tech press is that agents have levels of autonomy right so most teams don't actually build an agent in unleashed full autonomous from day one right so I mean I guess the analogy would be in in self-driving cars they have different levels of automation so I think most enterprise AI teams realize that with agents you have to kind of treat them that way too depending on depending on the complexity and the importance of the workflow what do you think so you go you go first you know very much human is involved and then less and less human over time as you develop confidence in the agent right but I think it's not good practice to

15:48just kind of let an agent run wild especially right now right so it's not because there's no who's the person answering if the agent goes wrong and that was a question that has come up often so this is the work that we're doing at a by-trade which is trying to create like a decision layer on top of the knowledge retrieval layer most of the agents which are built using just slash language motors lillims I think people need to understand this part which is lillims are fantastic acknowledge retrieval but they do not know how they make decisions if you think agents are independent decision makers and they can figure things out no they cannot figure things out they can look at the database and try to do something now what they do may or may not be what you like no matter how many rules you define across that so what we really need to develop is some sort symbolic language around how these agents are working which is more like trying to give them a model of the word around what is the cause and effect with all of these decisions that you're making how do we prioritize one decision

16:52where the work was the reason behind that so that incurred decision making reasoning here has been the same part so you you brought up the topic of observability right so there's two schools of taught here as far as agentic observability the first one is we don't need new tools we have the tools we just have to apply it to agents and then the second of course is no no we have we have this is a new situation right so now we need to be able to do more the observability tools have to be more capable because we're dealing with non-deterministic systems and so maybe we need to capture more information along the way right so chains of decision reasoning, phraseability and so on so where do you where do you fall in this kind of spectrum of we don't need new tools or we need new tools we don't need new tools but we certainly need new frameworks and especially a new way of thinking which is observability in Amelot's word fantastic if it's just about tools now people have to stop thinking about observability as just to visibility into the system and start thinking of it as an

17:56anomaly detection problem and that was something I'd written in the book as well which is now it's no longer about can I see what my token length is no that's not enough you have to do look for anomalies at every single part of the layer across a lot of metrics so your position is we can use the existing tools we may have to log more things we may have to log more things and then start building even simple use simple MN models to be able to do anomaly detection think of managing any machine any LLM model any agent as really like fraud detection pipeline so every single time you're looking for what are the simplest signs of fraud and that can happen across various factors but we need a more logging and again you don't need external tools for that you can set up your own lockers as well most of the people I know have been setting up their own lockers within their companies so you can simply use telemetry to be able to define a set use the general logs and be also to be able to define your own custom logs as well depending on your

19:03agentic pipeline itself which is you can define this is what it's trying to do and log more things across those things and then start building small machine learning models to look for what's going on over there so what is the state of where are we how many teams are doing this uh very fear very very few maybe it's just the top bits if the ones who are doing reinforcement learning training and using RL environments because that's where they're getting their data to do other but people who are not using for not using RL to be able to retrain their models so to say they're not really doing much of this part there's still depending very much on the exterior of the tools I'll get back to RL in a second but one topic you raised uh when you pointed out to transition from ML to LLM ops was the importance of thin ops right so which is for our listeners basically managing your cloud computing costs or in this case increasingly mastering token

20:04economics right so because basically uh it's one of these things that I think can bite you for example uh the first time you use cloud code you go oh man this tool is powerful and then boom you get an email with a bill I see that's why it's powerful right so I think uh and you multiply that's across the board to teams who are starting to maybe deploy some of these things and you see the importance of a thin ops so where are we Abby as far as tooling for thin ops in the age of generative AI and also the uh the practice of thin ops in the age of generative AI uh less than 5% maybe maybe 2% of the way there oh really yeah but obviously everyone's aware of it right uh because at some point when when you deploy you become aware and not enough people because I think alloy people just think about thin ops as cloud basically the cloud cost and there are different kinds of cost in the cloud it's also one of the things people are not doing enough is not

21:08profiling their models properly which is where are the costs really coming from are our models compute are which are they're taking too much route now that's we are we are we using reasoning when we don't need it right exactly now that's a problem we saw very differently that's very yes you can look at the fusion define your own custom kernels right now there's massive number of people who think we need to rewrite kernels for everything it's only going to solve one problem which is the compute bound problem but it's not going to solve the memory bound problem which is your data engineering pipelines are what's going to solve your memory bound problems and that's not where that's where most of the focus is missing so I've also mentioned it in the book as well which is data engineering is the foundation of first being able to solve the problems and then we move to like the compute bound problems do not start optimizing the kernels over there and then the third part would be the communication bound problem which is how do we make these cheap views talk smarter with each other which is how do we figure out the agent consensus and all of those problems now that's

22:09a communication problem and that would happen when there are different levels of bandwidth that everybody is dealing with the internet bandwidth as well the kind of serving speed as well different kinds of cost and every kind of transitioning from neck one wall to another belief will not boosting in like if we're not really hosting our own infrastructure then that's a different problem because it depends on which server do you get assigned your GPU zone again or the inner connects between yeah yeah yeah I want to give a shout out to Ray I'm on advice with the end scale but because Ray basically is built for these sort of pipelines because it can do fine grain utilization and help you decide between CPU and GPU all right so yeah just generally you don't think that the teams are taking token economics seriously like are people I guess not because how many people have I heard talking about caching for example right you know because if you if if if it's if it's a

23:11query that it's a prompt that the llem kind of answered before why do you have to go through it again right so I think plenty of people have started implementing kv caching but they don't really know again one of the questions people don't understand is how much do we store need to store in the memory itself and how much do we need to store in the caching which is the big memory question so that's the one I don't think people are able to solve a lot of people are storing too much stuff in the caching being that should actually be stored in the RAM itself in the memory and they're generalistic applications that don't really understand this agent doesn't really need access to the memory there's there's no point it just loads on the interpret really so I think the problem isn't really caching the problem is that differentiation of understanding for people yeah yeah I just threw that out as one element right because obviously there's many things to mastering token economics so you you brought up reinforcement learning right so a few years ago obviously people got into really let's do fine tuning right then they quickly realized and actually fine tuning became

24:18easy because basically there became so many services where you can just focus on label data and then upload your label data boom come back for lunch you have a fine tune model but then people realize okay so fine tuning actually fine tune but the model that results in pretty dangry is as good as my fine tuning data and then obviously rag and context engineering came into the picture now it seems like more people are again talking about reinforcement learning but in the context of valilems and there's a lot of libraries many of them built on rain for example but it seems like what's missing Abby is that fine tuning got to the point where I can sit down a domain expert and say produce label data and basically the domain expert is a first class participant in fine tuning as best I can tell for reinforcement learning the tools aren't there yet the UX hasn't been figured out in order to you know bring in the domain experts as you know the first class citizen in the

25:20reinforcement learning process which they need to be because a lot of the stuff really resides in the rain the big problem here and very very much to the point of what you pointed out which is the tools hungry there and one very specific thing I can tell you is most of the reinforcement learning environments that you're seeing are static environments agents are not learning statically they are learning dynamically if your other environment cannot adapt dynamically which is basically in 2018 to 2019 alert as like the open AI jm and a lot of reinforcement learning like Greece were coming off there is a line of work which is called curriculum learning which is basically adapting your model models difficulty to the to the results itself so basically now that can be used in reinforcement learning but I've not seen any practical implementation of using curriculum learning for reinforcement learning environment so people create these environments fantastic they work well for a little bit tight and then they're become useless so

26:25that's that's where even open AI entropy those companies are struggling with as well which is they paid heavily in contracts which are ear long contracts to say can you build this vertical environment can you build that vertical environment and that works went drastically but once the model learns on it then there's nothing else to learn and then you go back into the question of is this state of fresh really is this adapting with the wording that becomes the same red problem over again so maybe the problem is with RL itself maybe maybe we need a different paradigm is just too hard right so let me close by looking to the future the first thing is I guess this space is moving so hard this might be impossible for us to ask but if you look out that's a 6 to 18 months what are some things in the research domain that you think are not being talking enough about that might produce enough practical utility that we will start hearing about them in 6 to 18 months one is how to profile your machine learning models like the entire systems

27:27entering so a lot of people do not understand them as systems but all yes models so that's one thing which will make the massive amount of difference which is there a lot of AI engineers today but we don't have enough system design engineers that's where this is this is something that Ion Stoika at sky computing lab has been giving a keynote to us yeah interesting the second part is I've heard most likely I think I'm optimistic about seeing curriculum learning applied reinforcement learning as well where our Arnold environments can adapt in real time so when we train agents on them they're dynamically adapting as well so that's also one of the words which is being done by labs like sakana which are working in artificial life and all of that stuff evolution of any kind of machine learning model that you're seeing the third thing where I feel like the community is falling behind massively is on the data engineering side that's that's where we have massive gains to get so on the data engineering side I'm happy to say that I advise several

28:33companies in this space that are completely focused on tools for this new workloads and these new data types so last question for our listeners who are listening what mindset shift or what scale do they need to pick up in order to position themselves in their career for the next 18 to 24 months for anybody who's an AI engineer machine learning engineer or LLMOP engineer or MLOB engineer first learning how to provide your models start picking up array very quickly as a tool that to just get started on to see how distributed systems work you can pick VLLM if you want but start understanding distributed systems first and once you start understanding those systems then start looking back into models itself and with that thank you happy you can follow the work of Abby Aryan online at abyan.com that's abyan.com please join the

29:40thousands of people who subscribe to our newsletter which you can find at gradientflow.substab.com and we are listener and reader supported so if you can please become a paid subscriber thanks for joining us if you like the show please subscribe and read us through Apple podcasts or overcasts or tune in.com or Spotify and never miss an episode the data exchange podcast is a property of gradientflow and I'll be back next week and we will do this all over again.