HN819: Recipes for Automation – A Look Inside Eric Chou’s AI Networking Cookbook

0:00Sponsor Ovic Network Management discovers your complete network inventory, displays real-time performance metrics, backs up configs, generates compliance reports, and more in an intuitive, simple-to-use tool that scales as big as you might need. Get your free 14-day trial at ovic.com slash heavy networking. That's a-u-v-i-k dot com slash heavy networking. Welcome to Heavy Networking! If you sort your skittles by color and the colors by visible light wavelength, then then rank severity of a network outage by the name of the skittle flavor you found you tried. I'm Ethan Banks with Drew Conray Murray, and if you'd like to connect with us because we are real people after all, you can do that on the Packet Brushers Community Slack group or LinkedIn, visit PacketBushers.net to join the Slack or IRC because IRC lives on much like IPv4. And hey, we're talking about adding artificial intelligence to your network automation toolbox with Eric Cho. Eric is the author of the AI Networking Cookbook available now, and Eric is also

1:02the host of the Network Automation Nerds podcast, which you can find right here on the Packet Pushers podcast network. Eric, welcome back to Heavy Networking! Man, it is good to have you here, and dude, I don't know how you do it, but you wrote another book. You wrote another book. Would you give us the 10,000-foot overview of what this book covers? Yeah, well, first of all, it's good to be back. It's highlighted my day to speak to my esteemed colleagues, so thank you for making my, what is the Tuesday, much more beautiful than the rest of the week. And yeah, no, I didn't, I guess I didn't learn the first six times, right? That was fun, sorry. And I wrote another book, I took on the task. I think basically what it wants to do was, if I'm learning AI anyways, I want to document it, I want to do it in a systematic way, and want to bring people along with me, if they're willing to spare a few dollars to buy the book, to listen to my podcast or whatnot. So that was the basic premise. And little did I know, right? Like a 12 months later, or like, whatever

2:03however long between the planning through the actual execution was. Now, this is the book, the book was released in January. And basically it follows in same pattern as my previous work, Mastering Python Networking, I want to set up the foundational piece. So essentially, you know, for example, setting up chat GPT, what is the LA and what are some of the parameters that we set? How do you set up local models? How do you use it locally? How do you use it in a closed gap system? And then move on to some of the aspects, like setting up the front end, what about if I wanted to run, you know, some sort of decoupling the back end, if I want to just use API, then we use more into the more advanced topics, like MCP or, you know, building a co-pilot. How do I evaluate AI models, basically, right? So we all know, you know, a model could cost different costs on tokens. So if I could use a 3.5, you know, turbo, why do I want to use fully? Why do I want to use five and stuff like that? So go or you're laying down the foundation,

3:04moving up one layer and then going out into the more advanced topic, once we lose, lay those foundations down. So it sounds like you said, laying the foundations, meaning it sounds like that's the kind of stuff that are more principles and not tied to specific models or changes in the technology. Yeah, hopefully. So I think, I think, you know, one thing that that people would find out, there's no like OSI layer within AI, right? Like there's nothing that everybody just, you know, couple themselves into the physical layer and then everything that's, you know, more or less, the coming with the physical aspect of networking. So with AI and different vendors, you know, the open AI's have different, you know, parameters terms or terminologies, charging model, everything else, the system messages, then, uh, and Thropic versus, like, say, cursor and they have different interfaces. But yeah, you're right. So the aim is really just to set something up to get that first win to know, uh, back hello world, then, um, and then we go into

4:06more, more or less the other aspect. So for example, you know, uh, we obviously, you need to set up at account. So we need to pick something to set up at account. So we set out open AI and then, just to get that first win, we don't go off into, say, how do you set by on Thropic and set up five other, you know, like providers, we go directly into how do you change the temperature, meaning, how do you change the creativeness of your answer? How do you, uh, include your system message? So you don't have to type it into the problem over and over again. So yeah, so laying down the foundation, get that first win, and then chapter two will go directly into, now that you have that first win within the GUI, how do you use a script to automatically query that? And then we change direction and say, well, you know, now that we have those down, that you could, you know, go explore further. However, I know there's a certain amount of people who doesn't feel comfortable in talking to cloud-based LOMs. So how do we set up locally? And then we pick one thing, right? There's also, there many, maybe not many, but multiple ways that you could run LL locally.

5:08And I just pick one, Olana, and then just go from there and repeat and repeat, rinse and repeat, but doing it locally. So I think I, I hope that gave kind of a good overview of what I've made by foundation. Yeah, for sure. And just let's just set it up right away. Where can folks get this book if they're interested? You could get at most retailers. Barnes and nobos, if you still have physical bookstore, I don't know if they have physical bookstore, you know, selling the physical bookstore, because the last few times I've been to the bookstores, they're all very fiction-focused. So that's right up your your cup of tea, Drew, but for me, you know, mostly when I look at the the sales breakouts, most of them come from Amazon. So I think, you know, unable, most of the time people get it from Amazon, you could also get it directly from Pact. One thing that I do want to mention is we know your reading habits have changed. So if you, even if you get it from Amazon, there's a QR code in the book where you get a digital format from, from Pact directly, so that it's that's EPUB. If you use, you know, the EPUB formatting, if you use Kindo, if you use,

6:09you know, if you read a reading PDF, you could all do that. So, so yeah, thanks for that, that plug-in. You could get it at all the rebuttellers. Kindle. Eric, I bought your book on Kindle, via Amazon downloaded it, and I was reading it on my Mac, on the big screen, on my big screen with with the Kindle reader. Nicely laid out easy to follow everything, both the code sectors of which you have a lot, a lot of different code and or text screen output that's there, as well as your explanatory paragraphs that give you context on what's going on. And it's wonderful. It's very easy to read digitally, and also people really like paper because reasons, and that's fine too, of course, but if you're wondering, well, how this render in Kindle, pretty good by my eyes, I thought was an easy read. Yeah, I think for technical books, I think a very a little bit different than the experience that you read, like a fiction or, you know, biography or something like that, it's because there's a lot of code involved. So, just like all the other books, the code are in the

7:09GitHub repository. It's not guarded by like a pay or anything like that. So, even if you have about the book, you could go, you know, look at the code yourself. And what I also try to do, I don't know if other people have noticed, but I try to do a lot of explanation within the code comments, so that even if you just grab the code directly from GitHub or whatnot, you could actually realize the flow, you don't need a companion book to to know, you know, this is the section where we hit the response or something like that. However, you know, if you want to buy the book, yes, if you want more more information or if you want to have like a curated flow of thoughts, right, going from basic to more events, then yes, you know, go go read the book, yes. Now, you have a community that people can join if they want to keep up with you and keep up with each other as they work through the book, right? Yes, yes, absolutely. So, AI is moving so fast, I just recently decommissioned my, you know, Network Automation.community site and migrating everything

8:14to this new Network Automationers.com that you could sign up for free, or if you choose to, there's like early access for premium contents, so either way is fine, but you know, essentially when you sign up, you'll get notification on whenever I publish something new, and also you get invite to the Discord community where there's a real-time chat on people who are, you know, I guess, you know, more active on any one time for questions and answers. That's why I'll hang out most of the time, too. So yeah, please do feel free to join. Like I said, AI is moving so fast, and even in just that five minutes that we talked about, or eight minutes that we talked about this topic, right? We already mentioned there's multiple ways of running AI locally. I've only mentioned one, you know, a lot more. There's LM Studio, which I think it's a little bit more beginner-friendly, but let's code, like, let's developer mindset. So, you know, that's where I will try to supplement whatever that I have in the book to say, yeah, you know, Obama's great, but if you want something that's, you know, downloadable

9:17in a package that's API ready, then, you know, go for LM Studio, for example, and that's actually where I will start posting and try to make up for, you know, kind of the latency, which is inherently built into book publishing. Yeah. So that's network automation nerds.com, nerds plural. Right. So, no automation. So, two ends in between. They were going to mention nerds.com, and we'll have that in the in the short notes, but, uh, but yeah, you know, we'll. So, what level of network engineer would you say this book is aimed at? I think probably someone who's passed, like, you know, a plus or CCNA, and have a little bit of background in Python, because all the code are in Python. So, um, you know, if you want to just follow a query, that's great. You know, and that that's workable. But if you want to get the most out of it, obviously, you know, you want to be able to interpret the code or have AI explain the code to you. If you're willing to learn a little bit of Python, at least, you know, you don't get intimidated when you see, you know, spacing and co-blocks or indentation, that sort of stuff. So, I would say,

10:22to get the most out of the book, you want to have basic, uh, networking knowledge and basic Python knowledge. Yeah, and it is truly basic Python. It's not like this is super complex, excessively structured or millions of lines of coded all. I mean, they are straightforward examples, Eric. I wouldn't be able to write them if it's, that's a good complex. You're right on my level, man. Like, you know, you just have to cope with, uh, like that, that's the ceiling right there. So that that's seen me is pretty low as you mentioned. Because it's lots of us that are trying to learn something that we're doing it on a budget. What costs do I need to plan for it to be able to apply AI for for network engineering? Yeah, I think that's a great question. Um, when I first started writing the book, there was actually their free tier for most of a lot of the provider open AI, I believe actually offered a free tier and a lot of providers actually give you free tokens. I curse sure, for example, give you some free tokens to play with until you decide to buy the freemium model. But less at last, I think by the time the book published, even when we're doing the final editing,

11:27um, they stopped offering free, free trials. So you do need to, you know, actually, uh, provide, prepare a few bucks to, to get the, to assign a, you know, have a credit card verification. Same as, uh, Amazon AWS, right? So AWS actually asked for credit card information upfront, um, but, um, you know, they may or may not charge you. So I think that's kind of the, the mindset that you need to have, but just case in point, I think I spend, I put in $20 in the beginning of the book writing for open AI and I don't think I've used up all of it, you know, by the time I, oh, end with it. Okay. So you didn't sign up like 20 bucks a month, you just bought X number of tokens or something? Yeah. So again, I think that's a different charge model for different providers for open AI, at least on the API layer, you know, that's one thing that's explained in chapter one is that the, the open AI playground or, you know, platform that they're calling it out is different than the chat GPT that you launch your interface with. So when you, when you, you know, subscribe to chat GPT, for example,

12:30you actually have to pay monthly subscription. I think whatever, 1999 or, you know, whatever that price was for you to, you know, keep your history, keep your chat, prompt, active and so on. But for a platform where you use mostly, you know, API access and help you, you know, fine tune as I were explaining the book, all of that are usage based. So you actually put up, you know, say $20 and then you, you know, you kind of exhaust that token. And because we don't use that token, you don't use tokens that much, usually you exhaust token where you need to put in a lot of contacts, like thousands or even, you know, hundreds of thousands of words into one window, then use a lot of token to exhaust that. And therefore, if we're just using basic scripts, you know, a case in point is that, you know, you won't exhaust your tokens even if you do the whole book examples. However, you know, I don't use just open AI. I want to provide more variety. So andthropic, you know, actually have a subscription model where clock code is actually, you know,

13:31needs a monthly subscription cursor. If you do it monthly subscription, but again, you know, they're actually not that expensive. So I believe all of them are less than $20. So, you know, pick and choose what you want to do. Spend a few bucks is worth it. And at least, you know, me, I use everything. I use, you know, paid account for everything. Gem and I open AI, you know, cursor and andthropic and everything else that's, you know, video editing or, you know, all the other services can have that private account too. So I've tried it all. And to me, it's worth it. Now, you did mention running a model locally. Is that, I mean, in theory, I could download a model and run it for free if I'm running it locally. Is that, is that worth it? Yeah, I think it's worth it to try it. And I think it's worth it to to set it up just to know if it's for you. The models that you download. So, so let's go back and say, what is a model

14:32in the cloud offer you, right? So the model in the cloud offer you as a single interface where you can pick and choose the latest, the greatest model or or not. So, for example, you know, if you running in the cloud, you get to just using a drop-down menu, choose between GPT-5, GPT-40, for turbo 3.5, whatever. And same thing with inthropic, you know, Hayaku or any of the other models how you could choose. But in the locally, you don't get those models, right? Those are proprietary models that the cloud providers offer. And what you would do is you would set up these containers, right? Like these carriers of models and you would in the book, I will use a long-mizing example. You have two Docker containers. One, how's this the model? And the other one is the interface that, you know, interface with you on, you know, your chat interface or whatnot. And those models you will go, you will grab from like a, a, this quote-unquote open marketplace, right? And then so you could choose, you could download in the same format. And you really don't, your judgment on which

15:33model to choose really depends on user feedback. It's almost like going on Amazon and look at, oh, there's a thousand reviews. I, and that's, you know, 4.8. And I suppose that's better than others. Or they will explain to you that this is, um, I forgot the term they use. But essentially, it's like untethered, right? So, so I did, I did compare and contrast, you know, one of the book examples on, um, if you want to write code, let's go down though, like a regular, you know, model versus like a, a model with that specific trailer toer coding. And, um, and then, you know, let's go down with another model, which is not censored. And then look at all these answered. And so a lot of times, you know, you get, you don't even think about the things that you get from a curated model, like a chat GPT, right? Like they, or entropic, let's use cloud, for example, that, you know, when you ask something more sensitive, like, um, let me, let me go on like a little,

16:34little tangent here. Say, you know, if I'm depressed and I want to ask about how do I, like, in my life, whatever, right? Like really scary. These curated models are, are trained so that they don't tell you they try to tell you to go get help. Here's the way to, you know, kind of alleviate your symptoms. And in the on sensor model that you download from this marketplace, may or may not have done that. So I think that's kind of the guard rules that you have to keep in mind is that, yes, you could run it locally. You could, it's worth it to try it just so that you know the process. But whether you want to use it, um, you know, primarily where you want to use a production environment, the scope that you want to use it, that's, that's something that to be put more thoughts into. But it's not just the model you pick and run locally. It's a perform, my experience is it's a performance thing. So I loaded, I don't know, several different models, and at the end of the day, even with a Mac M4 Pro, that's got a good bit of GPU behind it. It takes a while to turn through my query and come back with a response.

17:35Yeah, absolutely. I think, I think you hit the same bottleneck as I did that, um, I think the most, uh, the biggest model I could run is that I had a reason of a species like seven billion parameters. And, you know, they're already 13 billion parameters and, you know, gazillions out in the cloud, uh, cloud models. So, uh, so definitely that you have to, uh, tailor your expectation on, you know, just matching your hardware. And then you could get a little bit more improvement on, you know, maybe like you said, like a Mac, mini M4 that has a ton of GPUs. But then you have to have that software matching that, um, there's like a little graphic interface, which I did not go into the, the book again, like things are just moving so fast. But there's a little graphing interface that you could see that when you run, say, LM studio, how many, uh, tasks are being offloaded to your GPU versus your CPU? So if your task stays on CPU for too long, definitely you're going to see that as slowness. But if you see your, your GPU, utilization hanging up, that chances are you getting more performance. So I think that's a

18:38combination of both your hardware limitation as your software limitation. And again, if you run it locally, you're on your own, you're on your own tech support. You know, like, you know, uh, there's nobody, the buck stops here right here, right new friend of the screen versus like, Hey, you know, um, I could, you know, point fingers and, and the switch happily switched to another model, uh, when it's like anthropical one that goes into your $20, right? Like, uh, I don't, I don't want to learn to be my own tech support. I want to endure the IT services. So, uh, so that's what that $20 would get you. It's these, uh, user nice to have. And these days, if you're going to get out new hardware, that's going to be a lot more than $20. So yeah, they actually have, I heard they actually have like a shortage of Mac minis because that's like the cheapest, uh, that you could get to run, uh, open cloud. And several, you know, naming changes after, right? Like it's called open cloud now, but they're, it's so popular that it's actually causing a little bit of a shortage on the, the Mac minis. So, yeah, definitely even, if you're for the cheapest one that you could buy recently, it's like,

19:40a couple hundred dollars. So before you wrote this book and after writing this book, has your opinion changed on the utility or usefulness of AI for network engineers for network automation? Yeah, forget before and after, man, you asked me yesterday and today, depending on the day of the week and the time of the day, my opinion constantly changes in absolutely on just the, the knowing, um, so, uh, I thought I would do something different this time before the book launch is like, I have like T minus five, T minus four, T minus two, and kind of build up the, uh, quote unquote, excitement for the book launch. And on one of the days was, uh, mastery is optionality. So I think that's the main thing that I've learned from just learning in general, learning AI in particular, that, um, once you learn something, you have the option to put it down or use it or not to use it. So I think that's the biggest opinion that I've changed is that AI is great for some task and, uh, I would, you know,

20:43personally not use AI for some things because I want to retain my humanity. You know, I want to still feel like a person like the thing I did, I still want to feel like I own it, that I wouldn't use AI for. But in order to get to that point, you have to learn it. You have to learn it enough to, to be able to say no, right? So, uh, so I think that's the biggest opinions that I have changed is that I want to learn as much as I could, so I make that choice, make that option and be mindful about the option versus, I don't do it because I can't, right? So I, I will learn how to run a local LLM model. And I choose not to because of X, Y, Z and everything that Ethan said. But, um, but not just, you know, just the optionality is, is what I'm after. So I think that's the biggest thing that I've changed before and after learning all these stuff and going, you know, zigzag and, you know, in whichever way that I went to bang my head and, you know, come back. So part of your learning wasn't just learning how to use AI, but also figuring out where AI could be useful and where probably isn't going to be useful. Yeah, absolutely. And

21:45it's just my personal choice. And there's something that I talked to my, uh, daughters a lot because, you know, schools is just, it being turned upside down with a high, that just the, that just beyond us, right? Like people actually use AI for all kinds of crazy stuff, including image generation. And school is one of those places where, you know, people or, or the students have haven't had like that prefrontal, uh, fully grown or like they haven't formed their opinions, uh, clearly yet. So, so I think, um, that's what that's, that's kind of related to what we talked about here is that, you know, I want to be able to be mindful about the choices I make or as much as possible, as opposed to just things that, you know, algorithm pushing in front of me. Well, Eric, I want to spend the rest of the show diving into one of your chapters and you, I think it was chapter three was all about prompt engineering, which early on when I first heard their camera, I was like, really, I got to learn how to say words, the robot. That doesn't seem like a skill. So I didn't expect you to spare you later on. So let's talk about, let's talk

22:51about prompt engineering. So explain what, why carefully crafting a prompt is an important thing. And do it in the context of how in the LLM actually works. Yeah, I think that's a, that's a great point that, you know, all of us have that initial resistance, right? Like if it's so great, if AI is the next big thing, why do I even have to provide context for it? It should just automatically know. But is it though? Is it? Because if you think about AI as a partner or as a colleague, for example, and very capable colleague that, you know, let's just say some, you know, you're in the network engineering team and you hire some, you know, superstar at network engineer and he or she started, would you just tell him or her to go, you know, work on production device? No, not really, right? The person that is very well versed in BGP OSPF and networking technology, but you want to give the person context, right? Here's how we do things where under strict restrictions on some of the security stuff, this is how we name our devices. Here's how

23:54architecture is and so on and so forth. So all of these, if I couple those into context, that is why you want to help AI to do is to give it as much context as possible. So you limit it from just this, you know, open-ended question into a more specific thing that you want it to do. So yeah, and then we should go into more details on that, but I'll stop right there and see that actually makes sense. Well, we can give it more context, but an LLM does not think like a human thinks that is in that sense in the context of reasoning or thinking, providing context isn't helping the LLM in that sense, but it is helping the LLM because of how it actually delivers a response to us vector database words that have a higher probability of following on to the prompt that we've given them. Is that a, I have that roughly correct, right? Yeah, so I think that's precisely because LLM cannot reason, it cannot understand like a subtle use of human language.

24:55That's why you want to give it as much information as possible, right? It's not going to read between your lines on, you hate JSON, so give me, you know, like, I don't know, YAMO file. So you have to specifically say, give me the JSON file, give me an YAMO file, and if you remember previous generations of, previous generations of AI interactions on the first time that you saw chat, chat box, right, chat GPT or any other chat box that you have, that's all you have. And the context you could give it is you have the copy and paste everything in there. If you want to give it examples of your router config, you got a copy and paste into that router config and say, hey, look at this router config and tell me XYZ, why do we have, you know, this accult, like what is this line, you know, 56 accult does or whatnot, you have to copy all of that in and you quickly run out of this, this context window that you have to copy and paste everything in. So the next step on, you know, that evolution is we have system messages, so you don't have to repeat your contacts every single time. So you could, you know, in the system message,

26:00you say, you're a senior network engineer, specializing in the organization, have a preference toward Python and JSON and, you know, you're very, yada, yada, yada, right. So that system message actually stays with all the chats across. So again, you're providing more contacts, you just don't have to do it over and over again. We're taking a quick break for sponsor, OVIC network management. And if you've not heard of OVIC, they've been around for many years and they have the goal of making it easier to run your network. Good visibility into what's going on, network maps, proactive alerting so that you're aware of a problem before the help that's not sliding up, compliance reporting to make the auditor happy, can think back up so you can go back to a non-goog state if you need to, you get the idea. Now here's how OVIC works. You deploy an OVIC collector, which discovers your network and then figures out your network topology and maps it all out for. You don't even have to know how the network is connected. OVIC is going to talk to your network devices and figure that out. All right. Now that OVIC knows your network topology, it's going to begin monitoring your network. And you don't have to tell OVIC everything you wanted to monitor.

27:03There are 64 plus pre-configured alerts built in, dashboards for every device, and OVIC can talk all kinds of different protocols. S and MP can talk APIs. If you've got devices with APIs, it can, it knows how to parse CIS logs. It can do net flow and do flow collection and more. So you get this really useful repository of all of your network telemetry in one place. There you go. Ethan, that sounds like bread-like green light network management, more or less. Yeah, but there's a lot more than that. You're going to get inventory tracking. You're going to get config backup and restore, as well as application identification and geodated through traffic analysis and OVIC scales too. If you have one campus or you're managing the networks of many different clients that you might have, OVIC can get it done. So check out oVIC.com slash heavy networking to see your live network map in minutes with their 14-day free trial. That is a-u-vik.com slash heavy networking. OVIC.com slash heavy networking. If you're in the market for a new network management

28:05solution, and now back to the show. When you were saying context, I was thinking specificity, as opposed to saying what went wrong on the WAN, I would say, what happened on this router located in California between 10 and AM and 2PM. But it sounds like the other version of context is giving this LLM a persona, like you're a senior network engineer, and that's also, does that count as that context? Yeah, or like you could tell it to returning the response in the almost voice, right? So yeah, that's part of the context, in my opinion, that's because you could actually say explain it to someone who's a very intelligent person, but not a network engineer, or you could explain back into as a senior network engineer. Could you give an example of what you mean then by a prompt with context and a prompt without context? Sure. So I think the context is half what you said about context as in the environment. So I want to say, hey, give me a configuration

29:10back in terms of iOS XC, or give me a configuration back in Junos. And then I want to attach a few files and say, here's the example of our naming convention, or this is an example of iOS, configure, what not. So those are the technical context. And then I would actually give it more persona-wise and say, hey, explain back to me as if I am a very intelligent business person, IE my manager, but not a very crazy about, you know, intrinsiccy of, you know, routing protocols and so that. Okay. So I think those are the ones that the personality context is coming back in the environment, right? Like, let's just say I'm writing a mob, so I need to break it down step by step versus I just want the configuration copy and paste that I could evaluate humane the loop and the copy back in. So I think a well, well crafted prompt would be example and context. So, you know, as much as you could cram in as you could and however you want

30:14to do. So I think a lot of times what the experience have improved on is how many of these contacts you have to do manually versus the contacts that's, you know, kind of implied. So none are the then obvious then the break between a chat with system messages then with agentic. So I think, you know, in a agentic world where, you know, if you have some sort of coding assistant, like cursor, for example, or a cloud cloud code, for example, that they actually, you know, if you launch it, then some of the contacts are implied. First of all, you want to write code. And second is if you have like Python's that you'll crawl through your folder where you locate your code, you'll see that, oh, you use Python. Oh, you has virtual environment. You have all these other stuff that is set up. So then it has all those contacts implied for you. So that I think that drastically improve your experience where I don't have to tell it, hey, activate

31:19the virtual environment. Hey, you know, I want to use Python and here's my standard on writing code. It's going to crawl through all that and say, oh, you always, you use, even though, you know, you use Python, but you use data safety, right? Like you, you were specified. I want to return a string. I want to return a number. It will look at your code and say, oh, that's, that's what you want. And then implicitly, that when it helped you write that code next time, it will write something that's data, data type safe or something like that. So I think, well, could we also say, Eric, that there's a global and a local context from a certain point of view. That is, I was using Cloud for a while. I'm not anymore. I'm using Gemini pretty much always, but I remember in my cloud configuration, I could give it a, what I would consider a global context. It was right in the settings. These are things you should know about me. I'm a network engineer. I know this. I have this level of expertise. I'm comfortable with these technologies and not these other ones and so on. So it always knew that about me when I would start a new chat. And then within the new chat, I could start, I could then have local context and describe a whole bunch of other things

32:22that I'm looking for from that interaction. Yeah. Absolutely. I would probably classify as a context, but in the sub category of guardrails, right? Like so, so you could definitely have guardrails on the global level. Let's just say, you know, we use the, our company, you know, we use poetry. So everybody should use poetry on, you know, package management. So in the global level, I want to imply in this depends on, you know, your, your tool of choice, it could be just this hidden dot cloth file or this hidden dot cursor file that specify, hey, you should always use poetry for package management unless specified otherwise. Then the next new hire comes in, you know, you don't even have to repeat that. So when he uses the same tool, that's a guardrail that's operated under on the global level. And on, on the local level, when you launch this touch subject, like just to say our team, you know, our team, you know, prefers to use, you know, plain vanilla JavaScript versus some type of framework like view or react or anything like that.

33:26So that on the local level, when you and I, the three of us are collaborating on a project, we want to have that local guardrail. And again, it's a hidden dot cursor file and the more specific wins that, you know, we'll specify our requirement, use plain vanilla JavaScript versus any type of framework when preferred online side overrided. So I think those are the, I think, I guess the guardrail context that we could specify that that is both on the, on the global level and on the local level. So we're saying with prompt engineering, then step one, don't be vague, be specific, give lots of context, explain yourself, imagine you're explaining this to someone who hasn't heard about any of this before. And yes, depending on the model and it's, it's a model, that's for writing code, you can make a few assumptions about what it knows or what it's going to assume about what you're asking, but generally speaking, be specific, write a longer prompt, take some time to think through all the facts that are relevant to the thing that you're asking the

34:26AI to do. And then you're going to get a good answer. If you're vague, you're going to get not such a good answer. Yeah, exactly. And if you feel exhausted just hearing those, then yes, me too. And the companies have heard you. So, so there's a whole kinds of guardrails or additional contact that they help you to be persistent. So for example, Clot, so there's that Clot IMD that everything that you talked about, you could either put in a hidden file, you could put into Clot IMD and we launched Clot, it's going to, you know, supposed to read that. And if you're using cross, cross agents, cross provider, supposedly, right, this is where like things get a little bit iffy, there's no like rfc that, you know, mandates you to do it. But if you put into agents.md, then all the, you know, Clot or and Clot or Gemini or whoever, like launch, this should look at this agent's IMD and follow the instructions and guardrail. Does it always

35:26work? I don't know, you tell me. I haven't tried it. But, but essentially, you could actually do, do all of that and try. And once you work it out, right? So, so here's the thing, prompt engineering is about iterations. So you would start on something and then you would work on something and, you know, find the most, the best match of, you know, verbosity versus the results that you would get. At some point, you're going to run out of things to say. And like, you could only be as verbose as possible as, as you could to a certain point. And then there's like diminishing returns, right? I will add a lot more examples. But the incremental game I get was like 1% better answer that I would get. So at some point, you find that point and then you want to grab everything that you've learned so far into a permanent file so that I could copy and paste next time or, you know, go into a more, you know, permanent location, you check that into Git and then everybody else in your team will have that contested. And so they don't have to see themselves. Yeah, it's really interesting. You put it that way about iterations because that's basically programming. That's coding. Yeah. You start with the first thing and you're like, oh, right, I didn't think

36:30about this or that. And then so you add that component to it or checking maybe. And oh, yeah, but then that would the code doesn't handle this corner case. And you got to add into this constant cycle of iteration until you're like, this is production ready. I'm ready to go. I wouldn't, it's not obvious that prompts need to be that way. But I have had that experience where it's like, oh, I did not ask the question right considering the answer I got. I need to tell it more stuff and then try again, you know, with more information or just just stay in the chat. Just go, okay, you gave me this answer, but I was really looking for this and consider these things and it'll go, oh, okay. And then it tries again and you get a more precise answer or more, you know, something more helpful. Yeah, isn't that live though, Ethan? He's just the nobody knew what we're doing the first time we do it. And then we just kind of keep getting better at it. And the, and you know, what is that saying? Don't compare your first day with somebody else is like hundredth day or something like that, right? So, uh, so yeah, it's all about iteration and trial and error. One thing I would, you know, kind of supplement what you talked about is a lot of times

37:33when I meet the the quote unquote AI critics or like people who say, I would never use AI because it hallucinates. Usually they're very loud and there are very many of them. Yes. And nothing wrong with that, right? Like you were entitled to your own opinion, but what I after conversing with them and trying to understand why they don't like it, what is the reason why it's not working the way you are? Usually is the fact that they try it once in messed up and they just, they just push it away, right? How, how dare it that you cannot add like two plus two is six. Come on now, right? Like hallucinate whatever than than I just, you know, did away with it. And two things I'll say to that one is, did you pick the right model? Because nowadays there's like multiple options than the default model usually is the cheapest and not that great. And so, did you think like the thinking mode, the deep research mode or whatever the math mode? So, first thing is, is the right model and the second is, did you iterate enough? Like, did you actually ask it the right way? Did you know like all of these things that we just talked about in the

38:36last 10 minutes? So, most of the time, once you get those two hurdles out, picking the right model, iterate the right way, usually they convert into, you know, it's actually okay. It's competent, right? You may not be the best, but it's competent. And I will use it in the right situation. Yeah, so that's that back to what you were starting talking about. Have you found one model over another seems better suited for networking and work automation, or is it just more about your prompting capabilities and your willingness to play an experiment? Yeah, so ask me next week, I might have a different question. I mean, I'm totally serious, right? I use GPT 40 quite a bit in the example, just because it's cheap, it's four, so at the time was the latest. And they decommissioned it, and just a couple of weeks ago, I think, because they were getting sued at various reasons. But so, yes, so right now, if I want to launch coding, and if I want to do specific things that I

39:43want to wrote cofer, I will use cloud. And let me, let me pull up my current interface, and I'll tell you, I'll use Sonnet 4.6, right? And then, you know, like, and then I will turn on the thinking mode on that's what I would do. But again, it's moving so fast, you know, be your own judge. Yeah, take it with a grain of salt. And I love the fact that Gemini sometimes, you know, sometimes it's just easy to launch Gemini within the browser, because you're opening up, and you don't want to, you know, explain things to them, and they have access to the tab that you're open. So those two are what I use the most right now, and that includes network engineering. And including live questions, live choices questions, should I, should I, not, to go back to school? You let go of the way that I could specify in a, and as part of my context that I want to answer back in JSON, or yeah, well, something like that. How does that work? How would the LLM even know,

40:46like, if it's a JSON response, what key value pairs to be, to be using? Yeah, you, that, that means me, man, like, it must be, must be the training that they're done, and the billions are dollar, they sink into the data center. But yeah, you know, that's, that definitely makes a difference. And, you know, I don't claim to be an expert in, like, the parameters, or, you know, how they do things in the back. I just claim to be a user that, this work for me, and I hope, you know, this work for you as well. So if you specify JSON, it's going to return to JSON, and if you say, this is, so some I always, yep. Okay. So I mean, okay, so the iOS, like, I've done that, like, I needed some quick configuration. I didn't really want to dig through syntax to remember exactly how to do a certain command. I'll just say, generate this, Cisco iOS, CLI would accomplish these things, and I'll give it the parameters and bang, it spits it out, it pretty much, it just works. It's not always perfect, but pretty much, it just works. So then you start getting into JSON, or yeah, and oh, and the training on that, though, there's a million examples out there in the

41:47Internet of Code that accomplishes these things. So you can kind of imagine, okay, that's how it learned that, that's how the model, but then you start getting into JSON and YAML and models that are out there, data models, I mean, not, not LLMs, but how would it, how would it know to parse that? I guess there's enough training data to figure it out is the answer. Interesting. Okay. Yeah, and I would tell you, because I'm in a mastermind group, and one of the member was a Django core developer, or Django steering committee, or I forgot what Jeff's title was, but, but essentially, I think something that he mentioned that's quite interesting to me is that, because he's well versed in multiple languages, multiple frameworks. And what he said was the, all the LL models get better answers on Django over other frameworks. Mostly because Django has been open sourced, and Django has been very meticulous about documentation, and about having open conversation with contributors, they're all, they open a GitHub issue, they'll talk back and forth, they'll say, hey, this is a

42:51Python, this is why we should think, do things, and so on. So when the model trained, they look at all these public available data. So it's almost like you give it, when they were, they were training, they have more information to begin with. So now when you ask something to that particular model who has all these information, is that they would give a better answer just because of the historical data that they have, the accumulation of these historical data and versus some other framework. So I wouldn't imagine that's the same thing. If all of your information behind a paywall, or all of your information, the histories were all private, and they have no access to it, chances are it's not going to know, it's not going to be as, as good of a companion as something that's, you know, Python, for example. So I was assuming every time you came to the prompt window, you're sort of starting fresh, but it sounds like you're saying, you can build up a collection of prompts like you have a collection of scripts so that you aren't having to every time go back and provide context, put in the guardrails, add a persona, figure out, you know, what kind of

43:54help, but you want, is that, is that what we're talking about here? Like precisely, people say have a repository of prompts. Yeah, and I think that's what some of the college kids have, you know, the early days of a child engineering, when you have to save everything in, in like the same single box, so they have a prompt when they're doing homeworks, for example, right? Hopefully, this is a prompt they have under no circumstance, you're giving the answer right away. You want to make me think and just give me all the data, right? Like that's a prompt, that's a parental prompt, but yeah, the parental prompt, the mom prompt, and then there's like the prompt of, hey, Friday night, I just want a quick answer, you know, have some beer with my friends. So anyways, so they would have all these prompts that saved up and depending on certain pace then. So definitely, that's one way to do it, but as what we talked about, right? There's also like these hidden files that you could, according to their documentation, this is what they'll look at and save you some typing, save your team from, you know, going off of, you know, the rails, for example, if back to our JavaScript example, if a fourth person shows up that, you know, I don't

45:00know, sky robot, right? He joined our team, like, and he wants to, you know, dig in on the code, and they say, hey, give me some JavaScript, it's not going to give you react or view, because we already set up that guardrail as, you know, plan vanilla JavaScript versus, you know, some code of framework. We've talked a lot about models that have been trained on publicly available data, the entirety of the internet and documentation and so on, so much of what's out there, but let's say I'm on a network operator, we got our own internal standards, and the model wouldn't know about any of these things, naming conventions, description fields, just the way we do certain things, and I want the LLM to give me answers with that information, as part of its context, is there a way to do that? Yeah, that's back to the agents that MD file, cloud, IMD file, you could put all of your, you know, naming convention standards in there, if you want it to be semi-permanent, and then you check it into Git, and so that, you know, somebody else would synchronize your Git, get that repository down in your automatically,

46:00have that boilerplate, or, you know, just kind of specify in your individual chat window, and say every time you start your chat, you want to have all these, always use, you know, airport code, dash, you know, like function of the device, and then, you know, routers or switches, and then your tiers, and so on, so forth. So I think there are multiple ways to do that, but, you know, once you have something that's set as a team, just like, you know, when you, when you have, welcome a new team member, right? Like, we'll use VS Code, so, you know, let's, let's go get your download VS Code, get your onboarding process, right? So you're onboarding via these templates, and these templates you want to, you know, save it in the a centralized location, and that will be like a private GitHub, or whatever. Well, you said, you said it, you said it, MD files, or markdown file, so that is that markdown file, markdown delimited, but essentially, it's plain English, where I'm saying, hey, I will let my naming convention looks like thus and so, and it's, or is it some structured data that's got to be in a very particular format?

47:02No, it's most of the time plain English, and you could, yeah, you could see a bunch of examples, and they'll provide examples on that on, I think it's just DomD, because it's, you know, I don't know, you could be top TXT, or, you know, you could do a plan mode, and so that's another thing that's, you know, kind of new, by the time the book was published, but now it's like common place where, you know, when you launch a code assistant, or, you know, I use code assistant the most, so I'll use that as an example, but I'm sure it's in other chats, interface as well, that you could have ask mode, or plan mode, and, and you actually, you know, ask before you edit, or, you know, I'm feeling lucky, edit automatically mode, so there's like full modes, so every time I start a project, I don't go to the edit mode nowadays, I go to the plan mode, and so once you start, you know, using the plan, do it actually have back and forth, I will always end up with, do you have any additional questions? And once we settle on about that, I say, hey, summarize everything that we talked about, and then save it into a file,

48:06and, you know, depending on maybe that cursor type defile, like cloud type defile, save it into that file, and then I will get checked into my centralized repository, so that everybody else has it, and they have this history, and then later on, if they found some mistake during that, that process, they could make their edit, they could check back in, and then next time I launch, I'll pull it down, and then, you know, there's some history behind it on why, why they change what they changed. So, let's say I've played around, found a model I feel comfortable with, I've been thoughtful in my prompting, I've now got some output back, how much can I trust this? So, I'm reliable as it, because that's the $100 question, right? That's right, I would even, I would tell next that, I would like to $1,000 million question, that how can I trust this? So, I think that's a great question, and I think that really the answer comes, really depending on you, and how the, the risk factor, like how much can you,

49:07is this irreversible or reversible change? So, that's always that human-in-the-loop factor, where at this point, I don't think anybody, or only the most risk-gravitating persons, would give AI that the total, you know, control, right? That's the same reason I haven't used an open client, because I just feel like I've gained enough too much control. So, in this case, you get that answer back, I think most people will still be that person-in-the-loop, because you could offload a lot of stuff, but you can't offload accountability, right? So, so you're still being accountable for it, and that's why, you know, we still don't have total autonomous driving, we still have like human-monitor driving there, so you, you cannot offload that accountability. So, why should AI, how do I blame you for this? Yeah, exactly, if you, instead of give blame, I do it like AI blame, right? So, true, and if you find a good answer for that, just let everybody know, right? But, right now, I think most of us still want to be that human-in-the-loop if we're responsible for it,

50:11and I would take a look at that config, or whatever that operation they, they told me to, and then I would, you know, ask myself, if I execute this, what is the risk factor? Can I reverse this? And then, you know, that to determine my next step, if it's like irreversible, then probably different. Okay, you've given us a way to think about it. That is, okay, you can 100% trust what's coming up at the LLM, you need to be smart, you need to be wise, and think before you, I'll go ahead and execute the code, for example. Yeah, hopefully. But, in general, in your experience, going back to how Drew phrased that question, I've written a really good prompt with a lot of proper context. Do you think what you're getting back out of the LLM's, generally speaking, is good, saving you time, and so on? At least for coding assistant, I think it's getting much, much better into a point where I, like, I feel like I'm wasting time double-checking AI, because it's so much better than me. So, I'll give you an example. I was trying CloudCo for the first time, like, not the CLI or the plug-in on your VSCO, but rather the Cloud client,

51:16that they now have, let me get that particular, so they have co-work and code tab. So, I was trying out that code tab for the first time, and what I wanted to do is, I've always wanted to have a YouTube playlist with, like, 40 videos. I always want to transcribe them, summarize them, and then see how much I want to watch them out. So, in this case, what I wanted was to download all the Autocon 4 videos, and I give it a playlist, Autocon 4 videos, summarize them, and then I will give a choice of whether I want to listen or watch this in entirety, or just kind of scheme through it, or the summary is good enough, right? That person, that speaker Eric Cho, I want to just, like, cross right away, right? I don't need to listen to that. For multiple reasons, but one of them is I was there. So, anyways, I launched the code, and I followed the instruction, I'm creating a space, and then just give it something. And without me, either, like, checking the code, because

52:20this is low risk, right? This is just simply doing, you know, whatever it is that I would do, except that it would take me hours, if not days, to come up with that script, and they would be broken without me doing anything, it accomplished that task right away. And maybe not, right, the first try, but it would, like, iterate, you'll find out what the air was, maybe it was missing a package, maybe that package was incompatible, or whatever it is, it just goes, and then just, like, a cute cat, just, you know, accomplished whatever it is that I would do, and I was very impressed. So, I think that is, you know, at a point where, yeah, most of the time, I think it's, I don't know, 99% there. If that 1% there, it's because I haven't explained it clearly enough for coding. Hmm. I guess that's been my experience too lately, where a year ago, certainly a year and a half ago, what you got back was kind of rough. It really, like, and, but as time has gone on, as Zidirator, we've had new models come out, what I'm getting back is pretty reliable most of the time.

53:23And if it isn't, I can prompt better. I can, as you were just talking about earlier, iterate on the prompt and, and approve the output that I'm getting. And I'm finding that it's scary good, like, I, like, I'm with you, like, I almost feel like I'm wasting my time here checking its homework, but at the same time, I don't entirely trust it because, you know, we had a history with hallucinations there, not all that long ago. So, can I really trust it? But it feels increasing like, yeah, yeah, we can. Yeah, exactly. So, I think the next thing, the next big thing that we should, you know, probably learn, do, or at least get familiar with the concept is AI and that's actually what my autoconfort talk was, is that, you know, the, the, the title was clickbaitish. I'm like, which AI model is the best, right? But if you actually listen to it, it's actually, you know, you want to evaluate the model. Like, you know, that's the thing that we want to do is we want to save those traces, what they called on how they came out with the answer

54:26that they've had come out that would never have time to look for, that would never have time to really dig deep into. We want to save in their location. And then we want to add a later point, evaluate these models on how good they are. And I'll give you a example on, it's actually in the book where, you know, we use the third model to evaluate the answers from like the turbo 3.5 and and the GPT-4 turbo 3.5 and 4. And supposedly 4 would be better, but, you know, the third AI actually evaluated 3.5 to be better. And the devil's in the detail is that because I have a better prompt for 3.5. So, so I think going back to it is that, you know, the next thing that we need to do is actually to figure out how to evaluate, evaluate AI, have a good test bed, make sure, you know, the blast radius is manageable, that we're comfortable with the amount of risk that we're taking and so on and so forth. It's less about the quality of the AI, but rather about,

55:29you know, what is the situational awareness that we're into to use this? In terms of evaluating output, do you think there's value in taking a prompt and running it through one or two or three different models and comparing what they spit out to look at how they're different or not different? Sometimes I just do it for fun, but most of the time I don't, most of the time I find that something that's good enough, that is, you know, get me 80% there and I'm happy with it because, you know, like, yeah, sometimes it's just that going to a doctor, right? Sometimes you want second and third opinion for the really important stuff, and sometimes it's good enough to go, okay, I'll take that antibiotic and you'll solve my problem if I go. So sometimes I would deliberately do it for comparison and just, you know, even within the same provider, I'll compare between, you know, GPT-5 versus, you know, four, for example, or, you know, I'll compare between Sonid versus Haikoui within Enfropic, and sometimes I do it just for fun, but I don't think it's worth all the time to just compare all of it, unless you do

56:34it systematically and you want to systematically doctor me like, again, if you want to e-vow, right? Like the same thing, everything else, constant, this is the only change variable, the model, the provider, and what do I get and kind of evaluate that? If you're doing it for that purpose, definitely, but if you just want to get like some code done like YouTube, you know, that YouTube example that I gave, that no, I'm perfectly fine. Just like, it got me the result I wanted, then I'm good to go. Okay. Security, Eric. So if I'm working with the public LLM, which has been the vast majority of our context here, context, but that's what we've been talking about. Oh, yes. I know what you're talking about, man. How do I think about security? So for example, I wouldn't upload router configs to the cloud unless I knew exactly how that router config date was being secured. Should I have that sort of thinking when I'm interacting with the public LLM? Yeah, definitely. So I think that's a, that's a great question. All the providers will, will want you to believe that they're absolutely secure and, you know, there's no,

57:36no worry on the leakage and all that, right? But, you know, five experienced credit check later. I found out that, you know, they're just, they're just kidding about it. So I don't have a good answer about that, but I would say the, the one thing that I would tell others and tell my wife, tell my kids or whoever that come up and ask me is that if you think it's important enough, you'll probably want to get the paid version versus the free version. And, you know, we could talk about like, you're the product or whatever, right? But, you know, the thing is that I think most of the agreement within the pro version is you, you have the option to opt out of being a train for those data like your, your response is being back. They absolutely need everything that you return, you know, to give you the, the best, you know, description or the best response, but don't, you know, paste in your social security, right? Don't, don't, you know, your banking information that should stay private when you launch Gemini versus like, make sure you're not

58:36locked into a bank account or whatever. But, um, but yeah, you know, I think that's a great question. At least in the pro version, that's why I pay for everything is like, I pay for everything and then I go into the settings to make sure that I opt out and everything I could opt out of. And what is the minimum thing I could give it in order to get the result I want? And sometimes the answer is not great. And like, uh, you know, I don't want to pick an open cloud, right? But essentially, I feel like some tools are just overstepping their boundaries and wanted to so much scope and permission that I'm not feeling comfortable in giving it. Uh, so that's an open end to question. But one thing that I would say is it may be looking to the pro version versus uh, the free version and then opt out of everything that you could opt out of. Okay. Well, Eric Cho author of the AI networking cookbook. Thank you for joining heavy networking today. And uh, remind people where they can get the book and then how they join your community if they want to talk about the book. Yeah. So anywhere that you not only get your books,

59:39a lot of times I see over a 50% of the readers get their books from Amazon. So I'll say Amazon in, uh, you know, and different formats. You can also order directly from packed. I would love to have you sign up for network automation, nerds.com. And there's like, when you launch the hero, like, I should have that joint. And then just type in an email address. You don't need to give your name. And uh, and then you'll receive a, uh, invitation link to the discord that if you want to discuss in real time. And uh, when you join the, uh, the community, you'll get a, um, you'll get email notification whenever I post something new. And that's how I aim to stay in touch on some of these stuff that we talked about that I didn't have the, the time with the space to including the book. Okay. Eric, thank you again for joining us. Thanks to you for listening to packet pushers today. Heavy networking podcast band we've been doing this since 2010. And a lot of you have been listening since the beginning, which when you tell me that it freaks me out. But thank you for doing so. Go get yourself a copy, Eric's book. Ask any of us that have our names on

1:00:42a tech book. Uh, they do not make much money for the author, but every little bit helps. So please support Eric's efforts and learn something new. I've been Ethan Banks, along with Drew, Conry Murray. We appreciate you listening or watching each and every week. Heavy networking is available as video now, if you didn't know, but it's available as video on Spotify and YouTube. And of course, audio as it always has been anywhere that you listen to your podcasts. And of course subscribe to network automation nerds with Eric Cho and search for packet pushers in your podcast client to see our entire lineup of shows, 13 active shows, putting out a fresh episode each week or every four tonight. Thanks for what you do as a network engineer. Hey, you got a hard job. You aren't, we know you were not always appreciated for what you do. You catch way too much blame and you make the absolute train wreck disaster apps that someone hands you work somehow. You do that because you're that amazing. So again, thanks. Maybe no one else cares, but we do. Last but not least, just remember, too much networking would never be enough.