The Greatest Time to Build a Startup (The AI-Native Advantage)

0:00The hallucinations thing is a lot of people don't necessarily understand what is likely to lead to that scenario. And that's one of the reasons why people get misled by like one shot examples. Oh, make me a to-do list app, but real work isn't like that. It's like, I've got a very specific internal library that you've never seen before. You've never been trained on, and I need you to get these specific details right. Ah, good engineering practices needed for a gentleman. They're certainly beneficial. And one of the things that the Dora report found in 2025 was that teams with high levels of development maturity went faster when they introduced agentic coding. Teams with low levels maturity went slower. I can think of some SAS providers that I've worked with who've had no competitors for 20 years. In the last six months, two have popped up replicating their feature set entirely using agentic coding. So there's a chance that you will fall behind if you do not stay on top of this stuff. What three tips would you give an organization who wants to roll out agentic software development across their company?

1:00Firstly... Back in November, we hosted the first ever in-person AI native DevCon in New York. This June 1st and 2nd, we're bringing it to London. It's two days built for AI native developers and engineering teams. One day full of hands-on workshops and one day full of practical talks on agent skills, context engineering, agent orchestration and enablement platforms and how teams are actually shipping AI in production. Join us at the brewery in London near the Barbican for all of that plus networking, parties, giveaways and a room full of people building the future of AI native development. You can also join us from anywhere in the world via the live stream. As you're listening to this podcast, you get 30% off your ticket with code pod30. Just head to AInativedevcon.io and we'll see you in London.

2:03In this episode, we're going to be asking the big questions, what are the hidden pitfalls that you don't even know might be ruining your chances of success with agentic development? And what are the most important factors that a developer needs to know about in using agentic coding tools as well as rolling it out across your organization? As I mentioned, joining me on this journey is Daniel Jones. Daniel, tell us a little bit about what recently. So, recently is a consultancy in North Europe and we help people with AI native transformation, whatever that means. And it's a thing that is kind of emerging as the state of technology evolves. We kind of help people with across three threads, really. One is agentic coding, so using AI in the software development process to increase productivity. Another is helping people build things that leverage AI. So maybe where your product has some ML features in or you're building agents. And then the third string of that is helping the non software parts of the business. So accounts, marketing, how can we introduce agents or workflows that make them more productive?

3:07So looking at AI, AI nativity, AI nativeness across all of those different strands. I love that the AI nativity, which is definitely celebrate that once a year. So Daniel, you work with a lot of large companies, enterprises talking about AI adoption and how those companies can best take on and roll out agentic development across large numbers of teams. Today, we're going to be talking about some of the advantages that can bring, but really talking about some of the pitfalls that people very often fall into when trying to adopt. And some of the best practices that we can add in. So what do we kick off by talking about people having good engineering practices to start off with? And where the pain exists whereby if someone doesn't have a good engineering practice and they use agentic development, where that pain could be exacerbated. So our good engineering practices needed for agentic development. They're certainly beneficial. And one of the things that the Dora report found in 2025 was that teams with high levels

4:09of development maturity went faster when they introduced agentic coding. Teams with low levels maturity went slower. And this makes perfect sense when you look at it through the lens of the theory of constraints. If you take one part of a system, speed it up massively, then you're going to create bottlenecks on either side. The interesting challenge there is that we don't know where exactly that tipping point is. So we're not sure how good you need to be in order for things to get better or how bad things need to be for it to be throwing petrol and dumps the fire. And when we talk about how good, what does good look like? Is it the documentation that needs to be exist? Is it just people following good workflow practices? What does good mean? So good can be measured in a number of dimensions and there are quite a few. So we've got things like test coverage. We've got things like alignment on what good looks like for coding standards. We've got things like the size of batches that they're working on, like how big and well detailed are their stories.

5:10We've got things like the path to production. What is their lead time to production? Can they get changes out quickly? Because if coding agents can chuck out, commit after commit, minute after minute, but it takes three days to get anything into production, which is quite optimistic some enterprises, then the tip of the branch is going to have moved on miles and you're going to end up with loads of merge, conflicts when something goes wrong in CI. So there's quite a lot that really would be considered just general good practice from the last 10, 20 years of software development that needs to be considered just because everything goes much faster now. So having those good pipelines, those good workflows, it enables that speed so that you don't just have one massively sped up component, which is the actual build of the code. And it's pointless having that if the rest of it is going to take time. Test, you mentioned tests. Why tests so important in a true agenda development? So tests are important for one reason,

6:11but there's some nuance here and maybe we can get into the ways that maybe some types of tests are less important or at least less important to humans. First and foremost, if you don't have good test coverage in your software, then how is an agent going to know that it's broken something? Now really one of the things that people who are new to engine coding need to realize and often that kind of miss is that your agent needs to be able to perceive something. If it can't perceive a problem, then it can't possibly react to it. So whenever you're getting frustrated with a coding agent, you need to consider like, could it have known better? Did it have the information to know that it was doing the wrong thing? And so if your agent keeps on breaking your software, well, did it get fast feedback from a failing test suite? If you've got low test coverage, then it's quite likely that it could break a whole bunch of things and be absolutely none the wiser. And that's not really the agent's fault because you haven't given it the information or the ability to perceive the fact that it's making a mistake. And is it actually the test runs that informed the agent?

7:15Yes, this is broken because the test failed. Or is it actually the agent using the test cases as context almost to understand, actually, this is expected behavior. This is kind of what we intend the application to do. That's a really good point. And it is both. Certainly the failing tests, like if your agent is well-mannered enough to actually run the tests all the time, then it should figure out that it's done something wrong. But using the test as documentation and guidance of what is expected is quite important. I think that, and this kind of brings me to, I mentioned nuance earlier, of maybe unit tests aren't for humans anymore. Maybe those are too low level. Like if unit tests help the agent figure out whether it's broken something, great, okay, agents go and write as many unit tests as you like. But acceptance tests, kind of outside in acceptance tests, that define what a user should be able to do with the system. Those are important for your ultimate safety barrier

8:18of this is what I need it to do. But also that guidance that you talked about of this is the expected behavior. This is what it needs to do. These are the features that we want to have, regardless of whether they work or not. This is what it is supposed to do. And that kind of then bleeds into things like spectrum development and where is the source of truth? Is it the tests or is it the spec? But these are all things that I'm going back to the previous point about the last decade or two of good software practice. Things like behavior driven development, you know, a good BDD test suite should have been describing what the software does at a kind of human accessible level anyway. So if you've got that kind of thing in your code base, then agents will be able to work with it better. And it makes sense, really, because if we as a developer are turning more into intent machines, where we describe and define the intent that we expect, and it's actually the agent's responsibility to effectively generate that implementation that describes the behavior we ask for, the tests we care about should be to validate the intent that we're asking for,

9:20whereas it should be the agent's responsibility to create the unit test because they're creating the implementation. So it will be weird for us not to care as much about the code, but then to care about the unit test for that code. It's nice to kind of keep those test and intent or generation levels the same. What about typical development hygiene, let's call it? So things like version control, those types of things. Is that as important with organizations doing the agenda of them? Absolutely. Version control in particular is critical to making good use of agents. Because sooner or later, your agent is going to do something that you don't want it to. It's going to commit a binary into Git, and then you're only going to notice like four or five commits later, and you're going to want to remove that. So if folks aren't able to do things like interactive rebasing, which, you know, to be fair, can be a little bit fiddly, and I found a little bit mind bending when I first learned how to do that. Then you're going to end up with quite a messy code base that will mislead humans and it will mislead agents. Another thing to consider when it comes to version control

10:23is that commits and branches are cheap, right? They're next to free, though, they add very little overhead. So making lots of small commits, which, again, is like people would argue, or I would certainly argue, is good development practice anyway. Maybe making even smaller, more frequent commits is better because then you can throw away the work that your agent's done. And when it comes to behavior change, a lot of adopting agente coding is not just knowing how the tools work and gaining new knowledge. It's unlearning all habits. Yes. So being happy to throw away some work that an agent has done because you've got plenty of commits that you can go back to, it's, you're going to be quicker throwing away something that an agent did, and then starting again, then you would have done typing on the keyboard yourself like it was, you know, 2022 or something like that. An agent's aren't precious. They're code. They don't care about throwing that code away, right? Exactly. Yeah, I often say to people that you should imagine you are the worst software manager possible,

11:25insist that you treat your agent a bit like a junior or a member of staff that you really hate and want to demoralize. They throw away its work, get three of them to work on the same thing at the same time and only have one of them have their work used. Like those, those patterns actually make you kind of more productive, even if when our AI overlords look back through our chat history in 20 years time, we're all going to be in trouble. It's always good just to add in the odd, I respect you, or, you know, very much appreciate, because, you know, we're not always going to be top of the pecking order at night. It's indeed. So, let's talk, continue talking a little bit about version control. Who should decide about the commits? Is this something that we just, you know, maybe as a hook to an agent or something which we just said, look, after every reasonable amount of code change, whatever that is, I want you to make sure you commit into a git repo or something like that. Who decides? I think that is a key indicator of how psychologically comfortable people are on their journey to adopt a genetic coding. And certainly to start off, I think folks are much more comfortable making commits

12:27than sounds like asking agents to go and do a load of stuff. I'm going to read the code, I'm going to validate it, and then I'm going to commit it. As you build confidence and build competence with things like spectrum development, it becomes more acceptable to people psychologically. So, I'm going to let the agent do the committing here. Anytime that we introduce a human having to take an action, like we slow things down by, you know, a several orders of magnitude. So, if your agent is sat there going, I'm done. Do you want to commit this and your attention is elsewhere? Then it's making way less progress than if you let it do that. But then there need to be some kind of guardrails around that and making sure that it's not just steam rolling ahead and definitely not doing any force pushing. Yes, it's funny actually because when I first started building with a genetic coding agents, I did manual commit, exactly the same. I did manual commits and actually I was able to go so fast.

13:27I actually realized, oh crap, I should have committed. I should have committed some of this a long time ago. The one thing I do love now, I always let my agents commit for me as they feel like changes are being made. The nice thing about that is you actually, when you go back, you can dissect commits and you can go back to a certain point very, very easily because it actually commits pretty often. And like you say, it's free. It's very, very cheap to do these commits. It makes software development going back retrospectively much nicer. You mentioned a little bit about safety. When obviously there's a YOLO mode, which when we do want to increase the autonomy of coding agents, we do want it to just go away and do some stuff without constantly asking like a child, can I do this, can I do this? What do we need to think about from a safety point, if you particularly from an enterprise, that it's not going to run a havoc on an machine? Yeah, I mean, end surprises are extremely sensitive to this kind of stuff. And one of the points that was provided much more friction

14:28than I realized was the use of containerization and people's level of maturity with that. Having come from a kind of cloud-native background where we were deploying things and Kubernetes and containerizing all the things, I kind of took it for granted that everybody would be familiar with things like Docker and Kalima and alternatives. Turns out a lot of enterprise developers aren't. And a lot of enterprises aren't necessarily set up for people to be able to run things in containers on their development machines. And if you don't, then you've got this situation where people can be running coding agents with dangerously skipped permissions, natively on their host. And I'm pretty sure those people are going to go hell and they keep doing that. So running things inside containers is definitely the way to go because then you can let it off the reins. And you don't need to be there to constantly say, yes, you can do that. Yes, you can run L.S., yes, you can cap that file. And I was optimistic about dev containers. So this kind of open standard

15:28that came out of Microsoft and VS Code supports where you pop some config in a directory and VS Code will detect that that config is there and then it will relaunch itself inside a container. Nice. That's all great. But the UX, like the config is, it fits into a sweet spot of hallucination that maybe we'll talk about in a bit where it's recent and it's not super popular, but it's popular enough. The agents tend to hallucinate the config, which is super frustrating. So dev containers are nice solutions to that, but maybe not quite mature enough. And I've certainly seen enterprise developers struggle with that and spend hours, if not weeks, kind of fine-tuning their dev container config and not being able to move forward and make progress. So that whole containerization thing inside MCP as well, I understand why it's this way, but it would be lovely if all MCP servers were run,

16:29like all the standard IO ones were running containers by default. We've had Docker for quite a long time. It would be nice if everything was just secure out of the box. And I was chatting with Oleg Shulya, actually a former colleague of mine who works with Docker, a developer advocate at Docker and I was chatting with him about Docker sandboxes. And they seemed like ideal for this, exactly this type of thing. And I think enterprises are really going to want their developers to actually start developing in this. When we talk about the setup cost, though, it's easy to develop in the sandbox, compared to obviously if you know, as a developer, I just want to go to a terminal, kick off, claw it and I can just play, use my local resources and things like that. How much is there if we're doing this in a container? It's not massively, massively problematic or massively building some in my opinion. And I think one of the things about enterprise developers is that they're often under a lot of pressure to deliver features, features, features, features and they don't necessarily have the time to invest in trying out different tools in different ways of working.

17:30And so if you're in that environment where you're not given freedom and space to explore new things regularly, and you're of the mindset, like, I learn one thing a year, I get sent on a course, my company pays for it, I'll get a certificate at the end and then I've learned to think, then it can be quite uncomfortable to be using new tools, like containerization, dev containers, to be using maybe a different terminal emulator if you're used to using an IDE, a big full-fat kind of graphical IDE and then somebody's saying, oh, okay, spin up something in a terminal emulator and then you're gonna need to authenticate with Cloud Code, but then there's line breaks in the URL and it can't redirect back to the port on local host because it's inside a container and it's not exposing that port and then you need to copy and paste tokens, those little bits of friction definitely put people off and it's something that's worth considering of guiding people through those just because you don't want people to bounce off before they've got to see the wonder of, like, how productive these tools can be. Yeah, absolutely. And really, I think it's the dangerously skip permissions,

18:34the yellow flag of un-can-elect clause or whatever agent you're using. Go off and do whatever at once, whereby those are the instances that actually you really, really need it in a container because at least in the other instances, you still get that choice of saying, yes, I'm happy if you do this or no. No, no, when you have access to that, that's fine. You mentioned Spectraven development a little bit and actually, I think the providing it with the right intent, providing it with the right instructions is very, very important. We can do that in a number of ways and maybe talk about skills and context in a little bit, but when we actually talk about the task that is required and I guess, previously, we would relate this to a user story or something like that. How important is it to provide an overly verbose or a concise user story? What are the important things that we need to add? The... Hello, it's Simon Maple here, one of the hosts of the AI native dev. I wanted to say thank you for listening to the show

19:35and to remind you that we post great content here every week. So be sure to subscribe to keep on top of our episodes. Also, you can send your questions to podcastatessel.io or ask questions directly in the AI native developer, Discord community, to continue the conversation, connect with other developers and explore more about AI native development. All links are in the show notes and the episode description. Now, back to the episode. The story that you can provide to your agent and the unit of work that you provide to it is hugely important in the level of detail provided. And something that I think we're exposing as we get agents more and more into the mix is the amount of knowledge in the head that is required or the amount of knowledge in the head that we all take for granted when we're doing software development. So Donald Norman in the psychology of everyday things but was renamed the design of everyday things

20:36talked about knowledge in the head and knowledge in the world. Knowledge in the world is discoverable. It's written down somewhere. So having your coding standards on confluence or something like that where maybe an agent could get to it, knowledge in the world would be a well written story with acceptance criteria so you could write a test from it. Knowledge in the head would be, well, I spoke to my product manager last week and I know how they want this to be implemented but it's not written down anywhere. So if you are missing that kind of information and there's lots of human context required or you're used to having that and just going, oh, I'm just going to ask on Slack for a clarification on that. That's a really bad spot for agents to be in and they're not going to be able to produce results that you want. They're not going to be able to implement things without making stuff up or asking a whole bunch of questions. So having a decent level of specificity there is important. And this is where, so with one of the organizations that I've been working with, we've asked them to connect their coding agents up

21:36to the Lassian MCP server and then start critiquing stories which the product managers are maybe raising an eyebrow out but it's putting back pressure on product on how well-specked are these things. And that's not necessarily because product is doing a bad job. It's just they're used to having a much more informal relationship. And something I've seen in quite a few organizations is you speed up the software development part and that puts pressure on product almost immediately. They're like, by me, we haven't got enough stuff figured out. We can't keep the agents well fed. We're going to need to massively speed up what we're doing here and also we're just so used to as humans maybe not going into all the tedious detail we need to because if somebody doesn't understand then they can come and ask us rather than meticulously writing out every single detail. I want to kind of like switch a little bit now and talk about the tools people will aim to use for a lot of these things.

22:37In the past, there's been very much a discovery kind of thing and developers have been playing with as many tools as they can just trying to get used to AI and switch mindset into and I can do this with AI approach. What do you see from an organization and industry point of view of how companies are trying to adopt a range of tools or focusing on specific tools? 2025 was definitely the year of tech leaders going, I'm going to let people use what they want. And I'm encouraging them to try stuff out and we've got somebody over here using GitHub co-pilot, somebody over there using Cloud Code. And I think part of that came from lack of knowledge and comfort from tech leaders kind of being like, there's so much to keep track of. I don't know what the latest thing is and I don't want to like, you know, bet on any one horse because I'm not sure. And then also a fear of, if I say we're going to use this tool, not only might it not be the right one,

23:39but it might be that developers don't like it and then they're going to push back because they've got a lot of pride in their job and the sense of identity from it. So 2025 was a lot of kind of soft touch, organic, disorganized, trialing of things. I think over Christmas anthropic did a great thing for their marketing of giving people free usage over Christmas. So lots of bored CTOs and VPs of engineering trying out Cloud Code and go, wow, this really works. So 2026 has definitely seen a shift in people wanting to be much more structured. I'm like, okay, we need, we can see what's possible now. We need to roll us out across the department, across the team, across the organization. And yeah, we're seeing folks try to standardize on tools. And I think there's still quite a lot of anxiety over what is the right tool to pick from an enterprise and organizational level. A lot of it sadly is driven by licensing and the logistics of how we pay for these things.

24:40You know, there's some speculation at the moment as to whether Cloud Code licenses are the subscriptions are gonna get revoked for enterprise usage. I've seen some chatter about that. But you know, if you've got a thousand developers and each of them need to expense an anthropic subscription every month, that's a huge amount of overhead as opposed to having an enterprise agreement. So we see quite a few people kind of opting for GitHub co-pilot just because it's part of their GitHub enterprise offering. Whether that's the right choice for every organization that is entirely different matter. And is the worst decision indecision here? I think though there's a strong case for that in that often when people are faced with transformative change, they either jump in with both feet and no plan. And we saw a lot of that in 2024 of like, we need to do some AI. What do we need to do? I don't know, just do some AI here, some budget. And definitely saw that happening at large financial institutions. And lo and behold, they didn't get any return

25:42on investment who would have seen that one coming. But then there's the alternative of we're gonna wait until this is settled, a settled science. And we're not gonna waste money getting up the speed on this. And the speed of change continues to increase as time goes on. So holding off, I don't think is a viable solution. Having some amount of budgets, some amount of time, some amount of slack in the system to explore and trial things out is the most sensible strategy because otherwise there's a very, I don't want to fear monger. But there's a decent chance that you will get outpaced by a startup who is embracing these things. I can think of some SaaS providers that I've worked with who've had no competitors for 20 years. In the last six months, two have popped up replicating their feature set entirely. And these are brand new startups who presumably are using agentic coding to just replicate their software very, very quickly. There's a chance that you will fall behind

26:43if you do not stay on top of this stuff. And then if we do get to a point where it's settled science, like do you want to be starting from a standstill because there are approaches, things like managing context windows as far as we can see. If model was carrying on working the same way that he do and the architecture doesn't change radically, then this is going to be an underpinning constraint of all agentic software development going forward. Even if we end up with software factories that build upon these things, you need to understand the fundamentals underneath. So that's a good segue. Let's talk about tips. And I'd love to talk about tips that you can give developers directly. And then we'll maybe talk about tips for roll-out across an organization. So tips for developers. And you talk about context. Let's go deeper into that. Context is, you obviously provide your messages. This is the task I want you to do. Context is really important. Now, whether that's skills or instructions about how your organization wants to develop software, those types of things, how much is the right amount of context?

27:44So looking at academic research, certainly the no-lima paper from 2024 suggested above 30,000 tokens of context, reasoning ability drops off by about 15%. There is another paper, the name which escapes me that was a repeating experiment in 2025 for GPC 4.1 era models. I've found between 30,000 and 60,000 tokens, again, reasoning capability drops off. That's not very much. Like there have been, oh, in Cloud Code, three releases a day, so things change all the time. But I have certainly, there have been points where I've opened up Cloud Code. No MCP service installed, no Cloud MD. And it's already using like 40,000 tokens just for its system prompt. So it doesn't take much to tip your model into the point where it's going to get confused. But one of the challenges with this is that when I was starting developing agentecoding training for people, I was on a video games website over Christmas

28:45and gamers tend to hate generative AI. I think it's going to be the re-enation of all things. And I was like, I'm going to try and educate people about why it's not all bad. And I mentioned in the comments on some website, why was I posting comments on a website that I was doing agentecoding training and somebody posted a very snarky response with like, duh, why do you need to teach people how to use coding agents? Like don't they just work? And I was like, you know what, that's actually quite a good point. And because these things just work, it's not until you have problems that you need to understand what's going on underneath. And context windows and context management is a good example of that. If you just use a coding agent and everything's working fine, you don't need to worry about context. And then, you know, conversation goes on a bit and then it starts to fall apart and you get more frustrated. But it's not obvious that's because your context window, the amount of context that you're using has gone over the effective limit or that you've distracted it by talking about too many things in one chat.

29:45So that is something that quite, you know, clever engineers who are new to this. It's not obvious. It's not immediate to them. Yeah, and I always say like a developer, they know when they're in the Goldilocks zone of context, whereby they're like one-shotting prompts because they're giving enough context to have that acceptable answer. They're not giving too much context so that the agent gets confused. It's just enough. How should a developer keep just enough context? Is it something that they should proactively monitor or something that they should react to if the agent is going straight? I was certainly of the opinion that people should be proactively monitoring this and also workflow habits of, you know, whenever you're at a clean break, clear context, whenever you need to ask a off-topic question and like my mind does not stay in one train line for very long. So I'm normally thinking about three things at once, like use a different chat, use a different app. Like I use warp, a agentic terminal,

30:48and if I need to ask something Linuxy command line, I'll do it in warp. If I'm doing something in cord code and I'm trying to be able to feature with cord, I'll try and keep that on topic. If you need to do research, I'll go to web browser and use a chat app like chat GPT or Gemini. So splitting up the conversations has the advantage of not distracting the model with too many concepts in one chat and not extending the amount of context that you're using. So clearing your context regularly, not going off-topic are things that are valuable to do. The other thing that kind of may be slightly different is not adding all of the MCP servers and not adding all of the instructions for every possible eventuality of how to write software into your agent's guidance or agent's MD. There was a paper that came out of Zurich a couple of weeks ago that showed that adding stuff to your agent's MD will reduce the quality of outcomes by about 20% over and having no instructions whatsoever.

31:50And it's interesting looking at things that have been fashionable over the months as we all learn about this. Like, you know, I remember certainly September time people talking about, but all the things in agent's MD every time it makes a mistake, stick something else in agent's MD. But what you're doing there is you are saying something that's largely irrelevant most of the time in the hope that it's relevant this time. So that's not a great thing to do. Then it was like, I'm going to add all the MCP servers for every eventuality and that causes its own problems. Like a lot of people don't realize how tall calls work and the fact the full menu of tools need to be provided to the model on every single prompt just in case it wants to use one of them. So skills and progressive disclosure are a much kind of better solution to this where instead of sending the model all the details about skill, there's like a little summary and the model can ask for more information if it wants to. And let's talk about that because I think that's really important. There's a couple of things here. One is like the behavior under the covers and the second is more activation,

32:50which is a global problem today, but it's something that we expect to get better. So you talk about adding things into the agent's MD. Anything that you add into the agent, MD file, the agent will fully read. It doesn't necessarily mean it'll act upon it, but it will fully read. When you add an MCP, again, there's a description there. It chooses when to activate it similar to a skill, I guess. A skill, like you say, there's a description there. It chooses when to use that. These are kind of more of the on-demand style approaches. So if the LLM agent feels like it needs to learn more about this thing or feels like it's doing something and it triggers a skill as a result of wanting to do development in a certain way, or wanting to write code to a style guide, it chooses to then use that skill, read that context, et cetera. So the overhead there is very real. If you have hundreds of MCP servers, hundreds of skills, it's going to get confused and it's actually probably not going to pick the skills you want it to.

33:51Do you have experience or advice in terms of, not necessarily what the right number of skills or MCP servers are because it's very subjective, but what advice would you give to someone who is thinking about adding yet another skill or yet another MCP to their environment? I think add them judiciously. Maybe have loads that you rely on but don't have certainly for MCP servers. Don't have them all added and registered at the same time. If you do need registered at the same time, then maybe selectively filter out their tools. If your agent doesn't support that natively, then maybe you could use an MCP gateway for that. For the skills, I think there's slightly less of a problem with that because the progressive disclosure, but you might end up with multiple skills that have very similar descriptions and then the model's not sure which one it wants to ask for more information on. So yeah, that could be a problem. And I know having spoken, I spoke to Amy Heineke

34:51from Tessel who was talking about this and some of the challenges that you folks face in making sure that the things that you're building get selected by the model and it's a thorny problem. So I gather. Activation generally is a thorny problem in terms of making sure that the agent chooses the right skill or the right, you know, the right piece of context at the right time. And it's definitely a problem that exists today. I think it would be less of a problem as time goes on and as agents get more used to using skills. But yeah, it'd be interesting to see how that turns out. So when we think about a user, a developer, part of a team building a certain part of, you know, a project that is much, much broader. When we think about context management and replicating their environment to share essentially the environment that developers environment with the rest of that team, what's the best way to be able to replicate that context, the context management from one developer to another?

35:53Is that something that should be more global, stays with the repository or is that something which should be more subjective to the developer how they want to work? So an interesting point and one of the places where team dynamics and the conflict of avoidance that sometimes is arguably necessary or allowed in large teams to proliferate can kind of create problems. In that if you've got lots of developers with different ways of doing things, then maybe in the old world pre-agentic coding then that was acceptable when you knew that, you know, Rachel over there has her way of writing code and Steve over there has his way. But if you've got an agent going over that code base and then it's going to be confused by two different code styles, then that becomes more of a problem. Generally, I would say that having things as global as possible and inconsistent is going to lead to better results. And as we move away from individual developers

36:54having their own idiosyncratic styles that they like to deferring more of the implementation to agents, does it matter if the code's written the way that you like it versus the way that I like it as long as the agent is able to deliver useful outcomes and features more quickly. So trying to standardize and make sure that these things are available to agents when they need them. Whether that is global and maybe you put that, you know, in confluence or a wiki and you give your agent an MCP service so we can access that, the problem with that is drift. And when it changes, then you're probably better having that in your code repository. So the guidance is pinned to a particular version and as that evolves, it's clear, you know, wikis are often where documentation goes to die. So yeah, you know, if you want to stay and after the date information, that's a good place to put it. So probably in the repository would be the main place to do that. And then when you get folks having context and guidance

37:58they're keeping local to themselves, I can imagine that leading to problems of if you've got conflicting information and then you've got one agent doing things one way and one person's machine and another. You know, those are the kind of patterns that we've generally throughout computing gone. You know, this is going to lead to bad outcomes. We want things done consistently and universally across a team or at least throughout a code base. Yeah, probably worth mentioning Tesla here as well, actually because obviously Tesla is that kind of like that package manager for skills and context. So actually does a lot of what you mentioned there. A lot of the like the versioning of skills and context, the ability to almost like have a manifest file within the repository that then pulls in the right context as per pinned in the manifest. Yeah, but I think what you kind of like mentioned is really interesting where by things like coding styles, I've never heard an agent argue whether it should use spaces or tabs, but there are things that are still important, the stack that we use, the methodologies that we of how we develop with test driven development or how we choose to develop,

39:00it's going to be important more than the cosmetics of the code, I think, which is super important. You mentioned kind of like hallucinations a little bit before as well. Do you feel like how do you see agents coding correctly? How do we use context to maintain accuracy? So the hallucinations thing is a lot of people don't necessarily understand what is likely to lead to that scenario. And so in the training that we provide, we get people to do an exercise with a really old model, so GPT 3.1, where we can deliberately mislead it and make it hallucinate things. And it's really quite interesting doing that. If you ask about Star Wars episode 10, like the model goes, no, that's not a real film. So when you're talking about big, well-known concepts, then it can detect a lies quite well. Like phantom menace, it's not a real film. Doesn't exist, does it?

40:01I would probably say more about a 7, 8, 9, at all. Let's not go there. Still scarred. If I was to ask GPT 3.1, tell me about the London thrash metal band, Simon and the Maples, then it will probably tell me now there's no such band. If I asked that same model, tell me about the thrash metal, stroke, hardcore band, Simon and the Maples, they were active in Camden between 1997 and 1999. Then it's much more likely to start spewing out a bio. And we did this with the exercise, about 50% of the time, moving from a capital city to a particular area and putting a date range in, where previously, the model would have said, no, no such thing. It's then like spewed out band bios, talked about like the break up of the band who fell out with her and all that kind of stuff. So that's an example of if you are adjacent to large concepts and very specific, then you're in the sweet spot of hallucination. And also, if it's something that the model is likely to have been adjacent

41:03to something that the model is likely to have been trained on. So I had the exact experience for what Tesseltile Registry was made to solve in though I had one of the developers that was training talking about, ah, I was using the latest version of some library. And my model was hallucinating. It knew that it was a new version. It wasn't given me the old API endpoints, but it was hallucinating new ones that didn't exist. And I was like, I've got a thing for you. Somebody has solved that problem. But it's very interesting because typically those types of challenges or problems, the specifics that are added by the user or by the agent in terms of, I need to do these very specific things. It's very detailed. So it's actually right for those types of hallucinations. Exactly. And that's one of the reasons why people get misled by like one shot examples of, oh, make me a to-do list app or make me a flappy bird clone. And oh, wow, this is really great. But real work isn't like that. It's like, I've got a very specific internal library that you've never seen before. You've never been trained on.

42:04And I need you to get these specific details right. So that element of detail, and this is why I think the band example is quite illustrative because when you start narrowing down to really specific things, that seems to be when it's more likely to hallucinate. Because we mostly toured around the Chelsea area anyway, so it's very easy to do. So we've talked a little bit about, we've talked a lot about agents, very little about LMs. And actually when we think about the hallucinations and things like that, it's coming from the LLM. So when we think about the roles of each and how we select and choose the tools, what should we be mindful of when thinking about that? And I think that's a really good point. And a lot of people knew to the technology. And I've probably been conflating the two because you interface with the agents and then really it's the model making decisions about which tool to use. The agent itself is not very smart. It doesn't have very much logic in it. It can't do. It's just a loop running around

43:04and the model is making decisions. And one, when folks are new to agentecoding get into this, they don't necessarily understand like what is the difference between the agents' responsibilities and the models. And so when we're doing training, after we've done an excite as with people, we go through on a whiteboard like, okay, what did we just do? And then draw it up or draw up all of the components where they live, where the tool definitions are, how they get to the model, all that kind of stuff, which helps cement it in people's minds. Folks that are coming into this new and there are a lot of them like in the last couple of weeks, I know so many senior engineers who have reached out to me directly and being like, I feel like I'm performing behind like, can you help me get up to speed with this? And they get up to speed really quickly and they normally jump in with Claude code or something a terminal based agent and they get using it and get it productive, but still without knowing what's under the hood. And so folks end up kind of worrying about which is the best agent to use and really there are functional kind of differences

44:07in terms of how large each other's system prompts are and how much of your context window, effective context window that leaves you with. But really it's more like a personality, it's more like different brands of cola, they're all full of sugar and fizzy that they just have slightly different spins on them. And really also the models, like if I use the lace GPT and if I use Opus 4.6, they're probably gonna be roughly equivalent in terms of what they can do. If I switch from Opus to HIKO, then I'm gonna notice differences but really the different foundation frontier models, they're all roughly capable and equivalent. So people see the brand names and they see the tools and they're like, oh, well codex is gonna be much better than Claude code. What I think is more important is ergonomics. Like do they have the features that you want that allow you to work in the way that you want to without having to sit there and babysit the agents?

45:09Do they support skills? Do they have the UX that you want? That is productivity is more important than correctness. Like the correctness is something you don't have much control over. Either the model is trained well or it's not. The productivity that you get from having the features that you want and the user experience that you want combined with the context management, that is a much bigger determinant of whether you're gonna get useful outcomes rather than which model you're using because they're all the frontier labs are trying very hard to keep up with each other. And it could also be around the way you as a developer ask your questions or provide your requirements because the models are gonna pick up differently, the models are gonna have different activation, ways of activating the skills using the context. And so it may be, and perhaps this is why there are so many arguments as to who gets the most out of which models and agents because people are asking things in a different way. So yeah, a lot of it could just be which agent

46:11actually works best with the way you like to work, the way you ask the thing. Hey everyone, hope you're enjoying the episode so far. Our team is working really hard behind the scenes to bring you the best guests. So we can have the most informative conversations about a genetic development. Whether that's talking about the latest tools, the most efficient workflows, or defining best practices. But for whatever reason, many of you have yet to subscribe to the channel. If you're enjoying the podcast and want us to continue to bring you the very best content, please do us a favor and hit that subscribe button. It really does make a difference and lets us continue to improve the quality of our guests and build an even better product for you. All right, back to the episode. Yeah, and one of the ways that agente coding is problematic here is because of the non-determinism, it is a breeding ground for superstition. And if you look back through the kind of annals of human computer interaction and usability engineering, then there are plenty of examples where when people get non-consistent results,

47:14there's a word for that. Then in intermittance, then they start to come up with superstitions of like, oh, it only works when I click on this thing first before doing the doodar. So yeah, folks can start getting all these superstitions about what prompts work and which ones don't, which is why the kind of work that Amy was talking to me about about running e-vows on things is so important to try and get some science into this. From my own experience and working with the hundreds of developers that I have, the whole kind of prompt engineering doesn't seem to be that important. It's more about context management. And as you say, how you communicate requirements, the whole like, how you phrase the question or like, if you give it a persona, I don't see that making too much of a difference even if it is something that is quite immediately engaging to humans of like, we like the idea. It's more about framing the context, making sure the right information is there that the agent has the ability to perceive

48:15when it's doing the wrong thing and whether your requirements coherent or not. So let's talk about the data as I think that's an important piece. A very often so much of this world is anecdotal in terms of, oh, yeah, I think the skill works most of the time, let's leave it like that or essentially going from gut feel of my experience was, how do we bring data into that, maybe through skills or context as well? Well, I gather that you folks have a... There's a loaded question. A skill is a package manager where you're rating skills both on how well written they are and how effective they are, which I think is a hugely valuable thing because so yesterday I was doing something in GoLang, I say, I called code, was doing something in GoLang for me and I installed the GoLanguage server plug-in and I said it off-running and I'm like, is it even used in this thing? Like I don't, I'm not paying attention, I'm doing other stuff whilst it's trundling away. Is it using, is this making anything better?

49:15And having more scientific data on that, I think is incredibly valuable, maybe going out to the far future when we end up with software factories which hold kind of separate topic of conversation where humans are involved much less. I'm excited by that idea because being able to run requirements through a software factory where you've got multiple agents collaborating and outputting finished software for a given set of requirements. Like if all of that is underversion control and has full observability, then you can run like methodological experiments in a way that you can't do with a human team because you can't get somebody to pretend they didn't just implement that story and then try doing it a different way. Whereas with software factories, we might be able to do that and slowly iterate and tweak prompts and tweak skills and which skills are we using and how are those skills written to be able to automatically start getting ourselves to maximum productivity off that software factory, but until we're there.

50:17So little visibility for the end user into what they're doing when they're activating skills, why how well those are working. And I think it's something where the kind of magic-ness of agentic coding tools works against the user. Like it's great when it just works, but actually we also need to be able to see that the dials and the readouts figure out what's going on totally. And I think there's a number of things that I find very important here. We talked about activation. And a lot of the time with activation, it's the way we write the skill. For example, anthropic, delivered a whole bunch of best practices of how to write skills, which was added into a blog and actually a Tesla review skill, essentially uses LLM as a judge to work out whether those best practices have been adhered to. And I think from the activation, realistically, the only way to actually look back in hindsight and say, how good is my skill from an activation point of view? Is to look through the logs and almost like say,

51:18how many times should this skill have been activated and how many times was it activated? And there's probably an amount of work there that can kind of look through that and provide that data. And then it's definitely one thing that we're looking at here at Tesla. The other thing I think is what we call task evals within Tesla, which is essentially run this with the skill, run it without the skill and work out actually, if you do use this skill, what benefit does it provide you? And for me, as a for writers of skills, for authors of skills versus or context versus the consumers, it shows you with data where your skill can be improved. And that for me, like whether it's 80% or 85% improvement with the skill, you know, that to me is useful information. It's a kind of like fuzzy feel good. But for me, the value here is for the author to go back and say, I can see what scenarios are failing. I can see what scenarios, actually my context doesn't provide much value for because it was getting it right beforehand.

52:20And going back to the point of overwhelming an agent with context, if an agent does well out the box already, does it need that context to actually just tell it to do the same thing the same way? It provides you that data to enhance and iterate to optimize the skill that you're creating. Yeah, and this is somewhere where I think from an enterprise context, we're going to see something new emerge. You know, in the last 10 years, platformers product, platform engineering, providing a deployment substrate to your engineers was hugely important. And you know, people looked at developer experience and that kind of stuff. But the kind of experiment running that you were describing there, that's something that the average enterprise developer doesn't have the time to do. Like, I'm going to do this story twice or maybe through the inclination. Like, I'll run this once in one work tree with a skill or once in another work tree without a skill, and then I'll look and evaluate that. They normally have too much feature pressure. So I can imagine a world in which, and in fact, I see this happening in some organizations,

53:21where there's like this kind of like, it's not quite a platform team, but like a DEVX ALA abstraction team, where they are building, they're working out which of the right tools to use. And much earlier, you mentioned about the challenges of an enterprise management, which skills are in use, and the whole package management thing that I know you folks are looking at. I have absolutely seen CTOs talking about that and asking the question very directly of like, how the hell do you manage all of these skills? Like, you know, it's just scripts floating around, moving between people's machines, like we need this to be auditable, we need it to be versions, we need to see what's running where because we're regulated. So I can imagine that as these tools start to get more uptick, we're going to end up with teams of people who are trying to provide a sensible kind of base level of abstraction for all the other developers to stand upon and to work out, which is the right skill to use. Wow. And do you think that's like a new role in the,

54:22it feels like an extension of the platform team, but perhaps not something that the platform team has existing skills around? Do you feel like it will be a role that maybe lives in the platform team? Or do you see that as a completely separate team? I think it would be nice if platform teams did that because really, you know, platform teams have tended to evolve out, well, I say you evolve out, evolve out of infrastructure teams, but then typically platform teams also have this injection of a lot of platform infrastructure teams were very kind of a ticket oriented, especially in the enterprise, like so is now ticket comes in, I do a thing, it goes out, they're not building something for the future that gets better every day. And then you have this injection of like a product management and software engineering. I was like, no, we're going to make durable solutions for this, rather than like doing toil and ticket comes in, ticket goes out, we're going to make a bit of automation to make sure that we never have to do that ticket again. So we've already seen like the evolution from infrastructure teams to platform teams and them introducing new skill sets. I think it would be nice to see the identity

55:23of a platform team evolving into how can we support developers with immediate future things running locally in terms of agents and the skills that they're using. Maybe the platform team becomes the software factory team in the longer term. And when we think about rollout for a larger organization, do you see that rollout coming from that platform team as they essentially are the enablers, right? Whether it's the developer experience, the platform team, all of these groups are enablers for larger enterprise development teams. Did you see that rollout spawning from that team? Yes. In the sense that a customer that we're talking to at the moment, they've got 12,300 developers. And so rolling out a consistent, sensible, auditable, regulatory, safe set of agentic coding practices, that's going to need coordination and that's going to need to come from somewhere central. If you've got people making up their own rules and ideas and pulling in skills and plugins,

56:23willy nilly across the organization that's going to lead to yes, auditing chaos. So that will need to be coming from a central place. And I see, I'm thinking of a particular customer here where there's a lot of people that were interested in the path to production and platforms were generally interested in tiny and feedback loops and value streams. And we haven't talked about value stream mapping at all, but that's a hugely important part when you're trying to adopt agentic coding practices working out how your software delivery function works as a system and what's going to happen if you speed up one particular part of it. But the kind of people that were interested in things like DevOps and platforms, they were always interested in productivity, getting ideas into production as quickly as possible. So then the product folks can learn was that the right thing to build. So it's a natural kind of progression for them to be thinking about how can we speed up other aspects of software delivery.

57:24What three tips would you give a developer who wants to be successful with agentic coding? One, get stuck in, start using something. Two, context management, make sure you understand it and three, perception, make sure that your agents can perceive what's going on so it can fix things. With the getting started, just pick it all. They're all much of a muchness, they're all fairly similar. So just get started with one, don't put this off until you think it's going to be settled science because everything is moving faster and continuing to move forward. With context management, it's not obvious what's going on under the hood. Clear your chats regularly. Make sure that you stay on topic. Don't give the model an excuse to be confused about things, don't add everything in the kitchen sink into your agent's MD because it's just gonna make things worse. On the topic of perception, if your agent lets you down and frustrates you by breaking things in CI, well, it needs to be able to see what's going on in CI. So add an MCP server so it can check what's going on in the latest build. Otherwise, you can't blame it for when it breaks things.

58:24What three tips would you give an organization who wants to roll out a gentex software development across their company? One, make sure that you're using a mature team who are enthusiastic and keen to adopt change. Two, they need to be well aligned have documented definition of what good looks like. And also three, they need to have good product management behind them where things are specific and well thought out. And there is a decent backlog of work to act upon. If you start off with a team that's already enthusiastic and performing well, they are more likely to go faster and you have better results that you can then repeat throughout the organization. And you've got the social proof of when people are looking at their peers of what has worked well and they can be impressed by those results. Folks need to be aligned around what good looks like. If you've got different ideas of what good code is, there's no way an agent is gonna be able to consistently come up with results that satisfy everybody. So that needs to be documented. Everybody needs to be on board on the same page what good engineering is. The product management side is something people often forget.

59:24Things need to be well specified so that agents can pick them up. As well as being well specified, there needs to be a decent backlog of work and a good understanding that underpins that what it is that needs building. Because as soon as you get your developers going faster, that backlog of stories is gonna run drive very quickly and I've seen that happening quite a few organizations. Yeah, amazing. Awesome. And as a final question, let's look a little bit forward. And you kinda, I think you've already alluded to some of this during the session. But looking forward, how do you see the future of agent development changing maybe the way we work as individual developers, but also as a workforce, as a company, the organization? The potential organizational change could be massive in that. And there's kind of incremental steps towards this where engineers are starting to get more encroach on the product turf by doing spectrum development. You've got product people starting to encroach on the engineering space with the idea of prototype as requirements. Those are all kind of incremental. But I think the big potential change here is software factories. You're already hearing about like high of mine startups

1:00:27where it's too slow to write things down because factories of multiple agents collaborating can deliver software so quickly that writing something down just slows the whole system down. So when we end up in that world and we're having conversations with customers about this of like, what does the world look like where you've got one product engineer per value stream? You don't have a two pizza team anymore. So many of the things about good agile software development might go out of the window there. So if we can end up with software factories, dark software factories where no one's reading code, no one's writing code, that reliably produce the right thing, then the future of the industry is going to look very different. But I really hope we're going to end up with much more bespoke usable software and an abundance of it, proliferation of unique solutions for individual people and teams where they don't have to wait for a SaaS provider to 18 months ago, can you please implement this feature? Like somebody can go and build it for you on a small scale almost immediately. And that I think is cause for optimism.

1:01:27Daniel, it's been absolutely wonderful and very insightful hearing some of the stories, hearing some of the advice and particularly the tips for people to really get going as a developer or as someone who wants to roll out a genetic development more within their organization. Super appreciate all your advice or your thoughts and it's been great having you on the episode. Cool, thank you very much, it's been pleasure being here. Thank you very much, Daniel, and thank you very much for listening tune into the next episode soon.