Trowser – The Software Tester's Browser with Rikard Edgren

0:00Right, and I think we're on. So another episode of the Evil Test to Show, and this time we have a guest. I like guest episodes because I have to do less work, hopefully. So we've got Ricard Edgren, and you're at thetesti.com, and you're the author of the little black book of test design, which I do have behind me somewhere, but you've got it free on your website as a PDF, so people can go off to the publication section and download that, which is a very dense, condensed book on testing techniques, thoughts, heuristics, all that kind of stuff. And you've also been working on a tool called Trouser, which is an AI browser tool, which we'll discuss in a minute. And hello. Hello. All right, so is that a good enough introduction? Oh, no, I missed it tonight. I missed legendary status from the last year's star conference, year star legend.

1:04What could be worth mentioning as well is that I have been doing testing all the time, almost. I started in 1998, and I did one year as a project manager, and I didn't like it, and then I went back to testing. But I've never become like a manager or stuff like that. I've been doing testing all the time, and that's what I like doing. So is that good? Is that good that you spent that's that narrow focus? I think that's the reason why I could write that dense book, or how I could create a testing browser, because I've narrowed down to the actual testing. Yeah. So does that mean you don't actually play the piano, because you only test? I can play it. So you do other things as well. I do have a lot of things as well. Yes. Yes. Okay. That's cool. So one of the interesting things about testing is just going back to what you said. When I started in testing, I started in testing, because it was a way of using a huge range of skills. Like I needed to be able to do development

2:07in testing, because we had to write our own test tools, because the test tools were too expensive. And although you say you did project management, but I didn't like it, you have no choice in testing, but to do project management. Yeah. Of course. Yeah. Testing is broad, of course. Yes. But the the center has always been the testing and finding out information about the sites or the software and testing. Yeah. So you learn a huge number of skills, even though your focus is just testing. Yes. Of course. Of course. Yeah. Yeah. I want to do become a developer. And then one company, you could start as a tester and then try to advance to developer, but I never could advance, and then I realized that I liked the testing. So it's certainly not that you never could advance, because you've written multiple tools and you're working on a tool now. Yeah. So it's that you chose not to advance into the programming space. Yes. Because I still believe in that concept of

3:12developer spanning all the rules. So that agile concept of developer where everyone is a developer. So you have a range of skills and you mix and mix and match on a team to get the best out of more people, but we tend we specialize to bring in that extreme domain knowledge and expertise. Yeah. And also I know some programming, but I'm a weak developer. So that's also the reason, should I why should I be a weak developer when apparently I'm pretty good at testing. So that's also into it. And now with the AI being so good at programming, we might end up with everyone being a tester and the AI can take part of the coding stuff. We'll see. So certainly. Yeah. What? I mean, so I find when I'm using AI, I have to do a lot of the testing. I mean, it can write unit tests and integration tests, and it can find issues as it's doing there, but it is very weak using the application to find issues. It depends on what your application is, I would say.

4:18And what you're looking for. And in my experience, it's a great compliment to the testing I do myself. So that brings us on a trowzer. Yeah. So how would you describe trowzer? So trowzer is a browser specifically made for testing. So it is a standard web you chromium in the core. So it can show any web page. But instead of having favorites and a lot of other stuff that the browser says, my tool has testing tools. So I have built in a lot of small tools that you want to use now and then. So you can you can test more quickly. That's one example. So this is designed to help you test rather than replace you as a tester. Yes. Yes, the design is to have a browser made for testing. So it's easier and faster to do testing. And it also has a rich API,

5:19which means that an LLM can also use the trowzer to do the same testing that I can do. Which also means that we can work on the same browser and look at what we see at the same time. And it also means we could have two trowzers with ARR1 and IR1 and then we compare the results. Or why limit to two? Three trowzers. Yeah. MULTY trowzers are doing MULTY user testing in on the same machine. Yes. So when you say the rest API, does that mean it's when you open up the browser, it's got an open socket, which is exposed as a rest API. It means that you can turn on an open socket at the port at your machine, at your local host. And it also has an API key that's unique for you. It's not wide open for anything. But since it has an API that can do almost anything in the browser, of course, it is very powerful and it's also dangerous in that cell.

6:22If you would go to a very dangerous website, it could do other things on your machine as well. Of course. So it's not as safe as other browsers, but it's also much more powerful. I mean, I can let the AI inject and it's gripped into the page and do anything and get access to the local storage and everything. So it's sharing the same browser session as you are. And it's a browser session that you have started and created. Yes. Rather than letting something like playwright open the browser, then you use it whilst it still has control. So you could say it's like a browser plus playwright, but it's embedding one, plus it also has testing tools. So for instance, I have the the X core library check accessibility it's built in and I built a lot of programmatic tests to check things on the page for annumers, amount of things. Yeah, I think X is quite good. I've been using X because I

7:24working on other test tool experiments on the side and one of those is has got X and playwright and lighthouse and I'm just figuring out how to combine all those. I just redesigned my website because of all the X stuff. So that might be why you didn't find the video on the podcast. Maybe I've been too excessively focused. But I remember you released a video showing browser, but then I couldn't find it on your channel. Oh yeah, I did like a demo. That's on YouTube. So if you have searched for a few tools, have you got two channels? Maybe I did a new channel called Trouser or something. All right, so you need to send the link to that so that I can stick it in the show notes because I had to look for it because I saw the Ricard Edgren channel and I had three videos. Quite old. All right. Yeah. All right. So you're going at a multiple channel mode on YouTube because you're going to be a massive YouTube probably not. But anyway, I did it because to me it sounds like such a good idea and I'm surprised

8:29we didn't have a testing browser earlier. We have a lot of developer browsers and stuff like that. We didn't have it and dedicated for testers. So that's why I want to do it. And it really helps me do testing of websites faster and better. So the demo shows what you can do with it. There's one small example just so people know how it could benefit them in the old days. If I would try to change the request or response on the website, I would launch, set the tech proxy, put my browser to that proxy port and I would do something, make it intercept, you know, stuff like that. Now I can just open the my network interception dialog in the browser and say, this is the reggae. If this happens, you should intercept. If this happens, we have this mock. I want to see how it hands the 500 error and stuff like that. So it's it's nothing spectacular new in trowelser, but it's convenient and it combines things so you can do it faster and then you will do it.

9:31And you decided to wrap Chromium. Did you experiment with just creating a Chrome plugin first or the G-Ghost straight to a browser? Yeah, so the story is that someone I trusted in the autumn, 2024 said that Cloud Sonnet is good at programming and I was not impressed with LLM's before that, not at all. But I trusted him. So I said, okay, let's try it. So I started doing small test tools that help me in my work, custom tools for the things I was testing. And it worked. It was good. It was so much fun to be able to create a tool that no one would ever create commercially, but I need that one right now and I could create it in an hour and it helped me. So then I saw that, wow, you can do this. So I had done so many plugins. Maybe I may be 50 different tools for my testing since then. And when I done all those, I was in the Sonnet,

10:33you could see that there was a limit. Maybe you experienced as well. When the potty's got bigger, it had problems. Things went wrong. It couldn't handle it with the context size and stuff like that. But then with Opus, I realized I don't reach the ceiling. I don't see the stop. So I was ready to try something. I won't do something more big than just a plugin for this. And that's when I came out with why not have my own browser. Wouldn't that be cool? And then it just went on from that. So I said, it's currently Windows only. Who are you writing it in? What programming language? It's C sharp. Okay. So in theory, it could possibly run on Mac with the .NET framework. Maybe I know it can run on wine on Linux. So with other emulators or wealth machines, it should be able to run on Mac as well. But I've been, as you've been long and in this as well,

11:35so you know that if you try to do something too generic, you will get into problems. So if I would have built a crossover trowel server for N platform, it would have been pretty weak with a bug everywhere and I wouldn't be able to test it. So I decided from day one, this will be Windows only because then it would be something good. And also with the C sharp and everything, it's fast. And yeah, we have to make interesting architectural choices when we build this. And so the fact that you're bundling it as Chrome probably gives you the ability to do things that you can't do from an extension. And probably more control over the UI. And it's certainly easier to install. It's just people get nervous installing executables with the Cance, the code. So you have to try and convince them that you've got a trustworthy app. But it's an interesting experiment, but also you are using it. Yeah.

12:38So it's not just here's a project I've created and it's not adding any value. No, it's more like I need it to do my work better. And then I'm also offered for free to others if they want to try it. Very few others are trying it. Extremely interesting. I think. So part of that is because it's kind of it's hard to find, but you have to know the test. I and it's buried as a zip on the kind of publications page at the bottom. So creating a new page might help because then the browser browser then you get some SEO benefits. But also that's why we're doing this to try and draw some attention to it. Because I mean, I have something similar that's not even close to as advanced, right? Because I wanted the ability when people are doing testing to track what you're doing inside the browser. That concept of taking exploratory testing notes. So I've got a Chrome extension which allows you to make notes in the browser and it will do things

13:44like take screenshots and it can do some checks and things. But it's purely an extension and currently it just writes out text files. And what I then do is I have the LLM point at the folder and then ask it questions about what I've done, what my coverage was, what ideas it's got for something else. You're getting that inside the browser as you test and feedback. Because I assume you can ask Trouser, what else can I do give me some ideas about what test? Yeah, I have the logging as well in Trouser and Trouser itself is pure software. There is no AI at all in Trouser, but you can connect to an AI. So I have like a chat window. So if I am somewhere, I could ask the AI, what should I test here? Or I could ask it, here's an input field, throwing all the dangerous payloads you know about to see what happens. So yeah, you can use it integrated as you want. I mean, well, I've also been experimented with is the Chrome AI APIs

14:51because inside Chrome itself there's some AI APIs for the prompt API. So you can actually use, so I created a LinkedIn Chrome extension, which as I'm going through LinkedIn, looks at all the posts, pulls out all the ones that have got links in it, shows me them in an extension on the side. So because I missed too much stuff on LinkedIn, but then I used the Chrome prompt AI that's built in to summarize those posts for me. So I've got summaries of all the posts that might be of interest down the side. And there's so much you can do in the browsers. And one thing I noticed with Chromium is it gives out slightly different browser headers to applications. So do you change the browser headers? Yeah, the, you mean the request headers? Yeah, they are super important in testing. So you can change them to whatever you want. Since I own the browser, I can change anything, and the AI can change anything. And I mean, Fiddling with headers is like the easiest way to

15:57provoke 500 errors. If that's something you're interested in. And yeah, you want to change the post body and stuff like that. And I also changed the user agent. So default, it says, browser. Because I've noticed that some websites when I'm automating them really don't like to be automated. So I have to change the headers in the automation in order to do the things, particularly user agents. One of the, so is this, when you've been building this, has this taught you more about how the web works in HTTP works. Oh, yeah. Oh, yeah. And is that then changed how you test websites? Yes, I would say that. I built it like a general testing tool for any tester who tests this web services. And I'm like a broad export tester. I have a look at usability and a little bit on security, not an expert, a bit of everything.

16:59Stop like that. And then I let the AI run the API. And I was like a stonch, wow, it can do exploratory testing. It can test something based on result. It do something else. It changes my, et cetera. It was fascinating. And then I realized that it was at its strongest for security testing, which it wasn't my intention. It was a general tool, but I found out, wow, it can do quite deep security. It's better than me. And then I needed to add some security features in trouser, because you don't want the AI to do everything by hand, because it would be too expensive. So after like a month with general trouser, I had several months with only security. And I learned so much about security and the web and HTTP and stuff like that. So it's like a new new area that I really like. So I mean, to start reading on request smuggling. I mean, if you haven't done that, wow, that's really fascinating. So are you now trying to use this in bug boundties? I have done that, yes. But they are so slow with

18:05with the responses, because they get so many things from AI. So I have done it and I got a few responses that were like, yeah, thank you very much. This is really interesting, but it was not exact what we're looking for and stuff like that. And I also did it on the VDPs, not for the money, but it's it's confined stuff. Yes. Right. That's cool. And I, before when I started looking at AI, I thought, okay, this is going to impact all career choices. So maybe I should go more in a security testing. Because I thought security testing is never going to get AI impact. And then one of the first big case studies that you see of success is security testing, where on hacker one for several months, the top hacker was an AI ball. And it's like, okay, we're going to have to figure out how to differentiate ourselves as humans and add value into this process. Yeah, but I think your gut feeling that

19:06security testing is a good way to go. I think it's still valid because you need someone to evaluate the results from AI. And you need someone to guide them and steer them and to see where they want to look deeper. And they are really good at some things, but they're not good at everything. But like a mediocre security tester, they need to find a different job. It's a good way. It's mediocre. It's a mediocre everything. Yeah, a mediocre everything needs to find a new thing, something they love more. They can learn more. So, so I didn't think because the security will be more and more of a problem when it's so easy to find the vulnerabilities, people will do it and they will do stuff. So the defensive work need to accelerate massively for any place that where your data is important. I did you see I last week, I think I was early this week or last week, hugging face got hacked. And then I think it was yesterday, it was said that the new

20:07open AI model had been the one that did that. You got to be really careful what you prompt it with. So when you're testing websites and you've got your browser, you're connected to the website. How you're connected to the web, how are you trying to sandbox or control browser? Yeah, so the first version of browser, this was in February and this year, it was a pure browser with nothing else, just a browser. And the very first feature I built was a scope mechanism. So you could enter the domain where you want to be. See if I said I want to be on eviltester.com, it could not go outside that it would be blocked. So I post the first feature I built because I felt this, that's a must if you let AI do it, it's a must you need anything can happen. So I used it for day or two and I never used it since then because the AI actually stays on target. And I mean,

21:08you're not really using it as an agent. I have an agent working in trowelser. So I usually use cursor, cursor is agent and it connects to the AI and it could do anything, but it does what I tell it to do. And I'm they have probably put a lot of effort of do just what I say don't go away because that's where you get the horror stories and even better reputation. So I have that feature to scope it, but I never use it. And have you been experimenting with other LLMs and other agents? Yes, I have. And which do you have you tried running it on local agents like Olamma? Okay. And do you get similar results? No, my computer is not powerful enough to be able to have the model that is needed in order to do testing. I think you can do programming, which is much easier than testing, you know, but the testing is so difficult to do is that you need,

22:12you need the cloud opus or better. I think. Okay. So how do you see trowelser expanding going forward? When I built it, I didn't know of any other way that you could go to websites, log in somewhere and let an AI do the testing. So when I realized what I had built and what you could do, it felt, wow, this is so, wow, I'm going to be a millionaire was a thought I had for an hour. And then I realized that's yeah, but that's because you like to experiment and see new things and find new problems, but most testers want to do the next year ticket and get the automation done. So they won't have use for it. So it's, I thought this will be huge, but this is not and it will not be. It will be like a power use tool for exploratory testers who want to

23:15expand and find new things and find new ways and work a bit faster and stuff like that. I don't think it will be a blockbuster. But I'm hoping that it will continue because I build new things all the time. It's so fast to do so. It will be new things and maybe it will be a hit somewhere for someone with it. So you're ever going to open source it. Well, it's free right now. So it's for free, but I am right now I'm reluctant to open source all the code because I don't know how that will be used. And someone else might take it and sell it or make a new version that's more of a hacker and stuff like that. I don't know. And also I'm not sure that I want other people to have opinions on the codes and stuff like that. Because you know what's what's taking the most time in software development?

24:17Tell me. It's to agree on what to do for people to agree what to do. That's what takes the most time. And with Trouser, I'm the only one. I'm the only one who have the code base. I'm the product owner. I'm the developer. I'm the tester. I'm everything. To agree is very fast. So I have removed that botning. So I can understand that I mean I tend to open source a lot of my stuff, but I find that no one really contributes or tries to change it anyway. And a few people do bug fixes, which is always useful. So I guess if it becomes popular, then people might try and steer it. But interestingly, that might also be useful in terms of getting new features in there. Yeah, we might end up there, I don't know. But I also have a small security concern,

25:19where open source is good for security if you have an active and big community. Because then people will look for it and fix it. But if I open source it and no one looks at it, except someone who wants to find the security floor and exploit it and stuff like that. Because since it's a browser, it's very powerful. So yeah, that's also in the in the back there. Okay. Because it'd be interesting to look at the source and compare notes with other things. Because I'm working on so many different projects. I don't spend a lot of dedicated time on anyone. But the Chrome extension I've got for exposure testing, the observatory on, I might well extend that at some point. And it'd be interesting to compare notes with other tools. But that's okay. I mean, I could give the source code to you, Alan, of course. That would be fine. Well, it's not like you're Ruby one where you could actually get the source code

26:23from the executable. I think the new AI are really good at reversing stuff. So they probably be able to find the stuff. Okay. Well, we'll see how it goes on. And if you ever do want to open source it, give me a shout. But also in terms of code quality and things, the LLM is really, really good refactoring code once it's given guidance on the architecture. And anytime I do in the AI development, I just load up the project with as many static analysis tools as I can associated with that language, get the prettier in there for the code formatting. So I don't do any of that stuff. And then it goes into the the build process in Git. So it's going through the CI process where all the unit tests are running. And the great thing is the AI will add all that for you. So if you got a lot of unit tests in your code, really good question. The amount of unit test is zero. Is that by choice? Yes.

27:25Right. Why is that? It is because when I build my test tools like in the year and a half ago, I spend a lot of time doing unit test and integration tests and I learned new frameworks for Yolva script and stuff like that. And it was really fun. But they never found any bugs. The bugs I found was when I used the tool for real. And so yeah, we need to change that. So I did that. Since I'm a tester, we need to have a unit test. Of course, we should have that. And I did that. And I realized I spent more time building the tests and maintaining them than building the tool. And now when the AI is becoming more costly, that would mean that I would like half or maybe I only can do one third of my trowelser feature enablements without knowing if there is value to it. Okay, I mean, I guess since you're still experimenting, unit tests can

28:29potentially delay it and it can slow down the development work. I do find LLNs are getting much, much better at coding. So I often use the unit tests in the code base. Not so much to find bugs, but to protect it from adding new bugs when it's working on a new area. And I find the unit tests very good at that. And also the unit tests when I'm reviewing them exposed to me the bad architecture decisions that the LLN has put in. And then I get it to refactor as a result of that. So what I have for trowelser is I have a smoke test. That's actually also inside trowelser as a quick test. Trowelser has like a script console and you have a quick test with code. So it runs a smoke test in 35 seconds or something executing a lot of the internal mechanism. That's all the ultimate tests I have right now. I will build when I feel

29:29when I feel like okay now now I have something here we are. This is where we are. I'm not going to change a lot in this. Then I will add tests and I right now I'm thinking about doing something that actually executes trowelser. Maybe it will be maybe I will have a docker instance of a web service that we will test programmatically in many ways and we have the expected results or an advanced internal mocking. So I mean I'm using trowelser to execute the features to see that all of them work. It's something like that but that's like a huge effort so that would be like the main thing to do. And right now I'm not finished with all the nifty features of the tool itself. I mean that's one of the things I always really good at experimenting and organically growing something. The danger is that organically grows a mess. So it's like you have to stop step back and fix the mess that has been created. Did you ever try and use open code? Not.

30:33So open code if you ever get concerned about cost and tokens is really good because open code itself is a free version of like Claude or codex. But you can sign up to open code Zen which allows you access to free frontier models. So a lot of the time when the big companies are making their models they release them in kind of beta which can come through open code. So I was using open code on GROC for a while. So you get access to these huge frontier models completely for free. There's kind of limits and you're probably helping train those models so your code is going through to them. But if I'm working on an open source tool I don't really care about that. Yeah. But that's an interesting augmentation and I find that open code is actually quite good at reviewing code. So I've got codex doing reviews. It's not brilliant. I hook up to code rabbit. Code rabbit is

31:34very good at doing reviews in CI. But open code I find actually really good at reviewing the code base and it's for free. Okay. So one of the problems with all the AI stuff is there's too many tools, too much to experiment with. And too is to try better. If that I would recommend top the quick look at that and see if it adds any value into the process. Yeah. Right now I'm using cursor. So this is my my hobby project. So I do normal work at the week and then on the spare time I do the trowelser and I have my own cursor license. And the opus credits I have it's less than a week and then I'm out of them. So I get new credits at the end of every month. So that's when I do like the major things or a big refactor or something. I do that and then I do small things with composure 2.5. Wait for the end of the month and soon I will take the next leap with opus.

32:34Yeah. So open code might help you fill in some of those empty weekends and cause you to not go out in the sunshine anymore. That would be good. Never see your children. I just work on this thing. Yeah but you know. If you just have you you think in the back of your head you have like this ID and the prompt forms without you know about it and then you just write it and then it can work for itself. So I mean I can see it spent time with my children. I can pay some some some some pros. Yeah so one of the benefits with one of the disadvantages with codex is you can connect to it remotely from your phone. So you can tell it to do something then you can go we pretend to do something else but still connect to your ear. So NAI in general how do you feel about it for software development? In general like big general I'm not sure if we are better off with AI than without it right now because it's used so many strange ways so I'm not sure if it's good for us or not.

33:38And I think that applies also for software development because you can use it so many ways but I also see that it can be extremely powerful and I guess software development is where it's at its best because the ones built it are developers so they built it for themselves and they have used it. So I think it's a very powerful tool but you of course need the more power the more you need to be careful with how you're using it. Yeah what the genie is out the bottle now so we're never going to get rid of it. No it's interesting that I mean even you you mentioned it in this where it's quite expensive to use so whatever plan you're on you only get it as short of your time but there are so many ways of using it free for smaller activities so the more experienced you are the more value you can get out of the free models and certainly you get most benefit from it

34:39with coding because that's you're using it to do so much stuff but as soon as you have an activity that you can split down into chunks like say you wanted to summarize a document and you can split down into chunks of pooling the text out the word document that can be done algorithmically then you can feed chunks into the AI in smaller parts and then use smaller AI models so it's not that AI itself is only good for software development but certainly AI in a kind of chat type interface that can review code and work autonomously is excellent there and it just might need a little bit support to do everything else. Yeah we also need to to make sure we do the right things I mean as always but now even more because it's so easy to do it so easy to do something that doesn't really have an value and you just do it because you can. So one thing that helps with obviously is testing because you check it as he has it done the right

35:44thing or has it done something wrong and are you when you're trying to use AI expanding your design skills and modeling skills and architecture skills. No okay and you said in a previous podcast that you you never use it for writing you never use it for test ID creation because you already have so many test leads and it's the same for me I mean the problem is not the idea the problem is it's more of execution time you have to stop so I would never use it to create a test strategy or something it would look good and someone who doesn't know the area would think wow great how good it is but is this not good and part of the good thing about it is that if you create yourself you know about it if I have a test idea we have 20 good strange test studies and then you decide I will do these five then you have done the analysis anything better so I would rather

36:45have the AI yeah you can create test studies or test cases but you have to run them yourself I'm not running them for you and then when it's interesting do do some testing here and show me the results so one of the things that has changed slightly from my opinion there is I have been let I use AI and I tell it to go and find bugs on the site and I don't know that I count it as testing I count it as coverage and exploring interacting that's fine but it's I would the reason I don't count it as testing is there's no coverage strategy there's no thought around everything that we could do so it can give me some information that sometimes is valid that I can act on and I'm sure Trouser can do that too you tell it go away look at this find some bugs and exercise it which for an experienced tester is good because you can let it do that in parallel while you're off thinking about something

37:46else but the danger is that some people see that see the result as bugs and go all right now it's testing and now I have to worry because it's doing all the coverage and when you get the AI to tell you what it has covered it's really small in terms of the scope of what it could have done and so I imagine that there's a at the moment there's still a rule for testers to work out what could we do what should we be doing and try to harness the whole coverage but AI still can can be used to find some bugs and sometimes I'll let it do that autonomously before I have a look at the system it's the same as for me it's a compliment to my own testing and I think that together we get a better coverage in total so it's it's useful and it's more attentive to details than me it's better than me on security testing it can work for more hours without getting bored

38:46and stuff like that so it really has value but I would never do it only the AI do the testing if if the result is important and also the the lens are very like they are pleasers so if you tell it we have this very nice website can you check that it's good it's a it's a wonderful website it's so much value and you have just a small tiny floor so there and if you tell it to find as many bugs as possible with high criticality it will show you oh we have 25 super critical stuff but not all of them are so one trick I do something is to give the opposite instructions like this is a super good software try to find some small thing over there or this is a lousy software there are bugs everywhere tell them about the most important ones and then you get a better total and I think I mean testing is testing is in the sampling business we can never test everything with contest samples over here and there and if we look at it with different

39:47perspectives we will get a better hole and the AI is a new perspective so it helps us yep so have you looked at try to look at any of the kind of commercial AI test tools there right there I know smart beer had like a webinar in March I think where they were going to showcase something that would change the testing world the revolution everybody has been waiting for and I was so disappointed I really hoped it would be something great but it was like test case on mass like you know test case test case test test test case or everywhere so what I've seen I'm not impressed and I I also see that most people are focusing on the automation thing and that part and I think that's handcrafted automation will be better for some period of time and automation is not all there is and there's also a lot of talk about the

40:49the pipelines the CICD and the preventing the defects and stuff like that and yeah it's interesting because I mean two vendors are in the position where they try and target unleash and so if it's a test tool vendor they're going to target what they understand of testing and they understand testing based on the tools they built before which have all been test case management tools so consequently all the AI builds test cases then automates those test cases and reports on it but why find interesting this and having tried some of the commercial tools when it goes away and creates loads of test cases it doesn't create loads of test cases it can create 10 or 20 or 30 and if I look at my codebase I have hundreds if not thousands of automated execution paths and I don't understand I think one of the things that you said AI is designed to please it's also designed to please not use too many tokens so I think it doesn't try and do

41:54the expense so it just tries to cartel it so you have to keep pushing it to go all right give me some more give me some more yeah but I mean the solution there is probably you should write your own tool that creates the test cases if you won't have the test cases we don't want to but other people might want them so if you know your system you can use AI to create a tool that creates audio tests and then that would be cheaper token wise and it could be repeatable as well I think if you can create a tool that does it instead of asking the AI to do it so we already have the the tool to do that in the sense that if you so for test just put them in as unit tests integration tests they run this part of the codebase they're automated that's what test means at that point and I suspect that the LLM driven stuff needs to stem from requirements which they they LLM land in terprits and then goes okay does the system meet this requirement

42:55and then it knows ways of violating the requirements based on heuristics or something else one of the things I'm trying to do at the moment with any way data which is a essentially data generator where you give it a schema and it generates data it's to have an anti schema so once you've got a schema it generates data which doesn't reach your schema and then you can feed that in that seems like a sensible thing to do and having something that looks at your requirements and then constantly tries to violate them on an ongoing basis because then it doesn't matter if it doesn't find anything but if it does it's information and I think the most of the test tools the AI test tools are focused on the wrong type of thing they're focused on trying to achieve some sort of coverage rather than trying to provide you some unique form of information that you would not have thought of or found yeah but it's quite hard to justify the cost of that yes we're paying like $200 for a tool that gives us one thing every week but later on the local models will be

44:03runable on your machines and the cost will be much cheaper so that's that's where you get to explode something that would be ideal unfortunately no one's got any graphics cards because they're all going in the data centers so the local I mean at the moment the local AI could be really good if you could get your hand on the kit so it almost feels like a conspiracy of trying to push us into the cloud-based AI by keeping it off our desktops even though it could be there but hopefully it will get there at some point I came to think about one other realization I had quite recently how you think about testing and because I realized since no one likes trowelser it must be I think different than many other people and what I realized is that many tool vendors and testers and teams and developers are focusing on testing the code that was recently written we did this code we're going to test this code my objective is to test the system and test the system that was changed by the code that was done the last weeks but I

45:10look at the system not that the code I mean for me code coverage is completely useless I mean quite often it's the code that wasn't there that was the bug and you can't have coverage that so and that's also what what trowelser is more it's like you look at the system to see what does it's not so conservative code because developers concern with that and I'm bringing a different perspective and so I think that's also quite a lot without all these test cases and the coverage and it's more on the code than on the system yeah I mean it's a different coverage model and when you're doing proper testing your coverage model is not code because that's that's a ridiculous concept that testing would focus on code coverage but it's really important to have code coverage to make sure that your code works and doesn't explode live but there are two completely different coverage approaches and that focus of the system is really interesting because the AI finds it quite hard to work out what the system is from the code base I think it's getting better

46:16like it it doesn't necessarily know what it's supposed to do I'm not sure I agree Alan the recently what I've done the last last month is tough like us typically on a website I will tell you do a reconnaissance test and tell me what this site is about the architecture and stuff like that and the results is good it tells me what the site is about it tells me quite a lot about architecture and like the API and stuff like that and so I think it's it has become good at that so it's getting better and I think what I mean is not so much what it can do what features it's supposed is but what the user wants it to be able to do and how they will use it and what the user need is it's very good at doing parts of that but sometimes it gets the wrong nuance for what is there and acceptable but having something that guides it like a requirement set or an aims or a goal set or something I think is also important but it's

47:21interesting that we haven't quite got that part yet but I think that's hard to sell for people but I think Paul's working on some of my Paul Gerard is trying to do some sort of requirement driven coverage types so that I guess brings us on to Eurostar you were at Eurostar a little over a month ago because you're now a legend because you were doing a Eurostar legends talk I think does that just mean you're old or does that mean that you're a legend it's very problem for me because for many years I tried to have the image of being humble and it's more and more difficult when you get that very long one yeah just the image not the actual okay and so in that talk and you were covering your kind of past experience on testing which is useful lessons learned and the future so are you able or is it a talk you're going to repeat or could you like cover some of it without

48:22giving away the answers if you ever repeat the time I don't think I would repeat it but I don't I do like one talk a year something like that so not that much so what it was was basically they wanted the instruction was to have something more personal about what I've done and what I see in coming and stuff like that so I told a bunch of stories from from my career and I can tell you one of the stories now because I don't want to summarize the speech but I tell you one of the stories my first day as a tester I had an extremely important test because I was we were testing a Swedish version of Microsoft Internet Explorer 4 service pack 2 November 98 and we were handed the test cases on paper back in days and I had like a veteran tester with me who was guiding me through first so I did the first test case and then when I was about to do the second test case and there's told me that in this test case you should not follow

49:24the exact steps written there you should get to the end result but you should take a different path use the keyboard do something else in between or use the accelerators and then I tell yeah but it says I should do this Microsoft said told me to execute this test yeah yeah but I can't do something else it's that's just what they say they want you to have variation and it's easier for that the right test cases this way so you should do it with variation and that's where we will discover more interesting things and I said okay let's do it and that small thing has formed my whole testing career to do variations to be observant to not follow rules strictly instead follow at the intent and the purpose and discover things about the software yeah I mean it's a good um notion we can automate the repeated path but adding variation you can add some variation

50:30in to the execution but the real variation is going to come from the human and adjusting it every single time and I mean it's good that you got that on the first day because I don't like people learn that until later when something sneaks through the nail so how do you see the future then for testing I would say that I mean the AI thing now is really changing things but it not changing that much I mean I you said in a podcast that maybe AI will force the world to rediscover what software testing really involves meaning that people have been doing quite bad testing here and there but now they are forced to do better testing because AI can do the weak testing and AI will produce more software that needs testing and I mean we have so many things that we know

51:31about I mean uh for instance uh in some companies I work with quality goals which are like this list of 15 sentences what qualities for our products that's something good to do then it's good now you could provide that to an AI to both the developer AI and the test AI and they will have a lot of stuff so my hope is that we have to do more of the good testing that we already knew about but we have to rediscover it as you yourself so cool so are you um so you're doing chisers to help you advance your abilities when testing these things um is there anything that people can do to get better at testing then I think the first thing to do is to understand what is important about your software and it's a lot of things that are important and to have that in the back of your head all the time

52:33and not be too focused on your year I think it's that will be the first thing I guess it depends how you write your tickets of course if they're too small and specific yeah but at the same time if you write this uh this super big that involves and you should be have the accessibility compliant and it should be easy to use and the words should be clear and simple to understand as no one will do it that way and so it's difficult and another thing that's uh is important about trouser is that uh I don't know if you heard no about get gigger answer uh he has tools to theorist heuristic and this is a general for society and what he says that the theories we have are shaped by the tools we are using so the tools form our model of the world so if you're using a test tool from a vendor with test cases and everything and the pass failed that forms your opinion about what what testing is so that's my hope with trouser since trouser is a tool people use the

53:40tool trouser and then they will think about testing more the way I think they should be think about testing so that's certainly true if the if the tool has capabilities and it covers an area that the person knows nothing about then they have they either have to ignore that feature or learn it and start using it and that's why I like people to open up the dev tools and a browser because instantly they see all this stuff that they may not know is appropriate for the web board what actually means and just opens up an entirely new world for people I met the tester once who said that I'll ask but why aren't you using the dev tools and they said well I tried that but there was so much noise over there no red things and alerts and stuff like that so if you aren't using dev tools you're not ready for trouser either I think we'll say trouser also have dev tools you get it built in with the with the chromium yeah I know I think it's a good and that's an interesting

54:43point to finish on that concept of augmentation which then pushes you further and it can be interactive augmentation like trouser or the dev tools or it can be static augmentation like running acts from the command line to open up a whole new set of information that you've never seen before and using tools not just to do something but to expand your knowledge about what it's doing and then build on it and then take that knowledge up to the next level so I think that's a useful thing and so I guess one lesson could be keep adding tools into your process to get new capabilities as long as you understand them and don't just lean on it as something to do yeah no so you can now you can create your own tools so that would be an encouragement to any tester whenever you're testing something you feel oh wow this is difficult oh wow this is boring maybe you need a tool maybe you should create your own tool because of web extension you can suck out all the network

55:48and find stuff and do reg X and stuff and the air will do it for you yeah really see yeah I think that's good idea as well and I'll say that you don't actually need to pay for AI tooling or to do it because you can use open I should be sponsored by OpenCode or not but OpenCode because you use OpenCode then gives you frontier models and you can use them so you can build tools without actually having to pay for any AI system or run a look at it and also chat GPT can be used for like Python scripts it want the Python script that looks through these files and find this stuff it can do that but it cannot do the bigger system with many files in world cool all right so and thanks for sharing your knowledge record and everyone can find you at the test i where i is y e dot com and there's a lot of stuff on there to read and we'll try and put show notes and things in there with links and in any last things you want to cover or say

56:49no it was nice talking to you Adam when next time we'll try and do you on the piano me on the guitar and then so we can do that all right thanks for guard thank you bye everyone bye so that was a chat with record Edgren you can find them at the test i dot com and we've got all the show notes over at evil tester dot com slash show