496: Test suite performance

0:00Hello and welcome to another episode of The Bike Shed, a weekly podcast from your friends that thought bought about developing great software. I'm Joel Kenville, and I'm Sally Hall, and together we're here to share a bit of what we blurred along the way. So Sally, what's new in your world? I had a fun time this weekend. My brother, who's one of my close friends in Lidz nearby, asked me if I could help him with his portfolio website. He's also in tech, but he's a designer and has just decided, like, hey, I haven't touched this in a long time and I should freshen it up and ask me for feedback. I gave him feedback and then ended up like wandering into his GitHub repo and looking at the code itself and giving him feedback. And then at some point he was like, I don't think that matters. Like, I'm not trying to get it.

1:00I'm not a designer who codes, like, I'm just trying to display the work I've done. It doesn't matter how well the website works if it works. And I was like, but it does matter. I do care about these things. Do you have a deployment pipeline? Right? Like, your commit messages are weird. You might want to, like, rebase this and squash them with this together. I think one of the things I said to him, I was like, you should add DS store to your Git Ignore. And he was like, my what? But then, you know, I started like reading through like the case studies and things that he has on his website. And I'm like, okay, I don't know what any of this means. And it all looks very impressive. So good job. Uh-huh. My mom thinks we have the same job, but we really, really don't. Is it bad that when you first said portfolio, I was thinking like stock portfolio? Oh, gosh. Although, why would you ever want a website for that? Unless you're hawking something. And why would you come to me for that? Website, yes. Stock portfolio.

2:00No. So yes, development and design or specialty. And you know, there are plenty of people who do both, but neither of us are really those people. What about you? What's new in your world? I've been on a project where I've had to do a pretty significant refactor of a subsystem with some really gnarly code that had been sort of evolving over time. So bits and pieces have been bolted on. And I've been really leaning heavily on LLM to try to explain why choices were made. And they're surprisingly good at code archaeology. So you're asking it why things were made, like asking it to explain to you past decisions that other people made? Yes, specifically by using the get log. Okay. So they tell me who, tell me the why and give me a commit shot. And it will do that to a certain amount of depth as well. So you know, it'll think for a little bit and it'll spit out a report and say, well, it was initially introduced to solve this problem.

3:01There was a race condition here and so this kind of hacky fix was added in. And then somebody else was making a change over here and they merged it with this thing. And so now the like hack has to play sort of double duty because it fixes a hole here, but also is load bearing over there. And then somebody wanted to extract that into a shared module and it moved and it'll give me like the three commits. And I was like, that was really useful context. And it's the kind of thing that I could have figured out on my own because it's just running git commands and scrolling through some craft that doesn't matter. But an agent in my code editor will do it near instantly. And so it took a thing that was valuable, but that I didn't do as much as I should because it wasn't worth the bother to now something that I rely on much more frequently. Hmm. Interesting. I can be such a cranky person sometimes because I'm just like, but I love doing that myself. And no one's making me not do that myself. I think for me, it was a little bit of a chore.

4:03The main thing is it's slow, right? I'm trying to get context to make a decision. And now I'm going to spend the next 15, 20 minutes digging through the git log, trying to understand something instead of getting a report within 30 seconds and then being able to make the decision, which is the thing I wanted to do anyway. Do you feel like there was anything missing from what your LLL gave you that would have been there if you had done it yourself or like any downsides to not having done it yourself that you noticed? Not really. I feel like the LLLM actually did a more thorough job that I probably would have done myself. Like it went deeper and got more context. It was relevant. Whereas I might have stopped at that first commit that I found. It also did a better job. I think of following things through like different file moves and things like that. I feel like I got a better outcome, at least on this particular refactor project. Maybe that's the thing to check in on at the end of it because we're getting close to the end of this season. Maybe that's a season three kind of thing, but like, hey, now that you've been using

5:05this practice semi-regularly, do you find that has the accuracy and the like thoroughness that you need or is it like, oh, it was useful in this one project, but overall, not the best technique. Do you like then verify that what it's telling you is real or have you gotten to a point where you just trust it? I'm mostly trusted. Occasionally, I'll do like take the commit shot and do like a get show, that kind of thing. If I want to dig into something particular, but I've not had any cases, in the case of where I have verified it, it's all been accurate. So I've not really run into anything where it's hallucinated, commits or anything like that. Well, that's good. It sounds useful for you. And my guess is because it's working with a very small sort of set of data that's like already there, it's much less likely to want to hallucinate something. I can see that. So yeah, I think the interaction of get and get history at LLMS is an area that I'm trying to tap a little bit more. Because I think there's a lot of value there.

6:06A lot of value that I get as a person, but that's still a little bit, sometimes slow and cumbersome. And I think that I can get a lot more value by plugging more machines into the get log because it's a really rich source of context assuming you're not my brother and you know how to make decent guess, get commits. Yes, yes, that is true. I would say I always write pretty in-depth get commit messages for humans to read. It also turns out that commit messages that are great for humans to read are also really good for LLMS. Another technique that I've used is if I want to load context into a session, sometimes I will just point it to a past commit and say, read this commit, this will give you all the session that you need and then we'll talk about what I want to do. And because I've written my commit in such a way that another human could read that and get all the context, it also works really well for automated tooling. Yeah, I was wondering whether you write commit messages differently using these tools,

7:07but it sounds like you don't. No, no, it's really, I think what's good for humans is typically good for LLMS. Any sort of like, why do we make this decision? What is this weird choice it was made in the code or why did I structure things in a certain way? Any gotchas, any maybe benchmarks I ran on change, alternative decisions that were evaluated but were we chose to not take that sort of thing. And AI is able to analyze a different sort of get a sense of what's happened in the diff better than a human can. I can stare at a thousand lines of diff and it will take me a long time to get a sense of what's happening. And so I need, I often rely on the commit message to give me a little bit of a sense of the what in addition to the why, whereas I think the LLM really cares mostly about the why and the human cares mostly about the why anyway. So making sure that's included in that and the commit message I think is what has the highest impact and we'll have the highest impact on your colleagues.

8:10Yeah. It's the thing that I push for the most on colleagues and clients is the, you didn't, you have a change, you tell me what you did, you didn't tell me why you did it or why why you went out this path and not another, not another one, can you please explain that decision? Yeah. Those are so crucial. I find that, you know, a good commit message has prevented me from reintroducing a bug a lot. You know, like, why are they doing this this way? This seems like a very strange way to approach this and then you go to the commit history and you're like, Oh, I see that they've listed out all the things I thought they should have done instead as things they tried and didn't work and here's why. Uh-huh. Well, would you like to talk about test suite performance? No, it's a topic you enjoy, it's a topic I enjoy. Yeah. Yeah. I've done a decent amount of work on projects that required better performance for the test suites. I gave a talk, uh, RailsConf several years ago titled your test suite is making too many database calls that was all about a particular aspect, uh, performance done a decent amount

9:14of work with parallelization as well. So I've touched a bunch of different aspects of, of performance. What are some areas of test performance that you've had to interact with recently? The test performance issue that I have dealt with the most tends to be like system tests. Hmm. A system test is inherently going to be slower than a unit test, but sometimes we need them and so like, when do I write it? How do I write it? Why do I write it? How do I make it less painful? Are you talking specifically Kepi Bar are driven? For the most part, yeah, that's where most of my experiences. How do you feel about the testing pyramid as a concept? I think it makes a lot of sense. I'm trying to remember exactly how that works, it's the idea of like, it's like, if we compared it to the food pyramid, like system tests would be like desserts. Right. Right. I do think that makes sense. I definitely, you know, many, many years ago, system tests were often the first tests

10:15I wrote, and I think that is less the case now. Do you find yourself writing kind of an outside-in approach, writing system tests first, and then letting that push you towards unit tests, or do you tend to sort of go inside out, starting with unit tests and then working your way up to system? I think I do both, I think it kind of, I think it also depends on like, am I working on a whole new feature? Right. In that case, I might do outside-in more because like, what is this user interaction going to look like? Or am I writing a test because I'm fixing a bug and want to prevent regression? That's fair. What about you? I do also mix. If I am sort of doing classic TDD, I tend to prefer outside-in, especially for new feature work. So you sort of start with a broad description of like, what are the parameters within which I am trying to build this feature, and then within that, I can play around and come up with an implementation that does what I want, and maybe parts of that implementation need extra tests for themselves.

11:18So I do really like that approach, but these days sometimes I find myself testing after. So that can be sort of a little bit of whatever. I find that, yeah, the idea of like, testing a little after these days, I feel like sometimes I feel like a spicy take, but I don't think it really should be. But I think the thing that most often will lead me to switch to like TDD if I haven't been doing it otherwise is going like, man, I keep doing these same like four manual setup things to see where my work is and to like work through it. I should really just put this in a test that I can run. The right-a-test with some like setup and then put a binding pry in it so that you can get exactly the state that you need to like manually check some things. Even just the debugging technique, I really do like that. Yeah, that's a good one. What is your spiciest test performance take?

12:18Because I know you've got some. I would say that it's almost like a Jason performance take, but one of the big causes of slowness in a test suite is going to be database rights. Typically this is going to be caused by using factory bot. I think most people don't really understand the value of factory bot or how to use it. And so oftentimes end up making the wrong trade-offs when it comes to performance in their tests, but also end up making tests that are really, really slow. I'm wondering if I am guilty of some of these trade-offs. Give me an example. Let me flip that question back to you. What would you say is the value of factory bot to you as a developer? Why do you like that tool? I like it because it allows me to focus on the bits of data that I actually care about when I'm writing a test. So if I need to create a new user and I really just need that user so that I can log into

13:22something, it's annoying to me to be like, okay, well a user needs to have a first name and a last name and a valid email and a valid phone number and all of this other stuff. And like having to write all of that out myself every time is so tedious and makes the code so messy and can make it harder for someone reading the test to know like what matters here. Right. Right. So I really like using a factory to be like, I'm creating a user. The user's role is admin. I'm not specifying anything else because nothing else matters. What matters about this user is the fact that they are an admin. Yeah. So an emphasis on locality, you want your setup data to be in your test, but you also have a certain like minimalism where you're trying to only have the data that's relevant to the test show up in your setup and nothing else that's just noise. Right. I think another thing that I read to a factory about for it that makes that I suspect might be some of what you think people are leveraging the wrong parts of it is same

14:25situation. I need a user so that I can test this, but all users have to belong to an organization. And then there might be like another layer of that, right? So every time you instantiate a user, you have to also instantiate an organization. Right. And that is annoying. And I don't care about the organization, it's not relevant for this part. So if I have a factory that does that for me, then I don't have to remember that in every single time, run my test and have it be like, this user's not valid because his organization isn't there and remember to do that myself. But I think because we've talked about this before enough, because I've done this to myself, that is where it gets messy and slow sometimes because every time you're creating a new record, you're going out to the database and all of those round trips to the database add up so that can get expensive. A, I would say mildly spicy opinion is that your factories, at least the base ones, should only have the minimum number of fields required in order to define on there for that record

15:32to be valid, to be savable to the database. I agree with that. I don't think that's terribly spicy. That is official factory bought recommendations. Yeah. In practice, most code bases I've worked on don't do that. Do you consider if you're making a factory for a model that has a belongs to association? Is that associated record part of what you consider the base needs to exist for this to be valid? Can you save it to the database? No. If Rails will yell at you for not doing that, then I will probably put it on the factory. If Rails will let me get away by not setting that, then I will try to not set that. If a user needs an organization because validation will fail without it and it won't save, it goes in the base factory. If a user needs an organization because nothing in the code makes sense without being able to refer to the user's organization, but for whatever reason the user is still valid with that one, which is pretend like that's a normal situation, then that doesn't go

16:36in the base. Correct. So the base is really the minimal set of values that will validly save to the database. Yeah. I think I have that type of approach. I feel confident about that. That makes sense. The idea here being that the part of the purpose of factories is like you said, it's hiding that noise of all the things that are not relevant. And let's say you didn't need to save to the database. Let's say you just needed a user in memory. You don't need a factory for that because you can just set user.new and set the one thing that you care about, maybe the admin flag, and then everything else doesn't matter. It's not relevant to this test and you can execute your test. The reason you care about all this other stuff that's irrelevant to your test is because if your record needs to be saved to the database, active record will enforce validations, or maybe your database has non-null constraints or foreign keys, and you cannot save to the database, even though those things are not relevant.

17:37And so what the factory is doing is it's handling all the irrelevant, but somehow also required stuff so that your test can only focus on the relevant things. I feel like approaching factories this way can also highlight sometimes errors in your validation, right? I've told active record that every user needs to have an email, and now I'm writing this test, and I want to create a user without an email, and it won't let me. Maybe that means that this actually should not be a required field. Errors are a post validation because you can create an instance in memory, and so almost anytime you're creating just an instance in memory, you almost never need a factory. Or like the build and build stubbed strategies that you can use if you need things like associations and all that built out. But for the most part, if you are testing an object and you're not persisting into the database, you don't need a factory. I feel like I tend to default to if that factory exists using build or build stubbed anyway.

18:40What are your thoughts on that? Like if we're not saving it to the database, then we don't have that particular performance hit. Depending on the strategy you use, there can be some like downstream things where it does save to the database, especially I think with the build strategy. So it's not quite as clean as you need, but also it is a little bit more extra overhead for things that if you don't care about, then there's no sense in doing. Now where you might want something like that is say you want a user instance in memory, not save to the database, but it does need to pass validations. The fact that it passes validations is not super relevant to the thing you're trying to test, but active record will get into the way. And so maybe then you'd use something like a build strategy or that's how I would tend to approach it. But I tend to view factory bought as a sort of a tool I pull out when there is a justified need for it.

19:41But my ideal would be just calling user.new in the test every time. Yeah, so it's a tool you use, but it's not like when you keep it hand, not your favorite knife. It ends up being a little bit that, just because in a Rails app, the most frequent thing that you're going to have to do is act on records that are saved in the database. And so that is something that I keep going back to. But if I can get away without it, I would absolutely use just a direct call to the constructor in my test. I feel like a third person in this conversation might go, but what about fixtures? And so I'm going to pretend to be that third person to go, but what about fixtures? So fixtures are a little bit of a different philosophy for testing. But they can feel the same role as a factory. Yes, yes. They're sort of the alternative to a factory. I should clarify that a lot of people associate fixtures with mini-test because that's the default Rails stack and a sort of like alternate stack that a lot of people like to use is

20:44RSpec plus factories. But those are all those pieces that are all interchangeable. You can do RSpec with fixtures, you can do mini-test with factories. There's not necessarily a tie-in with those two. Fixtures are just out of the box. They're faster. They are faster than factories and will almost always win any benchmark you go for. So if you were trying to maximize the speed of your test suite, fixtures are the way to go. But I feel like they're less good when you're working on the test in the moment for me because now you've got two different files you have to deal with or you're looking in different places and you no longer have everything that you need to know about your test defined in the test. What's interesting there is if you're talking about the even mystery guest data that will impact the outcome of your test, but that doesn't exist anywhere in the test file. It's a little bit of that like spooky action at a distance. Where fixtures are also interesting is that the reason that they're fast is that they

21:45basically pre-write all of your data to the database. So all the data you need for all of your tests gets pre-written once before the test suite runs. And so you only make one trip to the database or maybe one per table or whatever it ends up being. So that's really, really fast compared to making five queries per test or something like that across 10,000 tests. But what it does mean is that your tests are now no longer running with the minimal data set that you need in order to make that test pass. So your test isn't just saying there's an admin user and there's a normal user and they interact in this way. There's 20 different regular users and three or four different admins and maybe someone with some other intermediate role all sort of present in that data set. And that may or may not impact the result of what your test does. So to be clear, I could probably count on one hand the number of times I've used fixtures. They're not really a part of my workflow at all.

22:47But the things that pop into my head when I think about working with them are things like say you're on a big team, right? And you're using fixtures. And as I'm writing a test, I'm realizing like I actually need this user to be a little bit different. So my options are change the user during the test as part of like part of test setup being like user.update name is actually JoL and not Sally. Change the fixture or add a different record to the fixture and use that one instead. My first gut would be like, well, this isn't a huge change. And I don't want to add a whole second record just for this. And it's basically the same as the existing one anyway. So I'll just update this attribute to this fixture and it'll be fine. And then I break all these other tests because none of them, you know, they were all written within one expectation for fixtures. So then I add my own second object in there. And now we've got 14 different people have all added variations of an admin user that

23:49could actually all just be the same record. Right. Right. Your fixture files need a certain amount of curation. Which is certainly is also true of factories, but factories at the very least you can have for your base factories are like pretty hard and fast rule. That idea of like the only things that get set in your factory file are on your base factory are anything that's required in order to pass a validation. And so that's like a hard and fast rule that you can sort of apply. I'm thinking lean doesn't require a lot of different judgment from different people on your team. As opposed to what doesn't, doesn't count as a fixture requires a little bit more judgment. And at what point do you need to add something new as opposed to modifying something there? Yeah. Yeah. I think what happens with fixtures is they're a little bit of a shared story that your team tells. So you know that your sort of base layer has two admins and you know, maybe you name them admin one, madman two, maybe they are, you know, Joel and Sally.

24:52Maybe you even in this sort of shared story that you have, there's a bit of a persona to each of them and you're like, oh, well, Joel is a really strict one who like bands everybody and Sally is the one who is more open to things or, you know, there's things like that that become a little bit part of the shared lore and can't act as a shorthand in your tests if your team all sort of is in on this shared story. Yeah. Because you read it and you're like, oh, yeah, well, we're right, right, this test and somebody gets banned by Joel and then, you know, there's an appeals process and here's how it works. And then you can just sort of add a glance immediately, like download a lot of information. Yeah. But it's higher sort of global context. You have to know all of that. Yeah. And so now we're sort of moving out of just what is valuable for individuals or about a technical layer and talking a little bit about how do these technical systems interact with your team composition?

25:52Yeah. And different team cultures and makeups will definitely point towards different directions. Like if you've got a large team that changes a lot, it's not going to be reasonable to expect everybody to maintain the context of that story as they're building. And people will be trying to all sort of modify that shared story and diffusing that those updates to the shared story are a little bit longer. So a large team, a team with a lot of churn, a team that is growing rapidly might struggle a little bit more with something like fixtures. Or a team where you know that one up there, one of the places they need to improve is communication around something like this. Yeah. A small-ish stable team with good communication patterns will probably get a lot of value out of fixtures because that shared story does have advantages. A way that you can maybe sort of short-circuit some of that onboarding cost or churn cost is by using your test fixtures as your development data.

26:56And so you just sort of reset your development environment every day or every other day just to the latest, whatever is in that fixture file. And so now you get to sort of explore that shared story as part of your clicking around in the browser day to day. And so it helps the team stay up to date with that shared story. So in that situation, your fixtures are playing the role of what could otherwise otherwise be your factories. But you can also use them as an alternative to your like seeds.rb file where you're seeding the database with 70 million individual database interactions for your development environment. Yeah. So these are specifically development seeds. Yeah, development. Yes. Yeah. I think people do is they'll put a lot of like dev stuff in the seeds file and that's really meant for data that's required across all environments. So. Right. So yeah, I really mean more like whatever you're using for your often we will use something like a seeds file for like a development setup. And you can use the same things for your tests and your development setup instead of having

28:00to maintain two different sets there. Exactly. Here's my spicy take. Not every test that you write while you're developing needs to stay in the test suite. Hmm. Okay. So you use tests, especially in a TDD flow or even in a sort of test after kind of flow. I think both. But yeah, I think it's more often in a TDD flow. Sometimes I will do this when I'm testing a bug fix and I want to test a couple variants of it repeatedly to make sure that I'm hitting edge cases that don't necessarily need to always be tested in the test suite, especially if we're talking about a system test or something. Right. So there's this one very strange edge case where a person does this series of actions and the job script is something unexpected and it's a bug. And I want to make sure I can reproduce that and fix it. But and so I'm going to write a system test that does exactly that series of things as part of my work. But at the end of it, really what I end up changing is like, I don't know, like a small

29:03method on a model. Sometimes I find that at the end of the day, really what you want to the test you want to keep is the unit test for that model and not this complicated and weird user flow system test, which is not truly what the bug is. It was just how we discovered the bug or the context in which it happened. Yeah. I don't know. Does that make sense? Yeah. So every bit of code in your code base is maintenance cost and tests have to run and they add some runtime to your CI. Yeah. If it's not providing enough value to balance its cost, then you might consider cutting it. Yeah. I'm curious to get your thoughts here that have this sort of a little bit of a mental model that like to get your thoughts on about factories and fixtures. The idea is that testing with a fixture approach is a little bit of an integration test mindset. You're creating a sort of complete world with sort of, let's call it realistic data for

30:07a full system. And then you're just running a bunch of tests inside that world as opposed to more classic factory based approach, oftentimes emphasizes creating the minimal set of data that can recreate the conditions under which you were trying to prove something is true or false. And that's a little bit more of a unit testers mindset. Do you think it's fair to characterize the two sort of data tools as almost coming with philosophies that are a little bit different? What is more of an integration testers, philosophy, and the other one is more of a unit testers philosophy? I think it's a valid angle to view those through. And I don't think you're saying this, but I don't think that that's like a hard and fast. If you're using fixtures, you're automatically, you're at the test, you're writing a more like integration tests, or if you're using factories that test, you're writing a more like unit tests. But it feels like a good, um, short hand, yeah, mental model. So, um, you asked me about the testing period, but I haven't heard your thoughts on the

31:12testing pyramid. The sense that I'm getting in this conversation is that you tend to lean heavier on the integration test or the system test than the unit test. Is that accurate? No, actually, I do generally follow the classic testing pyramid. Okay. Yeah. I do tend to work sure from the top down so that like outside in approach, classically, but I will have much more code in, I try to push things typically into models and then unit test those, but I do like to open up by setting the parameters with a system test. Recently, I've been doing a lot more tests that are code that is in a system where I'm building just an API. Mm-hmm. And so in those situations, I tend to do more like request specs. Mm-hmm. And so those, I'm not necessarily spinning up capybara or like needing a browser equivalent, I'm just saying, if I hit, if I send an HP request to this endpoint, does it respond with this? Right. So I've not tended to be in that top part of the pyramid or the very point of it as much

32:14recently as I have in the past. With your outside in approach, do you ever, you know, you start with the system test and as you develop, then you've got these other tests going, do you ever then look back on it and realize that system test isn't actually valuable to keep? I have occasionally deleted system tests, especially if they're for like really sort of esoteric paths or things that I felt like wasn't providing a lot of value. Yeah. I, you want to balance that coverage, right? And that's always sort of the challenge in that at the end of the day, what's important is not like how well tested your units are, it's that the software behaves as a whole correctly. Right. You could have a whole bunch of perfectly behaved units that do not cooperate. But having well tested units sometimes makes it easier to reuse pieces or catches bugs or allows you to test edge cases that are just too expensive to test at the integration

33:16level or where there's just too much combinatorial explosion to test at the integration level. And that's the whole idea of the testing pyramid is you can have that volume of coverage at the unit level. I know that some people in the JavaScript community prefer a testing trophy over testing pyramid. So what does that look like? The idea there, I think, is it's a little bit more like a pinched middle where you have a decent amount of unit tests and then a decent amount of end-to-end tests and maybe not so many mid-level like integrating a subsystem test. That makes sense because in theory what you're doing in those mid-level tests would either be covered by the end-to-end tests or not really be adding value beyond what you've got on the unit tests in those cases. I think it might be a little bit also because some of those middle tests are really annoying to write. That is true. In terms of, yeah. And so like if you've got just like utility functions you can write unit tests for them

34:19relatively easily. And it's a little bit more set up but you can have a test suite that like pops open a browser and clicks around and does interactions and that has a lot of value. But testing at the maybe component layer and react can get annoying because you're having to like stuff out all these things. Yeah. I feel like I'm having flashbacks to trying to write tests of the component layer and react and experiencing exactly this. I do want to take a couple of minutes just to talk about the idea of parallelization because that's a classic way of increasing your test speed. Have you done any work of parallelizing a test suite? I haven't. I have worked in test weights that someone else has parallelized and I've enjoyed the benefits of that having happened but I have not actually done that myself yet. So I'm curious to hear how you approach that. So if you're using mini test on a modern version of Rails, you get parallelization out of the box. You might need to fiddle with a couple of configs, I forget exactly how that works, but

35:23it is built with parallelization in mind. Our spec doesn't quite work that way out of the box. Typically you have to pull in a third party gem, the parallel test gem, and then it will, there's a few different strategies it can use to parallelize your test suite. What's interesting is that parallelization is a more complicated problem than initially seems in part because you need to figure out how to chunk your test suite. And lines of code are not a great metric for what is going to be fast and what's going to be slow. And so what you might end up with is a parallelized test suite where one sort of parallel process is taking 10 minutes and your other five, each ran for one minute each and then just sit around waiting for that first one to finish. So sure, you shaved a few minutes off by passing some data or some work to those extra

36:25parallel tasks. But if most of the work is sitting on that one sort of unbalanced process, you're not getting the full value. Yeah. So are there tools that you can use to help you figure out how to chunk that up? So parallel test by default, it comes with a few strategies out of the box. One of the things it can do is use a timing file. So you run your test suite and it will record for each spec file how long it took to run in a sort of metadata file somewhere that it saves. And then the next run can read that and do the math and say, well, on balance, a file is probably going to run it about the same amount of time as it did last time. And maybe you made some modifications so that file is going to be a little bit off. But on average, that ends up being a pretty good estimation. So we can use that to split up the work. And that works decently well. If you've made modifications, that can be a problem. And if you have what I've sometimes heard referred to as a tentpole files, so specs that

37:30are really, really slow, those are typically your integration tests, that might be a problem as well, especially if it's a real outlier. Because ideally, what you want to do is take your total test suite time, divide it by the number of workers you have, and each worker takes one end of your test suite time. You've got five workers you want each to take about one fifth of the time. So let's say you've got a 10 minute test suite, every worker should process about two minutes worth of work. Well, what happens if one of your test files takes five minutes all by itself? Yeah, you're not going to get down to two minutes. Yeah. Question here, as someone who hasn't done this, is a test file, the smallest unit at which you can split up this work into parallel streams. So that's where the interesting question is, is the chunking things up. The easiest thing to do is to chunk by file. Once you start chunking smaller than that, I think you start having to like play less into just giving file paths to processes and having to like read and interpret those files to get like Ruby objects from our spec and then pass the examples around so it can

38:33get more complex. So in general, you're probably going to be working at the file level. You're probably working at the file level, although there are some things you can do to work at the, let's say the example level. We've also been talking about sort of pre splitting, using a timing file, something like that, you can also put all of your tests or all the files in a queue, sort them roughly by the amount of time. Maybe you have a timing file still, sort them all by roughly slowest to shortest, put them all in a queue, and then every worker just pulls from the queue. Yeah, that makes sense. And so that way, nobody's waiting doing nothing while somebody else has a big backlog work to do. You only get to take one file at a time, you process that, and if it takes you longer than expected, one great. The other workers will pick up the other things, and so that ends up being more of a self-balancing process. So there are tools you can run to sort of have your own queue system. There are third party services that will do that for you. Like Napsack Pro is the sort of more famous one, it just manages a queue for you where

39:38it'll like, and a timing file and all that, so you just tell it, hey, I've got so many workers, so many test files, and it'll assign files to your workers. That can be nice too, because it's easy to just scale up and down. When you are parallelizing to test, do you tend to sort of do the same approach for when you're running test locally, and when you're running them on CI, or are there variations and sort of what you think about for setting up parallelization for those two contexts? I tend to try to do something similar when I run a local versus CI, at least I'm running a full suite, from running a single example or just one file that I don't know. You can't parallelize one thing, and it's just not worth the overhead, right? Yeah. Another thing, part of what is expensive with your test suite is overhead, and so your Rails app takes a little while to boot, it might take five seconds, 10 seconds, depending on the app, if it's a very large one, I've seen a minute plus to boot a Rails app.

40:40Every worker that you have is going to have to pay that cost, and so if you say, well, I've got 10 minutes worth of tests, plus one minute of boot time, that's 11 minutes of total work, split that across five workers, I'm not going to do 11 divided by five. What I'm actually going to do is say they all pay that one minute overhead, and now each of them gets to have one fifth of that 10 minute test time. So total runtime is going to be three minutes per worker, one minute of boot time, plus two minutes of test time, for total of 15 minutes of total execution time summed it up across all threads, which means that you're going to start seeing some pretty heavy diminishing returns for your test suite as you start throwing more workers at this. And especially if you're saying, well, I'm not just going from two to three workers, but I'm like, I'm going to go to like 16 or 32, once your test suite time starts going

41:46like well below the Rails boot time, then you're sort of like every worker is like adding just a tiny little bit of increase to your test time while paying the full boot time. So again, with the example of the one minute boot time, now you've got something where adding a new worker is like shortening your actual test run, maybe by 10 seconds, but you're paying 60 seconds worth of boot time, and so adding another worker doesn't really move the needle much. And so that might be a time where you say, I'm not throwing more processing power at it. I want to look at reducing my Rails boot time. So you might do some more lazy loading. There's a variety of techniques that you can use to make that better. Yeah, that's a whole other episode. But there's an interesting math where you've got to do there, where the answer to a slow test suite can't just be throw parallelization at it. Yeah, like many of our problems, there's not a one-size-fits-all solution here. One thing that will help you with your parallelized test suite as well, make the total test time shorter, right? That's another lever you get to move.

42:50Maybe by making fewer total database calls, whether it's by improving your factories, switching to fixtures, using a combo fixtures factory as a approach, because there are a few strategies for doing that that we didn't have time to cover. But yeah, I would say in general, the levers you have to play with are parallelization, total test suite time, and overhead. So any sort of extra startup costs you have to pay. Yeah, makes sense. So you pull on those three levers and the balance that fits with trade-offs of your team and your app, and that puts you well on the way to improving your test suite times. I feel ready. On that note, shall we wrap up? Yeah, must do it.