Caching is a cop-out

0:00Caching is the process of storing the results of a complex operation that takes a finite amount of time that we perceive as a human as a long time to compute. It computes a lot of resources, CPU, RAM, and time, and storing the results in memory in another data store to skip that long running operation. It is a very, very, very bad practice and we started to see it in the last decade or so. I want to argue that this is a very, very, very, very bad practice and we started to see it in the last decade or so and always backfire. It goes backfire because it gives you a cup out to not look at the operation that you're doing and to avoid optimizing your work.

1:14So we hide behind caching. That's why in memory caches became so popular. Oh, what do you mean this SQL query is so slow and I can just cache it? Wow, that sounds amazing. So instead of taking time to understand why something is slow, we took the shortcut of executing it once or maybe per process and then cache the results somewhere. So someone else can consume the results inevitably with caching comes the stale results comes inconsistent results comes out of date management of the cache. How do I know that this cache is the latest? So you created new sets of problems by caching.

2:16Now I'm not saying caching is not good in all times. There are cases that are absolutely necessary to cache. I'm thinking of the CPU without the L caches, the 0, 1, 2, 3 caches. Our lives will be very, very slow. Our operations will be very, very slow. Although I could argue also that in itself is a cup out because the way we write programs is just we got used to just accessing the memory because it's cheap because we got saved by the CPU. And how the CPU caches the results to avoid unnecessary hits back to main memory. So the fact that we have the L0 and L1 and L2 and L3 caches is precisely to save us from this expensive operation.

3:18But the problem is we got used to it. We got used to doing all this operation and we almost never know that caches are being used. So back to this RAM example, we're limited here by the RAM, by the bus speed, by how fast we could perform a memory access to this DRAM by how much bandwidth the DRAM supports. So many factors here. So we're still limited. So in this particular case it's okay to cache. But even that I could argue that sometimes we over execute operations that are absolutely necessary. Unnecessary. So we keep hitting the same memory over and over again. So yeah, we no longer think about how memory is accessed as developers. We just assume that it's going to be fast and the CPU is saving us.

4:18I would argue that too is also an abstraction that is leaking in modern software. Most modern software. Currently the way they are developed, we throw functions all over the place. We create all sorts of calls between functions. And we never think how these functions are executed. We just assume that it's just a sure it's going to be a stack. And this is going to be the access from the text area in my process and it's going to be just going to pull that code. But imagine millions and millions and millions of code which could translate it to basically text code binary for that particular CPU. And as you're fetching pages, the poor CPU needs to cache them. But if you're not efficient in these calls, you're calling functions across this space of your process. It will call a function in page zero, it will pull it and it caches it in the TLB.

5:22And then the next function, it happened to fill in the same page that you were lucky. We just hit the same text area where the code is. And we just succeeded. But that is almost rarely happened. You know, that rarely happens. So we end up pulling so many pages. So we're end up hitting the memory so many times and we feel that bottleneck as a result. Because basically that's this because we lost that touch. We don't know how things work. We just assume it's fast. And that's even dangerous than an operation that is slow and let's go ahead and just cache it without understanding it. I'm giving you the best scenario in the CPU here. So of course, this is not the topic. There's something called profile guided optimization.

6:22Actually, there is a great video by Dave from he used to be a Microsoft developer. He's retired now. But he went through how they used to optimize windows. Was it 95 or 98? I can't remember. That the executable from windows at night, they take the executable and run it and see how the functions are called and then put the functions that are close to each other in the same four kilobyte pages. And that's because that's how CPU execute pages. Right. Thank you, Train. It's okay. You don't have to honk. So sometimes the train is just like the conductor like to honk for no reason. I don't know why this just feels like it. There's nothing here in this area. But anyway, so by optimizing that every memory access gets you one beautiful page and every subsequent calls are almost always in this area. So you see how precious these calls get. So back to our life, how we do things is when we execute a rest call or a GraphQL or any endpoint or a SQL statement.

7:37We just assume that you know what I'm not going to do that again. That's slow. That's just cash. We don't spend some time to think about how can I absolutely optimize that call so that I don't need it at all. I don't need a cash because eliminating cash is always the best case scenario. For you cannot sometimes you cannot do it like the case of the CPU you absolutely need. Elcaches because whatever is above you have no clue what they're doing. Okay, I can give you an example of something I recently worked on in my company. We used to have this expensive operation that happens frequently. And it was legacy code. It was code that is old and everybody just decided to cash it. And that cash unfortunately is executed pair an instance of the thing that I cannot I know this is vague but think of like your pair user.

8:47Every user has its own version of this cash and every process that you run into is going to see it doesn't know anything. So it's need to cash that information again. There is no centralized cash. So if every user that logs in it sends a request and if it lands on a process that doesn't have that cash it will start creating. It will execute that query. It will start initializing the user profile and start caching it in the process. But if another request hits another process completely different because it's a stateless system that it needs to do the same operation again because that cash didn't live in that process. So there was a whole project that all right how about we centralized the cash. Right. So let's remove the per process cash so that we don't execute the operation all over again. Let's just put it in centralized and then right is given to the picture of him cash. He was an example another cash example and then someone mentioned this is can't we just optimize that operation to not cash at all.

9:55How why are we doing that in the first place. So why don't you start stimulating and talking about this start breaking out the code is like wait a minute we could actually make this operation run in five minutes seconds. So the team spent months working it out the whole cash is removed. Removing the cash made everything better now the operation you don't need a cash at all. So because the operation itself was optimized to run much much optimally when you'll be tempted to cash especially if you don't understand the code like in this case. Right. Like the code has been there for a long time and we don't. People are afraid to touch it because of my break some other clients from you know from 20 years ago that is still using this and so it's very very delicate especially in the enterprise systems things are very delicate if you have like tons of clients that use it.

10:56I mean cloud by software clients right Python code and all sorts of you know JavaScript and other code. So you'll be tempted to do that but but allow yourself to be just maybe I can either optimize the operation or better not even call it at all about that that's even better never call the operation to begin with. And you'll be surprised at the beauty you could achieve give yourself some time to ask yourself the operation that I'm about to cash here is it can I eliminate it all together that's the best thing. Or can I optimize it to make a run so fast that I don't even need to cash now that comes with other problems as well because this the excessive calls to the database. Might not be as bad as cash so you need to make a trade off everything is a trade off at the end of the day so I just remember another example of this cases Amazon can use this got an outage during the pandemic.

12:09I don't remember what that does like the can use this thing I think is there there are events manager or queue system like forgot what it is but they got an outage because. They got a lot of load lot of requests because of the pandemic that kind of overload the system so they decided. To shut it down and restart it but guess what restart starting a single innocent of can use start started making. So there's tons of requests because they relied on this eager loading where they cash everything so the instance have everything cash but the startup became so slow because all of these requests are has to be executed in the beginning. But once you have it everything is cash so the start up became so slow because all of these operations that are expensive they are just they cashed all of that at once but you cannot escape it at the end of the day at the start up these operations have to be executed so they started up and then at the end of the day there's a timeout and that time I once it's reached the instance was killed I think was like whatever five minutes so that.

13:24Once you execute all this and they're killed and then the subsequent request spins up more instances of of these instance of can use us and then that also start hammering the database with these expensive queries they were that used to be cashed right that will they they are cashed now all of this. So that's how the data base was put so much load on the database on the backend that the database itself got you know the resources got maxed out and they stopped responding and that is completely the system completely melted down because I would argue that this might be the reason of excessive cashing without looking at these operations do we need to cash all of this stuff. Can we eliminate some of these are cost you can find answers that the answer is almost yes there are a lot of things you can eliminate can we optimize these queries that's the third question that you can do so these are kind of the summaries do we need this can I remove it all together can I optimize it so it runs faster or can I delay it perhaps if I can't do anything about it.

14:43And then if I can't do any of this then of course the answer is I need to cash it because I need the answer for this I need a quick answer but always give yourself this that it can I optimize this before I cash it so don't use cashing as a cup out try to give yourself some time to optimize these operations. And I'm going to see you the next one if you're interested check out my training courses in the description below become a better engineer look behind the curtain and then stand the internals and believe me once you understand these is like an understanding math then anything on top of it you can derive probably any software out there how it works you can understand any software even if you don't have the software I promise you that. See you in the next one.