Random Attention: How AI Gets Faster by Forgetting

0:00You know how you sometimes walk into a room with this totally clear purpose? And then the second you cross the doorway, you just go blank. Like you completely forget why you're there. Oh yeah, happens to me all the time. Right. But then somehow you can perfectly recall some random trivia fact, like how octopuses have three hearts. Well, it turns out artificial intelligence actually suffers from a really similar memory prioritization problem. Welcome to Intellectually Curious. Today, we are doing a deep dive into this fascinating paper from Salesforce AI research called Random Attention. Yeah, it's basically rethinking KV cache eviction for efficient reasoning, which I know sounds super technical, but the core idea is just wild. It really is. Our mission for you today is to unpack this breakthrough because it makes AI so much faster just by fundamentally changing how it, well, how it forgets. So to start us off, how does this memory problem actually show up in AI? Well, it manifests as a huge hardware bottleneck. So when AI modeled as complex reasoning, which we call chain of thought,

1:01it generates thousands of these intermediate tokens. And tokens are basically the tiny building blocks of words, right? Like the AI's thought. Exactly. And generating all those tokens so quickly just completely exhaust the GPU's memory, specifically this one component called the key value cache for the KV cache. Okay, so the AI is basically running out of short-term memory just because it's overthinking. How are researchers trying to fix this before? Mostly with really complex math. Uh-huh. Older methods would try to fix it by computationally scoring every single token. They'd score them, like grading them. Yeah, the system would run these heavy mathematical evaluations to decide which tokens were the absolute most critical to save and which ones it could just throw away as the cache filled up. Hold on. So the AI's memory management was basically eating up its own memory. Like it was wasting all this precious processing power, just trying to calculate what it was allowed to forget. That is exactly what was happening. The scoring pass itself became this massive computational burden. Man, that sounds incredibly inefficient.

2:03Actually, real quick, speaking of making AI more efficient, this show is sponsored by Embersulk. If you need help with AI training, automation, or software development, they are amazing. Oh, absolutely. Especially if you're trying to uncover where AI agents could make the biggest impact for your business, or even just your personal life. Exactly. You can check out Embersulk.com for all your AI needs. But, okay, back to the memory problem. Because scoring the tokens was slowing everything down, what did the researchers do instead? They tried this completely radical signal-free approach called random attention. And it literally just relies on two super simple rules. Okay, I love simple. What are the rules? Rule one, protect the original prompt at all costs. And rule two, evict the generated reasoning trace uniformly at random within each of the model's attention heads. Wait, attention heads, what are those again? Oh, right. They're basically the parallel processing pathways that the AI uses to analyze text. Okay, but you're telling me they just hit delete at random? I mean, if I randomly rig pages out of my own scratch notes while studying for a final,

3:06I would definitely fail. How does that not completely break the AI's logic? I know, it sounds like a total recipe for disaster. But it actually works beautifully because of redundancy. The researchers proved this using something called a planted-fact probe. A planted-fact probe, that sounds like a spy gadget. It basically is. They hide a specific fact in a giant sea of data, kind of like a needle in a haystack, just to observe exactly how the AI's working memory handles it. And they found that AI reasoning is just highly redundant by nature. Ah, so it just repeats itself a lot. Constantly. It restates facts in its text all the time. Plus, those multiple parallel attention heads we talked about, they all store duplicate copies of that same working trace. Oh, wow. So it's less like ripping pages out of a textbook and more like taking a group exam. Like, imagine 10 of your friends are all furiously scribbling the exact same scratch notes for the test. That is a perfect analogy. Right, you can randomly throw away like 90% of those notes, and chances are at least one of your friends still has the exact formula you need written down.

4:08Precisely. And the numbers behind this are actually staggering. A fact saved in just one attention head is retrieved only 3% of the time. 3% that's nothing. I know. But if that same fact is randomly saved across multiple heads, retrieval jumps to 99%. Since the original prompt is safely protected, that random deletion leaves enough scattered copies across the different heads for the AI to function perfectly. That is brilliant. So by skipping all that complex math to score the tokens, how much faster does this actually make the AI? The payoff is massive. Random attention matches the accuracy of the best scoring methods. But by abandoning the complex math entirely, it actually delivers 32 to 43% higher throughput when deployed in real-world systems. That is a huge leap in speed. Are there any downsides at all? The only real limitation is true needle in a haystack tasks. Like if a fact is stated exactly once and never repeated anywhere else in the working memory,

5:09it might get lost. That makes perfect sense. But honestly, it's just so fascinating that sometimes making an AI smarter doesn't mean giving it more complex math. Sometimes you just have to give it permission to be messy. Absolutely. It's a great reminder that highly engineered systems can really benefit most from simplified, elegant approaches. It really proves that our technological future is incredibly bright. Simple, elegant solutions like this are making AI so much faster and more accessible for solving humanity's greatest challenges. I completely agree. The progress we're making right now is just so inspiring. It really is. Well, if you enjoyed this deep dive on Intellectually Curious, please subscribe to the show. And hey, leave us a five-star review if you can. It really does help get the word out. Thanks for tuning in. Yeah, thanks so much for listening. I want to leave you with one final thought today. If a highly advanced AI actually functions better by freely forgetting its own intermediate thoughts, what does that say about how much useless information you and I are obsessing over trying to

6:09retain every single day? Maybe it's time to just remember the octopus and let the rest go.