Andrej Karpathy’s AutoResearch

So I actually spent like three weeks trying to perfectly tweak my Wi-Fi router settings manually.

Oh wow, three weeks. Yeah, literally three weeks. I was moving antennas by millimeters,

changing channels, running speed tests. Well, let me guess, failing miserably.

Failing completely miserably. And I was just sitting there, you know, wishing I could just run

100 automated experiments while I was asleep. I mean, we've all wanted to just hand off the tedious

stuff to a machine. Exactly. But what if you could do exactly that, but instead of fixing a router,

you were building advanced artificial intelligence. That is, that's a massive leap.

It really is. Today's deep dive explores this amazing project released by Andrej Carpathy,

called Auto Research. The mission here is to engineer AI agents that make the fastest

research progress possible on their own indefinitely. Without any human involvement.

Right. Yeah, it's a completely novel approach to automated model tuning. We're essentially

looking at an autonomous AI agent swarm designed specifically to train large language models.

And looking at their repository, it's, it's surprisingly stripped down like there are really

just a couple of main files driving the whole thing. That's the elegant part of it, honestly,

the simplicity. As the human in the loop, your only real job is to iterate on the instructions

in a single text file. The program.md file. Exactly. That text file contains your high level

directives. The AI agent reads that and its job is to continually rewrite the actual neural

network code, which lives in the train.py file. You got it. So you're basically programming the

organization's overarching goals and the agent acts as the researcher doing all the heavy lifting.

Precisely. The agent comes up with an idea, maybe tweaking the neural network architecture

or adjusting hyperparameters. It edits the Python script, commits the change, and just kicks off

a training experiment. And I noticed there's a very strict constraint on those experiments.

Every single training run lasts exactly five minutes. Right. But playing devil's advocate here

isn't a five minute limit way too short to evaluate truly complex architectural changes.

That is the core challenge of automated tuning for sure. But the strict five minute budget is

completely deliberate. It guarantees a perfectly fair comparison of radically different ideas.

To make sure those quick reads are actually accurate. Right. And to evaluate the results,

the agent checks a metric called Val underscore BPB. Validation bits per byte. Yes. Why use bits

per bytes specifically rather than say standard accuracy metrics. Because it removes dependencies

on variables like vocabulary size, it gives you this pure standardized measure of how well the

model is compressing and predicting the data. I see. So even in just five minutes, if that val B

B scores lower, the model definitively improved. Exactly. And the agent saves that code change

as a git commit. Yeah. The score goes up. It just discards it rolls back the script and tries

a new hypothesis. It optimizes perfectly for the exact hardware it's running on.

Completing about 12 structured experiments every single hour while you sleep. It's essentially

building a custom autonomous feedback loop for the code itself. Figuring out how to automate those

tedious bottlenecks is such a massive advantage. Huge advantage. And speaking of automating

progress, this deep dive is actually sponsored by Embersilk. Need help with AI training,

automation, software development, or uncovering where agents could make the most impact for your

business or personal life. They are definitely the ones to look into. Check out Embersilk.com

for your AI needs. But getting back to the big picture of auto research, it really changes the

paradigm of how we work. We are moving past the era of humans acting as slow meat computers.

Meat computers. I love that. Yeah. Trying to manually run research between eating and sleeping.

By structuring the program file to guide the train file, the AI accumulates these small wins,

getting incrementally smarter five minutes at a time. And every dot on the project's progress chart

is another successful accumulation of knowledge. That's an incredibly optimistic way to look at it.

It represents a massive leap forward for humanity. We are looking at the promise of exponential

autonomous progress. We are literally building the tools that will rapidly generate solutions

to some of our most complex problems. It's an unconstrained engine of discovery. Absolutely.

So what does this all mean for you listening right now? I want to leave you with a final

provocative thought. Think about the broader applications. Exactly. If an AI can autonomously

evolve its own underlying code in five minute loops, imagine applying this exact same autonomous

research method to other fields you're curious about. Like a swarm of agents iterating on clean

energy design or discovering new medicines five minutes at a time. The future is looking

incredibly bright. If you enjoyed this deep dive, please subscribe to the show.

Hey, leave us a five star review if you can. It really does help get the word out. Thanks for tuning in.