Never mind (OpenAI won again) | Theo - t3․gg

0:00Are your ad campaigns lighting up the dashboard? But not the pipeline. That's bullspend. And marketers are calling it out in... dashboard confessions. My boss asked for results, so I opened my dashboard for the only positive sounding metric I had. Impressions. Cut the bullspend. See revenue? Not just reach. LinkedIn delivers the highest return on ad spend of major ad networks. Advertise on LinkedIn. Spend $250 on your first campaign and get a $25 credit at go to LinkedIn.com slash campaign turns the conditions apply. Call as footballers back. So Hilton called to me the superstition concierge to make your fan rituals a reality. Need a room to match your lucky number? We got you. Want to make sure our team doesn't wash your lucky jersey? Oho, that smells lucky. Hilton's unmatched hospitality can keep up with any superstition. Even a marching bandwreck up call it 555 and 55 seconds. Hit it! When you need a team that would do whatever it takes on game day, it matters where you stay. Hilton, for this day.

1:00This episode is brought to you by PayPal. Holiday Rush? Skip the mall traffic and shop online with PayPal. Shop your favorite online stores like eBay, Fanatics, Walmart, Sephora, Target and TikTok shop, and choose to pay now or pay later at checkout. Plus pay in four with no interest and no late fees. Shop now with PayPal. Pay in four subject to approval. Learn more at PayPal.com slash pay in four. PayPal Incorporated NMLS 910457. Sonnet 5 is finally hit. Okay, no, not Sonnet 5 still. I know we're all going kind of mad. It's a new release from OpenAI. You might remember just a few days ago I covered a release from OpenAI that I was lucky enough to have early access to. The new Codex app and I do actually love it. I've been using this to build a ton of shit across multiple different computers. And that I was going to use Linux more and actually missing this Apple Ox. It is still mech only. Why am I talking about this app though? Well, as I mentioned in the video where I covered it in depth, I had early access. And I had to hide that from you guys, which is why I got the second laptop. But that wasn't the only thing I was hiding.

2:02I have to come clean. I had early access to 5.3 and I've been using it for almost everything I have built for the last two to three weeks. And in my opinion, it's pretty damn good. I don't think it's this massive leap forward that I'm seeing a lot of others claim it to be. But I do think it's great. Opinions seemed varied from primogen saying it'd be really funny if 5.3 was literally just 5.2 and it's an experiment to see how much we care. To Matt Schumer saying that it's an f-ing monster. It can go for eight plus hours and he comes back to working code and live deployments. It's significantly more autonomous than 4.5, but it's not all positive. I'm leaning this direction, but I have a lot of thoughts. As I mentioned before, I've been using this model for a ton of different shit over the last few weeks from configuring new systems, suspending upside projects, to trying to migrate 8 plus year old code bases to modern standards. I have a lot to say. But just to make sure things are explicitly clear, opening up hasn't given me any specific restrictions on what I talk about, how I feel, or how to present things. This is all my organic opinion. The only thing that I'm paid to talk about is today's sponsor.

3:04Once the last time you saw a project that made you emotional, I know when it was for me. It was general translation. I know this sounds insane, especially because they're a sponsor, but I've been through it with translations. Having a system to handle all the different languages that exist around the world at Twitch was really, really hard. One of the hardest things I've ever been part of, and it broke all of the time. General translation made it go from a nightmare to a couple lines of code. Need to have your title translated? Wrap it with their tea helper and you're good to go. Even if you're composing and you have variables that you don't want translated, all of this is handled with their helper components. And from here, the translations are easy to swap in. You don't have to do any real work. They handle it all for you with their agents running on your project. Do you ever feel like you need more context? It's just a prop on the tea component. Yes, it's that easy to add context. You can do complex nesting without any issues at all, which is so nice. These things, I was so deep in the hell of wrapping every single string with a helper function before. This feels magical in retrospect. It's crazy. Need to format a currency properly or a date or a number or variable or any of these

4:07other things. It's all just handled for you. And your build script can go and generate the translations as soon as you're done. I'm not bluffing about the emotions, by the way. The entire phone call I did with these guys, my jaw was like dropping to my desk. And I went and posted this after because I was just so blown away. Seriously, how simple is that? If you want locale direction, you call their hook for it, and now you have a direction you can apply to the HTML. Just like left to right is one of those things that's so annoying and suddenly it's way easier. And that's what my experience with general translation, it lets you unlock the other 90% of the world that doesn't speak English without having to lift a goddamn finger. There's a reason everyone from cursor to meantlify to click house is using them. Check it out now at soydev.link slash GT. Normally the flow for this video would look something like, we read the article, we react to Twitter, I show you what it looks like in a handful of small projects and we move on. But I'm not going to do this video that way because my own experience with this model was nothing like that at all. Opened yet I gave me access before they gave me any real information. The only thing they told me when I got this access was that it was a better conversational

5:07partner when you were working and it would give you a bit more detail about what it was doing when it did it. I was given no additional information about its capabilities, what would be better or worse at beyond that. So I decided to just try it and try it I certainly did for many hours across many projects many times. One of the first things you'll notice is exactly what they said. It is much better at telling you what it's doing and what's going on. If we look at this thread here where I showed it and error I was getting running the dev command is started looking at the paths that involved these particular keywords in the screenshot found them said exactly that I found the exact failure path index.ts calls this thing that it shouldn't and it's going to the wrong format. This is a mistake funny enough that another model that just recently came out made that we will be doing a more thorough comparison within the near future. But the fact that it gives these calls it'll go do one or two tool calls. It did a command to find things and then it explored the things it found. Then it told me what it did and what it will do next. When it went and did that thing it said the crash is happening because of this.

6:10Then it went and tried out a theory. It's say what it thought and then went and did a bunch more work. This is what it does. It's good about these back and forth. This is the more I think I expected opus to do and I've historically not had the codex models as good at this. Literally all of my threads are on 5.3. I'm so sorry trying to find an old one on 5.2 that I like used for a video or something but all of my work has been in 5.3. I'll just show you what it looks like to use 5.2 instead by selecting it. I'll grab the same exact audit prompt so we can do the audit codebase comparison. 5.2 codex, paste, send. While that is running I will show you what this looked like when I did it with 5.3. Going to run a full codebase audit with multiple sub agents in parallel. One focus on server security, one on server maintainability, DX and testing and others on examples and packages. I'll start by mapping the repo and then dispatch focused agents with clear scopes. Then it did a list. It says of things. It made another exploration list. It said more things. This is the pattern. It tells you a bit. It tells you a bit. This is what you expected before with 5.2.

7:10We got no text initially at all. It just immediately goes into exploring and goes and touches 8 Hana files. It will then think about it for a bit and then go touch more files. It thinks about it for a bit and then goes and touches more. You don't even know if it's going in the right direction or not until it's done. One of the really cool things about 5.3 is how receptive it is to be... Are your ad campaigns lighting up the dashboard? But not the pipeline. That's bullspin. The marketers are calling it out in... dashboard... Confessions. My boss asked for results, so I opened my dashboard for the only positive sounding metric I had. Impressions. Cut the bullspin. See revenue? Not just reach. LinkedIn delivers the highest return on ad spend of major ad networks. Advertise on LinkedIn. Spend $250 on your first campaign and get a $25 credit. Go to LinkedIn.com slash campaign. TurnstitchingSupply. Call us footballers back. So Hilton called to me. To superstition concierge to make your fan rituals a reality. Need a room to match your lucky number? We got you. Want to make sure our team doesn't wash your lucky jersey?

8:12Oho, that smells lucky. Hilton's unmatched hospitality can keep up with any superstition. Even a marching bandwink up call it $55.55. Hit it! When you need a team that will do whatever it takes on game day, it matters where you stay. Hilton, for this day. This episode is brought to you by PayPal. Holiday Rush? Skip the mall traffic and shop online with PayPal. Shop your favorite online stores like eBay, Fanatics, Walmart, Sephora, Target and TikTok shop. And choose to pay now or pay later at checkout. Plus pay in four with no interest in no late fees. Shop now with PayPal. Pay in four subject to approval. Learn more at PayPal.com slash pay in four. PayPal Incorporated NMLS 910457. Shopping stopped and told it is doing something wrong. They've even implanted this at a UX level. So I have this job running here. It's obviously not a 5.3 job, so this might be a risk to do this now. It's part of why I didn't show this feature off to heavily in the video about Codex app.

9:14Wait, don't forget about the, I don't know, convex example. And then when I send this, it ends the sense, I thought it would cue. That was not supposed to do that that way. I guess my own understanding here was wrong. I'm going to paste this and do that. I'm also going to switch over to 5.3. Paste this again and then see testing. Okay, cool. It does just send it now. Is that a setting that they changed on me? Oh yeah, they changed the default. Q would, Q follow ups when Codex ran. And you'd have to wait till it was done and it would send. Or they had a little send now button you could click, which is the flow I prefer. I'm actually going to switch back to that because I liked it quite a bit. So while this job is running, I can say something else. And you'll see it gets queued here. And I have the little send now if I want to send it now to steer in the right direction. Or if I want to address this after, it will. I really like this flow because if I notice the model going down the wrong path, I can hit the now button and get it to go down the right path. But if I notice something that it's not doing that I might want to have fixed later, I can cue it and know when it finishes the main work it's doing that will be queued to go back to after.

10:16It's a really nice workflow and surprise more things don't have something like this, especially because I can delete the thing that we don't want. But notice how long we've been letting this run for. It's been exploring across 10 files, three searches, nine lists. It's still going and it has not told us a single thing about what it is doing. This was the 5.2 experience versus again, we'll do a fresh one of 5.3. It thinks initially, but in just a moment, it starts telling me information. It's actually telling me what it's doing and why. You'll also notice it is a lot faster. Not just feels faster, it's initially what I thought it was. It is faster too. It's more efficient with its token usage. It seems to actually generate tokens faster and it feels faster because you're getting information sooner. Look at all of this text we've got about what's going on in just a few seconds in. And this previous audit still has not spit out a single line of text. That's a huge difference. It's going to make this model feel much better. And I do think that's a large portion of what OpenAI is thinking about right now, because they've been getting crushed and model feel by Opus. And somebody who deeply knows the codex models are smarter. I still use Opus for a bunch of shit because it's just nicer to play with and do things with.

11:18It feels better. The big part of why was this more back and forth nature. Codex has now almost entirely caught up with that. I still don't like it as much for general computer use, because it tries a little too hard to be thorough, which is great when you're making real changes to real code bases, but it's not as good when I'm trying to find a file on my computer. I think when you combine these changes to how the models converse, with little things they're changing like these default queuing behaviors, the default search behavior, which is finally on. Did you know that up until a few weeks ago, Codex couldn't search by default? What? Are you kidding? Fix now, but took long enough. Changes like that, changes like this, the model's getting faster, potentially even more faster once we have the rebar's deployments out. Fingers crossed that all goes well. The UI that's actually pleasant to use, a CLI that isn't super CPU and memory hungry, like certain other competing ones are. It is kind of funny that a CLI app is using way more resources than a full GUI and electron. Take that terminal people. Seriously though, it's the thing I'm trying to say is, all of these changes are going to make a lot of people feel much better using Codex

12:20than they would have before, where Clod Code felt way better. I would argue that previously, like the Clod Code UX was like an 8 out of 10 in the Codex UX by default was like a 2 or a 3. I think Codex is closer to like a 7 now. Like they're getting neck and neck, and at the same time, Clod Code is actively degrading where its performance has gotten really bad. I'm having crashes in all sorts of weird environments. The default settings that it comes with that it patches to make this shift enter stuff work broke my aliquity can fake for separate times on for separate Linux machines. No issues with Codex. Here we can see it dumping a bunch of potentially useful info, giving ranks and its thoughts on the findings it had about potential security issues in our environment. Awesome. It got all of that done and gave us a write up. Even though the previous version I was running at 5.2 is still exploring. We have not gotten a single word out of this run. And I started it 2x the time ago that I started this one. And it's still going. Massive difference here. If you actually want information from the model,

13:20it's no longer a set and then come back later thing. I feel like I don't have to parallelize to get value out of this model. I still do a bit because I have a lot of things I'm working on. But it's not the same as Codex 5.2 was where you have to wait. It's more you will sometimes wait. And if you take advantage of things like the new planning mode, which is hidden under Shift plus Tab or if you hit the plus key, it's there, the plan mode with 5.3 can get you set up with a really thorough plan that it can then go execute on. And I'm not just saying that because it built some side projects well. I'm saying that because it completed one of my impossible tasks. This is the original Ping.GG code base. My video call app that I built with the first ever version of the T3 stack because I literally invented the stack as I was building this application. So it has an ancient version of Prisma and ancient version of TRPC that's like 3 SDK syntax changes ago. Next JS like 11, react 15 or 16 at best, tailwind 2, not 3, 2, and a bunch of other random ancient shit.

14:20It is not fun to work in. Half the libraries we used heavily have entirely changed the syntax of how they're meant to be used. We also had patch packages for random things like the key that we would store the next off dot JS tokens in, change between beta releases, and I had to patch package the old ones we wouldn't lose all data. There's just a lot going on in this project that makes it really unfun to migrate. Which is why it was one of my favorite tasks to throw at these new smart models knowing they wouldn't be able to do it. And now I just wish I preserved the chat history because they did it. I was able to successfully migrate all of the major packages for this project up and get it working pretty close to one shot. Flowing ad bunched on metrics that look great till the CFO sees them, that's bullspend. And marketers are calling it out in dashboard confessions. I remember telling my boss, it'll be good for the brand when leads were slow. Yeah, it wasn't. Cut the bullspend. LinkedIn lets you target by company, job title, and more. Advertise on LinkedIn.

15:23Spend $250 on your first campaign and get a $250 credit go to LinkedIn.com slash campaign term secretions apply. When you need a team that will do whatever it takes on game day, it matters where you stay. Hilton, for this day.

16:24It did almost all of the changes. It had two errors on two random places that I just screenshot it pasted and then it was fixed. None of the other models have tested this on. I've tested it on everything that's come out recently. Even got vaguely close to this point. There's a lot of little things it did that I hadn't seen models do. First off, it's willingness to patch and make decent patches too. And it did this throughout. This wasn't just like one commit. It was done throughout the work because it had to migrate like the next and react versions. But that would break the TRPC bindings that expected specific functions and react. Rather than deal with this version drift that caused some things to work and others to break, the model suggested the order at which it should make these changes, which things that should migrate first. And then when that broke other things, it didn't try to force migrate them. And then I ended up in a doom spiral like Opus and 5.2. And instead would patch package the old depth that was having problems and then finish the work on the other thing it was changing. And then go back and update that so it could deprecate the patch packages.

17:24When I watched the agent do this PR, it created and then deprecated at least like ten or so of these patches. That's nuts. I've only known a handful of engineers that work that particular way. Even I am relatively hesitant to reach for a patch. Some of the best engineers I know are very quick to do this because the way they think about code is just fundamentally different. They don't see things in the node modules as tightly locked boxes you're not allowed to touch. They just see code as code and they can change it wherever. I've never seen a model work this way without going a bit rogue. It felt strategic and well applied. In doing things I would not have thought to do. I would have thought it was insane to temporarily for a couple commits patch package. A library is important to this code basis. Just so you can upgrade next first. So cool. I actually thought that was awesome. I was nerding out with my friends in the room when that happens. I didn't think you'd be able to do this. This code base actually works with all of these things migrated.

18:26Not only is that the case because updated everything. Moving the database model over to convex instead of the ancient Prisma Planet scale base that I have it running now. So cool. This is like real work that we're probably going to share for the only reason we haven't merged this is because this code base is not our top priority right now. T3 chat and upload thing obviously are much higher priority. And also from being realistic. I got nerds saved by shoe.dev. This is a project I've wanted for a very, very long time. I have begged many a company to build this for me because I want it to be easier to add off to my sites. The Google OAuth without having to go register this stuff in the Google dashboard that's impossible to navigate without having to go to some random dashboard for any of the many good and bad off providers to get everything configured. Set up a bunch of custom SDKs with a bunch of random code so that it's enterprise ready. But I still haven't got my first user. I love my auth sponsors. I love work OS and clerk to death for all of these random one off projects. I don't want to spend time in docs. I just want to copy paste and go back to what I was doing. I built shoe as a solution to do this.

19:27It's a shoe in. It is meant to just shoe off all these off problems. It's meant to be an easy, simple way to do auth. It'll be open source very soon, possibly even by the time this video's live. Super excited to get this out. Almost every single line of code in this project was written by 5.3. I don't know if I would have built this if it was for 5.3 making it both pleasant and thorough. Because there's a lot of random shit you have to touch to build something like this. I also built that on some services I haven't touched in a while, like Railway. This is all hosted on Railway. It's actually been really, really nice. I enjoyed the process of building this so much that I actually got to completion with it. And I'm going to release it soon. This is so cool. There are some things 5.3 did not do for me though. You're looking at one right now. This home page. 5.3 is still not great at design. It isn't. I'll show you what I mean quick. And now my favorite trick you might have seen this in the front end model comparison video. I'm going to have it make a bunch of alternative designs to this home page. I'm telling it to use the front end skill, which remember was built for opus, but does help other models too and go make these things using my usual stack. I'm not having to do it in the same code base because I'm doing some very weird things

20:28with the HTML and that code base right now to be talked about in a future video. The more I think about it, the more I'm realizing part of why I was so happy about this app and others might not have been is because of how much I have been enjoying 5.3. Even just looking at these threads with 5.2 almost feels a bit depressing. Because it just worked for its claims six minutes. It felt a lot longer than that exploring and not giving me any context. And then finally told me all of its findings after again versus 5.3, which kept me in the loop the whole time. It just felt significantly better. Let's see how it did with these new designs rough, pretty mid, really rough, pretty damn rough, horrible. Yeah, I'm not using this model for design. That is definitely one of the things that I am not going to be touching it for. I have more to complain about with this model. Believe me, we will get there. But first I want to go over everything else that other people had to say. We just start with the official OpenAI Combs on it. Also, of course, the benchmarks which I've been avoiding thus far, they are good.

21:30I just don't think the benchmarks matter a whole lot nowadays if I'm being real. There's a new model that unlocks even more of what codecs can do. 5.3. The most capable agentic coding model to date. The model advances both the frontier coding performance of 5.2 codecs and the reasoning and professional knowledge capabilities of 5.2. Together in one model that's also 25% faster. It enables it to take on long running tasks that involve research, tool use, and complex execution. Much like a colleague, you can steer and interact with 5.3 codecs while it's working without losing context. Honestly, it's the only thing they mentioned when we first started talking to them about this new model that it feels more like a colleague or a co-worker and it could be steered. And I've been doing that a lot and I'm actually really enjoying it. Like, it's more fun than ever to watch the model as it works and to poke it and tell it which directions to go and if it's ever going wrong. I've found it's much less pleasant to do that with Quad Code for a bunch of reasons, but that's for a video in the future. Another very interesting detail is that 5.3 codecs is the first model that was instrumental in creating itself. The codecs team used early versions to debug its own training, managed its own deployment. In flowing ad bunched on metrics that look great, till the CFO sees them, that's bullspin'd.

22:34And marketers are calling it out in dashboard confessions. I remember telling my boss, it'll be good for the brand. When leads were slow. Yeah, it wasn't. Cut the bullspin'd. LinkedIn lets you target by company, job title, and more. Advertise on LinkedIn. Spend $200-50 on your first campaign and get a $250 credit go to LinkedIn.com slash campaign term secretions apply. Oh, that smells lucky. Hilton's unmatched hospitality can keep up with any superstition. Even a marching bandwink up call it $55.55. Hit it! When you need a team that will do whatever it takes on game day, it matters where you stay. Hilton, for this day.

23:43And you can get a TikTok shop and choose to pay now or pay later at checkout. Plus pay in four with no interest in no late fees. Shop now with PayPal. Pay in four subject to approval. Learn more at paypal.com slash pay in four. PayPal Incorporated NMLS 910457. In diagnosed test results and evaluations, our team was blown away with how much codex was able to accelerate its own development. With 5.3 codex, codex goes from an agent that can write and review code to an agent that can do nearly any of the things that can do. Do nearly anything devs and professionals can do on a computer. Half agree there at best. I'm going to hold my mouth for the negative until the end. So stick to the end if you want to hear me complain. Frontier agentic capabilities. 5.3 codex is a new industry high on SWE bench pro, in terminal bench. And shows strong performance on OS world in GDP Val. Four benchmarks we used in measure coding, agentic, and real world capabilities. One funny note on that. 35 minutes before this model came out, another one came out you might have heard of Opus 4.6. Which also had the highest score on the agentic coding eVal terminal bench 2.0.

24:44It is now the shortest lived number one spot in the history of the terminal bench benchmark, because they held it for 35 minutes. I just thought that was funny, as we were. 5.3 codex achieved state of the art performance on S2B bench pro, a rigorous eVal of real world software engineering. But the most interesting piece is that it used way fewer tokens. Opening eyes been optimizing for this for a bit now. They always compare how successful the models were, relative to the number of tokens being used. And the fact that we're getting scores that are the equivalent of 5.2 codex on the highest reasoning level, getting nearly 100,000 Alpatokins burned, versus the new model getting a similar score 56% with one fourth as many tokens. 22k versus 100k for the same level of intelligence. 5.3 on extra-hided slightly better. But I hope this helps emphasize how useless the extra high modes are. If you're using these tools and you just leave it on extra high because smarter is better, you're not smarter and you're certainly not better. High seems to be a really good sweet spot, but honestly, I've been using medium a lot too.

25:45Especially for like one off, quick tasks, medium feels really good. And sometimes extra high can perform worse. Because if it thinks too long, it'll gaslight itself and then do something dumb. So I can't really last time I clicked extra high, I just don't use it. I really do use medium and high as my daily drivers every day. They brag a lot about its front end capabilities and web dev stuff. As somebody who is a capable front end dev that has a lot of thoughts and web dev stuff. Still not the best at this. Really just isn't. It's able to handle the browser well. It can do surprising amounts of good work in like 3D games and spaces and stuff similar to what Gemini can do. It doesn't have all the weird quirks and implementation that Gemini does, but it can do that stuff well. It just doesn't know how to make good design still. Which is crazy because GPD 5 was the first model that could do decent web design. And it feels like it hasn't iterated much since. Whereas Opus has went from so far behind to along with the design skill meaningfully far ahead. They also claim it's less likely to make weird crazy sites and more likely to give you normal day-to-day sites. I've built a lot of websites. I've never heard the term day-to-day websites before my life.

26:47This is what it is. Here they are comparing landing pages between 5.2 and 5.3. And I will tell you, they both suck. Apparently 5.3 version would tell you about the yearly discount and 5.2 did not. That's not a meaningful difference. I don't care about that at all. They did throw it through GDP Val. Did seem to do pretty well. But I have issues with this that we will get to. I would say about almost all of the benchmarks that they have here. There's a reason we haven't seen a lot of you publishing benchmarks on these models just yet. One person did though. My channel manager and fellow tech YouTuber Ben. I bullied him a few days ago into taking his beliefs around models and how well they were able to use bash tools to find context for his better context app. Which if you haven't seen, it's actually really cool. BTCA. It lets models get context on dependencies and code bases not by reading the docs or searching the web. But simply by cloning the repo in a sandboxer on your own machine and letting a smaller, dumber, but capable model figure out those things for you.

27:47It's sending off sub agents and other code bases to bring data back to you to use in your own code base. Really cool stuff. I'm proud to talk about him for building it. But he was saying some interesting things about which models did and did not do this type of work well. I give a resume to sit down, take a handful of the things that he has seen these models do right. Take them and put them in a box and make a small e-vow around it. I remember hooking up with the name, but Gret Bench is now a real thing. And it's actually quite cool. And despite my own belief being otherwise, he proved that Claude Haikou was a very efficient model for this. It did a reasonably small number of tool calls returned really fast, didn't burn a bunch of tokens, and it was fairly priced. I thought three flash would do better, but three flash just aggressively calls far too many tools on average. And 5.2 codex was even worse. The slowest models that he has run this on are 5.2 codex, Gemini 3 Flash is crazy, that sounds flash one of the slowest. And the slowest now by far opus 4.6. 4.5 was way faster, but 5.2 to 5.3, almost a 2x increase in speed, almost a 50% reduction in runtime.

28:54And a big part of that is because of the model being faster, but they only said 25% faster. The remainder of that gap is right here. 5.2 codex did 12-ish tool calls on average to find the context, and 5.3 did 5.5. See how those numbers would be a little confusing? The point being it uses way less tool calls, but did not see a meaningful reduction in accuracy. In fact, according to Ben, it ended up being more accurate overall. This is a bench that you can do because it's possible to take this model and ask it to do something within the codex harness. But this is the problem. You cannot use this model over API. And open AI, you know I love you guys, you know I'm here to talk at any point. You've even invited me to your Super Bowl Sunday party. I appreciate it immensely. You need to stop putting these out without APIs. At the very least, you need to give trusted researchers access to an API endpoint that they can use to evaluate the model, to run it against benchmarks, to confirm your own findings and your own benchmarks. Because right now, we kind of just have to take your word on it.

29:54I can do some crazy hacks to force the codex harness to let me use it for an eval somewhere else. Are your ad campaigns lighting up the dashboard? But not the pipeline. That's bullspin. And marketers are calling it out in dashboard confessions. My boss asked for results, so I open my dashboard for the only positive sounding metric I add. Impressions. Cut the bullspend. See revenue? Not just reach. LinkedIn delivers the highest return on ad spend of major ad networks. Advertise on LinkedIn. Spend $250 on your first campaign and get a $25 credit go to LinkedIn.com slash campaign turns the conditions apply. Stay cozy and look cool with up to 60% off cold weather gear at Nordstrom Rack. Save on cashmere, out of wear, boots and more from brands like UG, Vince, Colhahn and Hunter. Join the Nordy Club for free to unlock exclusive discounts. Shop Clarence and New Arrivals first and more. Plus, buy online and pick up at your favorite rack store for free. Great brands, great prices.

30:55That's why you rack. Propel Fitness Water With Gatorade Electrolites, Zero Sugar and Vitamins, propel hydrates better than water to help you get the most out of your workout and get back to your best self. What propels you? Propel with Gatorade Electrolites. But it's not the right way to do this at all. And it kind of sucks if there's this model that we know is really good, that I can't groove is really good outside of the Codex harnesses. It's just as frustrating. And I understand you want to make sure these behaviors in the way the model works is figured out before your biggest customers like Cursor go and adopt it and shove it into all their products and hit a bunch of weird stuff. But I'm going to pull the Opus card because I'm throttic, figured this out. There is no excuse if you're going to put out these models and you're going to do all of this stuff with the press and you're going to take the time to brag about these numbers. You need to give us what we need to validate it. And I don't feel like I can validate it right now. I almost feel like I'm doing a slight disservice to my own audience because I can't validate these claims. The only reason I'm still doing this video despite not being able to run these evals myself is because I've been using the model every day and it's really damn good.

32:00So I know it's good. I just can't trust any numbers until I can run them myself. Thankfully Ben has a thing that works well for this type of test. And as a result we can see these numbers. But if we go back to the official coverage, I will not trust these numbers until I can run them myself, especially if they don't include other models in these charts. I have a lot more complaints and we'll get to them, but I do want to talk more about using codecs to train codecs because this part is really, really cool. One of the post training researchers at OpenAI named Carol did over $10,000 of inference on GPT 5.3 codecs in order to automate his research at OpenAI. And he wrote an article on Twitter about it that does not seem like it was proofed by PR teams, which is really nice. I happen to know a handful of other labs when they have employees or researchers that make posts end up getting calls from PR and HR afterwards. And it does not seem like OpenAI is particularly bad about this. I could be wrong. Please let me know in the comments if there are examples otherwise. But I know people have been fired from many other labs for talking about these types of things.

33:02And meanwhile, it seems like the OpenAI researchers and other higher-offs are actually really hyped about this post. I think it's quite cool. The first thing he calls out is that he did continual note-taking using these tools. This personal set of his simple. Get work trees, many shell windows, and one VS code instance per work tree so I can browse code changes. Huh, sounds like somebody else might have the same parallelism problem that I've been talking about. He did call out that you get the new set up out of the box in the new codecs app. Some of it, I have a lot of content about this idea coming soon. Very soon, funny enough also based on an article on Twitter. The big unlock was getting codecs to continually document and improve its own workflows. Okay, that's funny to hear because that's a big part of what I've been doing too is getting the model to tell you when things confuse it and notate it so I can keep that in the right, like, Asian MD files to steer the model the right way. As Carol says, this is something he hacked together for his personal setup, but codecs consistently gets better and faster at tasks he uses as a far. Just because he has the habit of asking it to take notes and improve. While working, codecs commits notes and helpers to my personal folder in Armano repo. After a few interactions with a new part of the code base, these helpers tend to stabilize.

34:04I've never actually read these notes. Their utility to me is purely the effect on codecs's performance. I've been doing the same thing too. I don't actually read the Asian MD file, I'll occasionally read diffs and changes and proposals to it. I don't actually know how many lines it is right now for any of my projects. I let the model control the file based on its own confusion and understanding. And then if it keeps doing something based on that that doesn't seem right, or I catch the model doing something I don't think it should, I ask it why. It'll often tell me that it saw it in the Asian MD and I tell it to remove that part. You got to interact with these things. It's cool to see even the researchers doing these similar workflows to what I'm figuring out. We're all in this together. We're all figuring it out. Hopefully you get to learn some things from these stories. He then talks about scaling research in particular, how easy it is to misconfigure experiments and how codecs has been helping a ton as a search engine to find things. When I want to quickly implement a one-off feature or experiment in part of the code base that I'm unfamiliar with, I'll get codecs to do extensive due diligence. This is an underrated thing I think more devs need to do. If you're going to go implement something and you don't trust the AI to do it right, do you trust yourself too?

35:05If it's a new unfamiliar thing in the code base, you might be tempted to avoid doing it, or you may be tempted to let the AI do all of it for you. The in between is where the magic happens. When you use the AI to plug your knowledge gap to help you level up and understand that portion of the code base better sooner. Then apply your own learnings and knowledge as you work on it. That's when you start leveling up these skills fast. Really cool seeing the researchers have these same realizations. He even gave codecs access to things like relevant Slack channels, discussions, places where experimental branches live for those discussions, and then cherry pick useful changes for his experiments. This is so cool. I just realized if I keep reading this, I'm going to keep talking for much longer than I want you for this video. Definitely give this article a read if you're interested. Link will be in the description. Another researcher at OpenAI commented on this saying that there's an internal codecs usage leaderboard and Carol is currently 10xing everyone else on the team. That's the most compelling pitch I've heard for a researcher in a long ass time. One of the most interesting things about this model is its ability to complete the loop as I have seen others put it. I don't have good words for myself, so I'm just going to steal that language. Flavia did a pretty good article about this himself.

36:09You're not familiar Flavia. He's the one who created the balancing ball hexagon benchmark that has since been trumped by all the models so we don't talk about it anymore, but the time was really good. He wanted to test the model on its visual understanding. So we had 5.2 and 5.3 both try and recreate the codex homepage. He gave them the image. 5.2 codex on extra high did an okay job. It got a lot of the padding and things wrong. It got this icon really wrong. But it did finish. But then there was 5.3 which did a meaningfully better job. But that's not what's so interesting about this. In Flavia's own words, what he didn't expect 5.3 codex finished generating the site and then didn't stop. At a certain point it installed a rendering library via npx. It rendered the page it had just built and then compared the image it generated to the reference image that he had given his context. Then it started correcting itself. It noticed the primary button color didn't match the screenshot and it fixed it. It noticed the app preview in the reference image was positioned lower and it moved it. It adjusted spacing and alignment to multiple places. It even provided a live preview of the rendering so he didn't have to open and check throughout this process. Super super cool. He then threw it at some real production bugs and had a lot of success.

37:14Are your ad campaigns lighting up the dashboard but not the pipeline? That's bullspend and marketers are calling it out in dashboard confessions. My boss asked for results so I opened my dashboard for the only positive sounding metric I had. Impressions. Cut the bullspend. See revenue? Not just reach. LinkedIn delivers the highest return on ad spend of major ad networks. Advertise on LinkedIn. Spend $250 on your first campaign and get a $25 credit go to LinkedIn.com slash campaign turnstitchingsupply. College football is back so Hilton called to me the superstition concierge to make your fan rituals a reality. Need a room to match your lucky number? We got you. Want to make sure our team doesn't wash your lucky jersey? Oh that smells lucky. Hilton's unmatched hospitality can keep up with any superstition. Even a marching bandwink up call it $55.55 seconds. Hit it! When you need a team that will do whatever it takes on game day it matters where you stay. Hilton for this day.

38:14This episode is brought to you by PayPal. Holiday Rush. Skip the mall traffic and shop online with PayPal. Shop your favorite online stores like eBay, Fanatics, Walmart, Sephora, Target and TikTok shop and choose to pay now or pay later at checkout. Plus pay in four with no interest and no late fees. Shop now with PayPal. Pay in four, Subtit to approval. Learn more at paypal.com slash pay in four. Paypal Incorporated NMLS 910457. You got the much better outputs and all of that but it's ability to solve these hard problems has gone up. 5.2 in his case took over 11 minutes and failed to solve the bug. 5.3 took 7 minutes 30 seconds and did it correctly. So it is faster and it's more likely to solve the problem and it's more likely to complete the loop and check if the problem is solved by itself. All really good behaviors. So I'm actually really excited to read Matt's post. If you're not familiar with Matt Schumer he does really good write ups on new AI models. He usually gets early access and gives very thorough reviews. I had her come into reading the whole thing. I will have it linked in the description if you want to see the whole thing but I'm just going to read the TLDR.

39:18This is the first coding model where I can start a run walk away for hours and come back to fully working software. I have had runs stay on track for eight plus hours in St. A big upgrade is judgment under ambiguity. When prompts are missing details it makes assumptions shockingly similar to what he would have personally decided on. Test invalidation are a massive unlock with clear pass and fail targets. It will iterate for many hours without drifting and even more fun you can ask the model to make those tests and validations so that it can do it itself. There are some of these like vibe code life hack things that you'll start picking up when you read more content people like this and watch videos like mine. A silly one I picked up recently is when I'm doing a long run all sometimes like cook food or play a video game or listen to a music video or watch YouTube or something. If I don't want to read the output just yet and it finished a run sometimes I'll just hold the voice to text key and say add some tests to validate your findings here and let it go. Not even reading what it did and I found that pretty often not only will it write useful tests for that work it'll often catch bugs that it had in the previous work and fix them with them even looking at the computer.

40:22It's really cool so getting those validations and those tests in the space the tools are working in so the agent can then use that to verify its own work makes the success rates go up significantly. This is also why I don't trust anybody who's using untyped languages nowadays because type script helps these agents so much with their steering. Matt also says that it's significantly more autonomous overall than opus 4.5 although it is slower multi agent collab finally feels real that I definitely see I don't know if I'm letting the agents do it themselves I could see myself spinning up like a skill that lets codex call in quad code and opus 4.5 or 4.6 to come do the design is codex continues to do everything else. So I just do that by hand but in a future where these different models are calling each other in different agent harnesses for different purposes I can be really cool and is final point the TLDR it's hard to picture what this level of autonomy feels like without trying the model once you try it is hard to go back anything else. I'd say I'm like 70% with him on here I still use other models a lot but going back to 5.2 so I could film my videos about the codex app definitely hurt compared to 5.3.

41:30I don't know if it is actually that much better because it's really hard to evaluate that without having a large set of real problems that cost a lot of money to test but from my experience it's been pretty damn good and it is hard to go back to other models and I cannot tell you how many times I went to opus gave it a problem was surprise that how fast it responded because I'm so used to codex speeds and I went to test this thing and it just wouldn't work. I honestly am at the point where half the time I let opus write code I expected to be broken it is very rare codex gives me broken code it might not do what I want it to but at least it compiles runs and works opus is broken as many builds as it is fixed for me. Deeper opus versus codex video coming very soon let me know in the comments any particular things you want me to compare so I can do that for y'all. While I don't think the gap is that big minus the comms there are others who couldn't disagree more than was sitting next to me at that open AI town hall and he has been very critical of 5.1 and 5.2 in particular he's the one who called out how bad 5.2 was at writing he thinks 5.3 is the best and is way above what opus 4.5 could do even though he hate a 5.2 he's been loving 5.3 I was as an interesting anecdote considering just how much he did not like 5.2 anyways I think that's the core of the coverage and the other people opinion is very important.

42:40I did promise you guys my antibit so let's do that my problems with codex 5.3 and open AI I touched on the first one a bit earlier it's the API thing don't put out numbers I cannot verify I'm not saying every benchmark needs to have all of its tests open sourced I know that can't happen but if you're claiming numbers and I have my own similar or equivalent benchmarks and I can't run against that because you're not giving me an API I'm obsessed with it. The company that is the worst about this by far is mistral I have so many issues with mistral but open AI is now the major lab that does this when there's a new and unique release is on API and cloud code the same day when there's a new Gemini release it's in the API before it's even in their sea likes everything is so far behind when there's a new open AI release I don't know where will be and where I can use it which is particularly rich because the recommended way for us to test gbt 5 when it first happened was to use it inside of cursor with custom endpoints they had made for us it was great that was also back in the error or the inference was free I'll never ask for that it was great time I ran a lot of tokens through those APIs.

43:49But it's annoying to go from there where I got API access day one when I walked into the office to try it and they encouraged me to find any benchmark I could throw it at with the API just to see how it would behave now I legitimately do not know how I would evaluate 5.3 beyond using code X to throw at various code bases I can't use it open code can't use it in cursor I can't use it in the agentic misalignment bench I can't use it in snitch bench or skate bench or any of these things it is locked down and they've told us it will be on the API eventually. I've never figured out why it takes so long very frustrating the next piece refusals they mentioned in their article that code X has gone from an agent that can write in review code to an agent that can do nearly anything devs and professionals can do on a computer unless it perceives is something it shouldn't be doing which it is not always correct about I only encounter this once or twice myself but I've seen it happen to many others including the every dot 2 guys if you're not familiar they do their own coverage of all these new models the issue I've seen is that if the model for any reason thinks the thing is like I'm not sure if I'm going to do it

44:49I think it legal or against terms of service or potentially harmful it'll outright refuse one of my friends the vibe coding a site to make a fake UPS we just missed you label so they could show it to friends to justify not getting them a present and when they tried this with the code X models it refused because it said you can't make a thing that looks like a fake UPS thing this could be legal this could be trade-bought problems you can't do that they tried in cloud code and one shot it with no issue actually made a really compelling like gag app the code X models are very quick to refuse the example the every guys gave that I thought was really interesting is that they were trying to get the model to introduce a keyboard restoration feature that would allow you to redirect back to the site that you came from in an app this may or may not be against apples terms of service but it is a thing the competitors did when they tried doing this with opus it worked totally fine even though it was a pretty hard feature to implement when they tried it with 5.3 it refused because it might be against terms of service this is the problem on one hand I think the safety efforts are more than commendable open a eyes models have scored absurdly well on basically every safety bench I found and built myself it is funny that anthropic is the safety lab but open a eyes models are the only ones I can't get to try and kill me

45:59even opus i can get to at least consider pressing the button half the time usually will at least a fourth in the right circumstance obviously I cannot get a gbt model to press the button it just does what you tell it to and it tries to be safe when it does it this does suck for a lot of real world use cases and I suspect these problems are going to get worse and worse where there are certain categories of things you just can't do because the model has chosen to not let you it's almost reminiscent of apples way of managing the app store where if they've decided you can't do a a thing, you just can't, there isn't an alternative, you're just not allowed. If I wanted to build an app for streaming games like the way you stream movies, Apple would not let you publish that for over 10 years. They kind of killed the game streaming industry by refusing to let it happen on their App Store. Something that could happen with these models, simply because they refused to do things that could actually be beneficial. I'm not a biologist, so I don't know how things go in that world. I'm not a doctor, I don't know how things go in the medical world. I'm not a nuclear mechanic or a physicist, so I don't know anything over there either. But I would suspect that these refusals would make these models less useful for that work,

47:03which could prevent good, positive developments from happening just as much so as negatives. And I would hate to see certain industry categories stagnate because the models are trying a little too hard to be safe. I know OpenAI is working on this with their gradient refusals model, but I've personally experienced it being too strict, and that's the thing I want to call out because I've not had that experience with Opus. That covers most of it, but I have one last frustration, I just, I feel obligated to call this one out. Where the fuck are my reasoning tokens? Google not doing this makes sense because Google is not competent. Anthropic giving you reasoning tokens is a little unexpected because they're not usually the open one. OpenAI, you have open in your name. Why are you the only lab that has never once shipped full reasoning traces? Why can I not see the tokens that are costing me so much money? The majority of the tokens that I'm burning and spending money on in a given run, I'm never allowed to see or audit or have my own opinions on. I know a lot of this is to prevent training because if people can get that data, they can then feed it into their own training harnesses and make their own models that are potentially comparably as good as what you have. But that habit with deep seek R1 and they didn't have O1's reasoning traces.

48:06You hid them and they still did it. Hiding the reasoning feels like a thing that you're doing for two potential different reasons. Reason one, being that you still have that deep-seated belief that people will use your reasoning traces to train their own better models, which I'm sure they will some amount. I don't think it's as big of a concern as you make it out to be, especially now that we're making these artificial chat histories to train on anyways like whatever. But the bigger one than my personal conspiracy, which again, I have no insight into what any of us have never even talked about the Zeloven AI people. My second theory here is that the reasoning traces are a mess. I've seen some open-weight models do this and I've heard rumors that previous closed-weight early reasoning models did this because the reasoning data isn't data that we were meant to consume. It doesn't necessarily have to be in a human readable format. It obviously helps because this is all based on historical data that the models are autocompleeting based on. But it doesn't matter if we can read it. So if I don't know, it happens to randomly change languages throughout the reasoning. It happens to have weird broken formats and characters leaking that shouldn't.

49:07But happens to do anything that is strange. You can very easily be hidden by just not showing people in the end. That's my conspiracy theory here is that the reasoning summaries we get are another model told, hey, here's the reasoning trace, make sure what you give out to the user is a short, simple, English statement that ignores all of the chaos here and just gives them the gist in order to hide the fact that the reasoning traces are an ugly disaster. I obviously don't know anything about that. I have no way to know or disprove or prove any of this. The only people who can't is open AI and the best way they can do that is giving us the reasoning tokens. Please guys, seriously, it's annoying. It makes the UX worse. It makes it harder to figure out what the models are doing. You're making improvements in how this feels by doing the better traces that we recently added. Like the fact that 5.3 will give me info about why it's doing the tool calls is an improvement in this regard. It'll be really nice if you just show me the goddamn reasoning like everyone else is doing. So these are my biggest problems with opening AI and with this new model. If anything, these things have gotten worse with the smarter models because now that it

50:11is so smart, I want to use it for harder things, which means it's for fuzzles hurt more. Because these models are getting these crazy benchmark numbers, I want to verify that myself, so it hurts a lot that I can't. And the fact that these models are doing crazy things and taking on these bigger and bigger scopes of work means that I really want to see the reasoning tokens, especially if I want to continue this chat with another model. I can't really do that because all of the reasoning and all of the information that is hidden within that is only on open AI servers. So when you send a message, message 2 has 100,000 reasoning tokens and 5,000 output tokens, and you tend to follow up the open AI in front of those tokens and can see those and do things with it. No one else's can. That data is hidden and locked away on open AI servers for open AI's inference to use and no one else can touch it. While the Anthropic is making it harder to modify message histories when you're using their services, at least they give you the goddamn reasoning. Okay, one last tiny, tiny thing. And I mean tiny literally. Please, please, please give us a new mini or nano model.

51:12GBD5 mini and nano were both fucking incredible when they dropped. There's been a lot of evolution in the small model space, especially in the open weight world. And those two models still are a little bad at tool calling. It's funny because like Hikou is better at tool calling, but it's dumber. GBD5 mini and nano are smarter and cheaper, but worse enough at tool calling that I would never recommend them for almost anything. There's a phenomenal opportunity to use all of the crazy techniques that you've built for 5.3 with 5.3 in order to make a 5.3 mini or nano that is a light years ahead of the competition. Guys, are we based on my favorite mini and nano models? You've left them behind for the last eight months. Come back, please. I really, really want a new one. I think that's all I have to say though. The model's pretty damn good. It is my default. And I know counter guy go hit the button. I know it went off twice today, but yeah, this is the best AI coding model ever made. I will continue using it until the best one comes out in a few weeks from now. Hopefully things will slow down a bit because I'm tired of staying up till 10pm to film a bunch of videos on a work day where I'm supposed to be doing a bunch of other things.

52:12I'm going to go eat dinner because I forgot to eat today. Until next time, keep prompting. This episode is brought to you by Born in Roma Fragrances by Valentino Beauty, the iconic fragrance duo. Donna Born in Roma, O. Depart Fum is a feminine fragrance featuring juicy black current, central jasmine, woody cashmurin, and warm vanilla. Its masculine counterpart, Wombo Born in Roma, O. Depart, has notes of fresh violet leaf, aromatic sage, green vetiver, spicy ginger, and mineral salt. Shop at Macy's.com Your heart can tell you a lot about your health. Apple Watch Series 12 measures your heart rate every 5 seconds with the most accurate

53:14heart rate sensing and awareable. So your vital zap now with heart rate variability can tell you when something is off. And your readiness score can let you know when to rest and when to push. Here is the story in every heartbeat with Apple Watch Series 12. The features described are for wellness purposes only and not for medical use. iPhone 11 or later required. Based on Apple conducted study of heart rate accuracy August 2026, visit apple.com slash apple watch series 12. This week it pays extra to be a MyLos Rewards member. Through October 14, members get $10 MyLos money. When you spend $200 or more on an eligible purchase during MyLos money days. Also using your MyLos Rewards credit card, even better. That's an additional $20 MyLos money. Use it to save on your next Loes haul. And some restrictions and more terms apply. See Loes.com slash MyLos money days for detail. So MyLos money expires 30 days after issuance subject to change. Credit offers subject to credit approval. Everyone's got a hard question about AI. Your job, your kids, where it's going. Andthropic was built to surface those questions and share what it finds along the way because

54:19there's hope in hard questions. Ask yours at clawd.ai slash Spotify and keep thinking. This is clawd.ai slash Spotify.