Why OpenAI And Anthropic Are Pumping The Brakes

0:00Support for the show comes from Morgan Stanley's podcast, Hard Lessons. Some investing lessons only become clear after you see how a call plays out. On Hard Lessons, iconic investors sit down with Morgan Stanley leaders to go behind the scenes on the critical moments, both successes and setbacks that shaped who they are today. Watch or listen to Hard Lessons wherever you get your podcast. Energy of Ah-Ha, falta de concentración, nevlamental. You might be dehydrated. Gatorade Zero powders han sido diseñados científicamente para ayudarte a mejorar el balance de hidratación en el cuerpo. Hechos con la mezcla número uno de electrolitos, sin colores ni sabores artificiales, shop now online or at retailers nationwide. Gatorade, hydrates better than water, is it in you? Signos de dehydratación leve y moderada, consulta tu médico si los síndomas persisten. Megan Rapino here, this week on Why Are You Like This? I'm talking with Vice President Kamala Harris. We talk about her thoughts on public service, how DC has shaped her,

1:02and we find out if she's planning to run for President in 2028. Check out the latest episode of Why Are You Like This, where we get your podcast and on YouTube. How do you mark it's better? If money is evil, then that building is hell. And show no sign! That's another question. Show, show! Welcome to Prof.G. Markets. I'm Ed Elson. It is September 15th. Let's check in on yesterday's market vitals. The major indices declined, with chip makers selling off on fears of an AI slowdown, more on that in a minute. Meanwhile, CrowdStrike rallied 14% as investors piled into cybersecurity stocks. Brent Crude remained elevated at $105 per barrel, and finally the yield on 10 new treasuries topped 5% for the first time in three years. We will be discussing that news tomorrow.

2:03Okay. What else is happening? The AI apocalypse debate intensified over the weekend, and both anthropic and open AI have now officially weighed in. On Saturday, anthropic CEO Dario Amadei published an essay titled We Must Pace the Frontier. He wrote that we must slow the pace at which we improve the capabilities of AI models. He also warned that within six to 12 months, a swarm of rogue AI agents could take over the entire internet with a persistent botnet. Sam Altman also posted, quote, I agree with Dario that we need to pace the frontier. And Elon Musk concurred, posting quote, Dario is right. Altman also told Fortune that opening AI will not go public in 2026, calling this quote an ill-advised moment to go public. The White House, however, is not on board with slowing things down. President Trump, speaking to reporters in Ireland on Sunday, said, quote, whoever wins, AI wins. And called the people raising these alarms, quote, negative forces.

3:07Still, a slew of AI adjacent companies sold off on Monday on concerns that a slowdown would impact AI spending. In video, closed down, 3%, Oracle was down, 4%, CoreWeave down, 7%, and SoftBank, which is a significant investor in opening AI closed down, 15%. So here to break down what all of this means, we are speaking with Charlie O'Neal, co-head of Model Training at base 10. Charlie, thank you for joining us again on ProfG markets. You work in AI, you are an AI developer, you work with these models, you've been in this game a long time. Suddenly, everyone is very upset about this. And it's an interesting, it's interesting how the debate has evolved. But we're now reaching a place where the leaders of these AI companies are saying we need to actually slow everything down, which I'm not sure many people would have predicted. And now it's the President, President Trump, saying, no, that's the wrong approach. We need to speed things up.

4:09Where do you land on this? What is your perspective? It's really interesting to see the reaction to something that's kind of been like, I guess, like, linearly scaling for a long time in terms of the calls for pacing the frontier from people such as Dario and Sam. I think the best way to view what they're actually asking for is not necessarily, we're going to put a halt to capability development, we're going to put all these stringent checks and, like, you know, first party kind of things that are slowing us down. The best way to view this is we are going to keep advancing the capabilities of the models. We are just going to allocate a little bit more compute to making sure those models are safe. And then it's very interesting to see what they're so often so on today. Open AI and then probably planning on spending more compute probably than they were a few weeks ago. You know, there's rumors that OpenAI is going to allocate up to 20% of internal compute for monitoring and safety. So, you know, as they're doing these training runs, as they're deploying these models in the real world, things like the hugging face incident they don't want to happen again.

5:09And so if you allocate more compute to monitoring those models as they're doing their rollouts and as they're doing inference, then you're more likely to catch it. Anthropically is similar and some people have suggested there's going to be a much higher than 20% figure. So when you consider that and the fact that Anthropically in OpenAI really don't want to move off their, you know, model road maps, they want to keep training bigger and bigger models. They want to keep scaling the reinforcement learning they're doing on top of these big pre-training bases. They just have to allocate more compute to, you know, monitorability and safety research. Then I think we might even see the lives be even more aggressive with compute builders and securing compute. And it's certainly not going to be a, you know, a bearish sign for like the amount of compute the world is going to need over the next few years. So that's probably the best way to view it is like, you know, capabilities will keep progressing at roughly the same rate. It's just that we're going to allocate more compute on top of it to, to safety of monitoring. If everything that they are doing is sort of within their own power as you're kind of describing, why are they saying anything right now?

6:09What are they trying to get at? Because there are a lot of people, especially in government, David Saks has said this was the former AI's are and the president is saying this too. It sounds like what they're asking for is for the government to do something about it for that to be more regulation that is imposed on themselves. So what do you think they are asking for exactly in this moment? There have been some, you know, people like David Saks and so on saying that these labs are using this as an opportunity for regulatory capture to crowd out open source to shut down competitors, which are behind. I don't really think that's the case, to be honest. I think like when you look at the main things that these labs are calling for or at least instantiating on their own accord. It's things like third party evaluators and this commitment of compute to monitoring and safety. You know, the third party evaluators won't have necessarily any license or legal standpoint to shut down model development if they find things that they don't like. Like that's still going to be up to the internal labs themselves.

7:10I think like to be honest, like the best rate here isn't a conspiracy theory on either side. It's not the labs trying to crowd out open source. And you know, it's not some like, you know, kind of global cooperation between open AI and and throttic to crowd everyone else out either. I think it's simply a case of, you know, these models are getting very, very good. The close source models got there first, you know, in the next three to nine months, open source models are going to reach these capability points. And we're not at the point where that could have significant impacts on the world. Even if it's not malicious and you know, rogue AI is going off and trying to like kill humans, they'll probably be significant annoyances. Like the most likely scenario here is that things like the hugging face incident happen and the internet is overrun by like swarms of AI agents that are trying to get some like arbitrary tasks done that are not trying to be malicious or evil. And so like, I think the labs just recognize this. Some people have more extreme views, of course, but realistically, capability is going to progress. Like the most boring interpretation of this is probably the correct one, which is that the world will probably only accept super intelligence at a pace that it can exorbe.

8:15So the labs are telling down because they have to and not because they're plotting anything and the result is probably intelligence that's, you know, decently fast, decently safe, decently commoditized and it's spreading through best practices and even like distillation until the models are just basically a reasonable integration into the world. It sounds like you're not worried about this much at all and that is interesting because you work with open source models that is a lot of what you do. And of course, as you mentioned, that is what David Sacks has been accusing and a lot of people have been accusing anthropic and open AI of that they're saying, oh, now we need regulation because that way it might create some some level of regulatory capture, which might crowd out the availability of open source models and open source model providers. But you don't believe that is an issue. I wonder, do you think that this whole thing is overblown? Do you think that the Jacob Cox and tweets that really started this debate saying that AI could kill us all by the end of the decade is your view that that isn't something to worry about much either.

9:16I'm definitely not one of those extremists who think that, you know, the AI has a more than 10% chance of killing all humanity. I do believe that there are tail rest that like some people should seriously be able to do that. Some people should seriously be considering, but I think that what we're seeing with the current LLM paradigm is like, as I said before, a risk of swarms doing very, very annoying things to humanity and things that will be quite painful in the short term. However, I believe the benefits of AI to outweigh the annoyances and even the pain that those like, you know, short term things can cause and that at some point the world is going to have to harden to these systems. I think that the pace we're currently progressing out and particularly if we do things like, you know, dedicating more compute to monitor ability is going to allow us to integrate AI into a world at a pace we can handle. A really good example of this is like, you know, the cyber security arguments that the people have been making, you know, like we worry that when AI got to this point or even the point that was at six months ago, the world will be overrun by cyber security attacks and that just hasn't happened. A large part of the reason is that, you know, the frontier labs, the close-source labs have kind of been the canary in the coal mine, they've understood where the models capabilities are going to be at in six months for the open source.

10:24And we spent six months preparing these have been slowly released and, you know, like Greg Brockman describes using a stroke to like repeatedly harden, you know, all the vulnerabilities and opening eyes code bases and every single model that they release they do this with. And I think the world will look the same and cyber security is just one example, but it's also one example that's very important where we haven't seen this like massive pain play out and we have definitely seen the benefits so yes, I think there's there's there's risk to consider on any side of the spectrum. But, you know, I think I said firmly in the middle and I believe that the pace we're currently progressing at is a healthy pace and like I also have trust in like not only the closed lab leaders, but also the open source. Lab leaders to make sure that pace continues at an appropriate pace. Support for the show comes from Zbiotics. I know a lot of us like to have some drink strain vacation. I just got back from vacation myself.

11:33If you do, I have to tell you about pre alcohol by Zbiotics. Zbiotics pre alcohol probiotic drink is the world's first genetically engineered probiotic. It was engineered by PhD microbiologist to break down a set of TELDA hide and unwanted by product of alcohol metabolism. Make pre alcohol your first drink of the night drink responsibly and enjoy your next day activities. Zbiotics this old more than 14 million bottles and earn the trust of thousands including us. I am trying to drink less, but I still enjoy drinking and when I do my first drink of the night is in fact Zbiotics. I was using it before the response or try for yourself today. And if you're unsatisfied, they'll refund your entire order. No questions asked. Head to Zbiotics.com slash ProvG code ProvG for 50% off your first order. Zbiotics is a partner of ours. Support for the show comes from Morgan Stanley's podcast, Hard Lessons. Some investing lessons will become clear after you see how a call plays out.

12:34Hard Lessons iconic investors sit down with Morgan Stanley leaders to go behind the scenes on the critical moments, both successes and setbacks that shape who they are today. Here for iconic investors including Stan Truck and Miller, John Gray, Rick Reader and Gene Hines as they revisit the calls of work, the ones that didn't and the moments that changed their thinking. You'll hear in their own words how they built conviction, what happened when certain calls didn't play out as expected, and what they learned from those moments. Watch or listen to Hard Lessons wherever you get your podcasts. Energy Abaja, falta de concentración, nevlamental, you might be dehydrated. Gatorade Zero powders han sido diseñados científicamente para ayudarte a mejorar el balance de hidratación en el cuerpo. Hechos con la mezcla número uno de lecrolitos, sin colores ni sabores artificiales, Shop Now Online or at Retailers Nationwide. Gatorade hydrates better than water, is it in you? Signos de dehydratación leve y moderada consulta tu médico si los síndomas persisten.

13:36We're back with profg markets, one of the other conspiracy theories going around about why they're saying, saying all of this, saying that we need to slow it down, we need to be regulated now. CATURIAS percibillаются unos 100 $ para todos estos sitios 그다음ciados y a koşarko. ¿Qué se te presentes? ¿Qué pizzas así? Pues vende而且 el condicional de oxenamiento de government

14:36y esa será una comprendencia más lituativa. Desde todo el aspectoную Spencer, puedo que500, todas lasichiures, el devenir delito de su iPad multipliante es que sea verido. Pero era ex這種 que yo se widen... FunSean no ent pretende que esto es un triste riesgo wire, es que tiene que hacer el conc ME softly weld rails hetología de marzo de gigawat o�도, אבל el ampute- 커�zo aunes siempre Анtombra, echoным de manera similar, y también multimarde провino gesehen de

15:39como le pido Euststein por embargo, esto ya está. y no creo que tiene ningún nudo, de dondebitabe, ni cu залas y ag� No. You mentioned that the boring explanation is probably the most true, but a lot of the conversation has been anything but boring. I mean, this debate is exploding everywhere. For on tech podcasts, on cable news networks, and now it's literally occupying the mind of the president. What is it about this moment that is so triggering to everyone? Not just people who are afraid of AI, people who are excited about AI, people who are optimistic about AI. I mean, this has got everyone riled up in a way that I have rarely seen.

16:41What is it about this moment that explains that? The reason I saw this rule is not because there has been a discontinuity in terms of capability and advancement, in terms of what people have been saying about where these models would be at in terms of the press around these models. I think that most people really fundamentally involved in the technical side having been able to draw the straight lines on the scaling graphs and say, okay, at this point in time, we are going to hear. I think it's just a confluence of a lot of things and to some people looks like a discontinuity because it's managed to permeate the public consciousness in a proper way for the first time. I think a big part of that is the hugging face and all the discourse that that generated. And then you add things in open areas marketing around an astro of being AI. That's probably not going to help either. People have become familiar with this tone now and if you claim that your model is finally there, then they might see that as a phase transition or discontinued in of itself. But this misalignment research has been very clear for a long time. Go back to anthropics 2023 papers with much worse models, with much smaller amounts of RL, if any.

17:45And it's clear that models would behave poorly in incidents like the hugging face warm incident if you put them in weird situations. And the RL environments that the labs have been buying at scale some are good but some are also very poor quality, some are intentionally engineered to be impossible to do. And this is leading the models to do really weird misalignment things. I think that's pretty common sense and pretty clear. I also am not trying to downwate the risks that that comes with. Obviously the hugging face incident could have had real world impact. But at the same time, I don't think that there has been this discontinuity. So I'm glad that the world is recognizing it. But I do think that the reaction to it will calm down as people start to understand exactly how to interpret these things when exactly what it means. You mentioned how we have seen evidence that these agents do... can do things that they're not told to do. They can misbehave, they can get be misaligned. And you said that you expect that they might continue to do annoying things to humanity. To me, that word annoying is an important one

18:46because it is a very different description from what we heard from Jacob Cox and in his tweet, where it's not annoyances but catastrophes, worldwide catastrophes, civilizational destruction, etc. Is it your view that we will be limited to annoyances or is there some other outcome that is closer to what the Jacob Cox and tweet describes that you are worried about? What do you think that's not really in our trajectory at the moment? I think there is some path to dependency here. I probably agree with Jacob that there is a potential world we go down in which there is zero monitoring on chain of thought of models where we don't have any care in terms of what we are all the models on, where it's very, very cheap and there's like unlimited compute for essentially anyone to be able to train these models and in particular continue to train them from certain bases, where incidents could happen that would definitely not be classified as annoying

19:47and rather genuine, evil or malicious intent as much as you can anthropomorphize the models. The word cause real harm. However, I do believe that the path we are currently going down, that's not very, very likely. Again, I think that there will be times when the models do things and it will appear malicious intent, but the amount of compute with which we are running them at, how generally aligned they are in terms of completing tasks, they will show glimpses of misalignment, but on the whole, if you do look at a Claude or GPT model, it will generally try and do the right thing, like our line of training generally works. I think that we will see incidents, but certainly not large enough scale on over a long enough time horizon to cause really, really significant harm to humanity. If we keep going down this good path, I think that it's unlikely outcome. Do you believe that we have enough regulation or that the regulatory frameworks that exist today are good enough to prevent that, or do you think that there is something

20:49that needs to be changed in some way? Definitely. I think if we didn't have the dynamics we had now, where we have a drawfully in the sense that there are two labs very, very close to each other in terms of capabilities, and the dynamics that engenders with wanting to both be seen as the good guys and wanting to both pace the frontier in this particular case. I think that if it wasn't the case, let's say opening eyes and saying Sam doesn't necessarily have to worry about optics as much, I would be worried that regulation could slow things down to the extent that they need to, or at least provide the amount of oversight that it would need to. However, saying that doesn't mean that I think anyone has the answers to what regulation in industry moving as fast as this one looks like. At the moment, we do basically just have to trust the people developing these models to regulate themselves and have it oversight themselves. One thing that I would like to see is a little bit more public communication for the labs as well. One of the key examples of this is trying to understand how good the models these labs have internally are, because that gives us a really, really clear signal

21:51on how quickly to prepare things and how to prepare. Maybe regulation which enforces the labs to declare those sorts of things on key benchmarks would be a really good way to start. But again, I don't think we know what the whole issue picture is for regulation. I guess the thing that's kind of scary to me seeing what they're saying at this point is as you say, it does seem as though we're in a place where it's like we have to trust them to handle their models correctly to make sure that their models are aligned. But it seems as though what they're coming out and saying over the weekend is we don't even trust ourselves. Like we don't think that we know what we're doing exactly and we're worried and we think it could destroy things. And so we'd like for you to do it. We'd like for you the government to help figure it out. And I just want to play this clip from an interview with the president over the weekend where he was asked about, you know, what do we do if these bots take over and destroy humanity? And here is what he said. Some people say the worst case scenario with AI is that the robots,

22:52the machinery learns to obviously thinks for itself that's what it does. And that could turn against humanity. We have the whole rails. It's going to be fine. We'll always have something to stop them, right? We'll have a little gear. I really hope so. I don't like that. I really don't like that robot. We'll stop. But no robots are going to be a part of it. Robots are going to be big. But we're going to end up doing much better because of it. So the combination of their comments plus his comments makes me think, okay, no one's really in charge here. And maybe that's fine because maybe it's not a catastrophe as this researcher, ex-researcher seems to claim. But I don't see anyone really taking the lead. I think it wasn't necessarily so much a call for the government to step in and provide expertise and guidance. I think the labs are too smart for that. They know how, you know, the lack of awareness the government has about the capability of this technology, let alone how to monitor it and regulate it. I think to me it was more about leveraging the people who have actually really fundamentally cared about this problem for a long time.

23:53Like, of course, the labs have been focused on, you know, building capabilities as quickly as possible, even anthropic, which is very safety focused, has just been focused on scaling RL since the RL paradigm was discovered. And organizations like META, the UK Air Safety Institute, and others have, as well as Redwood, have really just locked in on this problem and have seen the full progression from the really poor models of four or five years ago, through to the models we have now, and have just developed really good science around how to monitor and evaluate these things. And I think that the labs are smart enough to recognize that this is going to be useful as they increase their, you know, monitoring efforts going forward. All right, Charlie O'Neill is co-head of Model Training at Base Turn. Charlie, always appreciate your time. Thank you. Thanks, Amy. Let's take a break from the existential implications of AI and return to our bread and butter, the financial implications of AI, almost specifically, the financial implications of anthropic.

24:54According to the Financial Times, anthropic is telling investors ahead of its blockbuster IPO that it has been profitable for two-street quarters, which is a very big deal because, as we've discussed on this show plenty of times, one of our biggest concerns about the AI business model is that it might not actually work, or at least that it might not work for the frontier labs. Why? Because of how expensive it is. Based on the financial documents that were leaked by Ed Zitron, we learned that OpenAI racked up more than $20 billion in operating losses last year. That is how much they're losing simply from running the business, and it was based on those financials that we started to wonder, does any of this actually make sense? Now, if anthropic is profitable as the headline suggests, then it would put this debate to bed. Sure, OpenAI might be a poorly run, unprofitable AI lab, but that doesn't necessarily mean that they all are. But that is only if this anthropic headline is actually true. And I have to say, I am a little bit skeptical.

25:56The first thing we should acknowledge is that the company is claiming to be profitable only on an operating basis. So that means they're not including things like fixed costs or depreciation or taxes at the same time. I am okay with that because anthropics fixed costs are not that high because they're not a hyperscaler. They're not actually building and buying the physical assets like data centers. So to be honest, operating profitability is fine by me. That is a good sign. Where I do start to get hesitant, however, is when I learned that they're only profitable on an adjusted operating basis, which means that they are actually changing their accounting rules to be different from standard accounting rules. And the changes that they're making could be anyone's guess. It could be reasonable. It could also be flat out ridiculous. But here is where I get especially doubtful. Supposedly, anthropic has told investors that its gross margins are higher than 80%, which is of course incredible. But that is only before it accounts for its revenue sharing agreements

26:58and before it accounts for the cost of training its models. I.e., its largest expenses. So there is no getting around it. Those adjustments are ridiculous. Now, the question is if those adjustments are also included in the company's calculation of its operating profitability. The question is if they are actually removing the amount of money they have to give back to their distribution partners such as Amazon and removing the amount of money they have to pay to build their models and train them. And then just telling us, screw it, we're profitable. If that is the case, then this story is genuinely meaningless. The trouble is we don't know. We don't have clarity or insight into any of the numbers because the company hasn't shared them. Everything we know is based on rumors. We will only truly understand what is going on when anthropic releases its S1, which I hope will happen soon. But until then, when it comes to the profitability of AI, I stand by what I said last week.

28:02And that is that I will believe it when I see it. Okay, that's it for today. This episode was produced by Claire Miller and Alison Weiss and engineered by Benjamin Spencer. Our video editor is Brad Williams. Our research team is Dan Shalan, Chris Nodona, Hugh and Mia Silvario. And our social producer is Jake McPherson. Thank you for listening to ProfGMarkets from ProfG Media. If you liked what you heard, give us a follow. I'm Adelson. I will see you tomorrow.