Can One AI Model Control Every Robot? | Sergey Levine

0:00Welcome to Sparks where we have deep conversations about deep tech, how engineers, scientists, entrepreneurs and VCs are shaping the next decade of technology. Our guest today is Sargele Veeenth, co-founder of Physical Intelligence, leading researchers in robot learning, one of the clearest voices as well as practitioner of making the dream of robots, physical intelligence in our world come alive. Sarge, welcome to Sparks. I want to start with one of the things you have talked, I think, in different forums, the so-called Morovax paradox. I don't know what is the origin story of that, but it's something, you know, it hits you that today all of us are using AI so much in our everyday life and yet in our homes, in our factories to a limited extent, definitely nothing outside. We barely see physical intelligence, which seems very surprising if these AI models are so smart, why don't we see them doing everyday work also around us?

1:00If you look at science fiction, I think people start imagining robots long before they were thinking about AGI and yet on the other hand, you know, we really almost don't see any real use case of robots around. So how do you process this? Yeah, that's a great question. I guess what I would say is this that it has actually always been the case with AI that problems that seem to us to be very cognitively demanding often end up being less difficult than ones that come to us naturally and there's a reason for this. Even before the current machine learning revolution, AI systems managed to, for example, defeat the world champion of chess long before they could understand even basic natural language. The reason for this, our brains are very capable and we've gotten very used to being able to kind of adapt them to things that they were never designed for, so to speak. So, and these things are hard and they feel very hard to us. It feels very difficult to solve a complex math problem or to play chess or to plan out some really elaborate strategy.

2:09It doesn't feel difficult to us to pick up objects or perceive the world with our eyes. That doesn't mean that those problems are not difficult. It just means that our minds, for a very good reason, became very, very good at those problems until they feel effortless. And that makes sense, right, because people have to survive in the physical world. We have to perceive the world. We have to interact with, we also have to interact with other people. We have to understand faces. We have to understand speech very quickly. So these are all things that our minds sort of delegates to portions of it that are not entirely conscious, that are not things where you can like, you don't get sweaty from how hard your visual cortex is working, right? Like it's not something you feel. But these things are very hard and they occupy a very significant part of the brain. So because of that AI researchers have always been, I think, a little bit misled into thinking that the problems that are easy for us should be easy for machines and the problems that are hard for us should be hard for machines. And to be fair, machine learning has made a lot of headway on those kinds of problems that we think are easy. But it's still tackling them sort of in a particular order, right?

3:14Like there's a reason why language models still, even though they're very capable, are at their best in settings that are more symbolic settings like programming, solving math problems. Like these are things that we can formalize more easily. And I think that the technology will progress and it'll get to those problems that are more physical, more rounded. We just have to recognize that those are some of the hardest problems. And is the is the crux of this is that there are things as you're saying through evolution, we have learned to do it over, I don't know millions or even tens or millions of years. But we cognitively don't understand the mechanism or algorithms about how those things work. For example, peeling an orange or picking up something from the floor and putting in the waste basket or driving a screw in the wall and so on. These things come very naturally. I think most people start performing these activities from the age of six, seven onwards and it becomes second nature to us. While we are performing, I think if I'm hearing correctly, we are saying we are innately wired to be able to do these things well, but we don't understand the mechanism that well to be able to replicate the machines. Is that the right way to think about this?

4:23Well, it may not actually be that we're innately wired to do it. It may just be that the kind of AI systems that we've developed, although they're very general, are not general enough. So there is an old idea in neuroscience that parts of our brain, at least the part called the new cortex, is actually fairly generic. And there are some experiments that have been done to study this. For example, you can take an animal, this is done with ferrets, and you can perform a surgery to reconnect the optic nerve to a different part of the brain. The way this experiment was done is reconnected to the auditory cortex, which is not the part of the ferrets brain that typically handles visual processing. And obviously the first thing that happens is that it has very diminished visual acuity, but then over time it gets it back, which means that the auditory cortex is learning to see. And there are a variety of other experiments that have been done that all have kind of this flavor of illustrating that different parts of the brain can sort of jump in and take over for another part to do its job, which means that it's not a set of specialized organs the way that like, you know, the different parts in your car, you know, the engine can't take the place of the transmission.

5:29But here it seems like there is at least a hint of a more generic computational substrate that can jump in and fulfill these roles. So it may not be that there is something inherently innate about circuitry for processing images of circuitry for processing sounds. It may just be that in AI research, we have not discovered a sufficiently general and broadly applicable approach that can discover those representations as well as the brain can, but perhaps we'll get there. Are you also then saying that this discovery of generalization to these kind of tasks will potentially require different type of architecture, new version of transform or something else versus just some kind of data and training and iterations will get us there. It's a very good question. I think at a technical level, answering that question requires us to kind of think very carefully about what role different parts of machine learning system play. I think it's very tempting for somebody to look at modern machine learning systems and say, well, the magic is in the architecture.

6:29But perhaps that's not the case, perhaps the architecture needs to fulfill certain requirements, but otherwise there are many solutions that do the job. So as an example, in the early days of deep learning, something that computer vision researchers got very excited about is that they saw that early convolutional networks would learn the level of visual features that look like oriented edges oriented, and these kind of features very closely match the kinds of features that are observed in the visual cortex of mammals. And they also seem like very good visual features for representing object positions corners, that sort of thing. But then after those initial results, people figure out that actually there are many algorithms that we yield those features. So you could run convolutional neural networks. You could also train deep, bolster machines. You can run sparse coding, could do all sorts of stuff eventually somebody figure out that even canes will discover the same kinds of features if provided with enough data. And I think that the moral of the draw from that story is not that architecture doesn't matter, but that we shouldn't think of architecture as the thing that determines what the system learns the architecture.

7:33It's like the processor in a computer, right? Like it's not like for different programs, you need a different processor. The processor can be slower fast, it can be buggy or can be robust, but it's not like the type of processor you have completely determines what kind of computer programs you run. The processor is supposed to be generic and capable enough to accommodate the requirements of the computer program, you run. I think in the same way the architecture needs to be powerful and big enough to acquire good features for the problem you want to solve. But a lot of it also comes from other places like data, like the algorithm that you used to train and in those kinds of things. I'm very tempted to ask a lot of questions about data, synthetic data, real world data and what not, but before we get too deep in the weeds, let me zoom out and request you to maybe share the broad perspective on physical intelligence. What is the, the thesis and the vision behind the company you're trying to build? So our main thesis is that we can make a significant, make significant headway on the problem of enabling physical intelligence, meaning AI systems that control physical, physical entities, robots, vehicles, that sort of thing, by approaching the problem in a very general way, meaning that a general model that understands the physical world, the relationship between cause and effect and all that stuff,

8:49is going to be more effective for controlling robots, controlling all the physical systems, then a more narrowly focused system that is specifically trying to solve a particular domain like warehouse automation, manufacturing, etc. And the evidence for this thesis, the reason that we believe this to be true is that this is basically seen in other areas of AI. So with language models, there used to be that for every kind of NLP problem, if somebody wanted to solve it, they would try to design their own NLP system, maybe their own learning algorithm, their own data set. And then language models came along and they provided generic solution that trains on all data from all these different NLP domains and actually solves each of them more effectively than the specialists in each of those settings. And we believe that the same is possible in robotics, that a general solution will be more effective and more specialized solutions, because it will actually have to understand the physical world rather than just overfitting to each domain in time. This is this reminds me of in my day job, I run a venture studio and we are in keeping a few different AI companies.

9:53Two years ago, the general consensus was let's focus on building fine tuned models for a particular domain, whether it's for finance or healthcare or education. And we tried all of that, but in the end, the generic models kept beating all those use cases. And today, at least, I think a lot of people are convinced that bar in the labor of fine tuning here and there and that you're doing mostly for cost reasons or speed reasons, but not for general intelligence. For general intelligence, today if you take fable or opus or I know the new soul model, that pretty much beats, whether you talk to this about psychology or about your personal health or about investing stock. It's pretty much a job done. I think you're making a similar argument for physical intelligence. There is a world in which you can build this generic model, which can and how generic do you think it can be? It can, you know, do your laundry, it can cook your meal, it can drive a car, you know, where does the how generic it can get? You know, what I would say here is this that people are so good at understanding the physical world that we end up using physical analogies, even in domains that are more abstract.

11:03So you will say like that company has a lot of momentum or the words he said hit me very hard, right? Like, you know, we use these physical analogies everywhere. We know exactly what they mean. The physicist, Richard Feynman, he did this interview towards the end of his life where he talked about kind of how he thinks about subatomic particles. And he says like, well, we say that they're like oscillating and vibrating and all that stuff. Like in reality, if you actually like understand the equations, you know that like it's not actually an oscillation, it's not actually spin and it's not actually vibration. But those metaphors are so powerful that they like really help us think about the problem and they actually lead us to drawing better conclusions that if we didn't use the metaphors. So there's something I think very deep about that. And I think it's entirely possible that not only will a more general approach to physical intelligence lead to a general solution robotics, but that it may also more broadly lead to a better solution for AI because it'll lead to AI systems that understand the physical world in a way that is closer to how we understand it.

12:05And they might be able to use those kind of physical metaphors, if you will, to reason about what's going on and that might make them more capable that might make them better suited for solving problems, even ones that are not inherently physical. And it might make them better at communicating with us because we'll sort of be speaking a common language, sort of speak. That is super interesting. So what you're saying is while all these models are trained on pretty much all the text that humans have ever generated. The text itself has a limitation. It cannot really describe the physical world beyond a point. And I assume at one hand, the other textual description of the real world is there or at least our experiences are to the internet. And these models also understand the physics equation somewhere. And I don't know, maybe you can clarify what understanding this context mean or maybe they've just memorized and they know how to solve the equation without really understanding what those equations really mean because to a large extent, you know, we can model and predict the behavior of how things happen with if you understand and of physics about the real world. So I think and the model has some understanding physics yet there are some limitations. The some level that I think what I'm hearing is that language of how real world works in some ways not inventory, maybe physics is too abstract.

13:18It requires too much interpretation, too much computation and every language is too much simplified. A lot gets lost in translation and abstraction. So somewhere like is that part of what you're trying to do? You're hoping that this language will emerge somehow? My guess is and this is a guess is that a lot of what makes language models work really well is not so much that they're trained on language, but they're but that they are trained on a reflection of human thought. Why? Well, because we think about things and then we go and write something on the internet. Maybe you post a comment on your favorite forum or you write an article and put it on the web. So in some ways like all this text ends up being a reflection of human thought and it kind of makes sense that if you train on enough human generated text, eventually you sort of reproduce a kind of a shadow of that thought process. The issue here is that this is only a very particular sampling of what people do because not all of our thoughts and are being reflected in text that we post on the internet and in particular those difficult problems according to more of experiments are exactly the ones that lead to outputs that we don't communicate to anybody else.

14:31And there's a reason for this like communication. Claude Chan had a great way of formalizing what communication is communication is when you and I believe certain things we're not going to communicate the thing that we both believe because there's no point in doing you already know that thing. So I'm going to communicate the difference. I'm going to communicate the thing that I believe that you do not. And I'm going to use the basis of our shared beliefs to make the communication as concise as possible so transmit as few bits of information as necessary so that you with your prior beliefs can reconstruct my message, which means that the things that all humans share are exactly those things that we never communicate to each other and never put on the internet. So you will never see an article on the web about how to you know control the neurons in your arm to lift up a teacup because nobody needs that communicated to them. And that's the part that's missing. So that's why it's it seems so compelling when we interact with all the lambs that they know all this stuff, but it's only the service level. Sorry to interject, but on the same this is an example of giving even if I want to I can't really describe how I'm able to lift this copy. I don't have a language or understanding of that.

15:34Yeah, because you never had to there was never there was never a reason to do that. So there is a it's a song, almost like a missing piece in this whole puzzle that needs to be discovered some other way if we have to be able to explain these machines. Yeah, and we we as people discover that from our physical experience. So if we in the AI systems with the same kind of physical experience, then they'll have that kind of richness, rich experience rich data from which to draw those representations. And is that the better you're playing out that keep putting you know, incrementally better versions of these models out there in the physical world and let them have the almost like embodied experience of what it means to fold a t-shirt. And after they folded enough t-shirt, they will know how to do it and hopefully it also generalize. I guess that's the other part of the better you're playing out there, you know, you are not really learning how to fold a shirt, but you're also learning how to operate in the physical world. And a lot of these kids, but we're constantly able to other tasks. So there's always with something like this, a bit of a distinction between sort of the the journey and the destination, so to speak, right?

16:35So besides the destination, we need to be very pragmatic about what the journey looks like. And we know that data is tremendously important for these things. Instead of trying to synthesize all the necessary data in simulation or in the laboratory, perhaps the best way to get there is to create a kind of a self-sustaining system, something where there is a model that people can use for whatever physical application they need. And maybe initially that model is not perfect that it makes mistakes, but it's good enough that there's a reason to use it in practice. And as it's used more and more, more data is collected of robots interacting with the physical world in all sorts of different ways. And ideally, there would be a kind of a zone of proximal development, a region where the model is not perfect, but can still gather experience and become better. And if such a thing is possible, then it's much easier to imagine how enough data can be collected to make all this reality. And synthetic data is has very little role to play here or are putting these robots in the simulation environments.

17:40But I think synthetic data has a role to play. The problem with synthetic data is that it's not a substitute for the real thing because ultimately what synthetic data does is it provides a way for us to transfer our knowledge to the robot. So we can encode how we believe the physical world works and we have some pretty decent beliefs about that. And that can get us somewhere that can help us bootstrap things that can get us a little bit of additional information, maybe speed things up. And since we ourselves don't know how we acquire representations of the physical world, we can't transmit everything that way because we don't know how to do that. I just for our listeners to understand your offerings and the business model from what I understand you are building this general purpose physically I brainstorm to speak. And if I'm building an application in future, it could be about some kind of home robot or industrial robot. It could be specialized in the purpose is almost like today if I'm building a tech company or healthcare company, I use a generic large language model.

18:41Is that how you envision people using physical intelligence to hopefully, you know, proliferation of this robotics startup where the brain is provided by yourself. So I think one thing I should say is that we don't yet have a full answer. So we have kind of a few guesses about how all of the logistics of this should work that we're exploring. But you know, I don't want to create the impression that we have too much confidence in these things. So what we're the way we're exploring this now is we are trying to see if we can provide access to this model to companies that are building robots for particular domains and that need essentially a kind of intelligence layer. That they can put in maybe with fine tuning or maybe with prompting or maybe with something else, maybe with reinforcement learning. And that would sort of be the missing piece that allows them to tackle their application. Remember working with a few dozen companies that are using our model in various ways, you know, some of them are working with us to extend it to handle new capabilities that it does not currently handle and some of them are using it directly for things that we already know it can do.

19:43But it's an exploration. So it's not quite as clean at this stage as you know somebody using the you know the GPT 5 API or something like that. So in some cases, the model works in some cases, it needs to be fine tune. And in some cases, it opens up a new research problem that we try to tackle. Let me take an example of a real robotics company. I was actually as we speak, I'm in conjunction with them of course, potentially investing their building this robotic arm for industrial use specifically in the textile space, you know, in India, we have massive textile industry. And just turns out, just teaching a t-shirt is is very complicated. Still required a lot of human text to be and they are building this robotic arm with their own custom model, etc. Now is your message to companies like this is fine, that's an approach you can do. But at some point these generic models can become so better that you can focus on the in some ways hardware parts of this and putting this thing in the real world while the generic model will come with lot of learning out of the box.

20:43So that job unit to focus on is is different than figuring out how to really train your robot become to do the job you're trying to do. Well, basically yes, but I guess what I would say here is this that you know imagine that instead of talking about robots, we were talking about having a person do a particular job, right. So if the job is specialized and it's not something that that person has already been has already learned to do, then there might be some kind of training that you would provide. But that training is not teaching them all of the basic skills of how to move their body from scratch. It's taking that that basis that they have from you know from their from their entire life and showing them just the difference just the additional thing that is needed to do this particular job. And of course that's much easier than if for every job you had to you know raise a child from from birth to do it right. And there's not only is that easier, but also we also know that that person is going to bring a degree of common sense and cognitive ability to that job.

21:45So even things that you didn't train them to do they'll still respond reasonably. So if they are working on let's say your text text textiles example and something goes wrong. Maybe that there's there's an accident or they drop something on the ground or somebody next to them needs some help. Like they know how to respond to those cases and you know you're going to train them to do that. So that's the kind of generality and capability that should in principle come from a true robotic foundation. Clear clear you've been added for last couple of years. Can you describe what kind of use cases today physical intelligence able to handle at what level of accuracy and other things that are surprised you to be able to see that this robot and will come to the form factor robot in a second, but you're just going to discuss on the use cases that your technology is able to handle. Yeah, so over the last year or so we've spent quite a bit of time on two kind of axes of performance that we thought will be particularly challenging. One is robustness and the other one is generalization.

22:47So for robustness we developed reinforcement learning techniques to improve our models from experience and we had a blog post about this last autumn in October where we described a model called pystarro 6 which was fine tuned with reinforcement learning for tasks like making espresso and espresso machine folding laundry and assembling boxes for packaging. And what we found there is that actually if we start you know if we initialize the policy from demonstrations and then we deploy it with additional ability to learn from experience we could actually get it to improve very significantly we could get above 90% on all these tasks and we could also improve the throughput meaning the number of successful completions of the task per unit time. So in many cases by more than 2x over the initial imitation learning policy and that was pretty exciting to us like I think you know the 90% is not good enough like the model for you to get better but it's very good to see that deployment that actually having the robot try the task allows it to get better and better through experience.

23:57So that's a very good start and we've been building on that since then. And if I understand correctly you have some of these applications probably in the field as well where these robots are able to full laundry fully accurate. We're working with with a number of partners and since then of course the best policies we have are significantly better than that like these are the results that we had in our in our previous publication which actually my colleagues just presented at our assess last week. But yes the determining factor really the way to think about this is there are laboratory experiments there are algorithm development so on but once the algorithm is solidified then there's kind of the industrial piece of this which is get a lot more data a lot more experience and this is why. You know what we look for in the experiments is the is the change is the difference like you know okay maybe 90% is not good enough but hey it was like 40% before so clear that's a big improvement because once we understand roughly how these things scale now we know that to make it really work we have to do the industrial piece of it which is actually actually deployed you know instead of having 10 robots our 100 robots all that sort of.

24:57Got it and you just to see and maybe you know you might have described this in the paper you're referring to how good are these things at folding laundry is what is available to reliability and how does even in help us visualize is just one arm bring the folding or be it two arms to fold it out is it would yeah it's typically two arms I guess to answer your question. If you set things up in a fairly controlled way meaning that it's folding clothing on a table with two arms and you know there's like a been on the left and a stack on the right it basically folds it every time the thing that changes is like is the fold above some kind of quality bar we have very high quality bar you know when I when I'm. Operators are evaluating these things they have a checklist of like does have at least this many wrinkles at least this at least that so you know according to to our bar it's you know 90% but even the ones that we consider failures are still full the clothes that's not folded as well as we would like. But there's there's another nuance here which is all the stuff gets a lot harder if you go from the kind of like standard generic setup of the task do something that is much more like what you want in your home because in your home you probably wouldn't want your robot to just take a bin fold everything and put it in a pile.

26:05In the top right of a table I assume you don't fold your clothing on a kitchen table right so maybe you would ask it like hey you know I want my shirts to be on coat hangers in my closet and I would like my you know jacket to be you know over there and I want you to fold the towels and put them in the store so you have some much more customized much more specific. This is something we're spending a bit of time on now where it's what once you start going to this more realistic setting you not only need skills that generalize much more effectively because you have to deploy them in many different settings we also need the robot to be able to reason about that entire task about the kind of customization about the plan that it needs to execute you know maybe it's going to put things in the closet with a coat hanger so it might as well grab like three or four of them so it only has to drive their once so all this kind of putting the pieces together in the right way this common sense is actually. Another level of the other of those that remains an open problem yeah there are some people who are Mary condo types where every table is labeled every shelf is labeled and so on very often so perhaps they can be earlier doctors where the robot is very controlled environment most of us are lot messier than that and I guess the generalizing intelligence need to be able to handle that let's talk about the form factor and maybe even before coming to form factors see most of us learned about the fact that we are going to be able to do that.

27:26So I think we are going to talk about robots and the from all these demos you know for lack of better I just can call it a cute demos robots dancing and back flipping and climbing stairs and. Racking things like that but you are focused on in some ways doing jobs and inside the shop or inside home that why not do some demos where robots juggling seven balls or whatever you know these kind of. Yeah yeah you know demos in robotics have a have a long history right like you know there are certain companies that have produced really amazing demos time after time and yet actually getting their systems out into the. Real world always seems to be a matter for next year so to speak so this is one of those things that goes hand in hand with more of experiments that. What is difficult to do is not just things that are physical but things that are generalizable so if you are willing to control your environment you know like in my laundry example if the dentist always on the left and the pile is always on the right or in the case of a robot doing acrobatics that you know it's always this robot on this floor in this in this room.

28:34You can actually get very very far you can do amazingly you know acrobatic agile behaviors and all that the part that ends up being really difficult is performing these kinds of tasks in any environment in any setting under realistic levels of variability and this is something that's very hard to get across in a demo because you know if I show a video of a robot putting dishes into a dishwasher well. You don't know whether it was trained on that dishwasher on those dishes or whether it's actually generalizing to a new setting so we've been evaluating all of our policies in new environments basically as a standard procedure at this point but it's not something that is at all apparent from our demos because if you look at a demo that's or fitting to a training set versus one that's testing and setting it looks like the same demo so generalization doesn't look. It doesn't look at first glance like a very appealing thing but it's actually the really hard thing. The product you're building I understand latest version is pi 0.7 which is I think more generic as opposed to pi 0.6 which was more in some ways domain or specific. Do you think of this as some kind of physical intelligence OS physical and large model what is the language you use to describe what you're building.

29:40I think OS is actually a really interesting analogy here. In robotics people have actually talked about robot operating systems for a very long time but I think that the mistake that people have made is in assuming that the robot operating system is like a computer operating system that it's a way for different processes to communicate. We can think of an operating system is something very different we can think of an operating system as a level of abstraction that allows you to build applications independently of the underlying hardware substrate. And if you think about an OS that way then really the right OS for a robot should be a brain right again to use the analogy of a human worker you know you the application for work is a description of a job or maybe a training program and it runs on an operating system which is somebody's brain. That is already configured to provide a level of abstraction so that you don't have to define all the you know hardware specific bits of the task and you can just describe it in this more abstract level.

30:40So in that same way I would imagine that for a robot the OS should be the model and the application should be a harness and a system of prompts that sits on top of this model and explains to it what the job is. And that your earlier question about how we could work with other companies I think that perhaps contains within it the answer that it's not that the OS by itself solves everything but the OS provides that level of abstraction so that everybody can sort of unlock their creativity and build the applications they want. And how far along are you far other companies to be able to get access to what you build so far so I can build my application on top of that's the kind of call this physical intelligence operating system. Yeah, it's something that's moving pretty quickly for us right now so so far we've worked mostly kind of in a in a one on one kind of partnership capacity where we work with companies where we're not sort of like throwing them all to them over the fence but we're working closely with them. You know we we receive data from them we look at that data would provide feedback saying like oh the data could be better if it was like this so that's been a very kind of hands on process but we are starting to distill the lessons from that and try to package them in a more scalable system and we still have exactly sure what that would look like you know I imagine it'll be some kind of API type system but it's probably not going to be quite as simple as like you know you boot up your robot and you you know type in pie that website and you get actions back.

32:05It'll probably involve some kind of fine tuning API something that allows you to customize them all to your application but we're still trying to get that right. Got got one of couple of questions around just what to expect in near future and I guess if I look at the broader AIC in there two dates people are eagerly anticipating one on the LLM so-called AGI and it's also it's always that looks like it's always 12 to 18 months away and they're just in same regime you know even though it's hard to define where they are. Even though it's hard to define what exactly AGI is but in the physical intelligence we are also looking for GPD 3 movement where there's at least one use case which has universal appeal is that also you look at your near term future that at some point there is going to be there should be a GPD 3 movement where there's an application which is quite universal some ways and which kind of announces that a physical a has a right. But that's a very good question I think about that a lot I don't think I have a single answer there because it may be that robotics would work out the same way it may be that'll be different you know at some level perhaps robotics will be most impactful when it's not widely visible like maybe the first really key applications will be from robots doing things that people don't want to do and that we don't see every day but it will still be tremendously useful you know what when I think about these things you know much as I like who robotics demos I mean the end goal is not to provide a really cool robotics demo the end goal is to make a model that really.

33:28And if in the process of making it really work we end up with a single moment that we can pin down or someone can say like hey that was really great I think that'll be wonderful but maybe we won't maybe it'll be that these things will sort of gradually slip from demos to to usefulness to utility to practice to practicality and then maybe one day you'll wake up and notice that there's all these robots around and no one quite remembered what was the moment when they started getting out there but there is also economics of this there's a lot of things that we can do and I think that's a very good question. But there is also economics of this huge amount of funding that's flowing into physically I space a lot of companies including yours have raised at this point billions or dollars at 10 so billions or also valuation which seems like lot of smart money is betting that some kind of breakthrough is on the horizon otherwise that kind of money will not chase this if the breakthrough and the universal application work let's say five years away it's at least the money is betting that breakthroughs are much closer. Well I think that's right and something that I think is also important to keep in mind is just how big of a difference there is in the approach between learning enabled robotics and the way that robotics has been done basically everywhere else up until now so robotics traditionally is very much a factory automation system that is very very heavy on human intelligence and very light on artificial intelligence so typically if you want a robot

34:54to do a job with conventional methods you need to get a system integrator to come out to your factory figure out what your processes are figure out how to automate them modify the robots in various ways maybe they need like you know special end defectors and special parts design the movements for them you know they refer to this as programming but it's not really programming it's really kind of memorization particular movements and all that so it's very very far away from what we would think of as AI and the change from that to end and learning enabled robots that are right. So I think that's the way we're learning our foundation models is an enormous change like it you know it's barely even the same the same type of device so I think it's important to keep in mind that this is not a case where there's an existing AI kind of ecosystem and we're replacing it with a slightly better one this is a case where the existing ecosystem is just completely different you know it might as well be on another planet. Given that you know most of the season by the upon open AI and topic and XS talk and I think they're all looking quite pretty so maybe I'm zoom you know they're probably very patient with the revenue expectations from AI in physical world but at some point people will start to look for which is a company get to first billion dollar revenue I think the the LLM economy became real when they're token revenues for this company start to exceed 10 billion dollars and then you can see.

36:18A future where it can be hundreds of billions of dollars and I think that all to get it there do expect you know I have any of robotics companies to get to say even a billion dollar revenue next to three years it's a good question the way that I think about this is I don't have any doubt that in the long run. This is going to be an incredibly important economic sector because well what we're talking about is machines doing work that machines currently can do like that has enormous value. What makes it very difficult to pin this down into a kind of like a more specific set of milestones like you know I obviously I personally have like a set of milestones that I would like to hit and I have my like aspirational hopes and so on but if I'm if I'm being very realistic about this like there are a number of different pieces that come together and. And you know it's always hard to predict with development of technology when a particular reflection will happen here we're talking about like multiple inflections that you know the model needs to be at a particular place there needs to be like the right initial sector for taking a foothold all this other stuff so it's one of those things where I guess like the implicit method like solving backwards from the eventual and and state to me very clearly suggest that this is going to happen but if I actually have to.

37:28I think I think it's going to be a very difficult process to pin down the specific dates for these things that becomes very difficult because it's not just kind of a linear progression towards something it's it's when the specific events slot into place so I think I've been very I've been very happy with how the our model development progressed and I think that it's getting to the point where we're already working to deploy it in real world settings if you go to dandelion chocolate and San Francisco for example you can buy a box of chocolates where the box was assembled by our robot like that already exists. But as far as like the first billion dollars I think I think that's an open question to me. Okay, but if you just look at it how people are placing the bets I would say the bet is that we should see in five years or less otherwise this level of investment will not make sense people believe. Yeah, I think that's a reasonable estimate for what is worth. The way that I think about it from a technology development standpoint is I'm much more concerned about sort of like priming the engine so that it spins because I firmly believe that the tech will get better and better the more. It's used in the more it's deployed so my top priority is to get it spinning right and you know I'm pretty sure like I have a good sense of how to get there as far as you know how all that will play out in the economics fear.

38:40I'm not the expert on that I have some of my ideas but honestly like it's probably something you know better than I do honestly on that note do you see yourself as a at the core more of a research lab with the intention of commercializing open a was also set up as a. Research lab even should they had a commercial product is that the right way to look at your very focus on commercial product from. It is so I guess the way I would put it is that it's kind of like a moonshot like you can say well you know in the 1960s the US had a program to go to the moon like was that a research effort well there was a lot of science that went into it also a lot of engineering and also a lot of industrial processes and the goal was to accomplish the mission and that's how we think about it that you know we don't wake up every every morning asking like oh. What is like the most clever research I could do I mean clever research is really nice I really like clever research but it's not just that it's also engineering it's also industrial processes you know how do you get the right kind of data collection so I think that the economic piece of it is really important but to me the economic piece doesn't come first but it's rather a means to an end that you have to have an effective self sustaining company to make something like this happen.

39:49Okay moonshot is a great framework I guess so it's a moonshot is the objective and whatever it takes you know it requires moving research forward you do that it requires figuring out go to market partners you do that and everything in in between I want to talk to you about form factors as well so again most of us there are two ways we are able to visualize robots one is humanoid form and there are plenty of demo videos and demonstrations out there and then there is this arm that you attach to a table and able to do bunch of stuff on top of that. I guess other form factor that's and for the only form factor in the people's houses the is a vacuum cleaner that's been around for more than a decade now and and some cute robots also that people have been selling for a few years where are you are as a company and individually what do you believe about you know how will this form factors evolve is there one form factor which is going to be more preferable than others or form factors will probably for it just like applications will I think that your comment earlier about operating system.

40:49I think it gives us like a hint here right because good operating systems they didn't make it a single kind of computer proliferate very broadly they actually made it easier to experiment and to innovate and now you can all you can have a computer in your pocket on your cell phone you can have a computer on your desk you can have a computer in your car you can have a computer in your refrigerator and all these are computers and I think with robots that will be the same way that once we have very general models it will enable this kind of proliferation of different kinds of robotic applications some of them might be a little bit more difficult to do that. They will use very generic form factors like maybe there will be humanoids or very generic cars but some of them might be very specialized and they'll all be able to leverage the same intelligence layer and I think that's actually really really promising I'll tell you a little story you know when we started the company we thought that we needed to do something very clever to accommodate different kinds of robot morphologies and I had like a I had a document with a long list of all the research ideas I had about how to handle different embodiments and we haven't gotten through any of those ideas yet because the most obvious thing.

41:49Of having the model output you know a list of numbers indicating the motor commands to each joint that was fine and right now our model outputs up to 38 numbers if your robot has less than 38 joints their product was zero and that's it and then it just looks at the image figures out what kind of robot it is and outputs corresponding actions and there's I think something really interesting in that that it makes sense and hindsight because there's a lot more variability in the world than there is in the robots like you know there's only a few dozen different robots where there's millions of possible scenes. So that piece handling different embodiments just didn't actually turn out to be that difficult most people listening to this episode really wondering when can I buy a robot how do I bring the robot home what is your guess so what is the first form factor or maybe you want to feel different form factors which are likely to become the dominant form factor that people have to start to bring to their houses. I think that the first form factors will probably have two arms and either a mobile base or static base meaning that if you if you live in a small home maybe you'll get two arms that you attach to a table and if you are fortunate enough to have a little bit more space maybe it'll be on wheels and we'll drive around and the reason that that's a really good combination is that two arms a lot better than one because then you can like hold something and then manipulate it and being able to move is really good if you can manage it under your space constraints because then your robot can do a lot more.

43:10And that's kind of that's what I call like the minimum package like that's basically your your you know original Mac and touch personal computer with a screen and a keyboard and maybe a mouse attached and between home usage versus factory usage where do you think bulk of the revenue in next four or five years might come from. So I'm not sure but I have some some guesses here and my guess is that if we want to not have a very difficult kind of bootstrapping problem there are many domains that are kind of in between the fully unstructured domain of the home and the fully structured domain of the factory. The factory the normal factories that currently use automation are in some ways not the right place either because there's already robots that work and they don't offer that much diversity anyway but there are a lot of things that are in between you know assembling electronics maybe that's something that can be automated disassembling them for electronics recycling impossible to automate right now right so you can imagine this kind of ladder of applications of increasingly more and more variability climbing all the way up to something that you could imagine seeing in your home someday.

44:18But it's a guess I don't know and it's also not another question that there could be like essentially like you know an iPhone kind of innovation where someone just figures out wait these are the features that you need for like a minimum home robot and out of goes. As you're putting these things together and working on newer version of the pie platform you think at this point you are having to push the frontier both on the science and research front as well as the engineering front or it's more engineering. And I'm a conceptual level most things are there they were just need to figure out a way to how to package it and make it work in with enough reliability in the real world it's absolutely both of those things and this is what makes it so exciting to me is that you know unlike with language models and other kind of generally they are the robotics world is wide open in terms of what the technology what kind of new technology needs to be developed so there's a lot of progress to be made by taking what we already know and building it correctly and there's a lot of hard work that goes into that but there's a lot of work that goes into that but there's a lot of work that goes into that. Also a lot of room for new ideas that could have a transformative effect so certainly when it comes to robots learning from their own autonomous experience we have good ways of getting started but it's not by any means a solve problem it's certainly not a matter of just implementing what we already know it's a place for experimentation and for new ideas and that's why I like this topic so much because there's room for creativity but also room for like just building stuff.

45:39In the aluminum world there is an immense concentration for all practical purpose in the US now there are really only three model companies and probably half a dozen in China and that's pretty much it. Where are we in this physical OS space today is it crowded are there huge number of containers or the bar to entry to Marshal a serious effort is so high that only few companies like yourself were able to really push the envelope there. Yeah it's a very interesting question so this is a place where I suspect that appearance and reality somewhat disagree so there's clearly a lot of action around robotic learning but I think that there is surprisingly little action around truly building general purpose robotic models so what I mean by that is that the typical kind of structure for robotics company is to build a really nice robot and then figure out how to endow that robot with intelligence and that's very reasonable because if you want to build like a robotics business like that's that's logical way to do it pick a domain design the robot design the AI for it.

46:47And while a lot of folks are talking about how they're building robotic foundation models which is to say models that use like you know LLM and BLM components to power their robot I actually do not see a lot of folks talking about how they're building true generic models that anybody could use for their own robots. Yeah and perhaps there's an economic reason for this or perhaps it really is like a bet that very few are taking us seriously as we are. Sergey you are a scientist entrepreneur you can you are struggling both the you put on research hat on to push the first field forward and then the entrepreneur had to see how we build usable products in the real world and they go to Martin so on. I'm asking this question on behalf of probably two sets of audience one is people who are interested in doing something with the robots in real life today last language model has really made my quality of life not better. And second audience is entrepreneurs who perhaps want to do some kind of robotic startup if you have to paint some kind of road map over next three to five years broadly speaking I know exact prediction exact dates and breakthroughs are in possible.

47:54But you have to still give some light and so on how next years are likely to find out I think we'll see the capabilities of robotic foundation models increased significantly. But I think that what we'll also see similar to what we saw with LLM is that building these things to that level of performance from scratch will get increasingly harder and harder because the models are getting bigger and more powerful. So I think that there's like that there's a good news to this but it's also something to be realistic about which is that it will increasingly be possible to get robots to do things that have not been possible so far because the models will get better. But I think we'll also see much more differentiation between the models that really work and the ones that don't and that will make it very important to use good judgment in selecting appropriate models. But I think most of this is a very good thing and what I really hope is that the robotics community will recognize the importance of not siloing things too much meaning that for robotic foundation models to work they really need to be built on a shared basis of data and it's one of those things where if you have a particular robotic application then getting data from your application included in the model benefits you most of all because the model will be better at that application.

49:11But it also benefits others because the model will get better and other people can use the model and I think that that's something that the robotics community will have to consider and each company each organization will decide what they're comfortable with. Yeah, that reminds me of a very important question which I missed earlier is all the aluminum companies also have their own robotics effort including Nvidia I think they announced something called G route I think N1.6 something like that deep mind has Gemini robotics and from what I understand both anthropic and open I have similar effort as well. Do this company have some advantage given that they have this massive amount of infrastructure or disadvantage given that the core intelligence is trained on the general text stuff that we discussed earlier. It's a complex question I don't think there's like a one size fits all answer but what I would say is this that robotics is a very big problem and it's also a very integrative problem. So I think that when thinking about how to approach robotics it's very tricky to approach it as sort of like a downstream application of some other kind of thing like I think taking a really good frontier model and then trying to get it to also control robots might not be a winning proposition as opposed to thinking about the problem from the kind of from scratch from first principles and setting up the right kind of organization like it's not just the technical problems also more important.

50:39So the technical problems also organizational challenge the right kind of organization that can tackle the research and engineering and industrial challenges of building a robotic foundation model. So you know personally I would like to see as much progress in the field as possible and I'm very happy to see that progress being made by any company and any research organization but having been involved in lots of different robotics research organizations both in academia and in industry a big company that small companies. So I think an organization where everything revolves around the robotics problem is going to be much more effective at solving this than one that approach is that more as the emplaced one application of some other kind of business. So I'm very famously said that at some point there will be 10 billion human-edred robots in the world you know and his dates are always very very aggressive but do you believe that feature and if you let's say we don't put data as a constraint maybe 10 to 15 to 20 years do we see this human-edred robots just all around us in office at home in factory or the formula I'm not first on exact form of human but these you know walking talking very dexterous two hand type of machines all around us.

51:49I think we will see a lot of robots and I think that once they once the intelligence is readily accessible the thing that's really exciting to me at least is that they might take many different forms both the human eyes I mean human eyes are cool I'm sure there will be humans but also many other forms that are more appropriate to the job and it may be that in the same thing you know today if you if you go through your day and think about all the instances where you interacted with the computer. There's just so many things that you never think about you know if you pay with a credit card the machine that you're that's processing your credit card payment has a computer in it your refrigerator probably has a computer in it right and we don't even think about these things and I think that physical intelligence devices with some sort of actual you know will become just as ubiquitous and we will interact with these things you know thousands of times every day and we won't even think about the fact that like hey you know I'm buying my groceries and the checkout counter has like an automated device that like packs my groceries into a bag for me or something like there'll be all the things everywhere and there'll be myriad different forms and we will not even think that they're there. My last question said Gaze again on behalf of a aspiring entrepreneur in the physical space I want to start a company what's a good place for someone to start find a very specific niche use case and start thinking about that or a horizontal problem which is not solved what's a good place to start.

53:06To be honest is not something that I'm that much of an expert in right so traditionally starting a company in robotics around a particular vertical requires dealing with a lot of really tough problems distribution channels and robotics are not nearly as easy as they are for for software or or internal companies right. So in some ways i'm kind of dodging the problem by focusing on something a lot bigger now you know I've been very fortunate to have a wonderful group of colleagues that can support this but I don't think that the path that physical intelligence is taking is by any means a good template for a more traditional robotics business. I really hope that you know and i'm working very hard to make it a good bet for myself and my colleagues and my friends but I wouldn't say it's the most pragmatic thing. I think you're and you're taking very research oriented approach which I think the both the very right entry and the capital required is enormous. I imagine there are a lot of people who are seriously interested and I guess maybe you can just I also obviously don't have the answer but I can add to it is one has to take a long term view if you are going to start a robotics startup

54:14that will take a 10 year horizon and not expect big breakthroughs in next couple of years and second or deep technical expertise this is probably area where having a deep technical background is probably an advantage you can't just throw with the applied a company's lot of people can you know figure it out as they go along is probably harder here for just resourceful on to on to go to without very strong technical performers to figure things out but that's my kind of you know preliminary kind of you wonder that. I guess maybe one thing I could say here that could be useful is that i think you know from having seen many of the robotics companies that we've worked with or that we consider working with. I think one factor that consistently stands out as something that differentiates folks that can get good results and ones that struggle with it is just the willingness and desire to actually work with robots like you know I sometimes talk to folks that are interested in robotics but the way they're interested in is like oh. You know I want to start a robotics company and maybe like I'll contract this person to add and that person to do this and so on and does nothing wrong with you know contracting piece out but I think that consistently the folks that can get robots to do something cool are the ones that actually love working with robots and I think if you have if you have that passion if you if you can't wait to get your hands dirty grab the screwdriver and get something put together.

55:31Then maybe like you're going to be able to do something great and if the thought of grabbing a screwdriver and doing something with a robot terrifies you then maybe you should try it out a little bit first like play around with it and see if it's the right thing for you and certainly you know even though our core mission at physical intelligence is around AI all of us are very passionate about specifically the robotics problem and all of us are eager to get our hands dirty play with the robot see what it can do and then you really develop a deep intuition for this. That's an excellent self-evolution test that aspiring entrepreneurs can do on that note. So I want to thank you for taking the time you are building a very fascinating company you are right in the middle of I think very exciting space there's a lot of hope and promise and the potential for profound impact to society to economy and and then probably make the life better for all of us so wish you all the best and we'll be cheering for your company from the sidelines. Thank you. Thank you.