Nvidia Cosmos: Building the Brains of the Future  ## Metadata - Author: [[Hard Fork AI]] - Full Title: Nvidia Cosmos: Building the Brains of the Future - Category: #podcasts - URL: https://share.snipd.com/episode/131cbbaf-339b-4dd4-8c26-a785c4165a68 ## Document Notes ## Highlights - **Consolidate Models For Faster Experimentation** - Try consolidating multiple AI models and modalities into a single interface to compare outputs and workflows efficiently. - Combining image, text, and audio models in one thread simplifies switching and experimentation. Transcript: Jaeden Schafer You can chat with the models all in the same thread. So you can switch between what model you're talking to in the same chat feed, which is super useful. And you get, of course, the image and audio models as well in there. So things like 11 labs, or ideogram for image generation, and some really cool stuff. So if you want to check it out, there's a link in the description. ([Time 0:00:56](https://share.snipd.com/snip/09148512-1b4d-46c7-bab8-0df2e15f9146)) - **Cosmos Reason Is Built For Physical Reasoning** - Nvidia released Cosmos Reason, a 7B vision-language reasoning model built for physical AI and robotics applications. - The model emphasizes memory and physics understanding to enable planning and real-world action reasoning. Transcript: Jaeden Schafer So they made a bunch of infrastructure for robotics, which I think is kind of cool. Cosmos is playing an interesting role in all of these new announcements. But basically the big thing is this new thing called Cosmos Reason. So this is a 7 billion parameter model. It's a reasoning model and it is a vision language model for physical AI applications and robotics. This is what I am incredibly excited about is the fact that these new models that are coming out, this isn't just another LLM for someone to chat with, but we're actually building tools Specifically built for robotics. And we know there's a lot of players making these robotics. There's a lot of these robotic startups that have raised a lot of money. We have Optimus from Tesla. We have Figure that's making some big strides. We have Boston Dynamics, which is doing some really interesting stuff. And so NVIDIA making a big focus, I think, and making these kind of announcements and building models for these, I think is a really big signal that they see a huge ramp up in this in the Very near term. And they want to already start before, you know, we like we don't have any of these AI robots on the mass market right now. I guess there's, you know, AI robots inside of Amazon factories, those types of things, but nothing that's a, you know, a personal use robot that a lot of these companies have been touting That you could see, you know, billions of these things being sold. We don't have them out yet. But the fact that Tesla or that NVIDIA is already starting to kind of put out models that would address those, I think really signals that they expect these to be coming out shortly. ([Time 0:01:15](https://share.snipd.com/snip/665feafb-45fa-4342-bc06-97f76a443914)) - **Synthetic 3D Data Powers Robot Training** - Cosmos Transfer 2 speeds up synthetic 3D environment generation for training robots by distilling spatial control inputs. - Nvidia positions synthetic data as the fastest scalable way to produce the datasets robots need to learn real-world tasks. Transcript: Jaeden Schafer Okay. So also joining this kind of existing batch of Cosmos world models is one called Cosmos Transfer 2, which basically can accelerate synthetic data generation, right? So it's able to really quickly create this AI generated data. And it's doing it from 3D simulation scenes, or basically what are called spatial control inputs. And it's basically a distilled version of Cosmos Transfer that is more optimized for speed. This is really interesting. So I mean, basically, what we're talking about here is the fact that these AI models, they're creating synthetic or like fake 3D environment simulation data. And why is this important? Because this is literally what you need for these AI robots to train. At this point, you could train an AI robot how to walk through a house by, you know, maybe putting a person putting a full body suit on and walking through the house themselves and looking Around. And like you could get some sort of cameras on them and you get some sort of data points from doing something like that, right? That's obviously not super scalable. And you could 3D scan the house or whatever. But if you just have an AI model that can produce 3D environments, 3D inside of houses or buildings or warehouses or whatever, and create massive data sets. That's the fastest way we're basically getting that data to help train robots. Okay, so during all of this announcement, this is at the SIGGRAPH SIGGRAPH conference. NVIDIA said that these models are meant to be used to create synthetic text, image, and video data sets for training robots and AI agents. ([Time 0:02:51](https://share.snipd.com/snip/129e761e-4c53-41ce-846b-1877f26134ee)) - **From Language Models To Physical Planners** - Cosmos Reason aims to serve as a planning model for embodied agents, usable for data curation, robot planning, and video analytics. - The emphasis on physics and memory marks a shift from language-only LLMs to models that must predict physical consequences. Transcript: Jaeden Schafer This is really interesting. So I mean, basically, what we're talking about here is the fact that these AI models, they're creating synthetic or like fake 3D environment simulation data. And why is this important? Because this is literally what you need for these AI robots to train. At this point, you could train an AI robot how to walk through a house by, you know, maybe putting a person putting a full body suit on and walking through the house themselves and looking Around. And like you could get some sort of cameras on them and you get some sort of data points from doing something like that, right? That's obviously not super scalable. And you could 3D scan the house or whatever. But if you just have an AI model that can produce 3D environments, 3D inside of houses or buildings or warehouses or whatever, and create massive data sets. That's the fastest way we're basically getting that data to help train robots. Okay, so during all of this announcement, this is at the SIGGRAPH SIGGRAPH conference. NVIDIA said that these models are meant to be used to create synthetic text, image, and video data sets for training robots and AI agents. Very exciting. Cosmos Reason, according to NVIDIA, basically lets robots and agents quote unquote reason. ([Time 0:03:19](https://share.snipd.com/snip/ff34fa36-e133-48f2-85cd-6632a3d6b698)) - **Simulation Replaces Costly Real-World Collection** - Synthetic simulation data can replace impractical real-world data collection like full-body suits or 3D scans for scaling robot training. - Generating massive virtual datasets accelerates training for navigation and manipulation tasks inside varied environments. Transcript: Jaeden Schafer And why is this important? Because this is literally what you need for these AI robots to train. At this point, you could train an AI robot how to walk through a house by, you know, maybe putting a person putting a full body suit on and walking through the house themselves and looking Around. And like you could get some sort of cameras on them and you get some sort of data points from doing something like that, right? That's obviously not super scalable. And you could 3D scan the house or whatever. But if you just have an AI model that can produce 3D environments, 3D inside of houses or buildings or warehouses or whatever, and create massive data sets. That's the fastest way we're basically getting that data to help train robots. ([Time 0:03:30](https://share.snipd.com/snip/86274574-a7c8-4b56-9525-6dcd85f43648)) - **Adopt Nvidia's End-To-End Robotics Stack** - Use Nvidia's neural reconstruction tools and rendering techniques to convert sensor data into simulated 3D environments for development. - Leverage their servers and cloud management stack to standardize robotics development workloads end-to-end. Transcript: Jaeden Schafer So very, very interesting. There's so much that was rolled out. There are also some new servers for robotic workflows um nvidia has something called the nvidia rtx pro blackwell server and basically this is a architecture it's a single architecture For basically as you're doing these this robotics development and kind of doing these robotic development workloads um nvidia already has something called djx cloud it's a cloud-based Management platform um and so this is kind of something that will go along with that. ([Time 0:05:38](https://share.snipd.com/snip/f9fa7497-9d5c-48c4-9e81-29b0ae2258b3)) - **Robotics As Nvidia's Next Growth Frontier** - Nvidia is pivoting beyond GPUs for gaming and LLM training toward robotics as its next major growth frontier. - Positioning hardware, models, and tools for physical AI helps Nvidia capture future GPU demand from embodied agents. Transcript: Jaeden Schafer All of these announcements come as NVIDIA is really trying to push further into robotics. I think it's looking towards what its next big use case of the GPUs they have are. NVIDIA has played an interesting game, right? Where basically they came out and everyone knew them as, okay, this is to make laptops better or computers slightly better. Some of their hardware, I mean, to the layman, that's what they would expect NVIDIA did. It became, okay, well, this is a kind of a performance thing, things like gamers, or if you need a really hardcore computer setup for maybe doing 3D modeling or something like that, you Going to want nvidia chips nvidia then kind of took the next step which was uh crypto actually crypto mining nvidia had a huge um windfall as kind of the whole crypto craze was taken off And just as crypto was kind of entering what a lot of people called crypto winter um and was slowing down a lot of the big crashes a number of years ago and a lot of people said oh nvidia's Like stock is going to completely tank. There's not so many crypto miners buying this. And all of a sudden, AI popped up and was kind of the next thing that NVIDIA really was poised to create chips for. The H100 being, you know, the very famous one that has powered so many of the AI tools we use today. And they really have been riding a high. But a lot of people are saying, hey, what happens next? How do they grow their company bigger beyond just training LLMs? And it feels like robotics is the next frontier. There's going to be such an insane demand. And NVIDIA is already trying to position themselves to look at that as they're kind of going beyond just AI GPUs and beyond just AI data centers, right? At some point, they're like, if that well runs dry, what's our next thing? And it feels like it's definitely robotics for NVIDIA. ([Time 0:06:11](https://share.snipd.com/snip/89e65429-e5b5-4d7f-a21d-b8051837ba86)) - **Nvidia's Pattern: Pivot Toward High-Demand Workloads** - Nvidia has repeatedly adapted its business from gaming to crypto to AI, showing strategic flexibility in chasing high-demand workloads. - Their H100 GPU success demonstrates how aligning hardware to emerging AI needs can create massive market opportunities. Transcript: Jaeden Schafer NVIDIA has played an interesting game, right? Where basically they came out and everyone knew them as, okay, this is to make laptops better or computers slightly better. Some of their hardware, I mean, to the layman, that's what they would expect NVIDIA did. It became, okay, well, this is a kind of a performance thing, things like gamers, or if you need a really hardcore computer setup for maybe doing 3D modeling or something like that, you Going to want nvidia chips nvidia then kind of took the next step which was uh crypto actually crypto mining nvidia had a huge um windfall as kind of the whole crypto craze was taken off And just as crypto was kind of entering what a lot of people called crypto winter um and was slowing down a lot of the big crashes a number of years ago and a lot of people said oh nvidia's Like stock is going to completely tank. There's not so many crypto miners buying this. And all of a sudden, AI popped up and was kind of the next thing that NVIDIA really was poised to create chips for. The H100 being, you know, the very famous one that has powered so many of the AI tools we use today. And they really have been riding a high. ([Time 0:06:22](https://share.snipd.com/snip/fe72f32d-ef4c-4c79-823e-60b732e0444a))