live. It is Thursday morning. It is 9:00 am Pacific time here on the best coast. I am coming to you today. What’s that? The hot coast. Yeah, it’s definitely a lot better today than it’s been. It’s it was over well over a 100 degrees the last couple of days in in Tijana here um south of the border down Mexico way where I fell in love and uh I know it’s not been it’s not been much cooler north of the border either but I’m here in beautiful sunny Tana Mexico and our stalart host Dr. Satia Malik is joining us from from San Diego, California. Indeed. And our guest, where are you coming from today, Natala? All right. Right on. But I like San Diego. When when I walked with Qualcomm, San Diego was greatest city to walk in. Yeah. Yeah. Except for when it’s 100 degrees. It’s It’s not Yeah. It’s not bothering me as much. I I’ve been, you know, going to the beach in the evening and it’s it’s really nice. That sounds awesome. Yeah. Yeah. Okay, let’s see here. Let me pull up our chat. Yeah, we got everybody in the chat. Looks like Zoom’s popping off. Howdy, everybody over on on Zoom. We’ve got Mthagor from Nevada Tahas. Please let us know where you’re joining us from in the chat, wherever you’re joining us from. We love to see the global OpenCV community chiming in here. Before we get started proper on the show, we got Kevin from Bazing Stoke. We’ve got uh Ataliba from uh Lwanda, Angola. We’ve got Stefan Sarnov from Vancouver, Canada. We’ve got Cad Pereira from the other Vancouver. I assume probably Washington. Uh we’ve got Hari from Helsinki, Finland. Um, Zachary Crane says, “Iowa is the new Tijuana.” Am I right? I don’t know about that. There’s not that much corn out here, surprisingly. Um, we’ve got Andy from Escandido. Pretty close as well. Got Andre from the Moscow region. Peter Wong from Sacramento, California. Carlos Nava from Baja, California. Got Bruce Wright from Calgary, Alberta, Canada. as the great Lance Storm would say, says he was in Anaheim last week. Um, when it was hot, it seems much cooler in Calgary. I’d imagine Calgary is usually cooler than Anaheim, but that’s that’s just my my ad hoc guess. Got Kashall from India. I’ve got Stefan De Wolf, excellent last name there, from Belgium. Um, pardon my French. We’ve got a whole bunch of AI assistants. JJ Laarea from New York City and a bunch of other folks. Let’s check over on LinkedIn. Hey there everybody. YouTubetubes and all of that good stuff. We’re live on Twitch, LinkedIn Live, YouTube, Patreon, and Zoom. And we are pleased as punch to talk with you all today. It’s going to be another one of those episodes that I think u I feel like we’ve had a string of episodes recently about stuff that I personally do not know a ton about. And so it’s really exciting for me and I hope it’s exciting for you all out there. We’re going to learn about the Nvidia Isaac sim. I know a little bit about robotic simulation from like I’ve I’ve played around with like the gazebo sim from Ross and stuff, but I’m not definitely not an expert and my understanding is definitely a bit out of date because of how quickly everything moves here. So, we’re very excited to have our guest here to put us under the learning tree for a little bit. I’ve got Arno chiming in from agitated France. We’ve got uh Mizraad from Iran. We’ve got Ankit from India. Love to see it. Thanks so much for joining us today everybody. I think uh now is a good time to get started. Satia, you want to say hi to everybody here? Let’s get started. Hello everybody. Welcome to OpenCV live. NVIDIA Isak SIM is an open-source reference framework for robotic simulation, testing, and synthetic data generation in physically based virtual environments. It’s a powerful system which can ingest data from multiple sources such as computer AED design, unified robot description format or URDF or real world captures. In today’s talk, we welcome Yoshi Cohen to take us along the path from OpenCV to Isaac, showing how OpenCV, CUDA, Gstreamer, and Deepstream fit into a modern physical AI pipeline, and focusing especially on how Isaac is used for robotic simulation, training, and validation. Welcome, Isaac. I’m going to welcome Isaac. Welcome Yoshi. Sorry about that. Yosi, but Isaac could be my second name. Yosi. Okay, it’s your second name after. It could be. It could be the last. All right, that’s that’s good to know. Okay, so first of But first, Satia. But I I am messing up today. Big deal. So, are we just going to cut it and start from scratch now? Go ahead. Right. But as always with me is Phil Nelson who is the director of content and creative. Uh Phil produces the show and if anything goes wrong it is usually Phil’s we’re adding a Yeah, we’re adding a caveat this I I I just messed up a lot but it’s okay. I will I will take responsibility for that one. You’ll you’ll just have to owe me one cuz it’s me. It’s me. It’s phi p. I am your co-host with the co-host the second banana who’s second to none. I’m also your plus one and only. You know it’s Mr. Nelson if you’re nasty. But you, my dear friends, can call me Phil. And I’m here to remind you of a few things we do on every single episode of this year program which you’re currently watching or possibly listening to sometime in the distant future. One of the things we do on this show is a special giveaway to you out there in the audience. Later on in the show, I will be asking a trivia question based on today’s presentation. and the very first person to answer that trivia question correctly will win the OpenCV University course of their choosing. So stay tuned for that in about 40 45 minutes from now. We’re also taking questions from you out there in the audience. Ask those questions wherever you’re watching. We’re live on Twitch, LinkedIn Live, YouTube, Patreon, and of course over on Zoom. Ask those questions. I’ll be monitoring those chats and we’ll bring up the best ones up on screen here later on in the episode. So stay tuned for that. Um it’s going to be a interesting episode today about robotic simulations. Something we have talked about in uh previous episodes, but it’s been quite a while since we did. I think the last one was about gazebo sim from Ross. And so really excited to learn from an expert in the field of the the state-of-the-art for robotic simulation. Uh Yasi. Um, but before we get started with the presentation today, I need to remind you all of one very important fact that Open CV is an open-source software library produced by a nonprofit organization. And as such, we depend on the support of our members and partners here. I’m going to take a moment to thank all of those members and partners at the top of the show. We would like to give our thanks to ARM, Futureway Technologies, Google Summer of Code, Qualcomm, Roofflow, Orbit, the British Machine Vision Association, Devpost, Jet Brains, The Edge AI and Vision Alliance, Open MV, Tang Graham Vision, Amped Software, Intuitivo, Weerun.io, IO, Intrinsic, BuyersDev, and Big Vision because every company needs a big vision. If you want to be as cool as these companies are, you can support OpenCV by going to the QR code I just put on the top of the screen there. Scan that code, check out what OpenCV sponsorship can mean for your organization, give back to software you depend on, and get a bunch of great perks as well, including bringing your stuff in front of the OpenCV audience here on OpenCV Live. We excuse me. Uh, one more bit of business here before we get started is I want to remind everybody about a new offering from OpenCV. An important thing we’re doing as part of our efforts to kind of futureproof OpenCV, make it so uh you can uh we can sustain OpenCV in the future regardless of how quickly everything’s going. Sati, you want to tell people a little bit about OpenCV enterprise? Yeah. So, OpenCV enterprise is OpenCV’s uh support system for enterprises, right? So, for example, if you are a large company uh and you have a system built on top of OpenCV, but it is not the latest one, right? Let’s say it’s not based on OpenCV 5, you are using uh OpenCV 4, OpenCV 3 and even OpenCV 2. In that case, uh you’re exposing yourself to a lot of security risks, right? So there are these security patches that come to us and now it’s increasing because of all the you know AI is not able not just able to recognize that these are threats but there are also uh threats that have been exposed because we are you know security holes are exposed because of uh AI and people are using those holes to do nefarious things right so we want to make sure that we are supporting our existing user base and of course OpenCV remains free OpenCV 5 Four, all of these versions remain free. Five is also we test for security uh problems and we fix them uh regularly. However, from for previous versions of OpenCV 3, 4, 2, etc., it is simply impossible for us to supported without funding. So, this is our way of supporting uh the older versions of OpenCV. So if you have built you or your company have built uh your software around OpenCV uh and you want to support OpenCV this is a great uh mechanism. So with OpenCV enterprise we have three uh tiers and it starts at 150,000 which is you know basic support for one version of OpenCV on one device and then it goes uh further up and the higher tier is basically we will do custom work for you and we will support you know we’ll add multiple things on our uh continuous integration system if uh your device needs to be added and tested for every new release as well as old releases. So that’s basically OpenCV Enterprise and if you uh are you know from a company please talk to your managers uh and if you’re a decision maker please please talk to us. You can email us at uh hellocv.org and we’ll be able to uh move forward from there. Yeah. Or you can scan that QR code that’s up on the screen, fill out that little quick little form and talk directly with the team. We are await your emails and we’re looking forward to chatting with you about how OpenCV Enterprise can help your enterprise. So, we have a Oh, let me hide this here. All right. Welcome, Yassi. Uh, happy to have you here on the show. Um, I think you’re muted here. Let’s unmute you. Yeah, there you are. Why don’t you say hi to the folks? uh let them let them know who you are and and we’ll get started with the presentation. So um I’m Yosi. I’m doing robotics uh tele operation. We started with tele operation before that computer vision and uh currently uh we are doing uh a lot of simulation work. Uh the presentation today uh is about simulation and robotics but also check the evolution of uh the computer vision or the optimization over the years meaning it connects it sees the optimization library such as IP and TSP, MATLI and other libraries over the years as they evolved into the Isaac cross and the Isaac sim. So we are not just concentrating on the Isaac SIM but also seeing it as a lay platform uh which include connection to lower layers meaning that Isaac Sim connects or relates to the ROS but it also relates to the deep stream and the deep stream could relates to the open CV and we see all the evaluation over the years and then connect it to the stack of tools which Nvidia currently uh provides which not include just the Isaac cross or the Isakim h it also include other many tools like world models like cosmos reinforced learning gym or lab and other tools I’ll talk about um one of the nice tools tools that was recently um exposed was Nvidia Grail. Uh one of the one of the uses of Nvidia Grail is how to use regular human motion when training robot because we have a lot of human motion captured but we don’t have a lot of uh robotic motion. Also the human motion is much more natural and it’s easier to learn from human motion than from robotic motion. But the question is how do I turn my motion my arm which could be in different length relating to a robot and to a robotic motion and in video grails closes this gap. I’ll show an example of Nvidia Grail. Also, we will discuss um how we train it. We could train it in a batch, fine-tune it or fully train a model which requires a much more sample and a different process for training uh the reinforced learning uh process in which the model talks to the simulation and we’ll see how to do that. Um, any questions or any questions or something that the audience would like to see in this demonstration? I think we’re just ready to go. They’re just excited to hear what you got for them. Okay, just bring up your screen share. Yeah, go for it. Just turned you on. We got a nice drasta effect here. Love to see it. Do you see it now? I do. So, can you is there anybody can see that full screen? Oh, yes. Can you see that? Perfect. Looks great. Okay. So, uh as I said, I’m not just talking about uh about the Isaac or the Nvidia tools, but also on the entire uh evolution of those tools. So, I’ll talk about low-level optimization. what we started with uh starting from 95 at least for me at that time I was at Intel and uh we created the Intel created the MMX and that was the first time I was aware of SIM command of optimization and how to uh optimize matrix multiplexion and later video into processing on a CPU. At that time you have to had a hardware cards to compress or decompress video and that was uh very innovative. Uh afterwards we’ll talk about crossplatform computer vision optimization. What are the different tools for crossplatform computer optimization and the standards? uh will start with OpenCV and then OpenCV over CUDA over ARM. Uh at Qualcomm I I talked to companies and did some dev relations for the first CV. It was 13 or 14 years ago. Yeah. So at that time there was no optimized GPU like and hexagon which is Snapdragon DSP optimized computer vision libraries and that was very innovative and it was used in several projects. Um we’ll talk about uh current video and sensor processing. Can I can I interrupt just uh so uh fast CV uh we actually support it as a backend now so people can write code in uh fast CV uh sorry uh people can write code in open CV and have it accelerated using fast CV we did a project yeah didn’t know that actually Qualcomm invested a lot of effort so it wouldn’t be exactly like open CV so there would not be any IP issues. Yeah. Yeah. But uh that also leads to problem that people have to learn a new thing now and fast. Yes. There was a lot of a lot of porting that people were required if you had an OpenCV code and then to port it to Fest TV. On the other end, it was it was working on iPhones and on Androids and it was great. Yeah. Um afterwards we’ll talk about video and sensor processing pipelines. What are the pipelines we have today to do processing? uh pipelines could be open CV as is a sort of a pipeline but uh it’s more like deep stream and draws which are more deepstream is certainly a pipeline and draws could be viewed as a pipeline but sometimes more of publish subscribe a publish subscribe network although that in many times in many cases the network is connected like a processing pipe and then we talk about robotic simulation. What are the simulation tools we have today? You you mentioned Gazibu which was the first tool I I learned robotic simulation with and got stuck in that first exercise when the arm doesn’t work because of a bug and then to fix it. if anybody tried that. Beside Gazibu, there are other tools that could have GPU uh acceleration that has real time uh rendering uh rate tracing rendering which looks much more um much more realistic. Uh so beside Gazibu we could talk about Mujuku which is older and physical. H we’ll talk about Isaac Sim and about Newton which is kind of like the the the merging of both Mjuku and Isaac SIM and then we’ll talk about how we train robot with reinforced learning and the importance of fault injection and recovery. We’ll talk a little bit about the different models, the VAS, the VLMs and the world models and how we merge models like VLA and VLM into single model like the group. So I’ll keep it short and if you have question don’t wait till the end just ask me. That’s right. So we we’ll start with the evolution. Um I’ll I’ll start with the mid 1990s. We have the uh seemed vectorization and optimization on MMX. This that’s for me was the starting point. erh libraries like the Intel IP. Intel IP beside optimization included a crossplatform and the crossplatform didn’t mean that it worked also on AMD uh but that uh it it could be ported between different uh Intel processors like you don’t have to write your code if you use the IP FFT you don’t have to rewrite it uh from the generation of the MME mix to the generation of the S SE or S SSE2 or three or four. H it was already you just uh replace the library it auto detected the processor and then uh your code stayed the same and you added the acceleration the same library could work on different processors. Yeah. In fact uh OpenCV back then uh when OpenCV was released the C API it directly uh worked uh exposed the IPL image right so IP is Intel’s performance primitives and then uh um OpenCV we used this format called IPL image right that was the standard image format uh and u you know that that was the first image format format until we moved to uh C++ where we use started using the Mac format initially it was the uh IPL image. Yeah. Mhm. Uh in the mid 2000 there were other uh optimization for other processors like the shark DSP. The most common one which was at that time TI was the same as Nvidia today at least for matrix multiplexction processing and video processing light computer vision and audio processing on the DSP mostly the TI64 processor and the V and later the Dainci processors. So uh they have a matt library optimized and also some computer vision uh primitives inside that and that was optimized for a specific hardware. In the 2010s, erh the there there was a movement towards erh both mobile platforms and GPU platforms and then we had the Open CV over ARM over CUDA and over Fest CV. Um if someone remember and if you look today on on Jetson platform they are still named L40 Linux for Tagra and the first Tagra was actually a mobile processor by Nvidia. M so let’s say that the the forefathers of the Jetson were actually mobile processors and at the time there were only three companies there were about three companies in the in the mobile processor TI Qualcomm and and Nvidia and I think that in 2014 there was only Qualcomm they were pretty much out afterwards were uh in 2015 and later Nvidia issued the um uh the Jetson family. H the Jetson T2 and in 2020 I think there was a Jetson Nano Nixavia family and later the Orin and to family and soon there would be the TX 2000 and 3000 which adds new smaller members like the NX Energy to the to family uh for those uh those uh processors are sockbased processor which has many acceleration units like DA for neural network acceleration video acceleration computer vision acceleration
like the open vx uh standard which they are not actually open standard. They were in the past. Um and there is a need when you uh optimize for them and when you write code for them to address all those um all those units and to use them correctly and uh the platform to do it is deepream. Deepstream is tied to OpenCV and you could uh use Deepstream and Open CV directly without uh any memory copy and hardly any buffer conversion. Um so those are I’ll talk a little bit about the deepstream and how to use it as the pipeline and then uh the robotic simulation um in the 2020 the Isaac Isaac is basically an enhanced ro by Nvidia which relies on deepstream. You could you could embed deepream inside of Isaac. Isaac Sim is basically a robotic simulation layer which knows the ROSS and run over uh Cosmos uh Nvidia simulation engine, the Isaac lab or the Isaac gym today uh which enables running the simulation in reinforced learning environment and the Newton as I said. So those are uh the um the different uh erh technologies or SDKs I uh previously reviewed. Um it’s not very exact but I could say that OpenCV is a in in the intersection of low-level optimization of libraries if you know the NASA or the IP or the TI MLY because they are all basic optimization uh of matrix multiplexion mainly. Yeah, the OpenCV that’s that’s what it does. Okay, so it it could be assigned to low-level optimization. It could be assigned also to uh crossplatform computer vision or crossplatform optimization with OpenCV over ARM over CUDA uh standard like the open VX partially the Open CL which could also reside in low-level optimization or par parallelization and the FET CV open CV could also be considered as part of the pipeline processing. which include pipeline processing like zeros which as I said is is more a published subscribe architecture just that it’s usually for video and computer vision organized like a pipe the deep stream which is certainly organized as a pipe and the isacross which is let’s say a ros with enhanced plugins by Nvidia Just like the deep stream is a Gstreamer with enhanced plugins by Nvidia. So that’s the relations that I wanted to show and see how everything is connected. Well, you already know what OpenCV is. Basically a matrix uh manipulation, image processing. Um it does have uh some uh option to run neural network with ONX uh with ONX architecture using the DNN. It runs on x86 ARM and GPUs like the CUDA and it’s a pluggable architecture. And if there’s something missing, not that there are anything that could be missing in in such a a perfect library, but if you want to customize something, but if you want to customize something, you could write your own plug-in for that. That’s very good in case you you want to connect something and you want to keep the architecture the same. We discussed that already. Um that’s basically the first part about the evolution. Any questions about that? No. the relation between Deepstream and Gstream or Isaac Ross and Ross and what are the enhancement between Isaac Ross and Ross. Okay, so we we we talked about optimization or or a process of optimization and now we have to think how we create a processing. Usually when we think about processing uh we have a source on on one end and then a result on the other. The result could be a movement of a robot. It could be a a box detector coordinates of the car I would like to follow. or the amount of people traversing an entrance or a line I would like to see in a security camera. Okay. So, so those are the processing and processing in computer vision is usually a camera or a camera source with RTSP compressed camera over RTP RTSP and then I have to uh process it is processing could be an ISP processing
debiring or decompressing the stream and and uh running some computer vision on it. In the past, the computer vision could have been uh some kind of uh filtering like Viola Jones filtering to detect faces and later we started using a neural network for that uh or anal network for car detection. Uh we could do that in OpenCV and we could do that with Deepstream. Uh Deepstream will work only on Jetson. It’s highly optimized to Jetson. Uh however, uh Gstreamer with plugs would work on any processors. uh in the past I use uh my own deepstream architecture when uh creating um uh a processing pipe for Raspberry Pi and that’s I’ll I’ll get into deep stream because that’s important and most of the processing computer vision platform today are Jetson based so I think it’s important important to learn the deep stream although that recently Nvidia issued the deepstream 7 for the tour and the deepstream 7.2 two for the NX and AGX orin and those libraries have skills that enables a non deepstream user to create deepstream pipe using agents which learn
so now you don’t have to learn let’s say a week or two or more uh to be proficient with deepstream you get into your cursor or clude h You open the DeepStream 7 Plus and you could create you could easily create a Deepstream pipe. The same goes for Isaac Cross by the way, but it’s very recent only in the last months. So, try it. At least I I tried for the deep stream and well it works and uh you are welcome to try. So Deepstream Ross as we said is a robotic communication library. It’s not an operating system. It’s not a robotic operating system. I don’t know why call it that way. It’s basically a robotic communication system. Erh, one of the things, one of the one of the advantages of using Ross, sometimes Ross does not have all the drivers to all the cameras and uh the control we have using Greamer and Deepstream over uh all all sorts of plugins and hardware is usually better than that of the ROSS. So you you could uh create a ROS pipeline connecting a camera to some uh uh processing neural network or computer vision. H but it’s better and uh there would be less bugs for different hardware and situation if you’ll use deepstream but how could I use deepream inside of Ross? I can’t. What Isaacross enables is using the deep stream as a ROS plug-in inside the Isaac cross. So h all the processing of the camera could be done in the deep stream. The deep stream pipe would issue some kind of a ros message. we could create a ROSS broker as the sync of the deep stream pipe and then uh connect it to the to the output of the ROSS pipeline. So we uh integrate a deepstream pipeline inside the ROS pipeline and that’s very comfortable when processing
uh deepstream. What we usually do is cameras RTSP row input. Um sometimes we use the Argus ISP which is a hardware ISP processors. If the camera does not have uh an ISP or you would like to save all the ISP or share ISP between cameras. Hardware decoder and the hardware processing and acceleration hardware. The streams is a very interesting uh plug-in. Um I had to create it from scratch when I used the open veno on Intel tablets. uh because it enables us to share the same model between different pipes between different cameras. Erh, we could use the same detector between different cameras, but we can’t use the same tracker because the cameras, let’s say I have four cameras, I can’t connect four cameras to the same tracker. Nvidia developed a special um component that multiplexes the camera and de multiplexes them between the tracker and so I could use a single uh model and then separated as part of my pipeline seamlessly and that’s that was very nice and when I had to create when I wanted to do the same for for Intel I couldn’t and I had to write something like the stream books on my Uh so that’s a very nice uh uh plug-in. Um also uh Nvidia is uh inference. Nvidia deepstream could run inference on the CPU on the GPU and on a hardware acceleration unit inside the Jetson which is called the DA deep learning acceleration. uh it is compatible with other architecture or separate uh hardware acceleration for neural network uh if you remember the Google corral or the Helio and I’m sure there are other hardware accelerators by other companies could you mention some
okay never Right. So, uh Ross uh Ross architecture um if we’ll if we’ll talk about different processing architecture uh in direct show on Windows uh we have h filters and pins. Uh filter is a processing and pin is a input and output. Um, in in Gstreamer we have peds and I think the name for the filters are plugins. Uh, in Ross the processing is done on nodes and it outputs uh messages in topics. So anyone could uh subscribe to a topic. I could subscribe to the row input of the camera. The communication uh there is a DDS communication which enables me also to share a single architecture between different computers. Uh that’s very nice architecturally yet uh it’s not something I use for real time. There are some ROS architectures like the H Zeno or the low latency but there are still it’s not the best way to uh connect different computers over low bit rate networks or latency sensitive network. uh will we have the sensor and actuator messages like the sensors could be a camera, it could be an IMU or a robotic actuator. Um that’s it. And now we’ll get to the interesting part uh which is the Isaac cross. We talked about deepstream. Dipstream is basically the Linux Greamer and Nvidia added some uh some plugins that will uh enable better processing on its specific hardware like the streams, like the uh decoder hardware encoder or decoder or um a specialized or or even the NVDCF tracker or the general tracker architecture that it provides and also the message broker if I want to take that pipe into uh the cloud into Kafka. Okay, so that was deepream as a layer of additional plugins over the Gstreamer. Why I discussed it when I’m talking about is cross because is cross is the same we have the basic Ross nodes and is cross uh provides additional additional nodes and capabilities to that architecture example slam or cu slam which is the accelerated slam and um leader segmentation which is a very processing consuming node and it’s better to do it on a GPU so they add it um and the use of all their accelerator and SDKs as part of the ROS. So is cross is basically ROS+. Okay. Yeah. So the this object detection etc these are u uh userdefined or they out of the box they come with some object detection and pose estimation capabilities. Okay. Um I’ll I’ll explain that. Um there is a an NV infer plugin. Mhm. The NV infair plugin is basically an empty gstreamer plugin which says which said the network I’m using is on this path and the way I want to use it and connect it is according to this configuration file which is in there. So by reading the configuration file I know which network that sorry the Jetson knows automatically which network to use and how to connect it and how to how to even scale the video. Let’s say I’m uh inserting 1080p uh video and my detector is 640 by 640 yolo. They are not connected. the deep stream automatically scales the video but then I have to tell him how to scale the video. Should I do it linear, billinear or other methods depending on the quality and the processing tradeoff. So um when when I wanted to do it uh uh in a in a server I I had a server in the past before deepream. I created a Gstreamer plug-in and plugged my specific uh detector inside it. Uh but I did a fixed plug-in with a single network just for me, right? Nvidia does a much more flexible solution and enable enable you by editing the config file to uh uh to use any sort of network with a lot of pre-processing post-processing uh that is required by the network right there were a lot of problem when using and even bugs till two years ago even less when using YOLO in the NV infer and we really didn’t know why and was a lot of problem between Nvidia and clients and us in the middle and in the end they found the bug and fix it if someone really wants today he could use directly the ultralytic sporting to to Owen and and Nvidia there is now uh porting from Ultralytics and not just from Nvidia, right? Quick quick uh note guys. Um I have a hard stop here in about 20 25 minutes. So if we could uh accelerate, you know, speed it up a little bit, that’d be fantastic. Yes, accelerate it. Perfect. Yeah. Okay. Okay. I I I like to accelerate stuff. Um so we’ll we’ll okay we we finished with the Isaac cross and let’s talk about simulation. Um okay simulation um we discussed the Gazibu you know it it’s a Ross oriented simulation zujuku uh it’s a physical simulation engine by Google it’s very good but the rendering of the images could be better so if we would like high fidelity visual simulation we’ll go to Isaac is good enough with its physics engine for simulating stuff physical. So, I’ll get the pressure when I’m pressuring an object. Yet, yet uh it’s good for simple grippers and hard objects. For deformable objects and tactile robotic hands, uh, Nvidia um recommends the Newton which has better h solvers for softer deformable object and for tactile hands. So the as I said there are four uh used uh simulation general simulation yet many robotic companies like Phanuk or ABB h have their own simulation studio something they created over the years. So when using their robots uh usually you’ll not use any of those uh simulators uh you’ll use the company simulators. Uh some of those companies are already porting to Isaac Sim. Uh the advantages of Isaac SIM um besides the highfidelity visual simulation and real time rate tracing is the connection to reinforced learning tools and also the connection to other uh simulators using a bridge for example real and I seen this week I think even today uh that a very very required bridge like the Unity to Isaac sim to to the Unity to Cosmos or Unity to Isaac sim was released this week. So it’s great news because it was something that uh was missing. Many simulation were done on uh on Unity and you couldn’t easily port them to the Isax. Um, one important uh, it’s actually not not a full bridge. It’s more mostly it’s more of a uh, URDF you are porting the assets.
Okay. Uh, in Isaac SIM uh, when we build the simulation environment, we have to take the assets. The assets are are the objects we are using and the environment, the desk, the factory environment, the different things I have to uh to take in the environment if it’s usually a pick and place scenario. There are a lot of problems when dealing with objects. Sometimes the entire simulation would get stuck just because the collider layer collider is the layer which wraps the object to provide the pressure because the object has let’s say some kind of a mesh but that mesh doesn’t create its behavior. You have to define also the collider around the object and around the gripper and there’s always problems in that. Beside defining the the different assets, you have to sometimes define the forces and the engines as part of the asset. For example, if the robot is an engine, you have to define its strength and its torque. All those that’s a long tedious process. uh also taking the entire digital uh twin environment and sampling it is a tedious environment and Nvidia recently provided tools that doesn’t fully automate this process but really assist in accelerating.
Um, OpenUSD is a standard for simulators and it was developed by Pixar. Very interesting. And now the Cosmos and Isaac when they are defining objects they are defining it using the Open USD.
I’ll skip that. Okay. So we could create a simulation of a robot doing a pick and place of an object. That’s the the most common task and the task that robots are great usually great in. And that would be used to fine-tune a model. Meaning that if we take an existing trained model, for example, small VLA, PI0 or Cosmos Edge Droid by Nvidia and just tune it a little bit. We could tune it without reinforced learning. Just showing the video with the robotics metadata many times and train it like we train a detector. That would work. So the simulation could run as a batch output a few hundred actions or episodes. It’s called episodes because we don’t train on an image. we train on the entire video on the episode and then insert those as a batch to the trainer. However, when we train the model from scratch or when we want to train it beside beside a a very specific task, we have to train it, not to tune it. And for training we are required to have uh the the robotic model connected directly to the simulation because the camera because when the robot moves the simulation have to show him the camera how it moves and then the robot learns what to do. It does something it got some reinforcement it’s bad it’s good and then he knows what to do at the end of the action. So that’s the way it learns. It’s basically a software in the loop simulation. H it’s much more complex and it takes a lot more resources and time to train a model this way. Usually the recommendation is to rely on existing uh models. Um as I said the models are uh VA’s VLM and world models and in many cases we use a VA which creates the action output. The VA is a vision language. We know LLM large language models. Yeah we know VLMs video language models and VLA are basically video language action. So it outputs the tokens of the actions the robot needs to do. Yeah. So I I like to say that uh people who get confused that don’t worry about VLS they are basically VLMs which are being uh you know uh you just the action part is basically more structured but essentially they are VLMs right um more or less we could say that um the action part is more structur the output is more structured uh but uh if you uh if you let go of that part right uh the output is more structured because it’s saying that okay you have to take this action instead of uh general free form text sure if you understand one you you broadly understand the other yeah yes is that fair to say yeah mhm one of the models as I said it’s pi zero and small va a model issued by nvidia the latest is groot groot is a model is basically a va model uh but with reasoning uh if you if we are using a PI zero or small VA small VA has hardly any it has a very small video encoder and it doesn’t have good reasoning it doesn’t know what it does it could hardly detect when there was a fail uh and it doesn’t understand things very well right so if we want to use it better uh we would use it with a VLM for reasoning. The reasoning model would tell him what to do and if there is any error. However, the reasoning model, let’s say a Quen or even a Cosmos is much more heavier than the small VLA. The the advantages of small VA is that it could run on a Jetson NX. We can’t run a large quen on NX or run them together. So we can’t do that. Pi zero has better h video encoder but usually we would like to run it with the Quen as Quen is kind of a a master to it. detects the errors, tell him how to do the recovery.
And uh the Groot is something like a a shared video encoder between the VA and the VLM. So that’s the Groot 17, the latest Groot model by Nvidia issu issued recently. And you could tune it, train it. It’s harder. um world models work differently. They don’t output a specific action. They we could use world models for any use. We could use them as detectors, as segmentors, as um as imageto image transformer and uh we could use them also for action. Uh, one of those models is a specific Cosmos model called Cosmos Edge Droid. Cosmos Edge meaning it could run on AGX or because the Cosmos Nano requires RTX 6000 that the nano version of Cosmos. So, Edge is much much smaller and would run on the AGX and Erh. So you could uh use the the Cosmos uh droid edge to run a robot actually a full humanoid also an arm but that would be a waste using such a large model just to operate a robotic arm. So we discussed that uh the Newton uh Newton uses it’s kind of like merging the physics of the Mjuku solver with the ray tracing erh speed and capabilities of the Nvidia is sim. Uh it’s relatively new in the market. I haven’t noticed many companies using it, but I’m sure that it could really contribute to um to humanoid simulation mostly humanoid and tactile hands.
How are we with the times? Got about uh 10 minutes. Okay, that’s it. We’ll have some time for questions. So, that’s basically it. I’ll I’ll cut it. Accelerate. Newton uh is developed by Nvidia, Deep Mind and um they all got together to uh build Newton, right? Is that correct? Yes. Yes. That’s very nice of them. Yeah. And Disney research also I think. Yeah. I just Yeah. And Disney research was also involved in Yes. Yes. Disney has a lot of robots. If you notice, there were a lot of release in GTC Taiwan of the Disney little robot walking back like that duck robot they issued. They have right. The duck robot is from Hugging Face. Was Disney also involved? I think it was Disney. No. No. No. I saw I saw a robot that could go in the water and also Oh, that’s fly climb climb walls also by Disney. And I know Yeah, that was a D Disney research project. Different from the uh small robot from Hugging Face. Yeah. Yeah. Micro duck. Okay. Yeah, that’s micro duck. That’s a different one. Yeah. Okay. Too many dots. Mhm. And um so we talked about uh how to train a robot. Basically, take a pre-trained robot model. Yes, the pre-trained robot model would not know how to do a specific task. If I want to move this object from one side to another or to put it in a box, uh, a non tuned model for that specific task would not work well. So I have to tune it to show it how to do it something like 100 times. Baby would learn faster, but that’s a robotic mind and we not there yet. So I have to train it. If you saw for example the robot hand when we we use it in hackathons here we are taking the one hand that I could control takes the object move it to the side do it a 100 times or let the student do it 100 times and then they hate you but afterwards they train an agx with a pi zero and then the robot lands and everybody ate a very happy and they don’t hate you anymore. So that’s you take the pre-trained model, you record the episodes of the training. Recording is done in the robot format which is now robot hugging face and video the same and then you train that model usually by zero. Um uh that’s how you train it. If you want to train using reinforced learning, I strongly recommend otherwise it’s it’s become very complex to collect all the data and send it to the model and get the model output and create all this software in this software in the loop glue to glue the model and the simulation and to make everything work. That’s a lot of work and the ISA club environment saved that from us. Um and I brought an example we did actually we did it on the on the first day that Nvidia issued the grail. uh we wanted to play with it and we wanted to see if uh it is it change if I’ll change for example the human being or its clothes or the color of the clothes does it affect the robot and the answer is of course no instead of just entering a video we created a prompt which changes the human to be sure that different people the the model would react the name and then we see the model the 3D uh skeleton in the body of the human and then the G1 skeleton it actually the work between this stage and this stage was done by us and what uh the the grail does it takes the 3D skeleton not the 3D human but the 3D 3D skeleton of that human and pass it to the G1. Okay, very nice. You see that we change we change the colors of the of the of the clothes and it still worked. And that was the grail. I think that that covers a very short um I’ll I’ll skip that uh since we don’t have time. So uh what we discussed we discussed runtime optimization uh perception perception pipelines action how we do action and robotic models and simulation and then the training we have only an hour so so I hope I I tried we didn’t dive deep into anything but I tried to cover a thin large layer of of the different tools we have this is very very useful because definitely I mean you covered enough and uh you know there are pointers that people can go and look at right for uh you know digging deep into it. So from an overview view perspective I think it was really uh very very helpful. Thanks. Yeah, I agree. Um I I definitely learned something here and we’ve got some next steps for for learning for folks out there in the audience. Um some some good comments about folks uh saying the same thing. So happy to hear that. Um, we’ll take questions here in just a sec. I’m going to do our trivia giveaway. Now, folks out there in the audience, if you need a reminder, one of the things we do on this show is give away something to you out there in the audience. I am about to ask a trivia question based on today’s presentation. And the very first person to answer that question correctly will win the OpenCV University course of their choosing. You can see what courses are on offer by going to opencv.org/university.
If you have won in the last couple of months, don’t answer and let somebody else get a chance to win and you can put those answers wherever you’re watching in the chat and I’ll pick the very first one. So, get ready to answer. On today’s program, we talked a little bit about the history of uh simulation and and mobile chipsets and a lot of interesting stuff, including an important release. Way back in the 1990s, Intel released IP. What does IP stand for? Answer, viewers in the chat. What does IP stand for? I’ll give you a hint. The first one is not Intel. The I is not Intel.
Answer in the chat. H. We got some folks answering, but they are not correct yet. There’s still a chance to win here. No, it is not informative path planning, but uh good guess there, Alberto. And I think we have Yes, we have a wiener here over on Yes, over on YouTube. The uh answer we were looking for there is Intel IP stands for say it with me now. integrated performance primitives because I for the longest I in fact learned about it today because I have been calling it Intel Intel performance primitives. Yeah. See I thought I I had to double check myself and and search because I was like wait a minute because it said Intel IDP and I was like wait would it be Intel Intel performance primitives. All right. So, our winner today was uh Lowry Time on uh integrated performance pros over on the YouTubes. Uh congratulations to our winner, Lowry Time. Please send one email to me. That’s Philopencv.org with the name of the course you would like and we’ll make sure you get that unlocked in your dashboard. That’s philopencv.org.
Um, we’ve got about five minutes left here, I think, for a little less than that for questions. Let me do a quick sweep of our questions here. Um, so, uh, Gassi, um, I have actually have a question myself, which is, so, uh, you said you released you created this demo the day of the drop, right? Um, have you seen anything um have you seen anything come out since the drop that has really like kind of blown your mind that you’re you’re like, “Oh, I didn’t even think of people doing that yet with with this new tech from Nvidia with in the in the last months LinkedIn was full of posts of people using the Astra model uh to program all sorts of very very nice robotic h solutions. So even people that has hardly any experience or no experience with robotics could do great things with the robotics and robotic skill that were when using the Astra model the Astro 6 model. So um looked at LinkedIn at least for me I see three four every days there are many great things for example porting between one day or less uh programming of doing a porting between a proprietary ABB simulator and Isaac that’s something that could take you a year in the past and people are just doing it like that yeah the speed at which things are operating here is is very uh it it can make your head spin as they say. Um another question here from uh the chat. So you uh again about that that demo. People are interested in it. Um about how long did it take the team to put together that that demo on day one. Actually I didn’t do it. One of my engineers did it and it took him something like half a day or less. Wow. That’s that’s wild. Totally wild. It uses several models because it’s not just the the the grill does only the 3D skeleton and he uses a prompt which created the video which changes the clothes and then a 3D modeling of the human and then 3D skeleton extraction and then from the 3D skeleton mapping of that 3D skeleton using real only to the G1. So at the time only G1 robot was supported. Now there are other robots supported. I think also the H2 Nvidia uh discussed a cooperation they had in in the last GPC Taipei uh with unitry. So the uh Groot model is oriented towards the H2 and the Grail model at the past was uh for G1. The past was two three months ago. Okay. Yeah. Uh again the speed is uh is pretty staggering. Um I think we we got all the questions. I think your presentation was was walked a fine line of uh being just in-depth enough to answer some of the obvious questions. Maybe folks will have some after the show, after they think about it, let it stew for a little bit. And if you do, you can send those questions to us. Newsletter opencv.org. We’ll pass them along to Yasi here. Send it to me. Yeah, send it to send it to our our guest here. How can folks uh reach out to you if they want to follow you on online or send you an email, Yasi? Uh LinkedIn. Okay. My LinkedIn page at Gangelia. You search for it. I think there’s just only one Yosi in Gangelia. Okay. Okay. So far. So far. So far. Um, that’s why I put my stake in the ground for Phil at OpenCV. Who knows how many fills there may be in the future. Um, all right. I think that’s all the time we have for today. Satia, you want to send us home? Yeah. Thank you so much, Yoshi. Uh, and, uh, it was it was an excellent presentation. I learned a lot myself. I actually took some screenshots also so that I can follow up. Obviously, we have the whole video that people will be able to follow up on. Uh and thank you Phil for organizing this show. Uh everything that you could do and perfectly. Still time for me to screw up, right? And thanks thanks our audience for uh being with us every Thursday. Uh we are really grateful for that. Thanks. Thanks guys. Yes, that’s absolutely true. Thanks so much everybody for joining us on this beautiful Thursday. We hope you have a great one out there. Take care of yourselves. Take care of somebody else if you can and have a great day wherever you may be. We’ll catch you next time. Same bat time, same bat channel, 9:00 am Pacific, Thursday. Adios. Thank you.