am. And you know what that means? It’s time for OpenCV Live. We have another action-packed, fun-filled episode for you today. We’re going to be talking about navigating drones in places where GPS is just blocked, denied. Um, so it’s going to be really interesting. This is especially a important topic here because of you know things like uh disaster recovery when uh the increasing number of uh natural disasters you know floods, earthquakes, hurricanes um being able to locate and uh and help people is very difficult if you’re reliant upon say things like GPS which can be uh you know heavily influenced by natural events. So, uh, we’re excited to have Aleandro here to talk about it. And we’re excited to have Satia back from the void, whatever whatever that was. When when it when it reapplies your your your smart background removal, it kind of does this like, right, pretty cool. Um, but please uh as you join us today, let us know where you’re joining us from, wherever you’re joining us from. We are live on YouTube, LinkedIn Live video, Twitch, and on our Patreon account. Um, I uh then let us let us know where you’re joining and also let us know who you got in in the World Cup here. Uh, I think I’m all in on Spain, obviously. You know, I think you’d once once Mexico is gone, you just go with the Spanish speaking countries. Like that’s the solidarity. First it’s first and of course first it’s first it’s Latam, then it’s Spain. Like Spain’s at the bottom of the of the pecking order. But well then Argentina is should be your choice. Ah, I mean they’re they’re like the Yankees. I can’t root for the Yankees like um but yeah, we’ve got uh uh Petro from Saudi Arabia. We’ve got Vernon from Glenn Allen, Virginia. We’ve got Masimo from Florence, Italy. We’ve got uh Doug from Pompano Beach, Florida. Muhammad from Mali. We’ve got Hussein says, “Hi from Pakistan.” Hi, Hussein. We’ve got Eliia from France. Hello, France. We’ve got Meduse from Italy. Uh, we’ve got South Africa in the house. We’ve got Mario from Mexico. Ola, amigo. That’s going to be a fun one. We’ve got Nevada Tahas chiming in. I’m told that’s how they pronounce it. There you got Andy Andy Matthews from England. Uh, bloody England. Um, I’m glad that they lost, but I’m I’m sorry for your loss. I I I contain multitudes. I can see your sorryiness. Yes. Yes. I again, it’s England is sort of like the Yankees, I feel. that, you know, the in baseball the best case scenario is that the Yankees make it to like the second or third round of the playoffs now and then lose. That’s that’s the best case scenario for baseball. I think with uh with international football as well, I think the best case scenario is that England maybe gets to the quarters and then loses. Like that’s that’s the best that’s the best case for everybody. Um yes, got a Neil from Neil. does the uh you know the way England played yesterday. Argentina uh it went so boring right after the uh after they scored the goal they just became so defensive and Argentina does something similar. So I never want them to be uh leading. They play their best when they are trailing. Uh the best comes out. They they like to. So I I’ve noticed this as well and I’m I’m not like an expert of course. Um but I one of the things I actually really did I really do appreciate about Argentina and specifically Messi’s game is that like it’s so casual, right? Like the he’s he’s just sitting there with like his hands in his pockets. He’s sort of like Columbombo, the old TV detective where like he’s just kind of sitting there, you know, looking around with his hands in his pockets and then like 2 seconds later he’s halfway across the field scoring a goal, right? Like he he’s so good at just making you forget that the greatest player in the world is right there and you’re playing against him right now and then there he is. He’s doing his thing. Um, but they do like to kind of sandbag for 85 minutes or so. I noticed like they’re they’re they’re kind of waiting and waiting and waiting and then bam bam bam and so it’s it’s been a it’s been a pretty good World Cup. What I mean what’s your what’s your opinion on like the the games and stuff so far especially to previous years before we get started? No, I think uh it’s it’s the best World Cup if you think about it because um first of all we thought that if you increase the number of countries the quality of the game will go down. I don’t think that’s the case. It was very entertaining. I cannot really say about the quality of the game because it has gone up so much. Um, but if you look the the matches were entertaining. The seats were pretty much full, right? Even though people thought that that would not happen. Yeah, I was worried about that. Yeah, but but I mean I’ve been to one of these games and it was completely packed, right? Within within 5% of uh the capacity. So yeah, it’s uh and and the quality of the game is very high, right? And the top teams also made it which is you know I would have like loved to have an upset but uh the top teams uh made it to the semi-finals. So I I’m loving it. It’s it’s really uh one of the best World Cups and especially for me I’m I’m enjoying it with my with my kids and it’s it’s a blast. Yeah, really great to hear. Um, I think, let’s see, we’ve got Carlos says, uh, “Hugs from horrible, terrible, extreme hot weather in Baja.” Yeah, it’s like 30° C in in Baja California right now. Um, Natalya and her dogs are all like just Oh, wow. sweltering down there. Luckily, I mean, they got a couple mini splits in the house, so I’m sure the AC will be on today. Um, keep it keep it set. You know, I don’t know what’s the version of 78 is. I’ll I’ll mom donny you guys and tell you to keep it at 78.
Anyway, uh now’s a great time to get started. Satia, let’s go. All right, let’s get started. Picked the perfect time to glitch. Hello everybody. Welcome to Can you Can you dump it and start again? Okay, I’ll start again. Hello everybody. Welcome to OpenCV Live. Autonomous drones only fly as well as they can figure out where they are. drop into a forest or a disaster zone and that gets a lot harder. The GPS signals they normally lean on just aren’t there. This week we will talk about uh autonomous drones with uh research engineers Alessandro Savilio Saviolo. Sorry, I I messed up your uh last name, but you’ll pronounce it right. About hunt, a navigation approach built for exactly the places where the usual solution fall apart. But first, we will as always with me as always is Phil Nelson, OpenCV’s director of content and creative. Phil produces the show. If anything goes wrong, it is his fault, including all the fumbles I made during this presentation. So, let’s uh let’s talk about uh just denied uh autonomous drones. Yes. Good morning, Satia. Welcome, Alessandro. This is going to be a fun episode. It’s me. It’s me. It’s phi lip. I am your co-host with the co-host, the second banana, who’s second to none. I’m also your plus one and only. It’s Mr. Nelson if you’re nasty, but you, my dear friends, can call me Phil. And I’m here to remind you of a few things we do on every single episode of this here program which you are most likely watching but possibly only listening to. I don’t know why. However, we’re doing a special trivia giveaway to you later in the episode. Wherever you’re watching, I will be asking a trivia question based on today’s presentation. And the first person to answer that trivia question correctly will win the OpenCV University course of their choosing. You can see what courses are on offer by going to opencv.org/ org/university. We’re also taking questions from you in the audience. So ask those questions wherever you’re watching on Twitch, LinkedIn Live, YouTube, or on our Patreon account or on Zoom. I will be monitoring the chats and bring those questions up as they make sense during the presentation. We’ll also save a little bit of time at the end of the show for those questions we didn’t get to. Um, before we get started though, I’m going to remind everybody here, Open CV is a nonprofit organization that makes opensource software. And as such, we depend on the support of our members and partners. I’d like to take a quick moment to thank those sponsors right now. Thank you. ARM, Futureway Technologies, Google Summer of Code, Rooflow, Orbit, The British Machine Vision Association, Jet Brains, Edge AI and Vision Alliance, Open MV, Tanggram Vision, Amped Software, Intuitivo, Rerun, Intrinsic, BearsDev, and Big Vision because every company needs a big vision. If you’re an individual, you can also support us and become as cool as the people you’re about to see on your screen on GitHub. Go to github.com/sponsors/opencv.
For just nine bucks a month, you can get your name up here. I’m going to thank these folks as well at the tiptop of the show. We’ve got Zion Khan, Loick, Mario Berseron, Alberta Beef, Rooflow, Bears, Dev,
Coer or Axel T81, DJ Greenwood on the one and the two, Techman, Dan Dogaru, Chonx Freze, Hugh I don’t know how to do that one, but I did my best. Stefan Sarnv, Luxronic AI, Alexander Vornov, Big Vision LLC, IPOP AI, Comet ML, Alexander Ismolof, and Masano Kita. Thank you so much. She’s a new one. I think our first uh Japanese um GitHub sponsor. So, thanks buddy. Really appreciate it. You can be as cool as these people by going to github.com/sponsors/openv if you’re an organization or a person and get your name right out here during the show. You can also join OpenCV’s official Patreon account and watch this show adree. None of the YouTube BS, no trackers, no nothing. Go to patreon.com/opencv for just seven bucks a month. DRM free downloads of every episode and ad free live streams of OpenCV Live. There will also be some future benefits that we’ll uh share with you all as we lock those down from our OpenCV members and partners. But that’s enough of my yapping. I think now we can go ahead and welcome Aleandro to the show. Welcome my friend. How are you doing today? Thank you so much and thank you for the invitation Phil and Sat good. Thank you. Yeah, I appreciate that. Who you got in the World Cup? My World Cup stopped in 2006.
After that, it was a bit of Yeah. disasters. So, I think that was the best World Cup in the last 20 years for sure. Okay. Just a little bit biased. A little bit just just a bit. Okay. Uh we’ll go ahead go ahead and bring up your screen share. You can take that was the one where Zidan had headbutted, right? Or Correct. Yeah. Yeah. The famous in Germany. Right. Right. Right. The famous Zedane headbutt. Indeed. All right. We’ve got your screen share. Let’s do it to it. Grammar that is. So, let me see if I can play alo this video. Okay, here we go. Okay. So I’ll during this presentation I’ll be presenting my uh PhD research that I’ve done at New York University and which is titled drone autonomy GPS denied environments. What I’ll be talking about is vision based navigation for agile quadcopter like you can see in this video and the task is to enable this system to quickly traverse a completely unknown environment be able to avoid obstacles find the target and track the target at a safety distance. This task is particularly important uh for applications like search and rescue missions. Over the years, we have seen how drones have become a very useful tool for first responders for rescue teams. For example, helping finding stranded people in the forest at night. Or we even see saw applications where the robot was actually throwing life jackets to people that were drowning in the ocean. However, one key limitation across all these missions is that these drones were always human piloted. So there was always the need for a human expert pilot at the right time and the right place with this robot and this limitation comes from the extreme complexity of operating in search and rescue environments wield with u autonomy with this aial platforms. For example, if we consider a wildfire scenario for search and rescue, we we need to build an autonomy stack that is able to continuously adapt to evolving operating conditions due to fire, wind, smoke. We need to be able to continuously react to evolving terrain, evolving obstacles due to, for example, collapsing trees or falling debris. And finally, we need to be able to fully navigate with only onboard sensing and onboard compute because often we have for example canopy smoke that is completely denying any external link like GPS or communication. The traditional way of solving this problem with the autonomy is to build an autonomy stack that is sequential. It first localizes the robot in the environment. Then it maps the free space. Then it plans a global trajectory along the free space and finally controls the robot to follow this safe path. This works pretty well in structural conditions. It’s also modular, interpretable and designed around safety. However, it relies on three fragile assumptions. First, we assume that our dynamics are constant and known so that we know our braking conditions. We know how to accurately steer the robot to follow the safe plan. And we can also hyper tune our controller so that it works very well in those operating conditions. Second, we assume that our obstacle maps are reliable and up to date so that we know what is free space. We know where obstacles are and so we can plan a safe trajectory without hitting any obstacle. And finally, we assume that globalization is available and accurate so that we know where we are and where is our goal. So planner and controller can steer us to towards the objective and also as we build the map online we can make consistent updates to our map. However, as we saw before in unstructured settings like the wildfire example, these assumptions break in particular, they can also break all together at the same time because we saw that operating conditions continuously change that our map is outdated the moment we build it. And also globalization fails completely due to canopy and smoke cutting that link. And so throughout my PhD, I worked on different methodologies, approaches in order to refactor, reframe autonomy to sort of get rid of these assumptions or at least minimize them. And I worked on different line of works. The first one was building continuous adaptation of our dynamics and control so that the system adapts during flight to operating conditions. The second one is reactive collision avoidance trying to completely get rid of the assumption of having a map or even building a map at all. So no mapping, no global planning. Can we link directly perception into action and avoid and guarantee safety? And finally, instantaneous relative navigation for being able to get rid of global pose, global trajectories all together and only navigate with respect to instantaneous relative frames that we define based on what we observe. So for the uh focus of today presentation and for the sake of time I will focus on these two uh line of works and by the end we’ll show I will show a few experiments also in the wild where we can see the full traversal reactive collision avoidance and also finding a target and tracking it dynamically. If you have any question feel free to interrupt me at any time. Um so I start with the first contribution that is reactive collision avoidance. So let’s start first by defining what we mean with collision avoidance, how it’s done and how do we uh basically solve this problem. So collision avoidance is all about ensuring that we don’t collide with any obstacle in the environment. As I said before, it consists by first mapping free space, planning and then control. So if we have the perfect map of the environment, then we always sort of be we are always able to uh traverse environment safely and reach the end. However, if we consider vision based navigation, often we only have a limited field of view from our camera. And so in a dynamic environment like the Wi-Fi example where we have falling trees, falling debris, if that happens outside our field of view, our map is outdated because we consider it static. And so for navigating and structuring conditions, it’s a very strong assumption to have a map that is always reliable. In particular, because also updating a map is an extremely expensive process. In disaster response, we have obstacles, objects that have different shape, size, and number. And so we need a fine grain, a map that is fine grain enough to capture all these different configurations. And that typically is quite too expensive to run on board a robot. So in this first contribution, we try to completely have a design shift of how we consider obstacle avoidance. Instead of building the most accurate map and updating it in real time, why don’t we get rid of it altogether and we try to focus on can we still enable navigation and safety safety and collision avoidance by only bounding oursel to local observations. This is falls under the name as I was mentioning before of reactive collision avoidance. uh prior works focused on either idolized perception but then as you see in the video for example in the center on the second plot on the bottom left even in a very structur simplified lab environment all those red blobs are false obstacle detections. More recent approaches also more exciting approaches from reinforcement learning world learn how to map directly depth images into velocity commands. The limitation here is that they tend reinforcement learning by default tends to overfeit the training distribution. And the task that we’re focusing on is navigating unknown environments. So environments we don’t have any data about. So using reinforcement learning doesn’t seem like uh the right tool to to actually solve an unknown problem. So let me present you a different solution that instead solves perception on motion planning control problems. On the perception side, we use monop estimation to complete our depth. We extract highrisk collision points and we map them directly into our it’s a model pretty controlled framework as safety constraints. So from perception side again we have a forward facing camera. This is what you get if you go in a park with a simple D435i or D455 real sense camera. Just state-of-the-art depth sensors that solves ste matching and what you get is a noisy depth estimate because you have a basin of about 9 cm. So you see up to six 10 m in that and so that’s why you see all those uh black areas and so we need to do something to to remove basically these bad layers. One thing we can do is uh use a foundation model for vision for detect for predicting depth. This was about a year and a half ago. The state-of-the-art my opinion was that anything in V2 and so but we can use the same approach with any foundation model. We predict from a RGB image directly a accurate generalizable fine grain relative depth. This of course has a lot of advantages but comes with the main drawback that it’s a relative depth. That means that we know how far the two obstacles are one closer than another. But we don’t know how many meters or inches they are from us. And therefore we may have the closest obstacle at 1 cm, 1 m, 100 m far away. And so we cannot use it for obstacle avoidance. To recover the missing scale, we use a second order optimization problem that aligns per pixel the relative depth with the sense completed with the sense absolute depth. And we obtain three coefficients A, B, and C that we can use to scale our relative depth into an absolute completed depth map. This is fundamental in particular in any real world setting because for example here it’s me going around with the robot handheld with a D45 D455 camera for frontfacing and on the top right you can see the simple output from uh the camera from real sense you can see how many blobs of noise you they come out when you do 3D projection on the bottom right instead you see in real time the completed depth from our pipeline you can See how we are able to get rid of completely of all the outliers, remove the noise and maintain the same absolute scale and almost even see further beyond our baseline. So uh all this is happening uh in real time on the drone itself or where is the compute happening? Correct. So this is all running on an onin NX 16 GB all running in real time on the system. There is of course a lot of optimization from this is like you convert to NX tensor key you do FP6 conversion and then you use smaller image sizes so that you high uh like higher frequency this runs at about 80 to 90 Hz on the on and it doesn’t consume all the compute because then we have all the pipeline I will present you after in the next slides but it’s quite efficient uh that’s very good actually and the engine v2 uses the vit small not the nano. Right. Right. Nice. And so now we have an accurate representation in terms of black space. For collision avoidance, we need to also convert this into some notion of also time collision because we depend also on velocity. If we are static with respect to obstacle there is no risk of colliding. So we can take our depth map and divide by the robot velocity projected across all array directions of our camera and we obtain what I’m calling time to collision map. This is giving us all information we need to avoid obstacles because every pixel of this map is identifying a 3D point in 3D space and how much time it takes for us to collide with that 3D point. However, it’s also specially redundant. Uh we can down simple it with mean pooling very efficiently. We can project the robot size. So we only consider collision with points that are actually causing a collision with the size of the robot. And finally, we can sort the remaining points and pick the k points with the minimum time to collide, which are the k points with the highest risk of collision. So now we have k points. We can for each point define a distance function like distance between the obstacle and the robot. We can formulate a safety constraint by using that distance function and in in my case I was using as a formulation a second order control value function formulation because it both ensures that we don’t collide with the obstacle. This is the typical constraint that you use the distance is bigger than equal to zero but at the same time we also dampen the robot motion as we approach the obstacle. So we are putting a regularization factor on the velocity and acceleration as we go towards the obstacle. So we can achieve higher agility and a smoother basically trajectory when we dodge it. Now we went from perception into K collision point. You define safety constraints. We can formulate for example a nonlinear model control framework. This is the general formulation. You are going top down. You are solving a optimization problem to predict a sequence of states x hat and a sequence of actions uhat over a horizon of capital n to reach a desired goal that is x goal that can be for example a 3D coordinate in GPS or for example a target location and this problem is constrained by a function h that is our dynamics a function g that is actually a set of functions that are the constraints over state and action. So for example maximum and minimum torque, maximum minimum thrust and finally we have K dynamically safety constraints that is what we introduced so far. Putting all together from perception down to motion planning control, we have a full framework that runs so synchronously between perception that can run at the frequency of our camera creates K collision points that we embed as K constraints into our NPC and NPC can just runs at its frequency of like for example localization frequency and what is important to highlight here is that we didn’t put any assumption whatsoever about the environment the obstacle shape size and number we are purely mapping perception completing it into k points, k spheres in the 3D environment and at every time step updating those spheres and asking the controller to predict trajectories that do not collide with them. So let me show you a few experiments to to prove these things. One question before uh before the results. uh what is the you know if you go back to the uh to the minimization problem uh what what do you use to solve this uh solve this constraint optimization problem and how fast do you solve it so um great question so this is uh OSTP problem which solves so there’s multiple ways you can code this you can code it in Python it will be super slow you can code in CS also fast enough but uh if you rely on frameworks like kazadia OS you can generate C code that you can then write here run on your embedded platform. You can also use libraries like OpenMP to do GPU parallelization. And so this actually is extremely fast. So without the control by function it can run easily at a thousand hertz on the embedded platform. With the control bar functions active it depends on how many we activate. I made an ablation study on the paper. If we have 10 active we can run at 400 Hz. If we go up to 100 active counter value functions, it goes about 100 120. Okay. So it’s it’s a matter of how many counter value functions we want. So that k parameters and how many are active in that time stamp and u but throughout the entire experiment from here later on it will be always 10 contrab functions and that’s kind of quite enough because if we go back for example to the grid of mean pooling uh like mean pooling is already discretizing a lot our image space. So one sphere would be large enough to cover like a sufficiently big portion of the image. So the grid for me for example it was I think if I remember correctly it was like 16 * 17. So that’s the upper bound of how many K you can put. Right. Right. And and uh is this optimization done on the CPU or the GPU? So most of the experiments were run still without open. CPU based uh you can like simply turn on a flag and rebuild it on the platform. It will deal also with GPU. I I didn’t use GPU enough because there is already extensive like work for like collision avoidance etc. I didn’t want to have any uh delay on that part like the MPC side is extremely fast like the bottleneck is always dealing with like perception like this. Yeah. Right. Right. Yeah. Why why use why overload the GPU when all the heavy lifting is done on the GPU and the CPU if it can already do it at that the speed then you should be using the CPU right correct? Yes. All right. Thank you. Thank you for the question s and uh so let me prove you all these claims and and show you some results both indoor and outdoor. Indoor localization comes from Vicon, outdoor from GPS and the task is always to take off from one position and reach end position. The robot has no knowledge about the environment. There is no tuning whatsoever from here to all the experiments you will see. We will always be using the same parameters and I think this is very important because in traditional stack instead you need to fine-tune based on different configurations maybe of obstacles of lighting conditions indoor outdoor different heights. And so we show for example indoor first everything is running on board the robot we task it to just traverse we change the environment configuration we change the lighting conditions and it’s able to traverse the environment safely. If we look at like also error plots or for example the yellow plots on the bottom here are the minimum value of the 10 controller functions that were always active and you can see that the value is always above zero. So we’re always practically guaranteeing safety throughout all these experiments. We test also this outdoor. We traverse a soccer field. Um we try to dodge the poles of a soccer goal first and then we bring it to a sparse nearby forest where we just go with the robot on one side of the forest, take a GPS coordinate, go on the other side of the forest, take off and let it go through. And so it’s able to autonomously dodge trees, leaves, hanging branches, and achieve speeds of up to 8 7 8 m per second. And there is no cable connected to the camera to to interfere with with the socket. Yeah. Another very interesting thing from this World Cup that we’ll be talking about for years. Yes, you need to collision awareness for that as well. And so what I showed with this contribution is that we can still operate even without having to build an expensive map without doing global planning by directly mapping perception down to control through safety constraints that are dynamically defined. Now the second contribution will instead focus on the necessity for a global pose. We will see how we can also get rid of that assumption. Let me start first by explaining how what is a global pose and how do we get it traditionally. So global localization is all about finding where we are and where is our objective. And this is obtained by defining a state that relates our frame, our body frame B into a world fixed reference that typically is just defined on for example where we take off, where we start the mission. And at every time step, we try to find this global trajectory in terms of relative position, orientation, velocities that link our body frame to that initial position. This is obtained in vision based navigation by fusing information from camera and IMU. Camera works by detecting at every time step a set of key points like you can see in the bottom video and tracking them over time and then fusing this information with inertial data from my view. The limitation what is it is that many key points at every time step go outside the field of view. Some may be for example position on a dynamic object. Some may be may have error due to motion blur. Some may have for example maybe position on reflective surfaces. So at every time step we always have some error that unfortunately accumulates over time. And so what happens as you see in the video the red frame it tends to drift after a certain amount of time. And so you always need some sort of global correction that can come from either GPS, it can come from loop closure or any other global corrector. And this is a very limiting aspect. It’s a very strong assumption if we’re flying again in a known and unknown cluttered unstructured environment because we’re limiting the a kind of environment we can fly into. We cannot fly indoor. We cannot where we need for example GPS correction. We don’t fly indoor underground under the forest canopy close to buildings or even if we use lamb we cannot fly high speed in texture poor environments for example and so in this second contribution I will focus on having a another design shift instead of requiring the the global pose let’s focus on what we can currently observe from our sensors let’s try to instead of building a global trajectory with respect to that world reference that we fixed at part of the mission. Let’s forget about that and let’s just focus on what we can currently observe and still ensure mission success. So this idea is partly inspired by prior works on visual servoing target tracking where you have a target field of view. You want to either track it or you want to intercept it. And beyond simplifying assumptions of perception like I want to find a red balloon in the blue sky, prior works have only focused on the tracking task. And this has two key limitations. First of all, we cannot do obstacle avoidance very cleanly because if you want to avoid an obstacle quite often your target will be occluded and so the moment you lose your target you finish your mission. Second is that we are focusing on search and rescue mission in a noncluttered environment and search and rescue demands that the target is not already in our field of view at the start of the mission. And so it’s completely missing the search part of the search and rescue mission. This uh that middle video looks like you’re you’re training a drone to chase Minecraft Steve.
Yeah, it’s very accurate representation of simplifying perception. It’s the all the outdoor images your videos you’re showing it just shows that you know you could just go for a hike and tell your advisor that oh I’m I’m I’m working uh collecting data or what not. That’s right. That’s right. Spot on. And so I will present in our own approach that redefineses how we want to build our state and our desired state. And it consists of two modes. One is for searching and one is for tracking. And in both cases, we are never building this word reference. We’re only defining these instantaneous relative frames. They’re I’m calling them at least instantaneous because we define them at every time step. So we don’t keep memory of them. And they’re relative because they’re only defined with respect to what we can currently observe in our field of view with our sensors. So let’s start from loing. Loing is defined we keep the same state components of position, velocity, orientation. Position is given by our barometer PZ. px and py are just zeroed out or you can leave like unbounded numbers below. We’re only focused on maintaining altitude in our position as accurate. Then we have velocity as optical flow and inertial data gives us our orientation. And we can fuse all this information with the traditional kma filter and centered calma filter in order to have one high frequency estimate of our state. And on the other hand, our desired state is to maintain a desired height, desired heading direction and desired velocity in that heading direction. Let me show you an experiment, an example that makes it more practical. So here we take off and we ramp up at 12 m height with the robot. We maintain a velocity forward of 7 km an hour and our heading direction is 0° with respect to the initial heading direction. having direction of zero means we want to maintain the same direction but it’s not necessarily the case. We can also show experiments where we for example follow the direction of maximum free space or for example the the direction of a low confidence target. And this is these are both experiments I will show you uh in a few seconds. Then we can switch in tracking mode. The moment we lock into a target, we can see a target in our field of view. We can switch from loitering to tracking. We maintain the same state components but now position velocity come directly from our object detection model and as desired state now we can define a desired distance in terms of XY so horizontal plane and a desired height with respect to the target. In other words, it’s like putting a cylindric area around the target and making sure we are always on the surface of that cylinder. So now if we continue the previous mission of loitering uh here we’re looking for a car and we have a transition at 0.7 confidence threshold from loitering to tracking. When it locks into a car it will switch into tracking mode and it will start moving and maintaining the desired offset of 10 m height and 10 m xy. And here the camera is frontacing with 45 degree pitch angle down. So the moment it centers in the field of view, it’s literally when it perfectly tracks it. So now what I presented you is a full redefinition of how we can define localization. We are defining it only based on instantaneous relativeity frame on local observation that we can every second observe. We can still the nice part is also that we can still use this framework with traditional GPS-based navigation. the moment the GPS is denied is corrupted or anything we can switch into lock mode we can switch into tracking mode and track these frames and so that’s why also one of the reasons why I kept the same state components of like 3D position velocity and orientation also we can switch between lo tracking based on detection confidence for example and the control of that the control I presented before for motion planning control for reactive collision avoidance can still work and still hold Even here we can still have the same NPC with constraint optimization with our dynamics function and our safety constraints for active collision avoidance. So now let me show you let’s conclude with a lot of experiments. First let me demonstrate tracking mission. So first we mount an mannequin on top of NTB so that we can move it around and we fly through a metallic thunder. This completely disrupts our GPS localization. But by using pure relative localization, we are able to smoothly track it. We test this at different heights uh different height offset offsets. Uh the previous experiment was at the same height as the mannequin. Now we add 6 m above the mannequin. So it goes about 8 to 9 m up in the air and we’re still able um what are the two things we are two views that we are looking at? Um sorry. So on the left is me falling with a pickup with a car and with a GoPro on top of the car. Okay. And so you see from the 3D perspective outside and on the right is the onboard vision of the robot. Okay. Okay. And then we can repeat the same experiment go on a nearby forest trail. Uh this goes about for almost a kilometer. And we can repeat the same mission, reach a speeds of up to, if I remember correctly, 50 uh kilometers an hour, which is the maximum speed of this ATV. And we can track it smoothly throughout the entire mission. And as you can see, we only have like 19 degree pitch angle for our robot. So it’s actually quite it doesn’t put much effort to follow the ATV. So we replace the target with now a pickup truck with good insurance. And we repeated the same experience. Now this was a model trained on Coco data set. So you have both the car and and the truck classes and then you have reflective surfaces and lighting like different shadowing. So as you can see there’s a lot of false detections. When you see the center bounding box with the green dot that is the output from the object detector. The outer bounding box that you see that zooms in and out that’s sort of like just a very let’s call it smart. It’s just a very basic mathematical technique in order to zoom in based on the last previous detection and zoom out when we don’t have detections. So this allows us to feed to the model just a small crop of the full image. So we give less clutter basically to the object detector and it’s much easier to do object detections. And so as you can see we were able to fly 80 km an hour pitch angles of up to 40 45° and follow the ATV both when it was speeding when it was slowing down.
Nice then. So this proves the tracking performance. Let’s now focus on lighting. For lighting, we are mostly focused on can we show generalization across different environment conditions in different heights, different lighting conditions and etc. different texture for example. And again we’re using the same parameters across different across all the the same parameters across all the missions. Here we fly for example in the forest in above the military compound from the forest trail all the way over a gr some grass vegetation all the way in the military compound and the robot is always able to maintain this zero heading desired direction. As you can see here if we overimpose and create an arrow from the for example the zero heading and then we see in blue the actual GPS coordinate that was flown. You can see that we are able to follow accurately the trajectory. Then we move into the dense forest. We take off randomly and here the heading direction is given by free space. So every bunch of seconds we take the largest open area in our field of view and we take that as desired direction. And that’s the entire point here is that we we mostly focus on proving reactive collision avoidance while loing. So we don’t really care where it goes as long as it doesn’t crash into a tree. And so on the bottom right you can see the completed depth and the red bubbles are the 10 contra body functions that keep being defined at every time iteration and you can see how the robot smoothly transitions and avoid every obstacle along the way and even here that we have three trees that create a sort of a bottleneck it’s still able to bounce back and keep uh the mission going. Finally, last two videos. Uh, first we want to show that we are able to loitering and track and then follow a dynamic target. So, we go back to the the military compound. We take off at different heights, positioning the mannequin with different also angles and trying to increase the noise and going in circles around the military compound in different orientations in different like directions. And we we able to track it quite consistently and perform this mission over and over. And finally uh probably the most exciting mission of like sort of a search and rescue mission. Here we take off in a approximate heading direction. We put a mannequin 7 800 m far from the initial position. We t the robot to just move along that desired heading direction and we condition our heading based on the detection confidence. So the moment you will see in a few seconds it starts having a detection confidence above 0.3 it will start also aligning its heading so that we don’t really need the assumption that we are perfectly aligned with target we want to move in that direction. We can also sort of doing a mini exploration of if I see the target somewhere in my field of view, I can bias that and switch into that. And then when it reaches a confidence of about 0.7, it switches from loitering to tracking and it find the target. It locks into the target and it stops and maintains the safety distance. This uh video reminds me very much of the shaky cam from the Evil Dead movies, the famous Sam Ramy zoomin shots. In their case, I believe they used a uh 2×4 with a camera strapped to it attached to a dirt bike. So, it’s a little bit different. Yes. And um so the message I want to leave you with across this entire more technical presentation is that sometimes we don’t need to spend too many resources building expensive maps and expensive global poses in particular in those environments where those are not required or extremely too complex to actually maintain. We can ground our autonomy, we can ground our navigation in a lightweight autonomy stack that is adaptive, reactive and it’s grounded on what we can observe from our sensors. Of course, this comes also with limitations um by we we are focused on openstructor spaces. If we take this indoor in a constructor environment with like a loop for example, we end up keep looping around and one solution for this can be in the future to create some sort of object relationship memory or feature based uh memory of what we already observed or our safety is also limited to what we currently observe in our field of view of our camera. So the moment an obstacle goes outside the field of view, we forget about it. And so something like a predictive model like in autonomous driving, we predict where cars and trucks etc. move around us and when even when they go outside the field of view. So predictive model would actually integrate also very well with the current contour function formulation. And finally we lack high level reasoning in terms of if we are indoor, we are head we have a desired heading direction of zero. We want to keep bouncing around given free space. It’s very hard to find indoor and red car. So some vision action vision language model that given an input image can tell us you’re indoor you’re looking for a car you should aim your head interaction towards windows towards uh doors that would be extremely I mean effective and that would also integrate very well with our framework as it would just bias the desired heading. These are my choice. Based on that first uh video there, it it seems like uh perhaps this would not be the best solution for saving you from the back rooms. Correct. Yes. That would be too much of a structured space. The the first we need the first drone that will rescue you from the back rooms.
So these are a bunch of publications that relate to the topics I mentioned and um that’s all for me from my side. Thank you so much uh for listening. Yeah, thank you. Can you go back to those papers real quick so folks can have a second to screenshot those? We’ll uh we’ll drop some of the uh we’ll drop some of these in the show notes as well.
Cool, cool, cool. Well, thank you for that. Uh, great on the time. Uh, you parallel parked this episode in, uh, the correct number of minutes. So, thanks for that. Um, now we’re going to do our trivia giveaway here and then we’ll do some questions from you all in the audience. I’ve got a few in the hopper already, but if you have a question, now’s the great time to ask it. post that in the chat wherever you’re watching and I will bring those up on screen in a few minutes here. Uh but first, it’s time for trivia. Um if you need a reminder, the way this works is that I’m going to ask a trivia question based on today’s presentation and the first person to answer that trivia question correctly. Anywhere OpenCV Live is broadcasting will win the OpenCV University course of their choosing. You can see what courses are on offer by going to opencv.org/ org/un university. So, uh today’s question is going to be uh in continuing with last week’s theme of a little more pop culture in the trivia. Um I’m going to ask a question about the World Cup. We talked about it here earlier in the episode. Zedane’s famous World Cup headbutt happened in what year and against what team? Answer in the chat to win. We need both parts of this question answered in one message. If you’re watching on Zoom, make sure your messages are set to everyone and not just admins. We’ve got some questions. We’ve got some good uh Oh, you’re you’re close here. You’re close here. I don’t want to give it away. I can’t believe that uh this answer is not instantly answered. This question is not There we go. There we go.
We’ve got I’m trying. Okay. I’m trying to decide. It’s a close one between YouTube and Zoom. It looks like It looks like Zoom was the one to get it first here. So, uh the answer we were looking for, we had a couple people say get the year right. The year was 2006. That was correct. However, a few people answered Germany. That’s not correct. I think it was maybe took place in Germany, but it was not against Germany. It was in fact against Italy. So 2006, Italy was the answer we were looking for. And the first person to answer that correctly uh was Masimo Lazeri. I don’t know. Gosh, I wonder how he knew the answer to that one.
But congratulations, Masimo. Uh please send one email to Phil at opencv.org. That’s me with the name of the course you would like. You can find all of the courses that are available by going to opencv.org/university.
Um, shout out to uh, Manuel on YouTube who missed it by just a bit and also Eric and uh, AGT Dash who were just a few seconds after our answer on Zoom. So, congrats. Um, now let’s do some questions here. We’ve got a couple of good ones. Let me pull up one from YouTube. This is always a question that’s on my mind when we’re talking about drones or edge devices. Um, how much power is needed from the Orin AGX when you’re running this? So, the that depends on the current board that you’re using and also the mode. For example, in if you use a NX, you have just 15 watts. If you use an R in an X, you go up to 25 watts. And recently in the last year there I mean for probably two more years um yeah probably two years because we graduated one year ago and it’s you can use some mode that is called super mode and you can use a carrier board and push 40 watts inside your car inside inside your and so I would say all these experiments were with 25 watts and but you can go up to 40 watts now with the super mode. Thank you for that. I’ve got Eric asks to avoid collision do you consider raw distances or are some filters introduced for avoiding collisions? We do only consider the distances coming from that completion. So we assume that that completion is consistent enough because our it’s still grounded on first principles how we get our absolute scale by solving stereo matching you know your absolute scale will be correct and so you can also uh measure what is the discrepancy when you solve your second order optimization to align predict the relative depth and absolute stereo depths. you can find also how much they are matching. And so if your error is small then you can directly consider those distances because you are confident that you only you’re still using absolute distances and you filled in those uh details like the all the noise and the outliers.
Yes, thank you and thank you Eric for the question. Uh we’ve got another question here from u I’m not going to try to pronounce this name because I will get it wrong but uh did you set any distance threshold to factor and what if the obstacle is wide? How does the algorithm handle it? Like for example where the tree branches are um the name is David I guess so.
It’s a great question. It’s what motivated our design. Um so typically when you do obstacle avoidance often you either have like voxil maps and you have these 3D boxes 3D cubes or you have assumptions about the obstacle semantics for example it’s a person so I can fit fit the person on a cylinder or it’s like a small box right so you put semantic reasoning in terms of what’s the class of the obstacle in this case we took a different approach instead of relying on semantics we are purely mapping pixels into 3D spheres And the size of the sphere depends on how close we are and how big is our robot. So for example, if another, for example, if we only have a small uh let’s say dangling branch from a tree with just a leaf, that will still become a bigger leaf. We will not get as close as we like as the size of our robot allows. If you remember that formula I was showing about maybe I can show it again quickly. Yeah. So yeah,
still a little time to get some questions in folks if you have any questions left. I have a hard stop uh at 10:00 today. So no problem. Yeah, you guys can continue. I’ll unfortunately have to leave at 10:00. When Satia leaves, it becomes the Phil Politics chat. We’re going to we’re going to leave it all on the field today. Folks, can you see also my mouse pointer? Uh I can’t see your uh there it is. Your screen. Yep. I just I just brought it up here. Mhm. And the mouse pointer. I can see it. Yeah. Perfect. So every time we have an obstacle, right, we we create a 3D point in space. And that 3D point, assume, for example, we are doing our loitering mission. So we don’t have this position. It’s literally just taking that and make inflating it as a 3D sphere in the environment. And that size of the sphere depends on this radius that we’re defining. That in our case was the robot radius. So every point will become as big as basically the robot radius size. So we’re always making sure that we’ll be always uh not too close basically by our size to that obstacle. But we’re not putting assumption. It’s not reason in terms of what is the obstacle shape or semantics. It’s reason in terms of I’m mapping that into a 3D sphere and I’m avoiding that and the size is dependent on my robot size. That makes a lot of sense. Like you definitely need to factor in the size of your drone. Um yeah. Okay. Thanks for that. Uh we’ve got uh Hussein asked, “Is this work published anywhere?” I think you already answered that. Um but we’ll make sure to put that link to the papers in uh the show notes on YouTube and Patreon, etc. Maybe. So this work originally this is called reactive collision avoidance something was originally published two three years ago and then also the works after was also using reactive collision avoidance for lighting and those were nova and then hunt more recently. So if you go all the way to the back it’s reactive avoidance and then nova and hunt build on top and explain how to also improve it further. Okay thanks you for that. I’ve got one more here from Jeff on LinkedIn says uh let’s say no GPS available. Can you approximate relative positive from last known GPS or last zero position uh this way to both approximate mapping distance from original point of a known location? So it’s a lot of words but uh did you get the question? Uh yes I think so correct me if I’m wrong. Um so your your question is we are flying GPS at one point GPS is denied and in loy mode can we still build that representation of how much we travel with respect to the last known GPS coordinate. Uh so the answer would be no in terms of that would still require you to build so basically put a relative or word frame on where you were denied and then build a trajectory from there and that would go against the point of like building loitering mode with instantaneous reality frames because then you start using like slam and vio in order to build that global trajectory. So it if you use just off-the-shelf sensors like drones have right not rocket missile I use or space robotics I use it becomes very hard to actually maintain that uh frame consistently over time right yeah in other kinds of applications where you’re uh very high uh almost right then you can uh then people use uh you know satellite imagery to map where you are right so they match satellite imagery to your current view. And I know of companies who can go, you know, and these are uh primarily for military use. They can go about 100 miles uh without GPS, right? And and but they are very high up and they can look at uh satellite imagery and compare it to uh that. Yeah. The limitation there is that you you assume you have those satellite images and you have a sort of a dictionary in your memory and you can continuously query it. Yeah. The problem is that what what if you’re flying under the canopy of trees, right? Right. No. So, this is a completely different uh problem. Yeah. It’s trying to like solve solve it from a different perspective and a different angle and if we want to stay like low key in like in low altitude environments at high speeds that I think we need to refactor and remove like slam and like global process, right? Uh so, unfortunately guys, I have to leave but uh thank you so much. Uh this was amazing. This was amazing presentation and uh we’ll talk about it later. Thank you. Thank you Syia. Adios Satya. Um that was actually our last question here. So uh I have one note. I mean obviously of course if you if you run out of GPS and you can’t use satellite imagery, you just have to shell out for the Pokemon Go data set and then you’re all set.
The most complete map of the world probably should write a paper on that. Yeah, somebody should somebody should maybe there’s some grant work in there if they weren’t shutting down the NSF funding. I think somebody could get some grant money to to work on that. Anyway, uh Alandro, thank you so much for joining us today. Do you have any final thoughts for the audience before we call it an episode? I think it’s sort of like summarizing what has been said so far in terms of grounding what we can currently observe is a great strategy in order to maintain efficiency and effectiveness in completely unknown environments but even when we don’t put any assumption about we don’t have the assumption of I have some prior maps I’m flying in a structure condition or I know my operating conditions or I have a satellite satellite image of my environment This is a great way in order to traverse that environment and maintain mission control basically and of course it comes with its limitations but I think there’s a lot of ways to to prove it and build on top of it. Awesome. Thanks so much for that great presentation. I you you presented very well and uh you spoke very eloquently on the subject. It can be very complicated but I think you did a great job here and the audience seems to agree. So yeah, thanks so much for watching everybody. We’ve got uh uh more open CV lives of course next week. Um same bat time, same bat channel. I am of course your co-host with the co-host. Reminding you one more time, uh OpenCV has some fresh new merch available. I’ll share the screen here real quick on our Spreadshop, OpenCV.spreadshop.com.
You can uh go over there and buy the brand new OpenCV open- source nonprofit no [ __ ] shirt. Uh scan that QR code there and that’ll take you to the official store. We’ve got a few folks that have bought these since we launched them a couple of weeks ago. Mine is in the mail. I’ll be wearing one probably next week or the week after as well. Um it’s a great way to support OpenCV and let people know where you stand. So, please uh scan that code or go to opencv.mmyspreadshop.com and see what we got. We got shirts, we got hoodies, we got the classic NPR style tote bag. Um there’s also some uh other stuff on there like, you know, we’ve got hats, the import CV2 command, the the alltimer there on shirts and hats and things like that. We’ve also got pet bandanas and things like that, stickers, buttons with the official OpenCV logo. So, check that out. get your gear and if you send us a picture of you with your new OpenCV gear, I will bring it up on screen here on OpenCV Live. So, uh, if you give me permission, of course. So, please check that out and thank you all so much for watching. Um, yeah, take care of yourselves out there, folks, and take care of somebody else if you can. And have a great day wherever you may be. We’ll see you next time. Adios.