Dr. Satia Malik, our stalbert host. Where are you coming from today? I’m back back in San Diego. I was actually in India for the last couple of weeks. Uh three weeks actually. Um so I’m I’m back now. And Dimma, our guest, where you joining us from?
Natalani, Israel. It’s like north to Tel a Viv. Right on. It’s a global community here on Open CV. We love having the guests from far-flung reaches of the world. I think this year I think our guest, if I I looked at the YouTube uh history recently, and I think we’ve had a little over 50% of our guests this year have been uh not based in the US, which is pretty cool. But folks out there watching, let us know where you are coming from. I’m here in San Francisco. Satia’s down in San Diego. Dimma’s in Israel. Let us know where you’re coming from in the chat. We’d love to see the global OpenCV community chiming in on this beautiful Thursday morning and we hope that you are having a great one. We’ll get started in just a few moments as you get into wherever you’re watching Open TV. We are live, excuse me, we are live on Twitch, LinkedIn, YouTube, and our Patreon account as well as on Zoom. We’ve got some folks over on Zoom saying hey already. We’ve got Stefan Sarna from Vancouver, Canada. We’ve got Nome from Ethiopia. We’ve got Venitesh from Shennai. We’ve got Andre from the Moscow region. We’ve got Lux Intel over on YouTube from Kosovo. How’s it going out there? Got Bruce Wright from Calgary, Alberta, Canada, home of the professional wrestling heart dynasty. We’ve got uh Utkur. Uh it’s a tough one for me from Sweden. How’s it going, Sweden? Shout out to all the Swedes out there. I was just at IKEA the other day [laughter] here, the the the Swedish embassy here in San Francisco. [laughter] That’s right. It’s It’s all right, though. I talked [clears throat] last time I was out of the country, I I talked with someone that said that uh the American embassy in their town was Hooters. So, [laughter] that’s I think I think I think Sweden got the better deal out of that one, you know. That’s right. And we’ve got uh Marco from Germany. We’ve got RFSI from Boston, East Coast, a slightly inferior coast to the West Coast, but we’ll allow it here on OpenCV Live. I think now is a great time to get started. Boss, you ready to go? Let’s do this. Let’s get started. Hello everyone. I am OpenCV CEO Dr. Satcha Malik. Welcome to OpenCV Live. This week we take a look at Clickins who run inspection for insurers, rental fleets, and car dealerships from ordinary phone photos. The models behind it are trained almost entirely on synthetic data. 3D vehicle models with procedurally generated dents, scratches, and defamations across uh makes color, lighting, background, and auto annotated at every pixel. In this episode, CEO Dimitri Gizerki will cover how the pipeline works, where synthetic data holds up against real world imi imagery, and where it breaks, and what it took to close the gap in production. If you have weighted synthetic data for your own inspection or QA problem, this is a practitioner’s view of what actually deploys. And uh you know this is going to be a very interesting episode because right before uh we started uh you know uh this episode we were talking about various things with Dimmitry and one of the things uh he mentioned was that it’s very difficult to do uh hail damage detection because hail damage is invisible to the human eye and they have a method that allows you to do this uh using uh phone cameras. So, I’m very excited to learn about these techniques uh in this episode. But before we get there, as always, with me is Phil Nelson. He’s the director of content and creative at OpenCV. He organizes this show. And if anything goes wrong, Phil, it’s your fault. Good morning. Yes. What a what a warm welcome I get from Satia on this show. I wrote that. I wrote the script. It’s fine. It’s me. It’s me. It’s phi. I am your co-host with the co-host, the second banana. who’s second to none. I’m also your plus one and only. You know it’s Mr. Nelson if you’re nasty. But you, my dear friends, can call me Phil. And I’m here to remind you of a few things that we do on every single one of these here episodes. This is episode 2. Jesus, what episode is it? 237, 227. I don’t even know. I lost my notes. But it’s fine. We’re taking questions from you in the audience. Use the Zoom Q&A button if you’re over on Zoom or just ask your questions anywhere in the chat. I’m monitoring those chats. We are live on Twitch, LinkedIn, YouTube, Patreon, and on Zoom. So, ask those questions anytime. We’ll bring them up later in the episode or try to flow them into the conversation if they make sense in practice there. We’re also doing a special giveaway to you out there in the audience as we do on every week. Stay tuned later on in this episode. I will be asking a trivia question based on today’s presentation from our illustrious guest Dimma who excuse me and the first person to answer that question correctly will win the OpenCV University course of their choosing absolutely free of charge. You can see what courses are on offer by going to opencv.org/ university. Now um got a little bit of business to take care of at the tippy top of the show here. Big announcements in the last couple of weeks. Satia, um, you want to talk to the folks a little bit about the latest moonshot here from OpenCV, OpenCV Enterprise, the newly released, newlyannounced, I guess, uh, yes, enterprise version of OpenCV. Tell them all about it, will you? Yeah. So, OpenCV Enterprise, this is the supported version of OpenCV. Uh the very first thing I want to make clear to everybody is that OpenCV will remain free for everybody for individual users even for corporations. It is completely free. Uh as you know we have Apache 2 license which basically says that please use it. Don’t you don’t even have to uh acknowledge us. Uh but uh you know don’t sue us if something goes wrong. So that’s the Apache license in a in a nutshell. I’m not a lawyer but that’s pretty much what it says. Yeah. Yeah. So uh but this enterprise version is the supported version of OpenCV. So what happens is that uh once you let’s say we just released OpenCV 5 right at some point we will stop supporting OpenCV 4 3 2 etc because you know we have a very small team and we cannot keep supporting the older versions but a lot of corporations they are still using I found out one corporation that is using one uh you know OpenC 2.4 4 uh in one of their products, right? So, wow, that goes back. That goes back, right? Uh the benefit of OpenCV 2.4 is that it was a tiny it had a tiny memory footprint and so uh it you know it works uh really well which was the style at the time because it was 20 years ago. [laughter] Right. Right. So, and now what happens is that there are security holes, right? Uh that get reported but we are unable to fix them for free. Right. So most of these uh you know uh the support basically comes uh with with security right we will basically fix uh as soon as any security is uh you know uh asked for any any uh hole comes up and we will offer it to all our enterprise users. There are other uh you know enterprises that also uh need they have their own forked version version of OpenCV right they they started with OpenCV but they forked it and we are happy to support those versions also. So basically it is uh and then some of some people may want special uh collaboration with OpenCV so that they can make OpenCV fast on their platform right companies like Qualcomm ARM they are already working with us so for them OpenCVs there are three tiers of enterprise one is LTS which is long-term support then we have the premier version which is slightly higher uh level of support and then we have uh the partnership version which basically says that okay we will we will make OpenCV work better right not just the previous versions but all the future versions also on your platform and we will actually have your device or your configuration in our continuous integration pipeline so these are for corporations but if you’re uh you know if you’re from a large organization which needs OpenCV support they want to make sure that you have a secure version of OpenCV sometimes you’re legally required to take u a supported version of opensource if it is available and if those are the cases please reach out to us uh you know uh at hellocv.org or uh or go to this URL and sign up and we will reach out to you. We’ll give you a link to have a 30-inut call with us once you have filled out all the information. So uh please uh check it out and let us let let your company know that this exists because this is not for individuals. This is for uh you know medium to large uh organizations and uh you can help by telling your managers that now we have OpenCB enterprise. So that’s that’s about it. Thanks. Yes. Scan that QR code I put up on screen. That’s opencv.orgenterprise.
It is not a secret new Star Trek television program. It is the latest release of OpenCV. OpenCV.orgprise.
It is something we’re excited about. We’ve talked about implementing I think for multiple years now. you and I have gone back and forth on this and with the OpenCV team and community and stuff and we’re really excited to have it out there in front of people. Um, as a nonprofit organization, we depend on the support of companies that use OpenCV. We put out free software that you can use however the heck you want. And uh, you know, lately it feels like we’re kind of being punished for it. [laughter] So, please please check that out. If What’s that here? Here’s a quick question for uh, for the audience. Which of these Google products use uh OpenCV? Google Lens, Google Meet, YouTube, Google uh you know even the main Google uh you know search uh app. Um which of those? And the answer is all of them, right? All of them. All of them. uh they use um uh OpenCV and pretty much you know any you know Snapchat any any app you can think about where there is yeah any image or video analysis being done they use OpenCV so it’s a good you know it’s it’s everywhere and we want uh to provide enterprise support so that all these organizations that are already using it benefiting from OpenCV can also have a more secure more stable version that is tested across you know multiple devices Uh and that’s that’s what OpenCV Enterprise is for. Yeah, we have it on on good authority that uh the Amazon’s lens feature on Amazon Prime video that shows you like what actors are in what scene uses Open CV as well. We we got the we got the Iggy on that from a a kind-hearted engineer over at Amazon. But enough of our yapping here. Welcome Dimmitri Dimma as he asked to be called. Uh, I love a good nickname. That’s a pretty good one. Welcome to the show. It’s great to have you on here. Oh [ __ ] Sorry, I muted you. Back. Yeah, thanks for having me. So, yeah, we’re excited to learn about how you’re using synthetic data. As I as I said before, we’ve we’ve talked about on the show off and on for several years and had some kind of deep dives and it’ll be great to see what you all are working on. Good. Let me go ahead and bring up slides here. Yeah, let’s do it. I got your Bringing up your screen share. There it is. Take it away. [snorts] Good. So, picture this. You’ve got a standard sedan, right? It’s [laughter] parked. It’s parked uh you know on a city street, broad light. uh and to the human human eye I mean it is uh undeniably there right? Yeah. So obviously we cannot miss it. So but if you run the life fit of a street with this car through the state-ofthe-art object detector like AI object detector let’s say YOLO or any version of YOLO. Okay. This this uh vehicle registers as completely invisible. Is that true though? How did how did you get this picture of John Cena’s car? [laughter] Yeah, we generated this one. Okay, this is a synthetic data. Okay. So, uh I tried this uh so um you know there is this uh there’s this company that I don’t know whether uh which company it was but they said that uh people wearing this kind of patterned shirt are not recognized and I ran it a standard yellow and yeah it nailed it right it nailed it. I have not tried it on this car but um I would be surprised if uh it doesn’t get it. What about jungle face paint? [laughter] Yeah, I came across that uh that article. It was it like was very popular last week. Yeah. And this is a designer that created this uh texture Yeah. texture shirt. But this pattern it’s a bit similar to what he did. Yeah. But this is the original one. Okay. That was created by the researcher. So I believe his work was some kind inspired by this thing. And this this patch is unique. So this patch we just generated it and put it on cover the car. Yeah. And and it make it completely invisible to YOLO including YOLO 26. Okay. Yesterday I tried this. Oh wow. It doesn’t see this vehicle. Okay. So and uh what’s wild about this like if you think about this that that this glitchy kind of psychedelic camouflage wrap that covers the car. this makes all this uh uh thing invisible. So h and this is like the problem the challenge that the today modern vision models must deal with. This is the level of intentional obfuscation that uh in reality requires complete rethinking of how we train our models today particularly vision models and uh and this is what actually clickens is about. Okay. And if you see this uh this one it’s we not only covered this with this uh uh rep but we also introduced synthetic damage on this car. Okay. So, uh just a few words about myself and we will move forward. I’m a technology expert, software architect with 30 years of experience. My major fields of interest are visual intelligence, ontology, I love ontology, uh domain driven design, computer vision, open CV of course guys, I love OpenCV, uh deep learning and graph uh algorithms, high performance and uh to mention but a few those are fields of my interest. uh previously I architected uh intelligence solutions uh worked with law enforcement uh with anti-fraud systems architected enterprise and startup uh solutions so quite a wide range of uh of solutions that I had experience with. I co-ounded a few companies, professional services company and one that dealt with transportation that but that was in the past. Erh and uh also co-invented a few patented technologies related to AI and computer vision and uh I’ve been for to the last 12 years I’m serving as a CTO and co-founder at Clickens. So I’m in charge of the technology and the scientific research and the solution architecture of clickens. So uh let’s see what clickens is all about. So that’s Satia just uh made this intro. Clickens is a AI uh vehicle inspections platform that is completely mobile seamless no hardware required. uh you just send the link and the user goes through the guided session and uh everything is also available through the API. So we can automatically or we can easily integrate seamlessly integrate with with any client any native client or other client that the users may have. So uh just to see it in action that’s uh as simple as that user takes pictures. Okay, no video. It’s just still pictures through the guided session. A user is is required to take pictures according to the protocol that uh uh is set for this particular session. Okay, it’s ab absolutely flexible, customizable. Uh the system has a safeguard. So if the wrong picture, wrong side was taken like in this previous shot uh you will see that uh red flag already firing that okay that’s the wrong side. So no tricks no hanky panky with with our system. Okay. And uh the uh nearly immediate response he gets automatically all the damages even a minor scratches segmented out and uh uh the result is is comprehensive conditional vehicle condition report that uh uh maps all the damages to to the specific panels. Uh one of the things that stands out for clickings we we measure each damage and as you can see it’s not in pixels it’s in absolute units like centimeters inches uh and uh all the damages are not only measured we use photoggramometry for that by the way not AI h all the damages are also positioned and and uh uh put on the body style map that uh shows the layout of the are and uh through this map we can communicate with the uh companies and map different areas that is let’s say have a lower granularity that parts the panel. Okay. So uh also we measure damage severity and tons of other things that we we do uh uh on uh during this inspection process and uh the user can see this nice HTML breakdown and drill down into the pictures and see the exact damages. Uh but also it is sell sent in the structured format to the user. So the customers will get a comprehensive report to their systems and they can present it to their users in a nice and convenient way. Okay. So actually the but the real life is in the real life nothing is ever clean. Okay. So as you see this is a real car that I pictured somewhere. I I don’t remember where. Okay. But uh this uh you see this livery uh I believe it will trick most of the AI systems trained on part detection okay on object detection because this pattern can be considered as an adversarial one but uh uh you never know until you you experience this the first time. Okay. So our domain is absolutely complicated. I mean this is extremely complicated. Yeah. So th those are real examples that came that that were sent through our system and we processed them. Okay. On the left you see the car that like this is a closeup of the car with the markings made by the adjuster. Okay. So this part must be replaced and you see those crayon marker markings. But you have a lot of damages here that must be dissected from the uh adjusters command comments. Okay. So that’s a very tough one. But if you look at this at the right side, you will see this uh uh uh car sharing livery texture that this texture hides or camouflages the damage behind this. If you look at this uh door, okay, at the lower part of the bottom part of the door, you will see a damages, okay, there are few damages here, dent and there is also a scratch. So you must be able to act in this chaos. Okay, so for it’s like AI cloak, you know, that’s it’s a a a it’s a very challenging situation for any vision model. Those cases are not simple. Okay, but it can get worse. Okay, look at this. Those are reflections. Okay, those reflections uh through those reflections, you must see the damage. So, you must be able to tell the real damage from a reflection. And the problem is that the the modern cameras, the standard standard cameras, they actually perceive uh the 3D environment through the captured light and contrast. Okay. So actually we got a 2D images that don’t have any depth or uh or any 3D interpretation but uh sometimes because of is of the three of the 2D nature of those photos some damages or some some reflections mathematically may be similar to damages. If you look at the dent or scratch a scuff, it can be visually interpreted as a for instance as a reflection. Okay. So the in fact you can in fact have a scratch on the wall which is getting reflected off the surface of the car and that scratch would be uh it would be you know almost impossible to uh take that you know notice that it is actually not a scratch on the car but a scratch on the reflected wall. Right. Abs. Absolutely. But it it can be even worse. I can tell you the real case. Usually you park, you take pictures on parking lots where [snorts] with a lot of damaged cars. Just imagine this gray car that parks next to this uh to this car has a damages and these damages are reflected on the other car. So mistake mistakenly you can pick them as real damages. Right? So uh and just imagine chasing after all these patterns if you work with real images it is endless process like you need to spend countless hours on acquiring those images annotating them. Okay. Yeah. And uh if you look at the annotator’s work you will imagine that that’s like it’s extremely tough. Yeah. because when you you look at the image you never know as an annotator what’s the ground truth is unless you are familiar with this vehicle. So uh we started our journey as I said in 2014. This is where when the company was established. So we started uh as an anti-fraud solution for insurance companies and then we transformed our business model to to vehicle inspections and we work in insurance area still work in insurance area and the automotive space. Uh so the first generation of our synthetic data as you look at the left side was like nonrealistic. Okay, it’s absolutely largely artificial. Okay, I must admit I did this when I had zero knowledge of 3D and simulation. Okay, that was my first attempt and exercise. But uh it was successful because the model that I put a very simple object detection model uh that was able to generalize even over those artificial samples and uh find the major panels on the vehicle. Mhm. So then it’s evolved into second generation of our synthetic data which you see in the middle and it is more realistic scenes still uh pretty rudimentary right and this uh uh still object detection. Yeah. But but when you go to the market and we started selling our solution we started talking to the customers and we realized that customers require fine segmentation. You cannot use bounding boxes because you need to be able to h to absolutely accurately create a list of detect defected parts. Yeah. And you cannot touch adjacent parts and the damage must be measured across each and every panel. So if a damage spans multiple panels, you need to measure the relative portion of the damage on that panel. Without fine segmentation, instant segmentation, you cannot do this job. Okay. So uh uh when we started growing up and we raised money, we hired professionals. Okay. So today we have a 3D team. Uh those are 3D ninjas that are absolutely you know uh 3D killers. So you see the quality of third of the third generation of synthetic data which we are having today and it is even more profound today. page is more sophisticated than those two examples, but you can see different patterns that can be simulated. A professional adjuster will not tell you if it is a real car or a fake one. Okay? And we can make sure that he will not. Okay? But again, this is not the idea. The goal is not to be as realistic as possible. The goal is to make our models learn and perform well on those uh things, right? So as you see the the uh the customers require an unprecedented level of accuracy in such a noisy domain with all those reflections and uh and all those uh uh things you know you still have to be accurate on each and every panel. You still have to be accurate on each and every scratch and uh uh still you cannot be 100% accurate. Okay. So we never replace human inspectors. We just empower them with our tool. So we give them the ability to tell okay you know this is a hallucination let’s remove it. Okay so it’s just one second toggle and this uh scratch is gone. Okay if AI let’s say mistakenly marked something but uh those are the things okay those that the things those are the expectations from us. So how do we measure the model performance when we generate synthetic data? Okay, that’s a very valid question. Okay, so the ex excellent synthetic data absolutely is not the same as an excellent model on real data. Yeah, it’s it’s like uh different things completely different things. So uh if you look at this so synthetic data examples okay on the in the first line on the left you see that those are excellent photo realism right so it looks like those are real ones. Mhm. Okay. But still you have uh your model underperforms on real in the real world. Okay. Because in the reality we face uh absolutely insane and bizarre patterns. So when you measure the distribution you see that there is a synthetic data distribution and there is a real world distribution. Okay. So essentially there is a gap between them. But you need to realize you need to to to to figure out what kind of a gap you have. If you have a data gap or a model gap and that’s pretty essential question. Okay. So if you have a data gap then you need to simulate more data. Okay. But if you have a model gap, oh, you need to stop and review your algorithms. Okay. So I will not deal with the let’s say our model algorithm because it should have a secret source and some parts of it are patented uh technology but still the model must be uh trained to ingest synthetic data and to process synthetic data to work on this. Okay, I will not tell you like it’s not a window dressing that it’s we we don’t use 100% of synthetic data for everything. It’s just absolutely impossible at this point. Yeah, but uh most of our data is synthetic and we use real data for absolutely uh specific patterns that we cannot simulate and we cannot reproduce. So how do you validate those gaps? Okay. So, we actually partnered with incredible guys from Latis Flow back like some years ago. This is a Swiss company. They are doing governance for AI. Okay. Very very bright minds there. So, we partnered there and they use their platform to analyze our synthetic data. So, if you look at how this looks like, okay, those are synthetic patterns loaded into the into the platform. Okay, this is a synthetic image of a car. It has certain characteristics, attributes and uh each and uh every image is actually uh represented as a point in this uh latent space. Okay, so it’s uh the computer vision models they don’t see pictures as we do. They see embeddings. Those are highly dimensional vectors with numbers. So usually similar pictures uh will be clustered together. Okay, if you look at specific embeddings, so in this embedding space, if you map this embeddings to this latent space, you can see that the similar images are grouped together. In this case, those are uh images that uh any expert will immediately tell you that those are front views, right? Yeah. So, so that’s uh that’s pretty easy. Okay. So, I picked this cherry picked this one because it’s it’s easy for a human to uh to understand this. So, uh have you uh by any chance uh tried uh this uh similar company actually it’s called three lines of code and they allow you to visualize uh the embeddings as well. Um very interesting. We had an interesting webinar with the Yeah, shout out to the homie Paul. Yeah, Paul. and unfortunately not but I would love to learn what these guys do similar uh yeah it’s basically uh so he won uh the one of the Kaggle challenges by removing data right so the data actually had two uh synthetic data sets and four u regular data sets and he found that one of the synthetic data sets uh when you embed it uh the embedding was outside you know when you cluster it was clearly outside the uh real domain Right. So, uh he removed all that data and he removed I don’t know maybe 90% of the data to win the challenge. Instead of using all the data, he actually removed all the data to uh win the challenge. Very interesting. Yeah. Garbage in garbage out. Yeah. Still true today. Exactly. But we are coming to this actually. So this is coming. This is coming. Okay. So uh what what’s going on here? Okay. Look at this embedding space and this is a synthetic data. Okay. So now let’s add real data. Okay. That’s exactly what you mentioned now. And you see that real data has a different distribution. Wow. Yeah. Okay. So what happens when you look at the cluster of the real data? Okay. That is definitely not synthetic. Okay. Yeah. So those are not represented in synthetic space. And uh when you look at this it requires a bit more expertise from domain expertise. But again that’s one is still pretty simple. Anyone like any professional will tell you okay what is common about those images? Any guess? Let me carefully see. Uh
it’s it’s tough because um it is see if anybody in the audience out there may be having the same trouble as me. We’re we’re very close to the release of Grand Theft Auto 6. And so all I can see in the in the early renders, okay, Grand Theft Auto 4, now Grand Theft Auto 6. They’re all in the garage. Maybe that’s that’s the Oh, nice. Yeah, you have a sharp eye. Yeah, exactly. Those are indoors images. Yeah. So those are inside the garage. Okay. And what happens that the these images are that stand out. Okay. And they have a different distribution. But okay. So what though real them real images have those and synthetic images do not have that. Okay. But what happens actually the question is does this affect the model performance? And if you look at at the at the slice, okay, so let’s slice this data set and let’s pick a few uh indoors images. Okay, those are positive examples. And this tool actually has also the ability to mark negative with X. Okay. So we explicitly tell them the platform okay those are positive and those are negatives. So it it can better classify the examples. Okay. And when you run the model on those on this slice of indoor images you see that the model performance degrades. Actually it drops by 70% 17%. Right? Okay. So it means that these indoors actually contribute 70 17% to the model accuracy. So what uh what’s next? Next actually we we are back home and the uh 3D team uh starts working hard and they say okay we need a lot of images okay and the idea is not to produce millions of images. It’s the idea is to produce enough images with different pattern that can cover and close this gap. So we created those indoors images in different context in different environments. Okay. And then we added this corpus to the data. Okay. Now let’s see what happens with the distribution. We are now adding the newly created data. And you see it gracefully joins the distribution of real. Yeah. So if you see here you see a mix of synthetic and and real images. So these images nicely join the distribution right? And if you measure the uh the matrix nice, if you me measure the performance of the model, okay, then you see that the uh let me just go back here that the model improves by 19% 19.7% overall but a very nice observation that the model improves on also on outdoors images. Okay. Plus 6.3%. Yeah. Okay. And it can be explained by just it was simply overfitting on the outdoors, right? Okay. So this solves problem for both indoors and outdoors. Okay. But you cannot chase after each and every bias all the time. So in this case, so this particular system also picks a automatic uh automatic ho highlights. Okay. So for instance uh one of the automatic uh picss automated slices comes to the picture and you see that uh if you slice this by this particular finding the model degrades to to 45%. That’s a very wild thing. That’s a very bad thing. Okay we have to analyze this. Okay. Do you think what’s the problem with this slice? Uh that’s a I mean I if you uh snow or what can you go back uh to the uh grid view? Yeah. Yeah. It’s it’s one particular vehicle type which is maybe not that common, right? It’s uh whatever that vehicle type is. It’s a Citroen cactus. Yeah. So it’s a Citroen cactus with this air bumps. You see those air bumps? Those are were particularly designed by company called Bus, a German company. By the way, we won the first digital damage recognition competition by bus. It was a world competition. So, we are familiar with this one. So, believe me, we we have created a lot of Citra and cactus. Okay? So, we reproduced this cars. Okay? And still, okay, I’ll go back. So still the model underperforms. Okay. We have a lot of synthetic cactuses. Okay. But but the model underperforms. So so what’s what happens? Okay. So here the collaboration with Latis flow. I’ll put it aside. Okay. And let’s go to the next tools that we are we are developing internally. Okay. And this one it’s uh something that uh that you guys uh can do uh without any buying any software or commercial system. So what we do we actually use here a a dino v3 backbone. Okay. So it’s a frozen backbone. It means we use internal weights coming with dynino v3. We don’t train dino v3. Yeah. Okay. So you can use it out of the box because it has the particular knowledge of the entire world or it knows all the objects. Okay, definitely cars. Okay, so using Dino V3 we actually create patches of real images. Okay, so it creates a grid of patches and we create a bank of patches. So whenever we analyze a synthetic dam synthetic image okay we actually compare we use a KN&N algorithm to to tell how how far our synthetic pattern is from the real one. Okay. So and uh this we call a realism map. So we actually measure the distance and this distance is represented by a heat map as you see. So look at this example and you will see that this entire air bump was flagged as unreal. Okay. Right. It means it was very far from the real world. So let’s look back at this the left side and you see our synthetic data. It looks pretty nice, right? Yeah. It’s pretty realistic. Okay. Yeah. But still there is a problem with that. Okay. So what’s the problem with that? The problem with that there are two problems with that actually. So first of all the problem the first problem is if there is anything like this in the entire real corpus. Okay and this is the first question that we need to ask and second question we need to ask if this particular patch is actually close to anything that uh that is real. So for for the first problem for the first question and we we absolutely must answer the two questions. So the for the first question the answer is actually we can use a train discriminator. Okay. So h this discriminator actually trained on those uh patches created by by uh Dino V3 backbone and uh this one tells us actually uh okay if you have ever seen this okay if you seen something you have seen something like this and this discriminator actually is trained to detect a a bad pixel a dead pixel or some artifact and it stops processing immediately there. So it says okay there is a pattern okay that’s that’s one is actually positive and the second one is actually a frozen feature KN&N it measures if it has seen this pattern in our data set so when you answer let’s say we have those patterns in our data set but discriminator says okay there is a problem with this patch so here actually is a uh we need to hand it over to the professionals. Okay. And at this at this point we hand over this heat map to this 3D team and they immediately spotted. Believe me the domain experts they immediately see the problem. Okay. So the problem is that this uh material uh has a lack of specularity. It doesn’t have Yeah. Exactly. So it should be a map. Yeah. And when we uh look at the reference images and we see the cactus, it is matte. Actually, it doesn’t have those micro, it has those micronormals that scatter the light on this material. So, making it matte and less reflective. Okay. So, this is the idea. And uh uh what you see on the right side is actually the reproduction of this material by our 3D team. So we recreated it. Mhm. And we fixed those materials and we ran it through the same procedure and uh these heat maps are gone again. This one this problem goes away. Yeah. But of course the next one will pop up shortly, [laughter] right? Okay. Yeah. So uh quick question. Uh you know uh you mentioned that you’re using a 3D team to generate all these images. why not use uh a generative uh approach where uh you know then you will not have this kind of uh problem. So here obviously the 3D team selected a material uh initially which was specular but uh if you actually used a generative approach then uh you would never get that right. You have excellent questions. You know why? Because I have perfect slides for [laughter] those. Okay. Sorry jumping jumping ahead. mutual appreciation society over here. [laughter] Yeah. So what what happens that first of all we started this 40 years ago. Okay. Yeah. No one heard about generative AI back then. Yeah. Okay. So we were pioneers in this space, right? I know that even Nvidia in their internal meetings and we partner a lot with Nvidia guys and they say they mention clickens all the time. Okay. Because we were ahead of time. We were ahead of their of their omniverse thing. We developed the same thing like many years ago for our internal purposes. Okay, we have our proprietary environment. But the the uh major problem with generative AI it’s not a problem with generative AI. Our system it can it built actually has a CAD CAD component. M so this component is actually the one that automatically annotates those images. Got it. Generative AI produces specially incorrect images. I mean recently they started improving. Okay. We we have seen this nana banana model like the latest version is absolutely mindblowing. Yeah. But still there are glitches here and there but uh this is the one problem. You must be absolutely pixel perfect with your annotations. Okay. And when you create generative AI thing, yeah, you never know where this lands you because it reconstructs the image. It doesn’t preserve the original geometry, the original uh pixels, the original h details that we want to emphasize, we want to particularly simulate. Okay. But if you want to bring more realism, that’s where generative AI comes to rescue. Okay? Yeah. So look at those. I picked two samples of synthetic data on the left side and we augmented them with generative AI actually generative AI and it was like a long journey to make sure that generative AI does not change the dimensions the position of panels and the all the engineering aspects of the vehicle but only only adds ad hoc objects those that happen in real world. Right. Here it also uh hallucinated some damage and put it a ruler which is very common in insurance by the way. But you you see those generated images are they were cherrypicked but uh this they bring more realism to and bring more noise real world noise. Okay. Sometimes it’s even noise of the sensors matters. Okay. Right. Today our latest uh latest synthetic data uh is has already all those things in inside. I mean we have all this uh distortion and uh all other things completely simulated but generative AI uh it’s a very good thing. The one problem I’ll tell you frankly it is very expensive. Yeah. So if you want to get a decent result, you need to use the uh top-notch model from Google, right? Okay. And if you use it, you will pay you will pay a big bucks for this. Okay. It just doesn’t pay off. So at this point, the the other problem with geni is that you do so much any model you choose, right? It’s actually not going to be perfect. So you build a lot of things around uh this model so that it it works decently and two months later all that work is useless because the new model that came out doesn’t have those problems right so that’s uh we started with we started with diffusion models like uh quite a few years ago before before it became a hype okay we started using diffusion models because one of our PhDs uh he’s actually his professor invented stable diffusion in Germany. Nice. Okay. So he he actually contributed a lot of his knowledge to this uh to this thing and uh we tried different versions of table diffusion but uh all of them were not specially specially correct. Okay, that they were not pixel perfect and of course they distorted the reality. So you couldn’t you couldn’t maintain the automatic annotations aside the those generated images but if it is completely reconstructed and generated you have no control over the objects and their positions and their and their masks. Okay. So this is like a kind of a dead end but uh we are trying to maneuver still we are using our core and sometimes we use generative AI to augment our synthetic data with even more chaotic and uh and uh bizarre scenes. Okay. So what is the size of the data u what is the size of the data set for these things? Are there like uh 100,000 images? Are there 10,000 or million images? A a good thing with synthetic data that uh whenever you have like 1 million real images annotated, you just roughly need maybe 100,000 of synthetic. So um synthetic data requires less patterns significantly less because it is absolutely generalized. Okay. So since it is abstract, it covers a lot of scenarios. So, but you can maneuver and you can simulate every scenario very fast. Okay, that’s something that you cannot do with the real data. Mhm. Because chasing, you know, it’s very difficult to find this uh let’s say this Mercedes or or this clever shuttle. Okay. Okay. Uh that’s very challenging. Yeah. So, so synthetic data comes to rescue but still for some edge cases we still annotate real data. Okay. But this is a specialized team which is like uh skilled and these guys do only this. Okay. So they just learned a lot of things through the tenure of clickings. Yeah. This is the same team that does this all the time. So they are extremely professional already. Okay. And we also have a quadrails on our side. We have a safeguards our QA team that uh checks those things and make sure that no bad annotation leaks into the model. Okay, I cannot tell you that it doesn’t happen. Okay, it happens but uh we are trying to to minimize those cases as much as as possible. Okay, so uh but this thing that let’s this sets the stage for the first slide that I showed you. Okay, though this adversarial attacks and this is something that is maybe underrated today or is not talked about uh talked through extensively but that’s another selling point of the synthetic data. Okay. So uh what if the real world is designed to fool the detector and what happens that those uh uh researchers uh around the globe what they do they actually do uh object detection in reverse. Okay. So they try to uh to find they use gradient descent to try and find the patches that uh uh that cause the uh actually in maximize the detector’s error. Right? So once they find those patches, it’s very simple to fool any vision model with those. Okay, those are real ones. By the way, those are three different researchers that I picked with permissive licenses. Okay. So, so that those are that’s why I I safely using this in in the presentation. So, we just use this patch and they covered the car with those uh with those reps. Okay. So uh but again if you do one of this and you cover the car it will make it will register absolutely nothing on many uh object detectors. Okay I don’t know about all of them. Yeah but but many of them will fail and uh let’s pick this one. Okay, that this is an example that the left side is is a synthetic car absolutely fake car simulated by our environment with some kind of synthetic damage. Okay, so we covered it with a with a pattern with a few patterns and ran this through the uh through YOLO. Okay, so this is a print screen from YOLO YOLO 26. Okay, interesting. So I have it running on my on my computer now. What’s the size of the model? Uh is it the I used the largest one? Largest one. Okay. So that’s Yolo large actually. And so Yolo is just thinking why does this guy keep showing me pictures of John Cena? [laughter] So the the the thing is that I don’t know uh I have not read that paper but it is also possible because YOLO is so popular that it was specifically designed to break YOLO. Right. You you actually mentioned that I believe so. Yeah. Right. The the the paper actually the original paper was specifically designed to attack YOLO 3. Okay. Okay. But again uh we just a bit uh uh we use that this pattern and uh we tried it on different angles. Uh it’s not the same on all the viewpoints. Okay. on some on some viewpoints where you see a lot of real car features like wheels like you know antennas uh the detectors actually detect the car okay sometimes they say that it’s a train sometime they say it’s a truck I have seen oven I have seen a kite okay I I have seen different things but uh it’s never a private car okay it’s never a sedan but uh uh the good thing here that uh the model, the same image runs through our system actually lands us on a very perfect segmentation and object detection. Nice. Why? Because our system is mostly trained on adversarial patterns. So when we architected this solution, we had this in mind that this system can be attacked. when you deal with fraudsters and that’s what I said we started with as as a counter fraud solution you must be ready for any adversarial attack and this is a I would say a significant advantage of the synthetic data so think of it as a let’s say as a biological immune system okay so if you see the same patterns all the time okay uh you you don’t have this uh vaccine in okay you will fail on each and every each and every attempt. Okay, but if you are actually simulate for this, you are synthesizing a vaccine for this. So we we are synthesizing this and we are prepared for even uh you know more bizarre patterns. Okay, I have a lot I have a folder called exotic patterns on my computer. So [laughter] I try different stuff and whenever I you know I travel and I see uh very interesting patterns I always picture them and run through the system to see that it doesn’t fail us. Okay. Right. That’s very important. Okay. So uh and the good thing that this is not only for cars. So this the same technique. Okay. I probably I didn’t mention that the core of the system is ontology. Okay. So we have the ontology of this domain. Yeah. So it’s not only the visual simulation, it is a visual simulation with automated annotations that come from ontology. And whenever we do inference, we also use ontology to infer on our ontology to make sure that we reduce the hallucinations of of the model. So it’s a comprehensive approach. It’s not only just uh you know render millions of images and solve the problem. So you don’t have to render millions of images. You just render enough images that close the gap and that’s why you need to Yeah. Also I mean for people who uh you know who are new to computer vision or uh who are uh learning uh what Dimmitri makes a very important point here is that it is not millions of images. The information uh I like to say the information is not in the number of images. it’s in the variance of your uh information is in the variance of the data right so in the extreme case you can think that oh if I have the same image copied million times it has zero variance right it’s the same image uh copied a million times so it’s not adding any more information right but if you have 100,000 images or even 10,000 images which provide a lot of variety then the model learns right and if you you may think that oh If you have more more data, it cannot be bad. It can be bad because you’re spending a lot of time and energy in processing all that data which which is not adding anything to your final uh accuracy. Right? So cleaning up the data to get the sort of minimal set. You just need to close the gap uh as Dimmitri said uh not just flood the thing with millions of uh images. Right? So this is a very uh common rookie mistake. people who are new they try to get uh you know they say oh if I had a million images no you don’t need a million images for a lot of things you just need the right uh amount of data absolutely go ahead I will second that yeah I will second that I will tell that’s absolutely correct and uh imagine that we would have rendered millions of of those citrine cactus with air bumps which are not realistic okay imagine if if we have done that we just spent our we wasted our resources. Yeah. GPU time and uh you know our team time and at the end of the day we would just overfeit on those patterns and we would never we would have never h address the real pattern. So it it’s sometimes it’s enough to have a even a few dozens of images taken from different colors with different colors and different scenes. Okay. so that the model will generalize on those patterns. Okay, of course they must be realistic. Okay, they cannot be like out of the blue but uh they should be the realistic and the same approach the same approach applies to any domain. Okay, so we tried this on boats, we tried this on two wheelers by the way all those are uh generated by our environment. Okay, synthetic environment. This B2 fighter is actually uh nice. Okay, I love this. All I can see still is Grand Theft Auto 6 [laughter] one track mode. It’s with Clickens logo by the way. Okay, so uh and you can also use it with aerial images and the satellite imagery. Okay, the same principles everything everything applies. Okay, so the good the I would say the the major advantage here is automatic annotation. Okay. So you can produce you you can use generative AI first but be sure that you have the tools to automatically annotate it. Yeah. Okay. Because uh that’s not straightforward. And uh this actually uh the takeaways here is that synthetic data is a control knob here. So it’s um become something that we dial that with uh generative AI we can actually even improve more we can introduce more noise more real world uh chaos into synthetic data still controlling the accuracy of the annotations and the accuracy of the model. Okay. Then we we need to be able to find the gap uh either using a third party software or developing algorithms like this uh uh realism maps that I showed recently. Uh robustness actually is a data problem. So all this uh adversarial wraps that blind the generic detector, it’s just another slice to render, simulate, label and train. Okay, so uh if you have this in place and you have models that can learn from synthetic data, that’s a very important thing because any model tends to do shortcuts, you know. Yeah, it will it will all of them are prone to shortcuts. Your job is is actually to make sure that model doesn’t do shortcuts on on any synthetic pattern so that it treats it as absolutely real. Okay. And of course the same engine can be applied to different domains. So actually that’s the end of the presentation. I’m curious with the new Astra model which can do amazing things with CAD. Have you found that your 3D artists can now create these 3D assets very quickly compared to before? Since it’s pretty new, we are still learning this. Okay. I can tell you that uh they announced that there they have a native integration with Blender. Yeah. Okay. And uh Blender is one of the tools that we use behind the scene, but it’s not the only one. Yeah. And Blender it is it does this integration through the MCP, Blender MCP. And I can tell you that we exhaustively tested this Blender MCP and we found it uh um like it is it is very limited I would say. Okay. Okay. So it’s it’s some tasks we resorted to human work. We tried to automate some manual tasks and at the end of the day we resorted to a human professional 3D expert because it he can done that uh faster and more accurate. Sometimes it is very difficult to explain the model the intricacy of the geometry that you need. Yeah. Okay. And maybe it’s not a model problem. It’s a human model interface which is sometimes uh challenging. Interesting. Okay. Yeah, I believe Astro I believe that we are we’re living in the era where everything is so dynamic and and we see new new models arriving like every week and uh I believe an amazing things are coming and definitely we will keep an eye on all those new models and see their capabilities and once they can uh replace any parts of the uh this sophisticated R&D uh of course we will consider that. Nice. Yeah. Thanks so much for the awesome presentation. I certainly learned a lot. I hope the audience out there did as well. And we’ve got some questions for you here. But first, it is time for our trivia segment. Before we get to our trivia question, I want to remind everybody out there that OpenCV is a nonprofit organization that releases open- source software. And as such, we depend on the support of our sponsors and partners. Um, I want to take this moment to thank them right now. Big shout out and love to our gold sponsors, ARM, Futureway Technologies, Google Summer of Code, Qualcomm. Silver members Rooflow, Orbit, the British Machine Vision Association and Devpost, and bronze members Jet Brains, the Edge AI and Vision Alliance, Open MV, Tanggram Vision, AMP Software, Intuitivo, Rerun, Intrinsic, Buyers Dev, and Big Vision because every company needs a big vision. I’m contractually obligated to say that at the end [laughter] of it.
If you’re joining us for the first time or you just need a little reminder, one of the things we do on this show is a special giveaway to you out there in the audience. I’m about to ask you a trivia question. Put it up on screen here about today’s presentation. I have cooked up a pretty esoteric one. I think I’m going to stump you guys. I feel pretty good about it. If you’ve won in the last couple of months, please don’t answer and give somebody else a chance to win. Type your answers in the chat wherever you’re watching. And if you’re on Zoom, make sure that your settings are set to everyone and not just hosts and panelists so we can make sure that we we see your answer. Um, without further ado, here comes our trivia question. So, uh, today we talked a little bit about synthetic data and how Clickins is using it for vehicle inspections. And at the top of the presentation, our guest Dimma said that there was no hanky panky. [laughter] What band had a number one hit in 1966 with the song Hanky Panky? I’ll give you some hints here. This is a Michigander. [laughter] Um he’s he’s from my hometown of Niles, Michigan. He had a band. He they also had a couple other hits. Mooney Mooney, Crimson and Clover, Crystal Blue, Persuasion, a couple of big ones in the late 60s and early 70s. Uh he later went on to be a part of some a couple other bands, but I’m just, you know, how long will it take people to to Google this or ask Claude and and does Claude get it correct? Let’s find out. So, what band had a number one hit in 1966 with Hanky Panky? It was not their original song. They It was It was a cover that was written by somebody else and became It’s kind of an interesting story. It was back in the days where you could have regional music hits like a song would get very popular in a certain part of the world uh certain part of the country and then that would lead to a band becoming big instead of now where it’s just kind of everything happens all at once it seems for a lot of these and we’ve got our answer in Zoom. Uh, Carito Gr, the answer we were looking for was uh, actually you’re you’re off. You’re off. Uh, Carito, you’re you’re this the last name is incorrect. And so, and we need the full name of the band. We got some close ones, Tiana. All right, Caro. He he he amended his answer. We’ve got the We’ve got the correct one. We had some close ones in there. The answer we were looking for was Tommy James and the Shondell’s. Tommy James and the Shaondell’s. A uh Tommy James grew up in my hometown of Niles, Michigan. One of the one of the claims to fame of that of that Littleberg. Um so yeah, congratulations to Caro. We got there. Uh please send one email to me. That’s Phil opencv.org and we will make sure that you get whatever course you want. You can see what courses are on offer by going to opencv.org university. Um, Caro says he hasn’t heard that song since his last visit to Lancing. Yes, indeed. Uh, Michiganders, respect, respect to Michiganders. So, we’ve got some questions here in the chat. I’m going to bring up one on screen here from one of our longtime viewers. Uh, Tiana on Zoom asks, “How do you identify meaningful images to reduce the amount of images when you’re training models like this?” It’s we talked a little bit about how important that data integrity is and the data quality is famously the May West once said back in the day too much of a good thing is wonderful. That is broadly applicable to many things in life but it is not true of data like this. H how do you think about the data and how do you inspect the data to make sure that you’re not wasting time as we talked about here with unnecessary or even uh harmful to your model data. So first of all you must absolutely control the distribution of your data. So since it’s a sulated data you have a full full command of all the makes models conditions and environments and whatever. So you must absolutely track all the parameters and you must distri manage the distribution. But it doesn’t mean that you need to have let’s say 50% of black cars and 50% of white cars. So you just need to follow the distribution of cars according to the specific geography. So if you’re deploying the system in the US or deploying the system in other countries okay we have the same system that serves everyone. It’s a multi-tenant cloud solution but uh the model must have representation of all of them all of those geometries and geographies and you need to control the distribution first and uh that’s uh something that I also mentioned this in the presentation you need tools for that either commercial or your proprietary tools today I believe you can use all those uh VLMs uh vision language models that uh um uh that that can classify images and they can help you with those captions and uh and embeddings. But uh that’s the job of uh uh embeddings. Okay. So you need actually to analyze this in in different latent spaces and and see which clusters pop up. So if you see the new cluster appearing and it’s under represented you need to address this because uh and you need to be able to track the model performance to to measure the accuracy and uh be able to slice the data and see the accuracy on each and every slice. That’s pretty tough uh challenge. So for this you need tooling okay and either your proprietary tooling or other tooling it doesn’t matter but you need some automation in place. It’s not that a human can chase after all those distributions. It’s just not not not a human task. Okay. So, we just need some automation and uh and analyzing analyzing the data. But uh you never have uh the proper distribution. You always chase after new trends and after new ge geometries, new vehicles released to the market. Our system works on any car. Actually doesn’t matter if this car is a new car that is a concept car. We we will detect the panels on each and every car even not released to the manufacturing yet. That’s due to the synthetic data. But uh you must be able to process all the different panels. So you need to know them. You need to represent them. So you have only doors in your system. Then you will fail on fenders. If you have fenders, you will fail on bumpers. So you just need to add enough data but don’t be that uh I mean you not you don’t have to be obsessive with that. Don’t create millions of images. Start with a decent number and measure the performance and see if you need more. Yeah great great answer there. Anything to add to that Satia? No that pretty much covers everything right. It is um like I keep repeating that the information is in the variance of the data not in the quantity of the data. And so we have to capture the variance of the data and uh you could do it incrementally right find where you’re making mistakes right uh and that you know unfortunately most of these tasks require a human to go sit with the data and analyze it figure out what’s what’s going on right so a lot of times people just look uh look at graphs and say that oh my my accuracy is this much what should I do without actually spending the time with data But uh most good u you know engineers I know they spend a lot of time um looking at the data and fixing the problems. So I’ll give you [clears throat] an example and I’ve given these examples before also Davis King uh he had a facial landmark detector and uh you know he had produced a model uh the paper was the same. So somebody an intern at OpenCV uh Google summer of code had produced the same model using the same paper right so uh but the accuracy was not as good right so I um I mean I finally reached out to Davis and he mentioned that you know uh it’s all about it’s the exact same code not exact same code but similar uh code but he had spent like seven days right he put on his headphones and seven days he looked at each and every uh image in his data said there are 68 points on the face. Each and every point was manually verified uh and he put it in the data and that made the the difference. Right? And I know several several um uh examples of people who uh basically produce spectacular results just because they spend time with the data. [clears throat] Yeah. Sometimes you just got to read all the papers as we as we said. In fact you know in imageet visual recognition challenge they have this human level accuracy. uh they say that it is 95% uh hum a human gets about 95% accuracy that human was actually Andre Karpathi right and say why would why would karpathy spend uh you know seven days or whatever huge amount of hours looking at the data because because he wanted to understand it he wanted to understand it right I think I think you give too much credit to humans because the professional adjusters we were told by one of our uh uh customers they measured the accuracy of the professional adjuster in Europe, okay, which is very pedantic and like at least 30 40 years experience, his accuracy in damage recognition was not above 80%. Wow. Yeah. Okay. Wow. Yeah. Yeah. [clears throat] Those insurance companies, I tell you what. [laughter] So, uh, thank you for those answers, guys. We’ve got a couple more questions here. A little bit of time left. Uh, Caro on Zoom asks, “Is there a contact email for Dimma can refer us to to communicate with the Clickins team? What’s the best way to get a hold of the Clickins folks?” Dimma,
dimclickins.com. That’s simple as that. Easy peasy lemon squeezy. Dima at clickins. Right on. Appreciate it. Dima. Yeah. Um, this one is a little bit similar to our uh previous question, but I think maybe we covered it, but uh this person’s working on a project that can use only synthetic data, but the test will be on real photos. Do you have any tips or guidance or advice for uh Mike Watson here on his project?
It’s actually depends on the domain. So if they have only synthetic data just for for the training I believe it can be anything medical or maybe insurance because they don’t share the data with with the companies uh and if they have access to real data uh let’s say different real data from a different domain they can measure as I mentioned using like Dinov3 and KN&N they can measure the distance uh between the their synthetic data and real data just to see if they are close. Okay, because they can be completely off. Okay, but if they measure the distance to real images even not the real ones that are used by this customer but real ones that used in the same industry, they can at least have some kind of a baseline. Okay. But if they are completely uh they don’t have any access to the real data. So they need to to train measure it’s you know it’s trial and error. You you train the model you measure it on performance you uh you see the shortcuts. If you see that the model doesn’t learn anything you will see the loss dropping dropping immediately. Okay. because due to the shortcut okay and when you train those models you need to identify those shortcuts and see okay let’s do a lot of augmentation okay let’s add augmentation maybe gener generative AI okay let’s augment those images to to look more real okay and see if if this closes the data gap so when you close the data gap you must be absolutely confident that you don’t have a model gap so that that the algorithm is perfect for for this uh for this type of data. So that’s a like uh multi-step process.
I I I hope it answered the question. Yeah, I think so. Um any anything to add to that one, Satia? Uh well, first of all, I think that you should get you should try to get, you know, beg whatever client it is to get you some real data. It is extremely difficult and you can get blindsided if you are trying to u do the synthetic data without access to any real data. I’ll give you a very simple example to just illustrate the point right let’s say somebody tells you about oh they want to do person detection it sounds like a simple problem right but then uh so you you basically train a person detector model it’s easy to do you get plenty of data and then when you look at the test data it could be a thermal image right it could be an image which is from uh looking from the top of a building uh down and you had not even thought about it right and I’m picking these simple examples but uh in In the real world the examples are much more complex right the situations where uh you could it’s sometimes beyond your imagination no matter how hard you think it is very difficult to come up with all the situations in which you will land when you see the real data right so uh I would say that you know u you know try your best but try to get some samples right from uh from the client which are representative samples even if they say that okay uh we cannot give to uh 10,000 images ask for 100 that u include some representative and some corner cases right so you need to have some idea of the real data otherwise you can be shooting in the blind yeah there is another thing that you can ask the customers to anonymize the data so let’s say some of the customers are actually not uh ready to share the data because of a privacy so in our space in our domain Those are the license plates. Definitely. Okay. So for license plates, it’s pretty easy to blur them. Okay. So blur the license plates and blur the faces. By the way, that’s what we do in our system for the customers that care about the GDPR and privacy for those one that explicitly ask. We do this. We aposcate the faces and and reflections of faces and reflections of reflections of faces. everything that okay that can reveal some human humans in the image but for the medical in medical it can be challenging okay what you cannot obiscate the MRI scan that’s just not possible so that’s I don’t think that we have a a meaningful advice to these guys rather than try and uh try and error and try to achieve some data maybe from a similar area. Okay. But real data to be a reference. Yeah, get as get as much real data or as close to real data as you can seems to be the advice here. So, uh, best of luck out there, Mike. Got a couple more questions here. Uh, Sedarth asks on LinkedIn, “Do [snorts] camera specifications like focal length, aperture, etc. matter for training data? Does your data set need different types of cameras, different focal length, different uh even uh you know uh different angles, right? Like a 30 degree lens versus 180 degree lens are going to produce very different data. When generating simulated images, do you need to have a data set for different types of cameras or um and how does your system work with this demo? Brilliant question. uh uh you absolutely must simulate sensors, camera sensors. So when we simulate when we do simulations, we actually simulate sensors, we simulate aperture, we simulate everything. We simulate atmosphere. Uh this is where the simulated data, 3D simulated data outperforms the generative AI because simulated data uses geometry and uses HDRI to uh to accurately deliver the physical aspects of this scene because the physics is preserved. We follow the real physics of the reflections, the phys real physics of the light. When in generative AI or all those uh you know language models that produce images, they have zero knowledge of physics. Okay, they can create reflections that do not resonate with the backgrounds with the back plate. Okay. So uh so those things are not yet resolved. Okay. And you see Lun working on physical AI. Yeah. Okay. and and the other companies working on on this space to make this AI aware of the physics of the physical world of the optics. But you absolutely must simulate data on different uh uh uh camera specification. Again, it depends on on your uh use case. What is your uh usual input to the system? Do your users use mobile devices or do they use digital cameras or maybe uh stationary cameras? Okay, so if your uh your system must deal only with stationary cameras, why why would you uh simulate mobile shots? Okay, so you should focus on on the stationary cameras but have a wide range of those. So uh when you do simulation, you actually control every aspect of the scene and everything matters. Distortion matters, barrel distortion, radial distortion, everything matters. The specularity, the ambience, the diffusivity, the reflection of material, the glossiness of material, everything makes sense. Everything is very important and that’s why, you know, it takes like uh 14 years or 12 years to to develop the tech because it’s a very very deep tech.
Indeed. Thank you for that thorough answer and thanks for the question, Sedarth. I think that is all the questions we’ve got here today. Yes, indeed. Thanks so much for the presentation, Dimma. It was great having you on the show. Um, I learned something as well. Yeah. And, uh, I hope everybody enjoyed my thirst for Grand Theft Auto 6. [laughter] And, uh, our trivia question about Tommy James and the Shondell’s. go listen to some uh some late60s American pop hits today out there folks. Uh you want to any any final thoughts for the audience uh here Dimmitri?
No I’m I’m happy to be here. Thank you Malik. Thank you Phil for inviting me. It’s absolute pleasure and absolute privilege speaking to you guys and the audience. uh so I would say that uh simulating the physical world I would say it’s the next uh standard or the baseline for the training data because we are running out of real data and uh humans humans it it becomes more and more complicated and challenging to annotate insane amounts of real data and we’re just running out of this data. Okay. So, synthetic data beames the becomes the next uh training data standard and it should be as uh real realistic as real one. Yeah. Thank you Satia. Yeah. Well, uh thank you so much Dimma. It was a pleasure talking to you and u the presentation was very very insightful. I’m pretty certain that a lot of our audience feels the same way. Uh amazing amazing presentation and uh thanks our audience for being here every Thursday and uh thank you Phil for organizing this show. Uh this is uh this is great. I learned a lot myself. Yeah. Thank you guys. Yeah. Thanks so much everybody. Thanks for joining us out there in the audience. Thanks to our guest Dimma. Thank you Satia for your your hosting duties today. We’ve got uh another one coming up for you next week. Same bat time, same bat channel. We’ll be walk welcoming the uh founder of smallest AI to talk about speechtoech AI technology. It’s going to be a new topic for the channel and we’re really excited to see what the state-of-the-art is in that. You can find us here wherever you’re watching. We will be there again next week for you. That’ll be episode 227. Until then, take care of yourselves out there, folks. Take care of someone else if you can and have a great day wherever you may be. Adios.