Transcription
Hey guys, today we're looking at something new. Machines learning to control physical tasks on their own. We're talking about AI moving dirt, building trenches, and running factories. And the tech behind it is a lot closer to Palunteer's world than most people realize. By the end of this video, you'll see exactly how Palanteer's software ties into this new wave of AI controlled machines. This is about large systems working together in real time using real data. That's Palanteer's core skill set. We're going to break down how autonomous machinery actually works. Why data is the biggest challenge in AI for the physical world and how Palanteer is already powering systems like this behind the scenes. Let's get into it.
This is this thing we keep coming back to which is, you know, the 1960s framework. People do what they're good at, machines do what they're good at. This is a lot easier way to work together basically.
That's right. And these are meant to spice into existing fleets. And so you uh and so maybe this thing starts as like a excavator doing heavy digging t digging tasks as part of a crew doing a heavy civil project in Texas. Um and then as uh as this progresses um you start to go to more capabilities and more machines and you you you still have huge amounts of work that has to be kind of like partnered and done. Um, and in fact it's expansionary because you're actually creating jobs that wouldn't be done otherwise. But you have an ability to kind of move higher and higher level and create almost like a operating system for a general contractor to define the work they want by the goal versus micromanaging the machines.
I want to go to the expansionary nature, but let's dig into the tech one time first. We've been talking about bunch about this. Most of our audience is not super technical. Some of them are. Let's just go a little more complicated here. Like like what are the actual AI breakthroughs allowing this? Like what are you using? What type of talent does it require? Like what does that look like in the tech stack? This is a perfect example of why Palunteer is relevant here. You need systems that can connect machines, sensors, goals, and real world outcomes. That means collecting data, organizing it, and turning it into action steps. Palunteer has spent years solving exactly this problem in defense and logistics. They build platforms that help organizations control complex environments. Whether it's a battlefield, a hospital system, or a supply chain. The same logic applies. In this case, the AI is being trained to control excavators and loaders. But behind that training, you need a platform that can understand the bigger picture. Where does this machine fit into the project? What's the plan? What's the goal? That's where Palanteer shines. their software handles this kind of operational control for governments and industries right now.
Yeah, absolutely. So, um fundamentally uh when you think about the tech stack, it's this like big chain that goes from um the hardware to system architecture of how to integrate on the machine, get the right coverage for safety purposes and everything else you need uh splice into the machine itself and be able to kind of like read the signal and control it. So, there's a lot of like hardware and system architecture elements. there's infrastructure, the operating system >> that's actually controlling the machine actually going >> yeah the feedback back and forth uh how do you like pass through data and then create your own signal that basically, you know, we'll dive a little bit more detail there um there's uh controls there's a large scale kind of like model in the middle that is effectively the autonomy stack right and the autonomy system >> so you're creating models of all sorts of things there >> yeah so so in this you can think of this as almost like this giant model that takes as an input everything from the camera light so there's cameras LAR IM Muse, GPS's, uh, uh, like tons of signals from the machine itself and you're, uh, and it controls itself from the human, uh, in a lot of cases where you're using that for training. And then you're piping that in, uh, and you're also giving it a goal of what you want the world to look like. Maybe it's the shape of a trench or a foundation that you're digging. Um, and the model is actually interpreting all of this and extrapolating out to a trajectory of what it should do. And you can >> here's what you do to get to this part of get this result. >> That's right. And so, and you can think of, and it's a complicated model. It's not just like one giant system. You can think of like embeddings for the vision uh components that you can borrow from open source LMS that already have captured kind of like what are the embeddings that are like very powerful for uh for you know cameras or LAR you can um uh so you can do a whole bunch of kind of like subtraining of these systems but at the end of the day what you're doing is you're trying to like replicate and predict what what a person would do and so that output is like a high-dimensional trajectory of what the body plus the bucket might need to do um in this machine. This is kind of funny because it reminds me of discussing like alpha go and alpha chess and stuff where you model it and you guys and there's no equivalent of like move 37 and alpha go where you just do something shocking. There's there's not going to be anything surprisy of like interpreting the patterns and the nuances and again in our case across tens of thousands of hours or hundreds of thousands of hours of data you start to pick up these patterns of how do you go manipulate the world to get it to where you need to go. And um and what's beautiful about it is that you don't even like this is not a problem where you could engineer a solution through rules and heruristics. you have to have this sort of modern approach which even 5 years ago or seven years ago wouldn't have been really, you know, possible and today that's actually the only way to solve these sort of problems and so you have this like big model that effectively is learning and then outputting this trajectory and um and the way to think about it is almost not too different from how a LOM has this like giant context and you you know giant foundation you give it some context and it spits out a sequence of words here it's it's somewhat similar except the sequence is not words. It's actually action space of you know this machine. Um and what's nice is that that architecture it doesn't matter whether you have a excavator doing digging or demolition or a wheel loader or a compactor or an agriculture machine like the um the foundations of that architecture the hardware carries over the safety system carries over everything carries over and then >> so it's a little bit like an LLM but instead of instead of letters or words you're actually modeling it based on real world.
This is exactly the type of system Palunteer was built to support. When you train a machine to act in the physical world, you're teaching it how to understand space, movement, timing, and goals. That's what they mean by trajectory. You can't just predict what happens next. You need to choose and decide what happens next. Palanteer is already involved in these kinds of challenges. In defense, they help coordinate supply drops, troop movements, and mission logistics using AI and real-time data. In healthcare, they help hospitals manage beds, staff, and resources based on fastch changing conditions. The same skills are needed to control large machines on construction sites or in factories. The AI can only work if the system understands the full situation. That means bringing in weather data, supply chain data, safety protocols, and human inputs. Those are all things Palanteer handles today. Things >> I'll put the trajectory. Yes. So it's like a high dimensional trajectory for um you know bucket and body and then you can do inverse kinematics to actually figure out what you uh how you actually execute that that world and so you're um you know so you're effectively if you think of the modularity of it um and then you obviously have a huge amount of infrastructure for like the cloud component for training managing these machines and everything else that's like very complicated and very important in this whole system and then simulation stack and everything that you do for offline development. But um you're effectively um creating this onboard system that is ingesting all of this data. Um the system splicing into these machines where we can basically now take existing machinery in a way that's totally reversible. It's not invasive and and in less than four hours we can take a machine and upfit it to be, you know, to have the system on it.
This is where the future is heading. Fast deployment of autonomous systems. But the real challenge isn't sticking a new system onto a machine. It's managing the data that comes next. Palunteer has already proven they can do this with legacy systems. They've worked with militaries and governments to modernize old infrastructure without ripping everything out. Instead of replacing systems, Palanteer adds a software layer that lets them work smarter. That's the same approach happening here. Autonomous machinery is about building smarter layers over existing equipment. But once the machine is upgraded, you need to connect it to a larger decision makingaking system. Who decides what the machine should do next? How do you track progress? How do you make sure safety and budgets stay on target? Um now you're able to actually um you know take that input and give it you know give a real intention and uh and then a lot of the interesting challenge becomes on how do you get the data to cap you know to really train that signal and how do you actually interface with it how do you set the intent in a way that's human understandable I remember one time we were talking about this you people are measured by their years of experience there's equivalent of like a level five year year or 10 year person maybe someone gets to a 10 year level sooner than or some some of But so so so what are what's the computer at now and what's it supposed to be going towards?
Yeah, good question. So the the interesting thing is for so for humans like excavators are one of the toughest machines. That's why it's like very high dimensional. There's like seven degrees of freedom and like you know you have all these subtle controls on the bucket, the body, the treads. Um so it's a lot harder to learn than a car or a truck to to drive. And so we've heard that often times it takes like four to five years to get really really good at this.
And obviously you can get decent early on. Um we're in the early stages of learning these capabilities, but it's moving quickly. Um and we can jump through these learnings astronomically quicker in time than a human can. A lot of it is actually based on the right structures of kind of like data and the you know the way you you you leverage it. Um and so the way we think about this is unlike a human which kind of brings up their competency uniformly and they're kind of a somewhat okay operator in everything um and then gradually kind of like build their way up. um we're collecting data on a wide range of tasks, but um what we really want to do is to get incredibly competent at uh particular deep areas >> to be the very best at a very specific thing that that maybe there's $30 billion of that to do in the economy.
Totally. Exactly. So, for example, just like heavy earth work where you're like loading dump trucks or trenching or digging or large scale like you know a factory where you have to like we're we're collecting we're working right now in a factory where um you're they have to move 600,000 cubic yards of earth to build a a paper factory.
This sounds terrible. It's basically for several months in a row this on this like multi-acre site just scoop the dirt loaded scoop the dirt loaded literally for months on end.
Months like yeah like 12 hours a day seven days a week. uh in the case of these sort of projects in Texas and Arizona, they have to stop working in the heat. Um there's just like all these like fundamental challenges and this is the part of the project that actually has like the most variance. Um, and so when you have like general contractors bidding on uh, you know, a project for Department of Transportation in Texas or Arizona or something like that, they're they're bidding they're winning the project by maybe like 1% and uh, the contingencies are are very heavy on like kind of the earth portion of, you know, a lot of these like kind of projects like this or factories and um, and you have problems like variances in talent, um, attrition, uh, you know, like like external elements like heat,
AI driven Machines are coming fast, but for them to actually work at scale, you need platforms that handle the data, the goals, and the outcomes. They're building the digital infrastructure that makes those robots useful, safe, and aligned with human goals. Thanks for watching. If you found this video insightful and would like more videos like these, consider subscribing. Cheers.