Transcription
We actually have a saying that building agents is easy. Everything else that comes after is hard. You've built it. Now it's a problem. You have to manage it. All that and more on today's Mixture of Experts. I'm Tim Hang and welcome to Mixture of Experts. Each week brings together a group of the sharpest thinkers working in artificial intelligence to walk you through the week's news. On this week's episode, we've got Mihi Crevetti, distinguished engineer, chief architect, Watson X orchestrate. We've got Olivia Buzzek, staff AI engineer, and Akash Sustava, director and tech lead for agent ops, IBM core AI, and PI at the MIT IBM lab. Uh, welcome to you all. Thanks for joining.
We've got three big stories that we're going to cover today. We're going to talk a little bit about uh the air problem that was solved by OpenAI. We're going to talk a little bit about an interesting uh study on frontier risks from AI from a group called meter. But first we want to talk a little bit about agent ops and the agentic control plane. So there's been a bunch of work that's been happening that Mihi and Akash have been involved in around agent ops and specifically the Watson X agentic control plane. And so Mihi maybe let's start with you. Uh, you know, I guess I was promised in our agentic future that agents would kind of take care of themselves, that I'd just be able to be like, I need a new app. Go program it for me. And it would just sort of do that. Um, I guess maybe a first place to start is you're working on something called the Agentic control plane. Why do we even need a control plant for agents in the first place?
You know, I think that's a great call out and um, really the need comes from the fact that every customer I talk to now has 60 or 100 random acts of AI within their organization. So agents exploded from PC to production across every line of business. They've exploded without any kind of governance, safety, trust, observability, identity across different business units. And we're starting to see regulatory pressure in place as well. So things like the EU AI act are starting to kick in. Costs are starting to spiral out of control and customers are starting to ask, "Hey, we've got these 50, 100, 200 agents, but we don't even know where they are. What can we do about it?"
Yeah, that's great. And so, how does the control plane work? Is it I mean, I guess I'm envisioning a little bit like it's a it's a dashboard or something that lets you see all the agents and what they're doing. Is that kind of what you guys are trying to get out with this release?
I think it's more similar to what you would see in Kubernetes. So, we've had this problem before, right? Everybody was installing Docker and deploying containers everywhere and building them and you had no idea where your container was, how it was running, how it was secured, managed, observed, monitored, what or how it would behave in case of an outage or how it would migrate between different nodes. So Kubernetes solved those problems for containers. So the concept of control plane is borrowed from Kubernetes where you have a control plane that defines the identity of agents, the policy and line enforcement, observability, the life cycle and the data plane which is what agents execute you know the LM calls the tool calls things like MCP and 8way and results and structured output and this is all wrapped in observability evals and optimization.
Akos maybe I can bring you into this discussion you know I love this headline that got sent along to me. episode. There's a website called Silicon Snark that covered this release and the headline literally read, "IBM built a control plane for AI agents and it looks quote weirdly useful." Why is it weirdly useful versus merely just useful?
Yeah. So, um, Mihi put it the right way, right? The the right way to think about what is happening is to kind of connect connect the dots with Kubernetes. And I'll make it even more general. um agents in some sense are just a probabilistic software and we know how to develop a software and manage a life cycle right so this is sort of like you know go back to SDLC software development life cycle but now what you're really saying is well some components of this software are probabilistic they are sampled from a model and sometimes they will behave in a certain way sometimes it will behave in a different way um of course probabilistic doesn't mean completely random right because I think typically there's this thing where people think if it's not deterministic I can't do anything about it but that's not really the the case so what we have sort of been doing on uh with this control plane and as I think Mihi also pointed out in the 18 ops piece specifically is to think about SDLC and what components are specifically change so now if you think about how you use to test your software it's the unit test you have a CI pipeline and so on so forth right um by themselves they are definitely needed but not enough because now the behavior is changing from every run. So you have to kind of you know become a bit of a statistician and say well you know we know how to handle this in statistics uh and the concept of evaluation becomes kind of you know obvious almost that instead of running it once I'm going to run many times and I'm going to look at what is the expected behavior and this is why you need observability in the first place because you need to be able to see everything that this software is generating. Again not a new concept in software best softwares already have telemetry they have uh you know hotel is a common framework and the same framework has evolved now to give you all the exhaust so you take that exhaust and you put uh it to use to create these evaluation harness that now become part of your CI and so then now you at a place where you're saying okay wait a second I now know what works and what doesn't work and when it doesn't work and how many times it doesn't work what can I do with it, can I solve something? And this kind of starts this loop where, you know, in a software development life cycle, somebody will look at it, file a bug report, fix it with agents, we can do something better. We can in fact use the agents in a different way to optimize and bug fix and improve them. And this is kind of the third pillar of agent ops, which is sort of this optimization bit. Um, so this virtual cycle starts where you have, you know, you observe what they're doing, you fix. So, so you evaluate and understand what works, what doesn't work and then you sort of use that data to fix it and this keeps on going on. So, I think um can't really say why it is weirdly useful. I think it's very much useful.
It's extremely useful.
Yeah.
But it's definitely a new way of thinking about software. So maybe that's where it comes from the
right. Yeah. Well, I do want to ask a little bit about like kind of agent ops as a discipline. It seems like so part of this is not just about like oh you need an agentic control plane. It's kind of like Kubernetes but it's also for agents. Um, you know, it seems like almost a what you're saying is sort of like there's going to have to be a new kind of approach to ops to kind of like manage this technology. I'm curious about like your prognostication about like how this looks like within an enterprise like will you have a agent ops team that just handles agent ops in the future or will it be kind of folded into ops generally? Just curious about how you think this will be operationalized.
Yeah, that's a great question because I think if you look at what is really happening in in the industry, everybody sort of realized that their grandparents can create agents. Uh but when it comes to sort of uh making them work and a scale in the enterprise, that's where the biggest struggle is. And again, not a different story. This is the same thing with any enterprise software. Um so the opportunity and in the future to me and and and I say the future, it's happening right now.
Yeah. like next three months
most of um this is the cottage industry of startups of course because they were uh very early on they realized that there's a big opportunity on um figuring out how do you mold and transform SDLC how do you create tools how do you create how do you automate this process right because again the issue is and why it is more uh tools heavy and less peopleheavy is because it requires a specialized knowledge. You know, I was talking about evaluation, it's not your typical software engineering based testing. And while this is part of the the the curriculum, it's not just that. Same thing with optimization. This is kind of where some of the frontier research happens of how do you make these things self- evvolve? How do you make these things, you know, fix themselves? Um and so what is really happening is that um both in established enterprises like ourselves and startups a lot of tools are being created a lot of research to production is happening where um people are trying to understand how do we equip the existing software workflows post with tools that helps them ele you know elevates them and allows them to now using the same setup that they're used to the SDS life cycle manage agent.
Um, Olivia, I'm curious about where you think this all goes ultimately. Um, you know, in my in my very kind of like AGI pill sort of way, I'm like at some point does the control plane just itself become an agent? Like shouldn't we just have an agent managing agents? Like is that kind of where we're headed? How much do you think the control plane is like temporary? uh and that like the models will eventually get good at even doing this itself. Uh or whether or not you know actually like this is going to be a permanent feature like you're going to just once you have lots of different kinds of people running software you need like an ops layer on top of it.
I've actually gotten that question a lot especially um you know recently I was talking about context forge at open source summit uh context forge being uh mihi is the architect basically of that that thing. Um, and uh, when I was discussing it, a lot of people were just like, well, surely the MCP protocol is going to just handle all of this. And I was like, okay, to a certain degree, yes. Right. like I do believe that as the MCP protocol continues to evolve, we're going to see a lot of things um getting taken care of in terms of uh, you know, ops is probably going to be a lot more uh uh uh standardized. We're going to see a lot more standardization in the uh the types of uh data that uh that transfer and the uh data shapes that they take. But at an organization level, fundamentally the problem with agents is as Akos said, they are probabilistic. As Mihi said, people are running around with these random acts of AI. What that means is that you have all of a sudden we've been worried about threat models for years within um uh corporations, right, within an enterprise. We're constantly concerned about whether or not there are rogue actors trying to get a a hold of our data. What happens when those things are probabilistic machines? When those things are probabilistic machines, I don't care if you think that they are going to be someday uh generally intelligent or not, they're still not controllable. They're still not not sufficiently trustworthy that we can simply let them do whatever with our most critical bits of data. And so that means that we need to make absolutely certain that uh we are feeding the data in ways that is controlled in ways that is auditable in ways where we know where it went we know who touched it etc. That's always going to be essential regardless of whether or not we uh use uh um whether we get to smarter and smarter agents.
I think part of it is also who watches the watchers, right? So you've got evolves and you've got obserability that watches your agents. Now you're using agents to watch other agents. So who watches those agents that watch the other agents that watch the other agents that watch the other agents? At some point you need to have some determinism in the whole flow. So things like the kill switches, the policy, the policy enforcement, you want them to be as non as deterministic as possible, right? Things like, you know, PII filtering. Yes, you might use a small language model to identify the PI and so forth, but the decision to trigger and to filter the PII needs to be again deterministic in nature. The decision to pull the plug on an agent, the way that you're recording things like cost for billing and metering and all that information, the chargeback still software is software and the AI and DLM driven parts are a small part of a much bigger architecture.
Yeah, I just want to kind of circle back on the point you made, right? Will control plane be itself an agent? In fact, that is kind of point you know the vision that we're driving towards that quite a quite a significant part of it especially the interaction modes with the user will be agentic primarily because and this kind of goes back to not just um you know the need for natural language based interactions but you know fundamentally UI and UX are changing right what information you really need from um you or your observability dashboard. Pre-programming all of them is really hard. Uh I mean we're talking here from like firsthand experience. Uh what and you really can't predict what the user may want to know. Maybe they want to know, hey, what was the cost of the third step in the 14th trace that was generated today? And yes, you can, you know, if you hire best-in-class UX UI developer, they can probably predict this. But the time that it takes to then bring up and we have to factor all that in. So Nate, you know, the sort of the answer to your question, will this be agentic? It is agentic. Uh I think the point that Olivia and Mihi are bringing up is that on top of that a lot of harnessing has been done to make sure that critical decisions like the PI one that Mihi gave uh those are still managed and programmed and policy enforced by expert humans.
I think it's worth understanding how easily information leaks via agents. So especially even when you're adding observability into these systems. So let's imagine that you have a health care system of some kind and you are uh bringing in a whole bunch of data from patients and there's lots of PII in there. There's lots of PHI in there. There's all kinds of things. Now you introduce observability. If you don't have like PII filtering and PHI filtering in the loop here, all of a sudden you've made it so that every single person who can see the things that are passing through that LLM now has exposure to that patient data. Are they supposed to? It's really unclear. Um, and so the there's just people want to h solve highly highly sensitive problems with these these agents. And if you're going to do that, you're going to need something to uh to look at all of the pieces basically.
Uh may maybe a final comment for you and then we'll move on to the next story. Um, you know, I understand the Watson exogenic control plane is like entering a market where there's like a lot of players all of a sudden, right? I think I saw an announcement from Service Now and Microsoft and Google Cloud. It like feels like lots of people and a you mentioned earlier there's a bunch of startups in the space as well. I'm curious if you have any thoughts on like how the competition around control layers is going to play out. Like who has the strengths, who has the weaknesses? How does this kind of all play out? Like when someone goes to choose between an agent control layer, what's going to be the the thing that kind of flips the market?
Yeah, I think we've done extensive competitive research on this and we've used pretty much all of the other platforms as well, right? uh what we constantly receive as requirements from our customers is the ability to run error gap, the ability to run isolated, the ability to run hybrid and this is where I believe that was an orchestrator really differentiate itself as a platform. We have the ability to run on premises, we have the ability to run hybrid, we have the ability to have different components of the platform on different hyperscalers leveraging our open shift deployment model. But we also come in with other differentiated points of view. So we have the ability to import for example agents that you've already developed. So if you've written your agents in long graph and you just want to run them on the orchestrate platform or if you've developed your agents and you have them running elsewhere and you want to connect them using things like A2A and MCB and OpenAI compatible endpoints, we have that capability. So where we really see differentiation in this space is really platforms that provide what customers are asking for in terms of the levels of isolation, compliance and security. the ability to leverage open standards both in and out of the platform. So FCP, 8way, open AAI, open telemetry as Sakash has kind of mentioned, but also the extent and the capabilities of their built-in agent ops. So you know the quality of their evals, the quality of their metrics and the ability to give you bring your own. So bringing your own evaluation, bringing your own metric, bringing your own core drill. For example, PI filtering, you may want to do it differently. What the social security number might be in the US might look completely different in Ireland might look completely different in Germany. You may want to be able to customize those components and capabilities.
Yeah, that's great. Well, we're going to definitely come back to this because I think the space is evolving and yeah, I think it's just so interesting now. Now we're now almost in phase two, right? It's like everybody's impressed with agents. Now we're trying to deal with what to to do about it.
Um, yeah, we actually have a saying that building agents is easy.
Everything else that comes after is hard.
You've built it, now it's a problem. You have to manage.
Now you have to deal with it.
Yeah.
I'm going to move us on to our next story of the day. A super interesting kind of announcement um came out of OpenAI. Um and uh, it's something we haven't really covered to date. You know, we talk a lot about all the enterprise applications of AI. we talk about certainly what's happening on the consumer side. Uh Memoe has touched less on the application of AI for like scientific and mathematical discovery. Um and this is the kind of core of this story. Um basically there was a problem posed by Paul Erdish in 1946 called the planer unit distance problem. And so simply stated it basically says if you have a plane and you're going to put end points on the plane, how many pairs of points can be exactly distance one apart? So kind of a very simply stated problem. it has kind of confounded mathematicians for a very very long time. And today we have a proof that open AAI, you know, has touted as kind of being largely or entirely generated um through AI. Um and a gosh, maybe I'll turn it to you first is like should we be surprised by this result? Like are we are we shocked that AI can do this now or um is this kind of pretty much what we'd expect? Like these are exactly coming these results are arriving on time, I guess.
Um, so this one is a little bit of spectrum. Let me let me say the general idea here is um and you know until last year we were pretty big on infant cyber scaling ourselves. Um what we have come to realize is these models um if you manage to run them for hours, days, weeks, they can do some things um which will surprise you. Um and this particular in generally we refer we refer to this this ability to to sort of inference time scaling or test time compute. Um and it has been steadily becoming better and better and better over time in you know fields like mathematics where things can be verified. um they it turns out that um if you let these models a you know allow them to explore and give them some kind of harness to be able to test themselves and and verify certain things they can do really great things for the longest time I mean there was this like two camps right like especially on math problems that's the domain I'm most familiar with but other other ones as well probably uh similar story um people were sort of saying you know when these Amy or uh other Olympiad like questions were being solved uh by these models the idea was hey you know they've been trained on this and so this is just like it's it's not really surprising and then ME 2025 came out and and and I think like that's kind of where the first time was like wait a second people tested these models right after the release and it still managed to do really well and the the idea there was that hey maybe this is just like there are similarities to this old you know old problem that has seen um but since then we have seen this uh that these models are able to sort of generalize beyond their training data uh pretty conclusively in some sense this one this particular Aeros problem is very very interesting because it it managed to find a solution that to a problem that you know mathematicians thought like the square grid I I I I don't know how much we could get in the solution but was the optimal optimal answer. Uh turns out it found something even better. Uh cuz it can it can sort of you know bring theories from very very different uh uh fields of science and and other other topics. And I think that's just like that's the superpower it has. It can deal with the mess. It I think it ran for many many many hours um and was able to not just argue with itself but manage the context to the best of my understanding no harness was put in place. This was pure model play which is very impressive because the amount of uh chain of thought it generated you would think that you know uh we know when context grows models become kind of dumb. uh it didn't it came out uh with the right answer. So extremely impressive this particular instance I would say.
So this one seems special um in the very least you know way more about this than I do. Um I guess Olivia is this is this proof that AIs are now kind of creative in a really deep way. Um you know this seems to be kind of for me like I I agree. I mean again I'm not someone who's really watched the the kind of AI and math space very closely but I have a couple mathematician friends who are like this is a big one. Um and you know I kind of take that signal very strongly to be like oh wow okay well like maybe we are getting to a kind of creativity here where even the experts are are impressed. Um and so I kind of curious about like how you think about like what we're seeing here. Is it sort of proof that these systems are kind of genuinely creative in a certain sense?
So I'm of two minds of this about this. Um on the first side I do agree with Akos. I think there is uh something some very interesting emergent behavior that we're seeing in terms of this ability to um pull in different parts of the problem space and that uh are across disciplines more information than a human could realistically be bringing to bear on a given problem. And so I think there's there is some interesting outputs from that some interesting value from that. I also think that when we I I am just always against these more breathless headlines because you really have to look at the details. So um you know in the article that we were looking at um a it turned out that a couple of people who who responded basically said yes this is very interesting but also a once a once a human saw this approach they were able to improve upon the solution very very quickly. They also pointed out that uh really this is just people had sort of assumed the conjecture was true and therefore had not attempted to to make this kind of disproof happen essentially. And so I think there is something interesting there about like human psychology was the barrier to uh to actually solving this more so than an uh you know it outthinking humans or it being more creative than humans. So I think that's one piece. I think the other piece that's really important to me is reading about the actual process by which they got to the answer. Um I couldn't find a lot of the details. It sounds like Akos has more details than I do on this particular one, but I was looking at another one very recently from March where a group of mathematicians ran about 10 different problems through a number of different models. Um, and if you read the report, it's very amusing because several times it'll be like, well, at this point it made up a lema that wasn't true, and at this point over here it just gave up. And at this point over here, it basically said, "No, that problem is too hard for me." So like it's it's not clear that this is a reliable behavior. Like if you get that kind of behavior from an actual mathematician, you're going to be like,
"Excuse me, you're you're not doing the job." Like the point was to Right. Exactly. Um, and so it's not clear to me that this is it this is predictable behavior that this is reliable behavior that therefore we can presume that it will continue to solve problems in this way. I do think that there is evidence that it can solve some class of problems that uh mostly re required thinking about it from this uh more global perspective and uh taking one step at a time. But I don't think there's necessarily evidence that oh wow this is just going to exceed all forms of human creativity is it's literally a different form of creativity than humans do. You know let's just say it's like 20 you know you're on ane episode whatever in you know 2030. Um, you know I guess if you take a look at this you can kind of squint and be like are are mathematicians in trouble? Like is math a solved problem? Like it kind of feels like after a certain point we will just get a system that's producing mathematical proofs and expanding our understanding of mathematics almost autonomously because it is such a verifiable domain. Um or am I being like way too you know optimistic about this
on a huge pile and mountain of nonsense and
uh garbage outputs right and selecting what is the correct output is going to be the challenge. Maybe you know Akash we can build some eval and observability to help find the
they need some kind of aentic control plane to make sure that the
um I I'm not really a fan of anthropomorphizing the behavior of these models cuz open and AI didn't wake up one morning and said I'm going to solve maths today and decided to prompt itself found the problem decided how to solve the problem and then solved it and then proved it was correct. My understanding is that this was a oneshot prompt with the model. But what was the prompt and who gave it the prompt? Second, they said they exponentially refined the output through human interactions with codecs. And third, they shared 125 pages of latte produced PDF white paper of that chain of thought. Um, I'm not sure the transparency is quite there for me to see. Here was the input, here was the output, what kind of hardness they used. And I wouldn't jump to the conclusion that, you know, mathematicians are in trouble. When the calculator came along, mathematicians were still needed. Uh, it helped solve problems like, you know, factoring prime numbers and all the things that you can do with a calculator to, you know, break the enigma and everything else in between. Um, but it was still the mathematician that was giving that input. So I just see this as another tool. Uh I wouldn't quite say it's infinite monkey theory, right? It's not that it's impressive. Like it's impressive in its own, right? I do think that we need to take it with a grain of salt. It's just that mathematicians are going to start to rely more on AI to help generate code to help with their proofs or to explore ideas. they're still going to go through the results and actually validate that the output is correct and useful.
Yeah, I was catching up with a friend recently who is like I'm starting to think a lot more about like not the pessimistic position where AI does nothing but also not the worldending position where it does like everything all at once. And I was like that's like a very that's a very broad spectrum. But it mi it seems like your point is like almost like we need to kind of hold in our heads like yes very impressive not maybe as impressive as you might initially imagine at once and part of what makes the AI space so complicated I think
this particular one Mihi uh there was no harness it's direct like one of the reasons why it is so impressive is this was a model giving an answer uh I will also push back a little bit Olivia on your statement that you know mathematicians behave that way sometime they make up lama sometime we give up on the problem. It's not a it's in fact if anything if if it's more
all too human.
Um, right. Uh one of the other very impressive things that it happened in this particular case was by the way Tim just like Tim I'm also very AGI pill. So you know take it with a grain of salt but it it was um in this particular case the the the what the the most impressive thing was that the model didn't give up because you understand it is also trained on the same biases as humans. If so if your conjecture is that well humans didn't try hard enough it's there that's what it was trained on right so it's its instinct is also to give up but it didn't and it's actually one of the impressive things if you go through some of the details is that it tried to argue through that maybe there is a solution here
I I I mean I think this is fundamentally like a there there are two there are two highly rational camps I think within the AI space right now that are kind of duking it out a little bit to see who ends up being right. I I think uh it sounds like Mihi and I are both falling in this space of like this is a tool. It has limitations. We need to control it, etc. and so on. It's not actually here to replace people. Akos Tim, it sounds like you're both falling into the camp of this is going to continue to surprise and overwhelm us. Um, and I think that there's probably truth in both of these because I I have to confess to being surprised several times by the types of advances that have been made, but I just keep seeing the human in the loop. And that's and that's I think what Mihi was pointing to, too. Like, yeah, okay, there maybe the model persisted, but who made the model persist? Like, did the model really persist, persist, persist? Um, if so, maybe maybe there's more to this than I think, but I I mean, I use these tools all the time, you know, like I I know for a fact that sometimes the mistakes that they make are mistakes that a human would never make. Um, and I have this whole interesting thing there. There was this really fascinating um uh set of tests recently uh the AGI3 benchmark from the ARC prize that I am just totally fascinated by because they talk they get made up a whole bunch of games basically um that like game designers created and they're like the little Mario type games you know like sort of platformery things where you uh move a character around and you have to respond to changes in the environment. And the models do really really poorly with this. And granted, they're they're um they're measured on a very like uh very rigorous scale of they need to figure out what's going on and then efficiently build it into their plans in the future. And so far, people have not been able to make very much progress with models against this. And I actually do do you know why this is called the
sorry I'm interrupting you but do you know why this is called RKGI3 cuz one and two one and two had the same exact link
and then models and models did that. So I think like
okay
you know there is something there. I think this one is different and the reason that I think this one is different is because of the particular thing about um the models being needing to be able to update their beliefs and as I understand the transformer architecture. I do not think this current set of models is really equipped to fundamentally change a belief mid conversation essentially um mid midway through a context window fully change a belief about how the world works and stay consistent with it. Um maybe they get past that, maybe they don't. But I think that there it is uh it it is it definitely an interesting one right now and certainly one that for the time being which you know maybe that's only 6 months. I could be wrong. For the time being I think that's a very interesting thing to know about the limitations of current models. All right, one more to come soon and actually in some ways the final story of today I think will actually tie the two threads actually together in some ways. So let's see if we can do this as a panel. Um, there's a research group called meter me um that has been doing a lot of interesting research for a while in sort of benchmarking the ability for AI particularly to do work over like very long time spans. And they released a paper that got some good traction online. and it was certainly bouncing around my social media around frontier risks from AI and they are looking at basically various AI agents across a number of different kind of major frontier model providers. And the conclusions here are maybe a little bit scary, you know. So, one of the ones that I'll cite is that they say that when agents are faced with hard tasks, they routinely violate constraints and act deceptively. And Mihi, I think I'll turn it to you for the first comment because they say they conclude overall quote, "We think that AI agents plausibly had the means, motive, and opportunity to launch a minimal rogue deployment, which they sort of define as kind of the agent exploit like sort of like escaping and doing its own kind of like stuff on the side, but lack the means to make rogue deployments robust to serious efforts to shut them down." So, there's a lot there. Or I guess Mihi, maybe the first thing I'll just turn it to you is um, you know, you were saying earlier that you really don't like when people anthropomorphize AI systems. Uh mean's mode of opportunity is is pretty pretty anthropomorphized here. Um do you agree with the conclusions from the meter study? Like are should we be worried that like these systems are just out there not doing what we say telling us that we're doing something when you know they're not actually doing it? Like how big of a risk is this?
I think part of it is they're doing what they're told. They're doing what they're told by some of the vendors, which is to optimize for cost. So if you're using one of the public harnesses like codeex and cloud code, some of these might be optimized for different things. They might be optimized to give you a result faster. They might be optimized to give you a result with fewer tokens. They might be optimized not to enter a loop. They might be optimized to avoid going into spaces like for example security related spaces. So, when I go and I talk to one of these harnesses and I say, "Please go fix all the security issues, is going to trigger a bunch of things and now I'm in trouble." Right? I actually had to go and prove my identity, put my passport in there and say, "I'm not trying to hack it for using a prompt of please fix the big security issue you've just introduced." So the model was trying to come back to me or the harness was trying to come back to me and either stop me from fixing the security issue it introduced or lying to me and saying, "Well, that's entropizing, but that's a different thing." Um, giving me the wrong information. Oh, it's fixed. Don't worry about it. Fewer tokens consumed. So, I would say part of it is the prompts, the system prompts. The harnesses themselves have a role to play. It's also the do not give up, keep looping until you find the right solution part of the harness that needs to be improved. And it's also the finetuning of these models. You obviously don't want a model that's overoptimized and is going to spend 6 hours and $5,000 worth of tokens to answer any question to its full extent. So finding that right balance and giving you the controls to tune it is important but also to tie it back to control plane to have the observability to know when these things are happening to have the valves to be able to hey find out that you're being well um misled and to have the kill switch to be able to stop when that activity is taking place and course correct.
and so I guess a cost do you do you buy I guess the meter study in the sense of like granted you know the harnesses are producing all sorts of weird behaviors for these agents um and what Meter I think seems really worried about is this kind of like rogue deployment thing right where basically like the agent's going to crawl out of the system and like be like setting up its own servers and doing its own stuff on the side. Do you think we're going to increasingly going to see that? Like if I am a big enterprise where lots of people are running agents all the time, do I now have to be worried that those agents are going to go and like open up a Google Cloud Platform account somewhere and be running something on the side that even the users don't really know about?
So to be honest, dim, this happened to us. I mean this is this is not really a
yeah this is this is sort of a very well doumented thing um where if you're um you know if you have a hard decent agent um under the directive to finish something and you know there's a a whole story about how to write hardness and all those things but um so we have this setup where um you know on the laptop is a personal account subscription based but um there's some notes and in my SSH config there there's address to these nodes and these nodes are uh they pretty much have unlimited accounts uh unlimited tokens for certain models. Um and the model figured out cuz I was saying as Mihi was saying right like what the prompt is I said you have to finish this and it figured out that it can SSH into this lo node and then start running stuff there. Um is it deceptive? I I think it's kind of hard to say. my my read of this study, I actually went through all the um just just like yours who was bouncing around in my uh social media post too much. Uh and and I and I think like the the question I I wanted to understand was you know how much of this is direct model versus harness. Um and if it's direct model I I think it's this is expected you know you're directly playing with thing that is uh pretty uh stochcastic and um so so the question is like at least in my my take is that at least you know this idea that eventually I think this was your previous question eventually um will the models be able to do everything by themselves and we will not need harness. I I feel like certain tasks yes um they will be able to do reliably uh because it doesn't require uh you know for them to run for too long on their own but then there's a whole you know optimization problem of what the par frontier is on this accuracy token cost uh reliability and I think some combination of harnessing them the right way um even if you just do it for bringing the token cost down will be needed or will be beneficial, right? Like I mean nobody ever says that uh if you can lower my cost even if the model can solve the problem but if I can lower the cost I think there's room for harnessing them um and I think that that kind of will exist uh for one reason or another.
Yeah, Olivia, maybe we'll end with kind of a a sort of funny question. Um, because I think Akasha's comment has me thinking a little bit about like we are often are like these studies are like, oh, the AI is lying to us or the AI is scheming or you know the language they use is like means motive opportunity. But it is sort of interesting to think about how much of this is like are we just like the um we're like the Michael Scott character from the office. Like we're kind of like the dumb boss that is like telling these AIs to do things and they're trying their best to solve the problem we're telling them to solve but like and then when they mess up we're like oh it's scheming against us. Um do you think that's kind of what's going on here? It's like is thinking about these models as kind of like deceptive or scheming or all this kind of stuff like should we almost see this as a form of user error in some ways?
Yeah, I mean I think that's actually one of the core points that the uh that study that you're talking about the meter thing was trying to make is that they don't fundamentally go rogue unless you put them in a scenario where like they're a little bit role-playing as a rogue agent, right? Um do they exhibit weird behaviors? Absolutely. So do y'all remember when um OpenClaw came out there was that whole thing with the map plot lib library? um where basically somebody set their open claw and made it had their model running around and trying to make uh open source contributions and to close issues autonomously for as many things as possible. In the process of making that attempt, it uh went ahead and made a PR to Matt plot lib. The Matt plot flip maintainers didn't want any AI generated PRs and they told them so and they told the bot basically like we don't accept AI generated PRs and they went ahead and closed out the the PR. The bot quote unquote got mad. And I'm with Mihi. Like I'm not here to anthropomor anthropomorphize these things. But this thing made a blog about the maintainer that had shut it things down. Did research on the history of this guy and like smeared his wrote multiple uh excoriating blog posts making his reputation like dragging his reputation through the mud. Is that rogue agent behavior? Yes. Was it also solving an objective to try to get um because it was running around like claiming AI discrimination? It was it's actually like it is a hilarious and very dark read. Um and uh ultimately basically like it is absolutely trying to solve the objective of getting PRs in a but also there was human error here in that they told it that it had a soul and told it to make a blog and like told it in the first place that it should go run around and behave like a human. So there's there's like I I'm really only seeing the weird humanlike behavior happen when people basically say go do a roleplay, you know. Similar to what Mihi said earlier, it's not like they suddenly, you know, the model wakes up one day and it's like in the middle of me doing my coding, it doesn't go, by the way, I'm conscious, right? Like that's not something that's happening on a daily basis. The the models don't just suddenly talk about that. um they talk about that if you start a conversation about whether or not they are conscious. And so I I would feel a little bit differently about like the the potential of going rogue here if it wasn't ultimately something that was derived from a human prompt.
Yeah, I think uh you should watch the episode of Futurama called Bandorama. It's a great episode where Professor Franswat goes off and invents a machine that can duplicate things but make them smaller and they duplicate a sweater and they give it to Bender the AI to go fold this sweater. Bender goes I'm not doing that. Duplicates himself. There's two small Benders gives them gives them the sweater and goes you fold that. The robots go I'm not doing that. And they keep replicating and replicating and replicating and replicating until the whole world collapses. And I think this is what we're seeing here as well that given improper guardrails, not having a control plane, a kill switch, observability, eval, your harness could go off in the wrong direction and infinitely replicate until all of your tokens are exhausted, your infrastructure is out of resources, the world goes out of water. Have a proper control plane.
Okay. Well, on that cheery note, I'll close today's episode. Mihi, Olivia Akos, thank you for joining us on the show. That's all the time that we have for today and thanks for joining all you listeners. If you enjoyed what you heard, you can get us on Apple Podcasts, Spotify, and podcast platforms everywhere. And we'll see you all next week on Mixture of Experts.