Transcription
Benchmarks don't capture everything about artificial intelligence and why it is so helpful to us. This video was requested by a member of my Patreon community. So, thank you for that request. Let's get into why benchmarks miss one of the most important things about artificial intelligence and how you can understand it as well yourself. And it is called cognitive offloading.
The TL;DR is that with cognitive offloading, basically this is something that all humans have done forever. We offload cognition to things like calendars, smartphones, journals, notebooks, that sort of stuff. We have found ways to compensate for limitations in our own cognition for literally thousands of years by carving in stones, clay tablets, those sorts of things. Our memory is not perfect. So we use cameras, we use notebooks, writing, that sort of thing. So we delegate cognition in the form of computers which do a lot of math for us and they also store a lot of data. But we also use things for alleviating working memory.
Now artificial intelligence turns this up to the next level because not only can we delegate cognition directly to a machine rather than say another human or a low-level cognition like basically just doing math. AI today can delegate other tasks like brainstorming and hypothesis testing and those sorts of things. And then the working memory relief comes from just the form factor of the chat. You don't have to remember everything in the chat. You can refer back to the chat later.
Uh and then what is more interesting, what came while I was researching this video was the search heuristic amplification. You know, kind of with that dialogue back and forth, you kind of know what you're looking for. And every time you get a new idea, you say, "That's right, that's not, let's test this idea." And so your search heuristics, your human intuition is actually greatly magnified by the artificial intelligence, even if the AI is wrong. This is one of the things that the benchmarks get wrong.
Um, and then of course there's all the technical stuff that's built in, RAG systems, um, and that sort of thing. But you are the one setting the direction and your big human brain that knows what you're trying to achieve is a really powerful partner. So I wanted to visualize it. Let me explain what this graph represents.
So in anytime you're working on a problem, whether you're using AI to write a novel, diagnose a medical condition, write computer code, basically what you're doing is you're navigating a high-dimensional hypothesis space. And so you start, you know, somewhere you start here. And the goal for what you want to achieve is over here. Whether that's a clever Python function or, you know, you're vibe coding your way to a new SaaS platform or a new game. Um, whatever it is that you're doing, there is the high-value solution that you're looking for.
Now, before you'd have to do it entirely yourself, and I'll show you what that would look like in a second. But with AI, you can basically draw a straight line from wherever you start. So, wherever whatever ideas you have and by the way, this is exactly how this video came to be is I did a very high volume or high-speed rather ideation session where I said, "Hey, I've talked about cognitive offloading before and a fan wants me to make another video about it. Let's figure out the math and science behind it." So, we were able to zero in very quickly on this high-dimen high-density high-energy slide deck. So, this is what it might look like with AI's help.
However, if you're just a dumb human on your own, you might take a very meandering trail because you don't have RAG built into your head. You don't have Python built into your head. Um you don't have the massive scale pattern recognition and generation that AI has.
Now, this is probably a little bit meandering, but certainly people that have used the reasoning models, the latest and greatest, you know, Gemini, even Grok, uh and but 03 and 04 to me are by far the best just because they have the most feature density. And so instead of taking a long meandering path that might take you days, weeks, or years, you just go straight towards the hypothesis, the optimal hypothesis space. So this is a this is my very very sophisticated visualization as to what it is what the difference is between doing it with AI versus doing it without.
Um because for instance you might not even know what data you're looking for but the AI you say this is the problem I'm trying to solve and it says you need this data and you're like cool thanks let's let's move on. So you can short-circuit a lot of the a lot of the detours that you might take as a human.
Now let's talk about kind of like the math behind this. So first and foremost, gradient discovery. Humans use a form of gradient descent the same that AI does. So gradient descent is basically how do I reduce my error. One of the hypotheses, hypotheses, sorry, of human intelligence is the free energy principle. So that basically human brains are trying to minimize future surprise. And minimizing future surprise means that you're trying to create a more coherent, more complete world model that is going to have a more accurate lay of the terrain. This is why I said that intelligence is a map.
Um so gradient descent is basically okay, how do I reduce surprise? Now what happens is you have search heuristics. You have many many search heuristics that you have learned over the course of your entire life where you have an intuition that says I think we should go in this direction. You can't explain it, but you say, "Let's just explore this direction and see what happens." And then you get a new data point. You say, your intuition says, "This data point is useful or this data point isn't useful." And either way, whenever you see that's useful, that isn't, you're able to converge more closely on the the correct direction. So, you're you're following a gradient of free energy or error towards lower error.
Um, and you have a whole bunch of search heuristics because one thing to keep in mind, this is just a two-dimensional search space. In, you know, coding, in novel writing, in medical diagnosis, you're looking in a high-dimensional search space, which could be 10 dimensions, 12 dimensions, hundreds of dimensions. That's too many dimensions to search with just map reduce. So, um, then what you have is even the negative examples.
So this is one of the reasons that benchmarks really don't capture the full value of AI is because and we we saw this particularly in earlier versions of chat GPT where just by virtue of of the ideas that it got wrong, that's bad ideas you're like no no no that's not useful. So you're able to very quickly discard bad ideas and knowing where the solution isn't is just as valuable as knowing where the solution is. So, like let's say you're in the middle of a map and you know all you know is my objective like you're playing a video game. My objective is north. It's not south, you know? So, don't go south, don't go east, don't go west, it's somewhere north, right? Even just having that cardinal directionality is still super useful.
Um, and that's what we're talking about with with the the landscape navigation. So, if you visualize that search space as a map, you say, "Okay, it's vaguely in the northwest corner, so let's go vaguely that direction." Uh, you get the idea. So, moving on.
Um, what are the dimensions of cognitive offloading? So, this was one of the things that came up while I was researching this video was, okay, it's not just the chat, right? It's like the AI is smart, but when you look at the differences in UX, particularly with um chat bots that are falling behind right now, namely Claude, um there's actually quite a few technical reasons that these the better chat bots, the ones that you guys love. I think the top three that you guys really like right now are OpenAI, Grok, and Gemini. I think Gemini is one of the most popular in my audience. So I I would say Gemini and and and 03 are probably in the top or the reasoning models.
So anyways, the reason is number one, working memory relief. So you have the chat window that means I don't have to remember anything that I said. I don't have to remember anything the AI said. So your working memory has a tremendous amount of offload immediately. Um, which that means that you can keep only the most salient facts in mind at any given moment and you can quickly cycle through them. You say, "Okay, let me scroll back up. What was this other idea? Let's revisit this idea. Okay, cool." And then you can discard the rest. That basically means that your brain is never working in a log jam situation where you're constantly flipping between books and web pages and those sorts of things.
And then the search heuristic amplification. So we've already talked about that so I don't need to redo that. The reflective dialogue loop. So there is more and more evidence coming out that using AI sharpens your metacognition. And so basically what happens is you end up with a virtual chatbot in your head because have you notice that the longer that you use these chat bots the more you can anticipate what they're going to say and eventually you can just simulate the whole conversation. So like if you use it for like you know therapy chatbot or even brainstorming chatbot you just say okay let me plan what I would say to the chatbot and then I would expect this kind of response. I don't know about you, but more often than not, I don't even need to go talk to the chatbot. I have it living rent free in my head now. So, but that reflective dialogue loop, even if you don't have it fully living in your head, you can still get to the point where it's like, hey, let me just spitball with you, right? Like the TL;DR, the colloquialism is you're just you're just brainstorming and and spitballing with the AI.
Um, and it also goes back to the transactive memory partnership. Um, so you're offloading some of your memory, you're offloading some of the cognition. Um but then also the idea generation. So for human brains this is this is one of the core ideas for human brains. Uh the executive function of synthesizing from scratch is very difficult and expensive. But picking between five options is actually way easier. So you say okay well instead of coming up with the five postulates or five hypotheses or five different you know mathematical propositions let me just pick the one that looks the best and then we'll iterate on that. So that that iterative recursive approach is something that no benchmarks really capture yet as far as I know.
Um so speaking of why do benchmarks really miss the magic? First and foremost, most benchmarks test AI on single performance. Um so it's it's testing the AI in isolation without human judgment. So that's number one. Number two is it's looking at only its performance on a single task or sorry a single pass on that task. Real-world problems are iterative. It's not like you know someone just sits down and says I'm going to figure out the next you know particle accelerator or or nuclear fusion reactor. It's hypothesis testing hypothesis testing research hypothesis testing. So all human progress is iterative. Whether you're writing a novel, diagnosing a medical issue, building the next, you know, space telescope, whatever. None of those are done in a single-shot progress or process, sorry.
So then there's also the time horizon blindness, which is that benchmarks don't really capture how accurate AI can become over time. So, what I would really like to see is a five five-round thing where it's like you give you give an AI five chances to correct itself um or 10 10 chances. I know that some tests have, but I want to see this really built in particularly on the higher stakes things because it's like, oh well, you know, it it hallucinates 20% of the time on a single task, but what if you build in like that kind of back and forth that human human use or even having a second AI agent that is that is correcting the first one.
Um, another thing is the tasks are static. Um, but you know like how do you benchmark design the next James Webb telescope? That is just a hard thing to test. Um so anyways and then the complementarity is human brains using the AI like you know okay sure you can benchmark a wrench on its own say like how strong is the jaws you know like what's its mean time to failure but in no way does that actually measure the ergonomics of the wrench right like what what is it actually like for a human to use the wrench so that's what benchmarks really miss the practical mechanisms that allow AI to do this and this is this is for all the people designing AI tools not not just OpenAI and Google, but for everyone is if you want to have a very powerful AI UX.
First, the persistent chat interface. This one's pretty obvious. Um, it gives you it gives you a memory trace so that you can remind yourself. So, you just offload all the memory. So, I have like 600 chats with Claude and 400 with OpenAI or whatever. So, it's like cool, I have offloaded so much of my own cognition and it's stored up in the cloud.
Next is retrieval augmented generation. So this extends the memory because then it allows it to search more broadly. OpenAI actually has three layers of memory right now. So there's the the old school you're like I've updated my memory. So it's got like a scratch card where it just keeps notes about you. Then it can also search across the rest of your chats, but then also you can have files that you upload in projects. So you've got file retrieval, you've got chat retrieval, and then you've got scratchpad. Those multiple layers of memory make the UX of OpenAI's models so much better than any other model right now. And I know some pe some of you people are like, "Dave, why are you why are you such a stand for OpenAI?" For those that have watched my channel for long enough, you know that I've had my criticism of OpenAI and as soon as someone else pulls ahead, I'll be like, "OpenAI is garbage, right? I I am I am a fair-weather friend when it comes to AI." So, I'm just saying 03 right now is the best for me.
All right. Next is Python integration. So it's able to run math, it's able to run visualizations and do all kinds of stuff. This basically says instead of run this, you know, write a script for me so that I can go run it somewhere else. It runs the script for you and it it goes and processes some data. It crunches some numbers. It makes a graph for you. That alone makes it one just much faster and you're building much more utility into the model which then allows you to just say I'm I don't want to write I don't want to write the code. Rewrite the code for me. By the way, think about where this is going cuz before too long, it's going to be able to do anything you want with Python. It's going to be able to do anything you want with Blender. So, you're not just talking code and 2D visualizations. You're talking 3D visualizations, meshes, video game scripts, all kinds of all kinds of stuff.
Uh, internet search. So, internet search is a kind of RAG. So, it's a retrieval augmented generation that instead of pointing at your own corpus, it's pointing at the internet. But still good internet search uh is this eliminates a tremendous amount of work that you need to do. So instead of having to go find a document and then copy and paste it, it just does all the search for itself. And by the way, it reads like 20 times faster than you can read. Actually, probably faster than that now. Um maybe a hundred times faster than you can read. So that actually really shortens the the the search loop for it's like, well, we need a piece of information. I don't even know where to look. It already knows where to look.
And then you've got the pattern recognition and pattern generation um which helps it to uh basically just be a great partner in a lot of different tasks. Um that one can't really be overstated. Moving on.
Um okay. So uh this channel is supported entirely by you so on my Patreon, my school YouTube channel memberships. So um you probably noticed that I've never done sponsorships. Um I've also moved this this card to the end. So, if you would, please consider supporting this channel in some way or other. Uh, link trees down in the description and a lot of the things basically every way that you choose to support, you get something in return. Um, on Patreon, you get a Discord community. Uh, I've got a couple of school communities. Um, and then, you know, if you if you join as a YouTube channel member, you also get access to the same extra content that you do on Patreon and school.
Um, so yeah, there you have it. Thanks for watching. Cheers.