Metacognitive Skill Learning in Humans and AI | Stanford

It's my pleasure to introduce Brandon Conan Smith. The title of his talk today is metacognitive skill learning in humans and AI. So, let's give Brendan a round of applause.

Well, thank you very much for that kind introduction and for inviting me to present my work to your lab. I'm delighted to be here. I'll be giving you an overview of my research, but I'll be trying not to go too in depth. So, what I'll be doing more is giving you a research story about how I've been trying to understand metacognitive processes in humans so we can import that understanding into AI to try to unlock more of its potential. So, I hope I won't be overloading you and I do hope to raise some interesting questions for discussion later. So, I'll be structuring the talk in the following way. I'll be giving you some background concepts. What is metacognition? What is skill? These types of things. Then I'll be talking about my metacognitive research. Then I'll be trying to explain how we can apply it and import these insights into AI to improve the design of artificial systems.

So I'll start with just a brief definition of metacognition. Broadly construed, it's the ability to monitor and control our own mental states. So we can perceive our own cognitive states such as attention. We know if we're distracted, we know if we're focused. We can perceive emotional states within us. We can perceive whether we remember something or not. And we can control these states. We can regulate our cognitive states. We can steer our attention towards a task. We can improve our learning outcomes. We can apply reasoning techniques. So metacognitive skill is the extent to which we can actually monitor and control our own cognitive processes. So this is crucial for both human and AI cognitive performance. It seems to be surprisingly an even better predictor of learning and performance outcomes than IQ. Those with strong metacognitive skills but average IQ can often outperform those with high IQ but low metacognitive skills. People with metacognitive skills can just be more prepared. They know what they don't know. They can apply their learning and attention towards difficult topics they need to master and they are more prepared for tests as a result. And this holds true for all ages and job types. It's being recognized by global educational organizations as a vital 21st century competency. There are different categories of metacognitive skill. As I said, there's the ability to control your attention, regulate some emotional state, apply metalarning strategies. These are what we'll be focusing on today. But there's many different sub fields. As I said, there's memory, meditation, therapeutic practices are largely about trying to teach people how to direct their attention, thinking, and emotional states in a beneficial way. But I'll be focusing on these three subdomains today.

Skill in general has to do with the ability for an agent to develop a high degree of control over some activity. And this often takes a large amount of practice. And there are these three domains. There is perceptual motor skills like tennis or driving, controlling things in the external world. Uh likewise with cognitive skills such as chess or math. And metacognitive skills as I've said has to do with controlling something internal to you such as your attention, memory, these types of things. And this is especially difficult for artificial systems.

So there are some useful insights that we can gather from this psychological and cognitive research. Now a big problem is that metacognitive skill isn't really well understood. Previously there's been no formal theory of how the cognitive mechanisms underlying metacognitive skill generate the phenomena that we can perceive. So this lack of a formal theory limits its application to humans, limits its application to AI. One of the problems is that humans have inbuilt metacognitive skill learning mechanisms, but AI do not. So we have to build them, which means we have to understand them first. So I'll be trying to answer Shraw's call for a unifying theory of metacognition. We're needing a good formal theory first. And this would ideally integrate research, distinguish it from general cognition, inform applications, clarify how improvements in monitoring and control occur. This is very important and I'll be going into some empirical validation for this a little later on. But the questions that have been guiding my research thus far are these, you know, what are the cognitive and computational mechanisms underlying metacognitive skill learning and how can we apply this understanding to AI design? So I'll be telling you about the first formal theory of metacognitive skill learning that I published and how we can apply these insights to the improvement of artificial system designs. So by telling you how I came about this through something of a narrative of my work in computational modeling, understanding the characteristics of metacognitive skill and the broader computational mechanisms of metacognitive skill.

I'll start off with computational modeling in terms of how we instantiated the empirical data on metacognitive processes by trying to implement them into computational cognitive architecture. For those who don't know, and many of you do, a computational cognitive architecture depicts the modules and components necessary for humanlike intelligence. Components such as working memory, declarative and procedural knowledge, perception, motor actions, these types of things. The common model and actar in particular abstracts out of the brain's neural connetos to drive something of a computational process model so we can understand how information moves through the brain and instantiate this as code and runnable simulations. So we reviewed the literature on metacognition in depth and modeled in actar the different levels of metacognition and metacognitive skill. These are just some visualizations of my code running to give you an idea and various metacognitive models have been instantiated in actar problem solving metacognitive threshold different types of learning strategies and we input meta knowledge associated with the temporal media lobe and production rules that are computational instantiation of procedural memory associated with the basil ganglia to get these runnable simulations that give us a clear understanding and help to ground our imaginings in these intuition pumps, so to speak. And we honed in on dual system metacognition as a way of understanding the different types of metacognition. And we applied the common model of cognition in particular to analyze this system one system two framework that was popularized by Danny Canaman or thinking fast thinking slow. We analyzed and tried to discern some of the misperceptions of this framework. And so we concluded that there are two basic forms of metacognitive information types. There are effective non-propositional metadata, metacognitive feelings that are fast, automatic, such as feelings of knowing, feelings of rightness. For instance, you know, if you know the name of a movie or the name of an actor, maybe you just can't recall the information itself at the moment. That's a a feeling or a tip of the tongue phenomenon. That is a signal that we have the knowledge in our memory banks if we just searched it a little more. And we have type two metacognum strategies that are slower. They are declarative. They are propositionally structured. They are concepts and often strategies that allow us to direct our processes towards retrieving this information. So for example, you might go through the names of different actors you know or you might go through the letters of the alphabet to then derive at a letter that primes you to remember the name of the movie. So this is how type one metacognitive feelings can trigger type two metacognitive strategies that allow us to retrieve knowledge that would otherwise remain largely inaccessible.

So I'll be starting with this publication that's hot off the press. that we published in Cogsai of this year, metacognitive skill as a domain. So this research draws largely from Joshua Shepard who has sought to understand the general characteristics of action domains. Shepherd begins by emphasizing control as a fundamental aspect of skill. Control involves the ability to align one's actions with a goal-directed plan in a flexible way to be able to address novel circumstances. So we'll be combining this with Nelson Narin's diagram on cognitive and metacognitive processes referring to first order cognitive processes that act on things in the world otherwise known as object level tasks usually performing external activities such as driving tennis you know the actual act of typing itself or cognitive skills such as chess or strategizing in terms of what you're going to write or math. And then there's the metal level which has to do with the second order cognitive process that monitors and controls first order processes. This is the metacognitive domain and this is the action level that we will be discussing today.

So I'll just be giving you one more example to sort of prime your intuitions. We have the motor level which refers to external behavior such as writing. If you're a writer, you would be engaging in the actual motor tasks that allow you to write by hand or type. If you are trying to strategize, you're at the cognitive level. you're trying to get clear on what you want to write, how to structure the paper, what type of ideas you want to express. This is a cognitive level skill that you can improve at. And then there's the metacognitive level which allows you to direct your attention or harness your emotions or motivation or reasoning processes towards the cognitive level towards the motor level that allows you to from a top- down perspective, these skills can support each other. And from the last two decades, motor skill and cognitive skill have been well researched as action domains as domains of skill. But metacognitive skill has been something of a missing puzzle piece. And so I'll be applying the research and the philosophical and cognitive literature to explain how metacognitive skill embodies the same characteristics as motor and cognitive skill.

So in brief, this paper bridges a literature on skill and metacognition to reveal the common characteristics that are shared between all three of them. I'll just be explaining a couple characteristics. In particular, it's goal structure and the knowledge types that have been used. Now goals are a common aspect of skill. Complex goals require sub goals to be achieved. This usually entails a hierarchical goal structure where you have sub goals that are organized according to their conduciveness to the main goal. How well do these sub goals contribute to the achieving the overall goal? So in the motor domain, so for example, if you are playing basketball, the overarching goal is to you know score the most points. You're trying to win the game. This is supported by sub goals such as shooting, dribbling, defending. This is supported by further sub goals such as ball handling and footwork in the cognitive domain. So, for example, chess, the overall goal is to checkmate your opponent. You're trying to win the game. The sub goals that support this are opening moves, attacking the opponent, removing their pieces. This is supported by further sub goals such as you have to avoid traps, you have to control the center of the board. And in the metacognitive level two, this is also seen where you have goals such as maybe an ideal learning efficiency or you're trying to control your focus. This is supported by sub goals such as targeting difficult areas, setting a timeline, choosing the right learning strategy, and supported by further sub goals like monitoring your comprehension, noticing confusion, directing your attention and learning strategies in the right way. So this is just one brief example of how these three domains share common characteristics.

The skill literature also distinguishes between two forms of expert knowledge. Now knowledge is what allows an agent to choose the right actions to achieve the goals. Very important. This is usually directed by two forms. Declarative knowledge and procedural knowledge. Now declarative knowledge involves propositional facts. It involves rules and explicit reasoning strategies also called causal models. But knowledge itself, knowledge of how to apply certain strategies, knowledge of how to control your attention or an emotion. It doesn't execute the action itself. This requires procedural knowledge. This has to do with more implicit representations, then execute the control of the action. It deploys the action itself. Same with metacognitive skill. You need meta knowledge that allows you to choose the right mental actions to achieve your goals, the right learning strategy, the right attentional control. And this is also executed by metal level procedural knowledge which acts out these strategies to then become more skillful. So whether it's tennis or chess or attentional skill, all three domains require the use of declarative knowledge to learn what strategies achieve the goal and procedural knowledge that executes them. And importantly, procedural knowledge is what builds up over time. And through practice, this knowledge then proceduralizes. The declarative knowledge isn't needed so much. You have these automatic procedure representations that act out the instructions to become fast and automatic and more skillful.

So we'll be going into this in depth with this publication also published at COGSAI a few years ago which give the formal theory of metacognitive skill learning that I was referring to earlier. So this refers to how these knowledge types interact to generate skill in metacognition and this paper was used as a foundation for further publications that showed how this theory clarifies metacognitive skill learning in different domains. So this theory of metacognitive skill learning relies on a theory of proceduralization. So we are employing a skill theory that has been very successful in explaining the cognitive mechanisms underlying motor skill and it's also been used to explain cognitive skill but it hasn't yet been applied to metacognitive skill. So proceduralization largely has to do with slow declarative knowledge being converted into fast procedural knowledge that's then increasingly refined. And this idea of proceduralization is a point of convergence among models of skill learning. This is one way of explaining how performance of any task over time and practice becomes faster. The actions become more automatic. They become more accurate. So in brief declarative knowledge moves into working memory. These are instructions. Say if you're driving or applying math skills or applying skills for learning, it moves into working memory and then allows you to activate these actions. Importantly, it operates within working memory. And then we have procedural knowledge. These procedural representations that are implicit. They operate outside working memory. They're computationally specified as production rules. And this procedural knowledge is what builds up over practice to become faster, automatic, and eventually replace declarative knowledge. So these three stages of learning largely involve declarative knowledge as I said. So when you're learning to drive, for example, you're getting instructions from the driving instructor who is giving you verbal strategies for where to put the key, how much gas, how to signal, and this in the beginning triggers procedural knowledge that then carries out the performance. In the intermediary stage, a procedural knowledge is paired with the instructions itself. And when it's repeated, it becomes more automatic. And through rehearsal, performance speeds up, errors go down. And in the third stage, in the procedural stage, you have actions that rely mostly on procedural knowledge. Declarative knowledge isn't needed anymore. These operate outside of working memory. They're increasingly fast, automatic. It proceduralizes. This allows for a type of cognitive reinvestment which is fascinating where initially you are using all your working memory to apply these instructions. But as the procedural knowledge builds up, it offloads the task from working memory and it frees up working memory for higher level control. And this allows you to apply your working memory to monitor the situation for changing circumstances which allows you to apply your skills more flexibly. And this is also what we see in the level of metacognitive skill. You can flexibly adapt to changing internal conditions as they arrive. So this cognitive reinvestment is where working memory can be redirected towards planning error detection adapting to novel situations. And in AI terms, by proceduralizing metacognitive strategies, it's something like freeing up RAM where once strategies are encoded into automatic routines, they become less computationally expensive. This improves both speed and flexibility. And this allows for greater learning and the greater application of these metacognitive skills towards novel situations, which we'll get into a little bit later.

So it was by arriving at this theory that I predicted that the hallmark telltale signs of proceduralization that we see in motor skill and cognitive skill should be seen in metacognitive skill as well. And this is this process by which skill learning develops through this power law function which was something that was quantified by Logan as the speeding up of reaction times which is generally accepted as a description of how skill acquisition progresses where there's initially a steep initial improvement that gradually levels off in a very predictable way and we see this log log function of the speed up curve that we saw in the previous slide and I suspected that this power law function that has been wellre research in motor skill and cognitive skill. I predicted that we should see the same power law of learning in metacognitive skill. And so I searched for this in the subdomains of metacognitive skill in the literature on attention and metamemory. And this is exactly what I found. I found that attentional skill also follows a power law of learning that reflects the characteristics of motor skill and cognitive skill as does metamemory skill. So both these teams of researchers discovered that reaction times during practice followed a power loss speed up that was typically characteristic of automatization in these other domains of learning. And we also see that this theory allows us to understand some confounding data in the empirical research that hasn't really been well understood otherwise. So the evidence for metacognitive skill has been present in the literature implicitly but has been unrecognized. So there's this paradoxical data in the skill research where high level athletes under high pressure situations, so for example, they're at the Olympics or they're at the World Cup, they have their performance decrease substantially as a result of the stress, also known as choking, we see maybe an athlete who is able to make a putt 10 times out of 10 under normal conditions under high levels of anxiety will have their performance disrupted and they'll miss. The interesting part is that research shows that athletes who are regularly not very self-conscious in their daily lives. They're more likely to have their performance disrupted under pressure. They're more likely to choke as a result of experiencing anxiety. When the research shows that athletes who are more self-conscious normally, they deal with more anxiety on a regular basis, they're actually less likely to choke under high pressure situations. They're less likely to have their performance decrease under pressure. And you might think, well, why would that be? That seems counterintuitive. You would think that people who are more regularly anxious, more self-conscious, would be the first people to have their performance decrease under pressure. So, what explains this? The researchers hypothesized that those who routinely experienced more self-conscious anxiety had greater practice at self-regulation. And it was this practice that aided their performance while under pressure. It was this practice that deployed some internal skill that regulated and stabilized their performance that allowed them to, you know, make the shot. So here we see the signs of metacognit proceduralization. We see an improvable internally directed skill that when practiced operates automatically and operates outside working memory and requires minimal attention. And this helps to explain something else. It helps to explain why metacognitive proceduralization has been unidentified so far. So for one, it's that it's invisible to observers. Researchers a very hard time seeing if somebody's, you know, practicing metacognition. We can see somebody improve at motor skill such as tennis. We can see somebody improve at math. They become faster. They become more automatic. They become less errorprone. But not so metacognition. you can't really see if somebody's focusing as easily or whether somebody's applying a meta memory skill or metalarning skill. And two, it's less perceivable to the performers themselves. Procedural knowledge, as we said, is not conscious. It operates outside of work memory. So when skill develops, it becomes an unconscious habit. This is also called expertise induced amnesia. So those who have developed a high level of skill often have a hard time consciously recalling the procedures they used to go about it. And this goes a long way to explain things. As meta knowledge is converted into imperceptible procedural knowledge, it disappears below the horizon, so to speak, and people lose conscious access to the strategies they use to monitor their own attention, to control it, to monitor their own learning strategies and control them. So, it's this dual imperceptibility that helps to explain why metacognitive proceduralization has been unknown to date. has been hidden behind two proverbial blindfolds, both the researchers and the participants.

And so now that the mechanics of this theory have been brought into the light, we have been using it to explain metacognitive skill learning in various subdomains so far. intentional skill learning, for example, emotional regulation, which we've discussed. And we've also been using it to explain how therapeutic strategies can proceduralize how detach mindfulness once it's become sufficiently practiced helps to detach agents from unhelpful or maladaptive thought and emotional patterns. And we've been recently applying it to the field of human computer interaction. So human computer interaction is the study and design of how agents interact with technology about how people engage with computer systems. It emphasizes better usability efficiency. So it typically models how experts go about tasks. GMS is one classic cognitive model that has been used in HCI. Helps to predict and analyze how users perform during routine tasks. So you see on the right there how we have some plan that a expert would engage in when they're going about some task. They select the different subunits of these tasks, the methods, the operators and they then go about the achievement of some goal- directed activity. But we see that metacognition isn't classically applied. It's been out of the picture so far. So the theory I've been talking about helps reveal where metacognition fits into HCI and how we can better apply it. So our metahci interface improves this classic GMS model by adding metacognitive processes like self monitoring, self-regulation, adaptive reflection. It helps reveal where users apply metacognitive skill and how we can better manage them and even train them to engage in complex tasks and decisions better is what supports both automatic type one and more deliberate type two metacognition. It allows us to see where they interact with these natural processes, where we might strengthen these and train these to allow for more targeted interventions.

So, so far we've been talking about how human brains have these inborn mechanisms for metacognitive skill learning, for controlling and regulating and monitoring our own processes. And the problem is that artificial intelligence lacks this natural ability. They don't have these mechanisms factory installed. So they have to be built. So the question is how do we import these cognitive insights into artificial intelligence to build better systems? So we'll be talking about this now. So AI has been making incredible advancements in the last few years. That's absolutely true. We see Boston Dynamics have been making some really surprising and encouraging advancements in perceptual motor skills. Of course, we're all familiar with LLMs and how quickly they're evolving, but it's the metacognitive domain where AI struggles the most and where it needs the greatest support right now. Now, there have historically been a trajectory towards increasing self-reference in artificial intelligence design. Processes have become increasingly self monitoring, self-controlling, self-learning. Few notable examples have been Google's alpha geometry, metar and quietstar. So for example, AI developers have been noticing that it can significantly improve performance. For example, Metarag has been improving its accuracy by using metacognition to detect and fix reasoning errors. Another example is quietar where the designers ingeniously gave this model an inner monologue which is a way of saying that it generates many different inner rationes and considers the best option before outputting the answer. And this is a metacognitive process that has doubled reasoning performance. And researchers have been increasingly calling for more attention to be directed towards building reliable, autonomous, safe AI. It's being considered is just critical and crucial to its development. It's being considered really an evolutionary step in the advancement of AI systems that we really can't unlock the full potential of AI and all the benefits that we can derive from it without building in these metacognitive systems that are as good or superior to human metacognitive systems. So there's this recent paper by Johnson at all, imagining and building wise machines, the centrality of AI metacognition, where the authors state that a lot of the problems that we're currently facing in AI are the result of these systems having poor metacognition and they focus on a few of these issues such as lack of robustness, its inability to be flexible or deal with unpredictable or novel environments, it struggles with ambiguity and uncertainty and out ofdistribution scenarios. Uh so another issue they talk about is explanability where many AIs operate in this blackbox paradigm where their decision-making is opaque to designers their reasoning processes aren't transparent and this is a real problem for trust and has real safety concerns and this goes into the third issue which is challenges for cooperation and safety and this leads into the ethical concerns we have where AI systems as they get more powerful there's a real concern that they will be misaligned with human values. They might misinterpret human intentions and this has a lot of risk associated with it. And we do have a few papers that engage with this that I'll be going into more detail during question period. So I won't be going into all the issues that the lack of metacognitive abilities are said to be causing an AI at the moment, but I will be talking about this one paper published at triple AI. I had the pleasure of presenting it at Stanford last spring where we discuss how these issues, these limitations are the result of largely an inability for researchers to really grasp how metacognition works computationally. The mechanisms underlying metacognitive processes in humans. It's hard to hit a target you can't see. And these crucial cognitive insights can be imported to help shed a light on what mechanisms they're missing and how we might build them.

Now, meta and AAI is a little different from how we understand it in human beings. In artificial intelligence, metalarning refers to a systems ability to improve and learn and adapt by adjusting their learning rates. We're using strategies usually have external programmers who are optimizing their algorithms for generalization and efficiency. But in human beings, metalarning refers more to these autonomous systems that have a broader ability to monitor and evaluate and regulate our own thinking. This full suite of metacognitive abilities that we then bring to bear in novel situations. It allows us to be aware of our uncertainty, confidence ratings, to adapt our goals and strategies based on changing and dynamic situations. And we propose that importing these insights into human metacognitive skill learning can be very useful. In particular, this theory of metacognitive proceduralization would allow system to automatize their strategy use and adapt over time which are then connected to feedback mechanisms that adapt these strategies based on use cases. This would allow AI to also engage in a form of cognitive reinvestment which we talked about. This frees up working memory for continuously monitoring situational changes for novel circumstances. Allows them to dynamically replan and adapt their strategies based on environmental changes. This is related to basian qlearning. This reinforcement learning which I'll be happy to go into more during questions. So at triple AI we did propose this metalarning reward function where briefly metacognitive strategies would become automated as they are used and rewarded over time. So for example very broadly the artificial system would identify some learning task. It would select the appropriate learning strategy for that task. It would then apply this learning strategy. It would evaluate its effectiveness. Is it useful? If not, it would result in a negative reward. If it is useful, it would result in a positive reward. And these rewards or punishments would feed back into its memory of strategies where the useful strategies would become more automated with time. They would become more nuanced, more refined, more precise, more contextually appropriate to the circumstances. They would then be able to be more adaptable across novel changing dynamic circumstances. Um, and through the updating of its strategies and its choices based on evidence, it builds up this use case history which allows for more adaptive improvements under uncertainty. And I'm just speeding up as a result of me getting closer to my allotted time. I'll be happy to go into more detail about that during the question period. But briefly, the improvements this would allow for would be its ability to be more autonomous, the capability of these systems to be able to monitor and adjust their own learning strategies, to be more flexible, to be able to dynamically select strategies based on changing circumstances. It would be more efficient. It'd be reducing cognitive load and minimizing computational cost associated with these strategies.

So one of the papers we were discussing earlier on building wise machines calls for the benchmarking of wisdom. They call for developing assessments that reasonably measure metacognitive capabilities in these artificial systems. And I believe that importing some of the psychological assessment tools that have been used to measure metacognitive capabilities in humans can be reasonably adapted to assess if AI is on track to developing human level metacognit capabilities.

So just to be mindful of time, I will conclude by saying that I hope I've given you some good reasons to believe why understanding the mechanisms of metacognitive skill learning in humans can be very useful in helping us to build and train analogous metacognitive capabilities in AI. I think this is an extremely promising field of research. I was asked to leave a lot of time for questions afterwards. So I will be happy to begin the discussion. Thank you very much.