Sitemap
Open in app

Sign in

Medium Logo
Write
Search

Sign in

AI Advances

AI Advances

Ideas in depth. Where exceptional thinking takes shape.

Can an AI Learn the Planning Algorithm? Part 3

The Biological Perspective

Malcolm Lett
40 min readAug 23, 2026

--

This series looks at how autonomous and adaptive artificial reasoning agents can learn human-like strategies for searching, planning, and reasoning.

This is the third part of four, following an intro article on Meta-cognition for Reasoning AI:

This article focuses on studies of human reasoning capabilities and their limitations. I discuss ideas from rational and bounded-optimal theories of behavior, the efficiency reasons for hierarchical planning, and a developmental perspective on knowledge representations that gives a clearer way to identify what information is available to AI models for reasoning with.

Contents

· Introduction
· Resource Rationality
· Meta-level Markov decision processes
· Hierarchical Planning
· Representation Redescription Theory
· Meta-cognition
· Not Covered
· Part Summary
· Citing
· References

Introduction

The future of artificial intelligent designs do not need to be predicated on human mechanisms. We are free to find solutions that are more appropriate to the substrate that we work with — that of silicon electronic componentry. The success of artificial neurons is testament to this fact. That too the success of the almost biologically implausible backprop algorithm.

Nor do we necessarily want to replicate all the same limitations and cognitive biases inherent to human intelligence. However, the high training sample dependence, the high energy consumption, catastrophic forgetting, and poor adaptability shown by artificial neural techniques are also testament to the fact that we have a great deal to learn about intelligence.

Humans are still our best and most examinable example of highly adaptive intelligent system design. So, while our longer term designs may eventually diverge, in the near to mid-term we still need to take inspiration from biology wherever we can.

This part of the series looks at several lines of investigation into the biological cognitive and developmental processes underlying adaptive human reasoning. Many of these ideas have influenced the development of AI solutions, yet they remain less well known within the AI community at large. So I want to give them the air time that they deserve while also reframing the discussion in terms of the low-level meta-management processes needed to support agents that learn to plan.

I’ll start with a discussion of how brains balance accuracy against efficiency, leading to subtler forms of search than what classical AI focusses on. I’ll describe how cognitive behavior can be understood in terms of Markov Decision Processes, enabling a formalized analysis of different architectures. How hierarchical planning too can be formalized, by viewing it in the context of a meta-optimization problem: the need to maximize the efficiency of planning over the long term. I will explain the difference between information in the system and information for the system, and show how this enables us to find fundamental limitations in the reasoning capacity of AI architectures. Finally, I will summarize trends in Meta-cognition research.

Resource Rationality

Bounded Optimality and Resource Rational Analysis are two closely related fields that examine human behavior in terms of the need to strike a balance between optimizing behavior and optimizing the use of limited resources (Lieder & Griffiths, 2019; Lieder, Callaway & Griffiths, 2024).

They combine bottom-up neuroscientific and cognitive architectural design theories that set constraints on the kinds of computations possible (e.g., attention, working memory, Bayesian brain theories) with top-down psychology and behavioral economics theories that enable us to define what optimal rational behavior is. This came in response to the conflict found between earlier rational theories of human behavior (Anderson, 1990), and the growing body of work that found that humans are routinely sub-optimal compared to ideal models of rationality (Simon, 1956).

Fully optimal behavior can only be achieved by considering all possible decisions/actions and all possible outcomes, given the information at hand at the time, and by accurately balancing uncertainties, utilities, and costs. Accurate calculation of probabilities is usually an intractable problem and approximations must necessarily be used. Even with such approximations, calculation of all possible actions and outcomes (or even just most actions and outcomes) is still time consuming and requires significant computational resources. Thus the act of deliberation carries its own costs, in addition to any costs associated with overtly carrying out the decided action or decision:

  • Time: longer deliberation may not be appropriate where time is of the essence (e.g., when deciding how to minimize the impact of an imminent collision).
  • Energy: computation takes energy, longer computation takes more energy.
  • Opportunity cost: time and other resources consumed spent deliberating over one problem are unavailable for deliberating over another problem, or may delay or forestall some other more profitable activity.

A resource rational agent balances these cognitive/computational costs against the benefit of deliberation through a combination of the following mechanisms:

  • Heuristics: adaptively select the kind of computation being performed and the amount of information taken into account. “fast and frugal” heuristics tend to be less accurate, while slower and more costly heuristics are more accurate.
  • Stopping criteria: only perform further deliberation while the expected benefit exceeds the expected cost. In some cases this means not deliberating at all.

An ideal resource-rational agent behaves according to an optimal balance between the benefits of overt action, the costs of overt action, and the costs of decision making and deliberation. In practice, agents can only ever approximate that balance, but behavioral studies show that humans approximate it quite well.

The next few sections discuss specific results from bounded optimality and resource-rational research.

The mathematical theory of bounded rationality

Press enter or click to view image in full size
Two models of control. Bounded rational control recognizes more constraints and costs.

A rational theory of human behavior assumes an optimal policy, 𝜋*, is one that maximizes utility, as in equations (1) and (2):

Press enter or click to view image in full size

Here, a is the action taken, p(o|a) is the distribution over likely outcomes, o, having taken a, and U(o) is a utility function that factors in the reward from the outcome, R(o), and the cost of performing the action, C(a,o). The 𝜆 just enables a choice of relative weight between reward and cost.

A bounded rational theory extends this in a number of ways, as in equations (3) and (4):

Press enter or click to view image in full size

First, it imposes bounded constraints, 𝛱, on the possible policies, reflecting limitations due to inductive biases and the computational and representational power within the system. Second, it includes the costs of calculation of the policies themselves (C-comp), which includes direct costs such as the use of time, memory, and attention, and indirect costs such as opportunity cost.

A bounded-rational system thus has fewer resources than the ideal system, and yet has a more complex problem to optimize.

Amortization of optimization

One part of the solution is that, unlike for many RL techniques, real-world biological organisms do not calculate optimal behavior de novo (from scratch) at each moment. Behavior is the result of an optimization search carried out over multiple timescales, including evolution, societal learning, lifelong individual learning, rapid day-to-day learning, and moment-to-moment decisions (Lieder & Griffiths, 2019; Lieder, Callaway & Griffiths, 2024).

Press enter or click to view image in full size
Behavioral optimization over multiple timescales

The set of cognitive strategies and heuristics that an individual uses over different scenarios is known as an “adaptive toolbox” (Gigerenzer & Selten, 2001). At the individual level, the tools in this toolbox are learned through imitation, teacher-led instruction, and self-discovery. At the evolutionary and developmental level, the cognitive architecture underlying those heuristics gradually evolves and matures, changing what decision making heuristics are possible, their efficiency, and their accuracy.

This means that an individual’s behavior will only approximate ideal resource-rationality during any given moment-to-moment act of decision-making and planning . The degree to which they approximate the ideal depends on evolution and lifelong learning.

Factors include the amount of world knowledge the agent has obtained (e.g., an infant’s world model is severely deprived in comparison to an adult), the set of heuristics they have learned so far, and how well developed (i.e., optimized) those heuristics are for the task at hand. Another factor is the accuracy of the individual’s heuristic for selecting appropriate heuristics. For example, it has been suggested that some gaps between observed and ideal resource-rational behavior in lab settings may be the result of poor heuristic selection (Krueger et al., 2024).

Another important outcome of the amortization of optimization is that moment-to-moment heuristic calculations do not need to optimize over all possibilities from scratch. Rather, anything that can be offloaded as biases and learned pre-calculations should be, so that the minimal amount of computational effort is spent during live planning. From the perspective of artificial RL agents, this means for example storing priors and thresholds that focus and restrict search spaces, and learning task-specific value functions for repeated tasks.

Heuristics for decision making

It is difficult to directly measure the heuristics that people use for decision making and planning. Some lab experiments seek to externalize those heuristics by forcing the otherwise internal decision making and planning into overt action (Lieder, Krueger & Griffiths, 2017; Krueger et al., 2024; Lieder, Callaway & Griffiths, 2024). This is done by presenting participants with a computer interface that initially hides information, but allows the participants to discover as much or as little information as they wish by selecting items to be revealed.

By observing what information participants reveal, the sequence order, timing, etc., experimenters can identify the strategies that participants follow, and identify the trend for different strategies under different scenarios.

Experiments vary by the stakes (usually some small or large monetary reward for completing the experimental game satisfactorily), costs (a penalty for each item of information revealed), and dispersion of rewards (how uniform or variable the costs or rewards are). The experiments are designed so that it is also possible to model the ideal resource-rational behavior: the behavior that optimally balances reward maximization against explicit experiment-constructed costs plus assumed cognitive costs.

Press enter or click to view image in full size
Choice heuristics. The identified three base heuristics. Others appear to reflect combinations or variations of the TTB and SAT approaches.

Combined with earlier work, these experiments have yielded a small set of identifiable heuristic behaviors, including:

  • Take-The-Best (TTB): choose between alternative options based on the one single attribute that is the best predictor of the outcome, ignoring potential influence of other factors.
  • Satisficing (SAT): consider alternative options until one is found that is good enough, and then stops searching. In other word, stop as soon as some threshold of “goodness” is met, and then ignore all other options.
  • Weighted Additive (WADD): consider all the available information and compute the expected payoffs of all alternatives. This corresponds to the optimal strategy under assumption of infinite resources.
  • Satisficing-TTB (SAT-TTB) combines elements of TTB and Satisficing: inspect only the payoffs for the most probable outcome (like TTB), but terminate as soon as a choice is found where the payoff for the most probable outcome is high enough (like Satisficing). This approach doubly reduces the amount of information considered.
  • SAT-TTB+: start by inspecting some or all of the payoffs for the most probable outcome (as in SAT-TTB), and then inspect additional payoffs for the second-most probable outcome from one or more of the most promising options.
  • Random selection: pick something without any consideration to value.

Both TTB and SAT (and their combinations) ignore information about some alternatives or attributes. Thus they solve some problems very efficiently, but make mistakes in others. WADD is not technically considered a “heuristic”, as it represents the strategy of searching all possibilities rather than making heuristic assumptions. Random selection is almost never resource-rational according to optimal models, but humans do tend to employ it in some situations, such as where the stakes are too low for the participant to care about the outcome, where they are under too much time pressure, or they are simply too tired to bother.

Press enter or click to view image in full size
Externalized thinking experiment: in addition to the total return from the path taken by the spider to an external node, the participant also receives a penalty for each revealed place value. This is believed to reflect an externalization of thought processes where additional thinking reveals more information about the nature of the problem, at the cost of additional time. Variations include modifications to the stakes for the participant, time pressure, and the distribution of rewards. (adapted from Lieder, Callaway & Griffiths, 2024)

The work also identifies the contexts under which different heuristics are resource-rationally optimal. Some broad conclusions are:

  • In a scenario where information gathering yields rewards, and the variance of those rewards varies in different ways as the search is extended, then different search strategies are optimal depending on the change in variance (Lieder, Callaway & Griffiths, 2024). Breadth-first search is optimal for decreasing variance (possibility of highest rewards closest to start). Best-first search is optimal (greedy or A*-style) for constant variance (all steps receive same variance of reward). Depth-first search is optimal for increasing variance (possibility of highest rewards closest to target).
  • Use fast-and-frugal heuristics (like TTB and SAT) when one outcome is significantly more likely than others (Lieder, Krueger & Griffiths, 2017).
  • Use simple heuristics (TTB, SAT-TTB, and random choice) that require minimal computational effort when the stakes are low. (Lieder, Krueger & Griffiths, 2017).
  • Use more complex heuristics and full search (SAT-TTB+, WADD) when the stakes are high (Lieder, Krueger & Griffiths, 2017).
  • When deciding the best choice or action by simulation, the optimal number of mental simulations depends on the ratio of the time needed to execute an action over the time needed to simulate it (Vul et al., 2014). For many everyday decisions, this translates as optimality being reached with a very small number of simulations. For example, if it takes at most three times as long to execute an action as to simulate it, then it can be resource-optimal to decide based on a single simulation.

Meta-level Markov Decision Processes

A common assumption in RL design is that there are two interacting systems: an external environment and the agent’s body. The body performs actions that affect the state of both the environment and the body. And the environment may have its own dynamics that also affect the state of the agent body. The agent’s cognitive processes are included in that system only to the extent that they are tasked with optimizing body action.

In contrast, the resource-rational perspective is that correct analysis of deliberative behavior requires “including the structure and resources of the mind itself in the definition of the environment to which cognitive mechanisms are supposedly adapted” (Lieder & Griffiths, 2019, p. 14).

This suggests that we need to extend those RL assumptions in some way. The most common RL formalism is that of a (fully or partially observable) Markov Decision Process (MDP), discussed in Part 2 of this series. It treats the environment and body as a single unified (but factorizable) state, with actions limited to those of the body. One option would be to extend the unified state to include the cognitive system, and to extend actions to unify across both body and cognitive actions. While theoretically sound, this model would be difficult to reason about in practice.

A simpler model is to factorize the behavior of the overall system into a hierarchy: an “object-level” MDP formalization for interactions between overt body action and the external environment, plus a “meta-level” MDP (meta-MDP) for the cognitive processes that choose the body action (Hay et al., 2012; Lieder, Callaway & Griffiths, 2024). Deliberation itself is essentially a sequential decision making problem under uncertainty (Russel & Wefald, 1991), and thus the MDP formalism can be conveniently applied at the level of cognitive state and actions.

Note that I’m using the object-level / meta-level terminology as used in the bounded-rationality literature, where “meta” refers to the act of deliberative control over overt action selection. This is different to the meaning of “meta” in meta-management, which is one step further up: meta-cognitive control of deliberation. I’ll try to keep these different meanings clear from the context.

Press enter or click to view image in full size
Object and Meta-MDP. The object-level MDP models overt action, while the meta-MDP models the deliberative process that chooses each object-level action. Computational operations, c, in the meta-MDP change belief states, b, ending in a terminal action ⟂ where the final belief state is translated to object-level MDP action, a. Computational costs are captured as r-meta, with the final computational step receiving the expected object-level reward. This cycle repeats for each object-level MDP action.

Formally, the meta-MDP is represented as follows:

Press enter or click to view image in full size

where:

  • Scripted B = the set of possible probabilistic beliefs
  • Scripted C = the set of cognitive operations (i.e., actions) that can update the agent’s beliefs
  • T-meta = a transition function modelling how cognitive operations change the agent’s beliefs
  • r-meta = captures the cost of thinking and the reward the agent expects to receive from the environment when it stops deliberating

The cognitive operations are effectively actions, but in computational space. To enable interaction between the meta-level and object-level, the cognitive operations include a special action, ⊥, that terminates deliberation and translates the current belief into an object-level action, a.

Many of the experimental papers I have cited in relation to bounded-optimality and resource-rationality use the meta-MDP formalism to train ideal models of resource-rational deliberative behavior in lab settings. This enables them to probe deeper into the heuristics by comparing human behavior to those ideal models. For example, planning can be treated as search over branches of plausible choices, and the associated search strategies can be factorized into a selection rule (which step to expand next?) and a stopping rule (when is the result good enough?). By careful adjustments of lab experiments, we can explore the selection and stopping rules that humans employ under different circumstances (Lieder, Callaway & Griffiths, 2024).

Another consistent and powerful theme from the meta-MDP formalism is that it offers a way to examine how to perform the resource-rational optimization described earlier in equations (3) and (4). Both extrinsic and intrinsic costs can be modeled via RL techniques like the use of value-functions, and model-free and model-based techniques can be used to explore how to optimize both short-term and long-term activity.

Intrinsic rewards

This also highlights the need for additional reward structures at the cognitive level. Extrinsic rewards are typically sparse, with a reward being received only after a long trajectory of overt actions have been taken. This sparsity already creates a difficult credit assignment problem. However, if each meta-MDP trajectory is guided by the expectation of a single reward for a single overt action, and that overt action is unlikely to receive an immediate reward, this makes the credit assignment problem all that much more difficult.

A number of different research areas have yielded suggestions for the kinds of intrinsic reward systems that may be at play within human behavior, for example:

Hierarchical Planning

The need for hierarchical mechanisms for planning has been intuitively obvious for many decades. Single-level RL mechanisms compute trajectories over sequences of moment-to-moment actions and struggle to solve problems of credit assignment, error accumulation, and irrelevant detail. By contrast, humans plan over high-level representations and break problems into high-level steps, rarely needing to consider more than a few steps regardless of wall-clock horizon length.

How to do that well in an AI RL setting remains a very active area of research. As my focus for this article is primarily to identify ideas outside of well-trodden RL research and as methods for hierarchical RL are described and discussed at length elsewhere, I will not attempt to cover it in any depth here. What I will cover is a small selection of results that have come out of behavioral and resource rational studies and which provide guidance for the future work in hierarchical RL techniques.

Heuristics that simplify planning

The intuition behind hierarchical planning can be described as: because humans appear do it, it must be a good way to simplify the problem. And because humans appear to follow some common techniques (e.g., high level abstractions and task decomposition) then those techniques must be good strategies for simplification.

The bounded-optimality and resource-rational perspective gives a clearer and more rigorous way to view these optimizations — as resource-optimal approximations (Solway et al., 2014; Correa et al., 2020; Correa et al., 2025). In other words, the existence of hierarchical planning and the hierarchical strategies employed by humans occurs as a result of a multi-timescale optimization over architectures, representations, and behaviors to maximize return under resource constraints.

As outlined by Correa and colleagues (Correa et al., 2020), humans have been found to employ several optimizing strategies, broadly categorized as modifications to either the search process or the problem representation. Examples of search process modification include: limiting the search depth, pruning away unpromising paths, directing search using model-free value estimates. Examples of modification to the search problem representation include, task decomposition and hierarchical planning.

Both of the latter can be productively viewed in terms of optimization.

Goals as search optimization

From the point of view of maximizing long term reward, there is no reason why people should have goals. The most optimal outcome is achieved by considering all possible trajectories forward from a current state and selecting the action sequence with maximum expected return.

Press enter or click to view image in full size
Trajectory optimization options. Option A: from initial state s₀, find any trajectory to anywhere that maximizes total return. Option B: select the highest return goal, g, then find the best trajectory to it.

Goal-selection is a resource-rational optimization that reduces the computational burden (Lieder & Griffiths, 2019; Correa et al., 2020) and enables additional modes of decision making and planning, such as backward planning, which may have benefits in certain circumstances.

It achieves that through factorization of reward-maximization planning. Reward maximization can be seen as simultaneously considering all possible outcomes and all possible trajectories to those outcomes. Instead of jointly maximizing over both, it can be more efficient to treat those as two separate stages of computation. The first stage identifies the outcome with the highest expected return, while the second stage identifies the best trajectory to reach that outcome.

The factorization has a drawback in that the first stage requires a pre-computed value function to integrate out the many possible trajectories to different outcomes. However, that can be cached, and the subsequent calculation of trajectories becomes considerably simpler.

The second stage receives a well-defined start and target and can heavily narrow its search and use different techniques for exploring trajectories (such as forward search from start, backward search from target, and sampling). Additionally, as the major consideration to globally-focused expected return was incorporated in the first stage, the second stage needs consider only local relative value (e.g., whether one trajectory is more plausible or less risky than another) and can focus more on dense intrinsic rewards (effort, duration) than on sparse external rewards.

Furthermore, the rewards and relative value estimates used in the second stage will tend to be partially independent from the global value function, enabling more efficient reuse of second stage policies across different global contexts.

Goals from successor representations

The first stage’s need for a policy-dependent but trajectory-independent value-function fits well with the idea of Successor Representations (SR), introduced in Part 2. SR factorizes value estimation into a delay-discounted future occupancy calculation, M(s₀,sₖ), and reward calculation, R(sₖ), as per equations (6) and (7) — see Part 2 for detailed explanation:

Press enter or click to view image in full size

By combining goal-based planning with SR, the full reward-maximization problem is factorized into three semi-independent parts: rate of reaching a potential goal state, trajectory-independent global rewards, and trajectory-dependent local rewards.

Formally, we have a system where the overall policy, 𝜋(a|s), must choose an action to take, a, from current state, s, by evaluating equations (8) to (10) below. It first identifies a goal state, g, by maximizing over successor occupancies, M(s,g), and global rewards, R(g), and then computes the best trajectory between states s and g, subject to expected local rewards, r-loc, and a discount factor, 𝛾.

Press enter or click to view image in full size

As shown in research on successor representations, this can enable rapid adaptation to changing environment rewards, even if it at the cost of some inaccuracies (Momennejad et al., 2017; Stachenfeld, Botvinick & Gershman, 2017).

Bottleneck states as sub-goals

One way to improve the efficiency of planning is to reduce the number of options available at each step. If a long-horizon problem is to be decomposed into sub-tasks, one must solve the problem of how best to decompose that problem. Assuming that the environment changes slowly, a globally optimal strategy is to pre-compute a small number of possible sub-goals made up of states that occur disproportionately frequently across many different trajectories (Solway et al., 2014).

This happens at bottleneck states, such as in doorways connecting rooms, bridges across rivers, and major road intersections. Using the terminology of network-theory, bottleneck states occur at the sparse inter-community connections linking otherwise densely connected communities (Schapiro et al., 2013), as in the below diagram.

Press enter or click to view image in full size
Example state transition graphs used in bottleneck experiments. In the Towers of Hanoi experiment, people tend to follow the trajectory that crosses only one boundary, despite both options having equal length (adapted from Solway et al., 2014)

Across a wide variety of experiments, behavioral research has found that humans employ bottleneck states for segmentation of their environment in both space and time, and use them as sub-goals for planning. For example:

  • Using doorways (rather than corners or other arbitrary points within rooms) for planning navigation through multiple door-connected rooms (Solway et al.)
  • When navigating virtual towns with bottleneck points, people plan hierarchically, selecting bottleneck points first, before planning within-region routes, even when they cannot directly observe the town map (Solway et al.).
  • When tasked with identifying natural segmentation points within apparently arbitrary streams of non-uniform patterns, people tend to spontaneously select bottleneck states from a withheld generative graph structure such as the one above (Schapiro et al.; Solway et al.)
  • fRMI studies find that closely related events (those that tend to occur closely in time) have more similar neural activation patterns than unrelated events, irrespective of the visual similarity of events (Schapiro et al.)
  • In Tower of Hanoi problems carefully constructed so that two equal-lengthed sequences both solved the puzzle, 72% of participants pursued the strategy that crossed only a single bottleneck state instead of two (Solway et al.).
  • In experiments involving techniques that break Euclidean maps (e.g., bus-stops, highways, teleportation portals), people have been found to reason about graph networks of topological connectivity between these states, rather than as strict Euclidean maps (Solway et al; Warren et al., 2017; Peer et al., 2020).

The result that people reason about topological connectivity is theoretically supported by work on cognitive maps in the hippocampus, which found that place cells are poorly described by models that assume they represent a Euclidean map (Stachenfeld, Botvinic & Gershman, 2017). Successor representations (more specifically, latent successor features as predictions of expected future state) were found to better predict the behavior of place cell firing in different environments. This suggests that neural representational similarity follows transition structure rather than Euclidean distance.

Identification of bottleneck states is more formally known as a k-way normalized min-cut problem, and can be found through eigen-decomposition of the spectral graph over state connectivities. When successor representations are available, the eigen-decomposition can be performed over the matrix of successor representations, simplifying the process and suggesting a biologically plausible mechanism for bottleneck discovery (Stachenfeld, Botvinic & Gershman, 2017). This suggests SR as a strategy for bottleneck discovery in AI systems. Alternatively, artificial NN models have demonstrated the ability to find bottleneck states through simple regularization techniques (Pertsch et al., 2020).

Bottleneck states as sub-goals offers an additional way to optimize the goal-based planning factorization discussed earlier. Rather than learning value functions over all possible states, it is better to learn the value functions of bottleneck states and to select between those as sub-goals. Thus reward maximization is further simplified to the task of selecting between a small set of bottlenecks as goal or sub-goal, and then computing trajectories between bottleneck states.

Representation Redescription Theory

I switch now to a developmental perspective, as it tells us something important about representations of knowledge. From developmental psychology, the theory of Representation Redescription (RR) suggests that humans employ many different representations, even for similar knowledge (Karmiloff-Smith, 1992; Karmiloff-Smith, 1994).

Colloquially, RR draws a distinction between knowledge in the mind and knowledge for the mind. More concretely, knowledge can be broadly classified as implicit or explicit. Implicit knowledge is embedded as part of a behavior but not otherwise accessible. Explicit knowledge is accessible for inspection, spontaneous on-demand theory generation, and verbal report. RR claims that knowledge is initially obtained in an implicit form, and gradually becomes more accessible for high-order reasoning through a process of gradual modularization and “explicitation”.

Press enter or click to view image in full size
RR representational processes. Three recurrent phases that create or revise new representations.

RR proposes the following stages of knowledge acquisition, consistent with both observational developmental studies and controlled lab experiments on humans:

  • Phase 1: information acquisition from the environment. This phase is characterized by being externally driven — experiential, teacher training. It culminates in behavioral mastery, but that mastery may apparently be contradicted by a poor ability to articulate the skill verbally.
  • Phase 2: internal representations become the focus of change via system-internal dynamics. This phase is internally driven. It includes, for example, self-initiated theory generation. The individual can articulate their knowledge verbally, but that knowledge may at times contain errors due to the lack of reconciliation from phase 3.
  • Phase 3: internal representations and external data are reconciled. The individual tests their new knowledge against the external world and revises that knowledge to account for any errors.
Press enter or click to view image in full size
RR representational forms. Multiple forms of representation, each serving different purposes.

During those phases, knowledge representations transition between at least four different characterizable forms, with each new form adding to the existing rather than replacing them:

  • Implicit (I): This is the primary form of procedures or action patterns for responding to stimuli in the external environment. The encodings are often sequentially specified and inflexible. New representations from each learning event are initially stored independently, even if closely related, and so no intra-domain or inter-domain representational links are established. The procedure as a whole is learned, but its component parts cannot be independently reviewed. Examples of this form can be seen clearly in the consistent sequential pattern for drawing a human figure typically taken by 4–6 year old children (draw head, draw body, then draw limbs), even when asked to modify the drawing in an unfamiliar way (Clark & Karmiloff-Smith, 1993). This is knowledge in the system, but it is not accessible for the system to do conceptual analysis.
  • Explicit-1 (E1): Redescription of Implicit procedurally encoded representations into a new format that is now open to intra-domain and inter-domain representational links. Simpler and less special-purpose, but more abstract and more flexible; for example, identifying a link between a zebra and a zebra-crossing. The internal knowledge representations enable more flexible behaviors, but their content remain inaccessible to conscious introspection, manipulation or verbal report.
  • Explicit-2 (E2): Re-represented into non-linguistically encoded representations, probably spatial representations. Now consciously available, but not sufficiently well-formed in order to give verbal report of it.
  • Explicit-3 (E3): Re-represented into a cross-system code that can be converted to verbal report. The knowledge is now arbitrarily queryable — both the whole and the parts of a given concept are understood and can be examined from many different perspectives. It’s possible that some linguistic knowledge is directly translated to level-E3, skipping E2.

One particular benefit of the more explicit forms of knowledge stands out in the context of the earlier discussion on resource limits — that of on-demand knowledge generation (Lombrozo, 2024). Knowledge gained from any given experience has a potentially infinite number of ways in which it will be relevant to future contexts. This introduces a resource-optimal learning problem: how and when should learning be performed to maximize the number of contexts in which the knowledge can be flexibly applied, while minimizing the resource expenditure for that learning?

In the front-loaded extreme, the agent immediately embeds as many different forms of the knowledge as it can. This would require considerable up-front time for representational computations, consume substantial amounts of storage, while most of those different embedding forms would never be required. In the end-loaded extreme, the agent simply records all raw event information verbatim and leaves the adaptation process to on-demand processing. This avoids the up-front computational overhead, at the cost of extensive computational overhead in the future where the need for rapid response may not offer the luxury of lengthy computations. It also carries excessive storage requirements.

Get Malcolm Lett’s stories in your inbox

Join Medium for free to get updates from this writer.

The more efficient strategy, as is often the case, finds a balance between up-front and on-demand computation; but not just through careful tuning of computational degree. Rather, the change in knowledge representation from rigid implicit forms to more easily queryable explicit forms enable arbitrary and efficient on-demand knowledge generation.

Representational review of RL

A concrete look at the mathematical representations within RL frameworks brings these discussion points home. If an agent is presented with an object to interact with, a P(a|s) policy representation suggests the actions that should be taken against it, a P(s’|s,a) forward-model representation suggests how the object will change if an action is taken, and a M(sₖ|s₀) successor representation predicts what state the object will most likely take in the future. None of these representations support a descriptive account of what the object is made of, how it relates to other objects (present or not currently present), or on-demand knowledge generation through inductive and deductive reasoning.

Press enter or click to view image in full size
What different RL components can tell us about an object under investigation (soccer ball from openclipart.org)

All of the above three representations can be described as first-order or implicit models. This is true even for RL’s most rated model-based transition models, and even when they pull a PR stunt and rename those models as “world models”. These different representations can support substantial adaptability, but they suffer a number of limiting characteristics (Clark & Karmiloff-Smith, 1993):

  • Network learning is example-driven. The internal representations are changed only such that they should mirror the statistical properties of the input/output pairings observed from the environment. No deeper conceptual re-assessment occurs that could produce Explicit-1/2/3 representations.
  • Knowledge of rules is emergent. Behavior is the result of a complex interaction between many thousands of sub-symbolic representations, optimized for asymptotic fitness. There is no high-level symbolic representation of the rules actually being carried out.
  • No introspective self-observation. The agent becomes the literal manifestation of those emergent rules, and ideally this should offer the opportunity to identify the aforementioned high-level symbolic representations, however the models have no ability to observer their own representations or behaviors. Thus they remain ignorant to what could be extremely beneficial knowledge, unless some external agent offers the information.

Classic AI devoted much effort to symbolic representations, inspired in part by these very problems. The “unreasonable effectiveness” of connectionist models like RNNs and LLMs have greatly challenged those earlier inspirations for symbolic AI, but the current connectionist techniques fundamentally follow an Implicit knowledge formulation. The LLM can action a long description of a concept, but it cannot truly inspect its own conception.

An example of how Implicit-only knowledge can be a problem is seen by applying the Deep Thinking perspective (see Part 1) to multi-layered sequence architectures like RNN & Transformers (especially the decoder-only style of transformer that is popular today).

Press enter or click to view image in full size
Shallow high-level thought problem. Deep-thinking conclusions at one time step are not accessible to the full sequence of deep-thinking at the next time step. Affects both RNNs and decoder-only Transformers.

These architectures have a vertical stack of layers, with information propagating up the layers within a given time-step. The whole stack is iterated, with state at each layer transferring laterally to the same level at the next time-step, and then propagating upwards. In such architectures, all layers are able to directly or indirectly process state representations held within the lowest layers. However the representations within the highest layers can only be attended to by themselves (state moves laterally and upward, never downward).

This makes sense when we conceptualize the stack of layers as increasingly high-level abstractions. However, it’s a problem when we conceptualize the stack as a sequence of deep thinking. There is no opportunity for the entire stack of layers to iterate a full depth of “thinking” against the highest layer’s representation. This creates a shallow high-level thought problem, limiting the capacity for long-horizon abstract reasoning.

The ideas from RR suggest that we can gain a lot more flexibility, with much fewer resources, by incorporating a wider mixture of representations. Particularly those focusing on higher-order and/or explicit knowledge representations that support arbitrary querying from different perspectives. I also expect this to be important for meta-management of systems that learn to plan. Such systems must produce their own internal reward structure, and uncovering that reward structure is itself a learning exercise.

Unfortunately, the earlier work on RR leaves open the question of the redescription process (how implicit representations become redescribed as explicit) and form (what form the explicit representations take), offering only some general discussion about connectionist vs symbolic ideas (Clark & Karmiloff-Smith, 1993).

The meta-management feedback loop described in the introductory article is one possible solution to this problem. It takes a high-level representation of the cognitive state and makes it available as a first-class sensory input at the next time-step, enabling low-level layers to attend to it once again. But that is just one option. I leave a more thorough search for more recent implementation ideas to later work.

Meta-cognition

RR provides one viewpoint on the larger study of Meta-cognition, which examines the ability for individuals to review their own knowledge, memories, and physical and mental abilities, and to use that information to adapt their behavior (Winkielman & Schooler, 2012; Fleming at al, 2012; Koriat, 2007). Meta-cognition “adds flexibility to cognitive processes, making them less dependent on external cues” (Fernandez-Duque, Baird & Posner, 2000, p. 289).

Meta-cognition should in theory offer the strongest insight into the low-level meta-management mechanisms needed to support human abilities for deliberation and learning to plan. In practice, the study of meta-cognition is extremely difficult. Much of the research has focused on what can be objectively studied via overt behaviors as proxies for internal cognitive processes. The result is that theories of the underlying processes are somewhat fragmented, with the most useful insights coming from very focused areas of research — such as in the case of the resource rational search behaviors discussed earlier.

Another problem with meta-cognition research has been the difficulty in converting those theories to practical AI solutions. Most attempts have focused on symbolic AI, where hand-crafted rules and data structures are manipulated via hand-crafted algorithms. Some systems incorporate aspects of learning and connectionist AI, primarily for conversion of perceptual space to symbol space.

While useful, these approaches still focus on the high-level cognitive behaviors, without a deeper understanding of how those high-level behaviors emerge from lower-level processes. My focus is on understanding and replicating the processes that biology employs, and it is hard to imagine how those symbolic approaches can be translated.

That being said, there are plenty of opportunities to take inspiration from what we do know about meta-cognition. Many of the advances in reasoning AI have been motivated by attempts to replicate these high-level behaviors via architectural and prompting techniques. And by analyzing the gaps between AI behavior and acknowledged human meta-cognitive behaviors we can identify direction for future research.

These final sections briefly summarize meta-cognition research, link to cognitive architecture research, and discuss how a common theme of work on meta-cognition and cognitive architectures (error detection and correction) brings us back to the topic of meta-management.

Human meta-cognitive behaviors

Press enter or click to view image in full size
Layers of study. The study of Meta-cognition is diverse and influenced by many research disciplines. It tends to focus on behavioral studies of meta-memory, meta-learning, meta-knowledge, meta-control, and theory of mind. I see Meta-management as encompasing theories of the low-level mechanisms underlying those behaviors.

The study of meta-cognition covers a wide span of different cognitive behaviors, broadly grouped as follows:

  • Meta-memory: meta-cognitive awareness of memory. Memories appear to be encoded or are accessible in at least two forms: one for their specific content, and a second form that identifies metadata such as source (Fernandez-Duque, Baird, & Posner, 2000; Benjamin et al., 1998)., certainty (Carruthers & Williams, 2022; Paul et al., 2015), or mere familiarity — as in “feelings of knowing” and the “tip of the tongue” effect (Rosenthal, 2012; Shimamura, 2000).
  • Meta-learning: individuals can assess their own learning progress. They use that to make judgements of certainty and to take corrective actions to improve their learning progress. This has been primarily studied in terms of its contribution to successful learning in school-like settings (Paris & Winograd, 1990), selection of strategies for memory, learning, lifespan approaches (Marković et al., 2021; Shimamura, 2000), and learning higher-level objectives (Timmermans et al., 2012).
  • Meta-knowledge: various aspects of meta-cognition appear to depend on developing new representations about first-order processes and knowledge. These new representations are assumed to be both (i) at a higher-level abstraction than used in first-order processes, and (ii) about those first-order processes and knowledge, referred to as second-order or higher-order representations within much of the literature. In addition to the work on RR discussed above, other research has looked at classification of first-order outcomes into knowledge, hope, fear, regret, etc. (Cleeremans et al., 2007), and how the absence of knowledge may be identified (Fleming et al., 2012).
  • Meta-control: while some aspects of meta-cognition merely enable reporting on first-order processes, others elicit direct control over those processes as a result of that meta-cognitive awareness. Examples include trading off between exploration and exploiting existing knowledge (Marković et al., 2021), control over working-memory and attention (Whitmarsh et al., 2017; Shimamura, 2000), balancing effort vs benefits of possible behaviors (Carruthers & Williams, 2022; Marković et al., 2021; Peters, 2010; Fernandez-Duque, Baird & Posner, 2000), planning (Marković et al., 2021; Cleeremens, 2020; Fleming et al., 2012; Fernandez-Duque, Baird & Posner, 2000), internal conflict resolution (Shimamura, 2000; Fernandez-Duque, Baird & Posner, 2000), maintenance of cognitive homeostatic needs (Peters, 2010; Shimamura, 2000), and emotion regulation (Shimamura, 2000).
  • Theory of mind: meta-cognition forms the basis for theory of mind and its ability to support social cohesion (Carruthers & Williams, 2022; Cleeremens, 2020). One concrete example is that it enables a group to identify the individual who is most certain about some decision point (Cleeremens, 2020; Fleming et al., 2012; Cleeremans et al., 2007).

Cognitive architectures

Attempts have been made to translate the observed and theorized behavioral aspects of human meta-cognition, as well as some theorized underlying mechanistic aspects, into artificial cognitive systems. This gives rise to Computational Meta-cognitive Architectures.

For detailed reviews see that of Cox (2005), Caro et al. (2014), Nolte et al. (2025), and Bohre, Wagle & Patil (2026). More recently, there is a growing interest in incorporating meta-cognitive capabilities into LLMs and Large Reasoning Models (LRMs). See reviews by Liu et al. (2026) and Chuprov et al. (2026) for details.

Press enter or click to view image in full size
Simplified version of Meta-cognitive Integrated Dual-Cycle Architecture (MIDCA) (adapted from Cox et al., 2022)

Meta-cognitive Architectures form a subset of a much larger research effort into Cognitive Architectures — both the effort to develop high-level theories of human intelligence and to build artificial versions thereof. For a wider survey of cognitive architectures, see that of Kotseruba & Tsotsos (2020).

Historically, development of cognitive architectures focused on symbolic systems, but more recent efforts typically combine symbolic methods for high-level processes and connectionist methods for perceptual processing. Only a small number of architectures have been proposed that employ only connectionist solutions, for example that of D20 (Daglarli, 2020). All of the small handful of current meta-cognitive architectures that follow an auto-meta-management architecture use symbolic reasoning (Nolte et al., 2025; Marshall, 2006; Mascarenhas, et al., 2022; Goldberg, 2022; Peller-Konrad et al., 2023).

This tells us something of the difficulty of reverse engineering how the fully-connectionist brain produces so much apparently symbolic reasoning.

Fixing and learning from mistakes

A feature that is common to many meta-cognitive architectures is that of error detection and correction. The process typically follows a sequence as follows:

  1. Prediction: make predictions of expected outcomes before carrying out behaviors.
  2. Error Detection: identify an error based on large discrepancy between expected and observed outcome. Alternatively, a potential error may be identified due to a conflict between predictions made by different control strategies.
  3. Explain: examine the error in order to generate a theory that explains its cause.
  4. Goal: generate and transition focus to a new goal for rectification of the failure, either for the immediate instance or to prevent future repetitions
  5. Act: carry out the new goal — which may involve further evaluation of possible strategies for current and future rectification.

This process can be applied to perception (e.g. when we briefly mistake a person for someone else), general regulation of ongoing cognitive processes, errors in memory retrieval, and errors in conceptual reasoning (Caro et al., 2014).

Some work has sought to concretely identify the kinds of error that can occur during lengthy deliberation. One such list is as follows (Beaudoin, 1994) — repeated from the introductory article:

  • Oscillation between decisions. Wasteful re-assessments of decision points, leading to a meta-stable (oscillating) but stagnant state (ultimately achieving nothing useful).
  • Insistent goal disruption. Repeatedly getting distracted by competing goals that have been previously disregarded.
  • High busyness. Attempting to multi-task between too many goals, leading to poor outcomes.
  • Digressions. Choosing to deliberate over some sub-goal, and then losing track of the “big picture” by forgetting to return to the overarching goal.
  • Maundering. Getting stuck deliberating over the details of a goal without making a decision.

The sequence of error detection and correction (prediction, error detection, error explanation, corrective goal selection, and corrective action) requires a mixture of different kinds of process. Some are automatic background processes that are always monitoring but only influence deliberative cognition when something goes wrong. Others require explicit deliberation and access to mental models of causative relationships. These align to automatic and deliberate meta-cognitive processes theorized in human meta-cognition (Shakarian, 2026).

Press enter or click to view image in full size
Timing relationships between cognition and meta-cognition

Explanatory analysis of past mistakes is only one possible state of a timing dimension to meta-cognition. Meta-cognition can also occur in anticipation of future cognitive performance (e.g. predicting the inability to find a suitable plan, or predicting plan execution failure), or immediately during the object-level activity (i.e., providing a high-level overarching meta-control of ongoing cognitive activity) (Cox et al., 2022).

Taken together, this begins to tell a story of the lower-level meta-management processes underlying error detection and correction. Predictions of outcome. Automatic monitoring. Cognitive alarms that change attentional focus. Implicit and explicit representations. Differentiation of meta-management processes into before, during, and after.

I leave that as a thought for now, but I will return to it in Part 4, particularly to the differentiations of automatic/deliberative and timing.

Not Covered

Avenues abound for gaining inspiration from efforts to understand human intelligence, that we might replicate desirable aspects of it. People have been questioning and theorizing on what makes human intelligence for thousands of years. The topics that I have not even mentioned fill numerous libraries around the world.

I will briefly mention two such areas.

Neuroscience

There is extensive neuroscientific work investigating how different brain regions contribute to different aspects of cognition. There is still much that is uncertain and debated, but also some growing consensus. Many theories of human intelligence and consciousness try to align their functional components to specific brain regions (or interacting systems of brain regions) based on that consensus. There is much to be gained through this avenue, as evidenced by what we have already gained, but also many ways in which we can delude and derail ourselves in the effort.

My approach for now is to avoid assuming any explicit correlation between function and physical form, but ultimately we must if our theories are to be grounded in reality.

Non-computational accounts

I tend towards the more computational approaches for understanding human behavior and intelligence, but there are many non-computational approaches that offer insight. One such example is that of Ecological Psychology, which, among other things, focuses on direct associations between the affordances offered by the environment (simplistically, availability of options for interaction) and the actions taken on the environment. Some work has shown success in RL agents using this idea of affordance as an alternative way of encoding information about the state of the observed environment (Horton, Chakraborty & Amant, 2012; Kim et al., 2026).

The Enactivist 4Es model of cognition (embodied, embedded, enacted, and extended) emphasizes how cognition must be viewed as part of a system of interrelated interactions between environment, body, and mind. Essentially, if we have any hope of understanding intelligence, then we must understand how evolution found the solutions that it did. And to understand that we must take into account the whole dynamical system.

Part Summary

This article looked at four areas where biology can inform us of designs for adaptable and intelligent systems, and which help us in identifying potential problems in our current AI solutions. Efficient and appropriate planning must trade-off the cognitive costs against the long-term benefits. Hierarchical planning is a resource-optimal way to simplify an intractable search problem. Multiple representational forms are required in order that agents can have knowledge that is both in them and for them. And all of these processes, and more, must be collected together into a cohesive architecture.

The resource constraints on silicon agents (particularly those floating in the proverbial cloud) are considerably different in degree and form to the constraints on biological brains attached to wet and hungry human bodies. However, many of the same principles apply. The researcher-driven evolution of artificial reasoning architectures must design systems that adapt their own computational behaviors if they are to adapt their external behaviors to their ever changing environments.

The last part of this series will focus on my own ideas for pulling together the various topics discussed in these articles so far.

Acknowledgments

Unless stated otherwise, all diagrams and images are the work of the author, with AI help for the images.

Citing

For attribution, please cite this work as:

Lett, M. (2026, Aug). Can an AI Learn the Planning Algorithm? Part 3: The Biological Perspective. AI Advances. https://medium.com/@malcolmlett/can-an-ai-learn-the-planning-algorithm-part3-b3718af62202

References

--

--

Malcolm Lett
Malcolm Lett

Written by Malcolm Lett

302 followers
14 following

I study Human and AI intelligence and consciousness

AI Advances
AI Advances

Published in AI Advances

116K followers
Last published 12 hours ago

Ideas in depth. Where exceptional thinking takes shape.

Help

Status

About

Careers

Press

Blog

Store

Privacy

Rules

Terms

Text to speech