Accepted at NeurIPS 2026 · Reinforcement learning

Learning When to Think

Adaptive Internal Computation for Reinforcement Learning

Virginia Tech

Paper coming soon Code coming soon
One policy, different depthsUp to 8 steps
Animated conceptual example: a routine state acts after one refinement; a hard state continues to depth five before acting. These are examples, not measured policy depths.

Example only; the pictured depths are not measured policy outputs.

12benchmark tasks
5policy families in the transfer study
8maximum refinements in the main setup

01 / Motivation

Give each decision the computation it needs.

Iterative policies can refine a state representation before acting. A fixed-depth policy spends the same number of refinements on every state, even when some decisions are routine and others need more thought.

Adaptive Internal Computation (AIC) learns a stopping decision alongside the action. It can stop early or continue refining within a capped budget, then executes one action in the environment. The paper asks when this state-dependent allocation helps, and separates its effect from other changes in training.

Core ideaLearn where to spend internal computation, not just how much to allow.

02 / Method

One policy learns to act and stop.

An encoder produces a latent state. A shared residual block refines it repeatedly. At each depth, action and value heads read the representation, while a halting head decides whether to continue. Stopping at the maximum depth is forced.

Figure 1 from the camera-ready manuscript: a state encoder feeds a shared refinement block, with action and value heads at each depth, learned halting heads, and a forced stop at the maximum depth.
Updated Figure 1 from the camera-ready manuscript. The main configuration caps refinement at eight steps. View full size ↗
1

Refine

Reuse one residual block to improve the representation without adding a new block of parameters at every depth.

2

Decide

Sample a stopping depth and an action as a joint policy, so PPO can assign credit to both choices.

3

Budget

Penalize expected depth and supervise value predictions at every depth with a shared rollout target.

Intermediate value differences are diagnostics, not labels for the best stopping depth or counterfactual gains.

03 / Experimental results

Performance depends on the task and the budget.

The twelve-task benchmark spans control, manipulation, navigation, language, gridworld, and rare events. At the reported eight-step nominal cap, AIC has higher mean reward than the listed fixed comparator on nine tasks, ties on one, and is lower on two. These are operating-point comparisons; realized computation is not matched by the cap alone.

Three interactive simulator runs

Watch the task. Compare the policies.

Choose DoorKey, Memory, or StackCube. The 20-second videos illustrate the environments, not policy rollouts. The graph shows fixed-policy and AIC reward means reported in the paper.

DoorKey · 0:00 / 0:20

Find a key, unlock a door, and reach the goal. About this environment ↗ · License

Interactive reward graph

Fixed policy vs. AIC

FixedAIC

DoorKey: AIC has higher mean reward (+0.0211) and uses 3.36 refinements on average versus 8 for Fixed-8.

Six plots compare AIC and fixed-depth reward across nominal budgets 1, 2, 4 and 8 for PickCube, StackCube, AntMaze, RareEvent, FourRooms and Puzzle.
Figure 2 of the camera-ready paper: five-seed means with ±1 standard deviation bands across six benchmark tasks. View full size ↗
+0.0185

PickCube

AIC minus the listed fixed comparator at nominal B = 8.

+0.0191

AntMaze

AIC minus the listed fixed comparator at nominal B = 8.

−0.0216

FourRooms

A default-setting loss at nominal B = 8.

04 / Mechanism

What actually drives the gains?

A higher return can come from adaptive allocation, intermediate supervision, extra capacity, uniformly useful refinement, or different training data. The paper tests these explanations with targeted early exits, a yoked data study, and an allocation-by-supervision factorial.

Intervention

Truncate selected states

Forcing high-depth states to stop at depth one often hurts more than truncating a count-matched random set. This indicates sensitivity to chosen refinements, though the amount of compute removed is not matched.

Factorial

Separate allocation and supervision

PickCube and AntMaze show allocation-dominant gains. StackCube has mixed effects. RareEvent is supervision-dominant.

Adaptive stopping and supervision factorial · reward means from Table 5
TaskNeitherSupervision onlyAllocation onlyFull AIC
PickCube.421.434.472.488
AntMaze.451.458.505.518
StackCube.394.413.441.455
RareEvent.558.589.560.591

These factorial means have no seed-level intervals and should be interpreted within this study, not substituted into the main benchmark.

05 / Theory

Why adapt depth at all?

Under explicit assumptions, the paper shows an allocation advantage at equal expected depth. If easy states are already solved at depth one and a small fraction of hard states need depth K, an ideal state-dependent allocator can reserve depth K for hard states. Any state-independent stopping schedule at the same expected budget retains a positive expected shortfall.

Theorem 1 · intuition

Spend one step on easy states. Save the deeper computation for hard states.

This establishes an opportunity for allocation under the theorem’s value-profile assumptions. It does not guarantee that AIC learns the ideal stopping rule.

06 / Limitations and open questions

Better allocation remains difficult.