Bounded Autonomy: A Budget-Constrained Framework for AI Agent Operations Toward Safety-Critical Process Environments
Give the agent a budget. Not a blank check.
Mohammed Imran Khan1,2 , Manju Khanna2 , Abhishek Shivkumar1 , Tuhin Sharma1 , Adinarayana Sakala1 , Nirav Kumar1 , Pranjalya Singh1 , Darshika Pundir1 , Atrik Ghosh1
- 1 Red Hat, Bengaluru, Karnataka, India
- 2 Amrita School of Computing, Amrita Vishwa Vidyapeetham, Bengaluru, Karnataka, India
DOI 10.2118/236401-MS
// Summary
Instead of approve-everything or free rein, agents get hard limits on actions and how much they can change the process. Safer, faster, and cheaper to run.
Bounded Autonomy constrains AI agents along four dimensions - tool calls, cumulative process perturbation, simulation timesteps, and checkpoint intervals - calibrated from one tightness knob. Pre-execution LangGraph middleware rejects over-budget actions. Across 7,100+ experiment cells on OpsEnv, tight budgets cut reactive safety-check rejections ~89% and token cost ~85% while completion retention centers on 100.8% (95% CI 90.9-111.5%). ReAct runs ~7× faster under tight budgets.
~89%
fewer safety-check rejections under tight budgets (reactive)
~85%
lower token cost with completion essentially unchanged
7×
wall-time reduction for ReAct under tight budgets
100.8%
pooled completion retention (95% CI 90.9-111.5%)
// Explainer video
0:00 / 0:00
Bounded Autonomy explainer
Space = play · ← → seek · M mute
// How it works
Budget B
4 limits from one tightness knob
Pre-check
Middleware rejects over-budget calls
Execute
Update counters + checkpoints
Rollback
Sim-only restore when needed
// Problem
Agent governance in process plants is usually binary: full human approval (too slow) or free rein (too risky). There is no graduated way to quantify how much autonomy an agent should get, or to cap its physical impact on the process.
// Approach
Four-dimensional autonomy budget B = (tool calls, perturbation, timesteps, checkpoint interval), derived from a single tightness multiplier on each scenario's Nopt. Middleware pre-checks every tool call, executes if within budget, updates counters, and checkpoints. Simulation-only rollback restores state when needed. Advisory layer below SIL 1 - always behind BPCS/SIS.
// Notable contributions
- ▹ Four-dimensional autonomy budget including physical process perturbation - not just tokens or API calls
- ▹ Pre-execution LangGraph middleware that rejects over-budget actions before they reach the process
- ▹ Empirical safety-completion trade-offs across 4 architectures, 4 budget levels, and 3 LLMs
- ▹ Adaptive budget controller that redistributes remaining budget from completion, safety margin, and utilization signals
- ▹ Checkpoint rollback evaluation (simulation only) with 1,295 triggers and 33.3% completion-improvement rate
// Method
// Figures from the paper
Graphs as published
Exact figures extracted from the SPE manuscript - architecture, heatmaps, completion, wall time, and the rest. Click any figure to enlarge.
Fig. 1
Bounded Autonomy architecture
Fig. 1. Budget enforcement middleware intercepts every tool call. Over-budget proposals are rejected before they reach OpsEnv; approved actions update the remaining budget and may create checkpoints.
Alg. 1
Budget Enforcement Middleware
Alg. 1. Pre-execution checks on tool calls, perturbation, and timesteps; one-step safety simulation; checkpointing on interval Br.
Fig. 2
Mean completion by architecture and budget
Fig. 2. Tier 1, static mode. Individual run scores overlaid. R&E baseline at 0.704. Tight budgets keep completion essentially on par with unconstrained.
Fig. 3
Overshoot effect under unconstrained budgets
Fig. 3. (a) Under unconstrained conditions, actions and completion are weakly anti-correlated for ReAct (r = -0.22). (b) Paired per-scenario comparison: tight budgets often match or beat unconstrained completion.
Fig. 4
Pareto frontier: completion vs safety violations
Fig. 4. Tier 1, 960 static cells. Marker shape = architecture; fill = budget level. Unconstrained reactive agents sit at high violations; budgets pull them down the y-axis.
Fig. 5
Mean safety violations heatmap
Fig. 5. Tier 1, static mode. Unconstrained ReAct/Reflexion average ~9.6-9.7 violations; tight/minimal budgets cut that to ~1.1 / ~0.9.
Fig. 6
Mean wall time by architecture and budget
Fig. 6. Log scale. Tight budgets cut ReAct wall time ~7× and Reflexion ~6.7× by stopping wasteful action loops.
Fig. 7
Per-scenario operating regime under tight budget
Fig. 7. Tier 1, static mode, pooled across architectures. Left: mean completion. Right: mean safety violations. Optimization (A) stays near-zero violations; some disturbance (B) scenarios remain harder.
// Key findings
Pooled completion retention centers on 100.8% with a worst-case CI bound of 9.1% loss - budgets do not tax task success on average.
Unconstrained agents often take ~5× more actions than necessary; excess actions can be counterproductive.
Architecture matters: reactive agents benefit most on safety and speed; Plan-and-Execute already uses short sequences and retains ~98.9%.
Some multi-step optimization tasks (e.g. A04 under ReAct) can lose up to 73 pp under very tight budgets - tightness is a knob, not a free lunch.
Perturbation is a distinct governance signal from tool-call counting - process impact, not just compute cost.
// Concepts
Budgets
four limits on agent autonomy
Perturbation
cap cumulative process change
LangGraph
pre-execution reject middleware
OpsEnv
20 CSTR scenarios
// Slideshow
Bounded Autonomy
10 slides · native playback with animations
Preparing…