Implementation
I built a simulator with persistent gates and a SQLite sandbox. Their 15 controller variants use the same basic planner but differ in how they check evidence and update memory. Some check the suspected failure, some check randomly, and others verify the next action.
Results
Each agent had to complete a sequence of orders while handling unreliable observations. I measured completed tasks, incorrect memory changes, and how much correct information remained supported by evidence.
- Alice completed 46.6% of main episodes, compared with 50.8% for action verification and 44.6% for immediate revision.
- Incorrect edits per episode were 0.31, 3.38, and 14.63 respectively; supported retention was 90.5%, 99.5%, and 97.3%. Selective revision reduced incorrect edits but did not improve completion.
- All controllers failed when observations remained corrupted: their shared planner still waited for confirmation from the faulty channel. Checking the suspected failure did not clearly improve completion over random checks.
Evaluation details
The evaluation contains 38,400 controller episodes across 2,560 scenarios. The main comparison has 1,920 episodes per controller; tests with combined failures or an unavailable explanation are separate. Success means completing all 24 orders.
Related conditions come from 40 independent base units. The paired bootstrap resamples these units, rather than treating every repeated episode as an independent sample.
Limitations
These are small symbolic environments with initially correct memories, mostly hand-designed failures, and a working backup tool. The study does not test language-model agents.