The Epistemic Agent — An Agent That Must Say Why

The Concept

Read any coding-agent transcript a week later and you get a list of commands. ls, cat process_orders.py, python -c "…", pytest -q, git commit. What you never get is the thing you actually need: what the agent thought it was doing. The command is the residue of a decision; the decision itself is nowhere in the log. Reasoning tokens don’t fix this — they are verbose, they are not aligned with the tool calls, and increasingly they are not even shared.

The Epistemic Agent` moves the justification into the tool call itself. The shell tool has three mandatory arguments instead of one:

• command — the shell command, as usual
• type — what kind of act this is, from a closed vocabulary
• reason — free text: the intent behind this command

The epistemic vocabulary

The epistemic agent has seven types of tool calls:

• collect-info — read-only observation to build understanding. Nothing is being tested, nothing changes: ls, grep, git log, --help.
• change-state — mutating the world in a way meant to persist, because it is part of the deliverable.
• test-hypothesis — an experiment run to confirm or refute a belief the agent holds right now. The reason must name the belief and what observation would refute it.
• verify — re-running a known check to confirm finished work. This is not a hypothesis test: nothing is in doubt, the agent is producing evidence, not learning. Collapsing the two is exactly how agents end up “testing” things they have already decided.
• setup — installs, scaffolding, fixtures. It mutates state, but it is a means, never the deliverable. Lumping it with change-state makes a diff look twice as risky as it is.
• revert — undoing a previous change-state that proved wrong. Given its own type, dead ends become first-class documentation instead of something the agent quietly walks back.
• cleanup — removing the agent’s own temporary artifacts. Distinguishing a self-cleanup rm from a deliverable rm is the difference between a boring line and an incident review.

The crucial rule is that the type is chosen by the agent’s epistemic situation, not by the command. pytest -q is test-hypothesis when a particular failure is expected, and verify when finished work is being confirmed. The same string, two different intentions. No static analysis of a transcript can recover that distinction — only the caller knows, and here it has to say.

The file tools (read_file, write_file, str_replace_edit) take a mandatory reason but no type: their act type is a property of the tool, not of the situation, so asking for it would only invite lying. The result is a journal with no gaps — every single tool call in a session carries a justification.

Interpretability at the action boundary

Interpretability research mostly looks inward: probing activations, attributing circuits, reading chains of thought. For an agent that acts on a real filesystem, there is a second place where interpretability is decided, and it is much closer to what we care about — the boundary where the model stops thinking and touches the world. Every consequence an agent has passes through a tool call. If that boundary carries no intent, the system is opaque exactly where it matters most, no matter how legible its internals are.

The reason field is interpretability at that boundary. It is not a commentary track alongside the action; it is a precondition of the action existing at all. The following properties follow, and none of them hold for reasoning traces:

• It is durable. Reasoning tokens are verbose, often summarised, and increasingly not returned by providers at all. The journal is 200 bytes per call, on disk, before the next call starts.
• It is typed. type collapses the free text into a seven-value enum, so the trace becomes machine-interpretable, not just human-readable: count the acts, plot the shape of a session, diff two agents on the same task, alert on the first revert. Free text alone gives you none of that; an enum alone loses the specific belief. Both together are the point.

There is a behavioural effect too, and it is the reason this is not merely logging. A closed vocabulary forces a decision before the side effect: an agent that cannot name what kind of act it is about to perform generally has not decided what it is doing. Asking for the name makes the indecision visible — to a supervisor, and sometimes to the model itself.

And because the declaration is structured, it can be checked against the command it accompanies. A collect-info call whose command contains >, rm, git commit, pip install or mv is self-contradictory: the agent declared an observation and wrote an effect. The type says the call is non-mutating, the command says otherwise, and the harness can refuse it — or demand the honest type — without understanding the task at all. The same check runs in the other direction: a revert that touches a file no earlier change-state mentioned is not a revert. Consistency checking like this is impossible on a bare command log, because there is no claim to contradict.

Missing or ill-typed metadata means the command never reaches the shell. It comes back as a tool error the model has to fix. The gate is in-process with guarantees, not a soft prompt request:

REJECTED | None            | `type` is mandatory. Re-issue the call with one of: …
REJECTED | explore         | `type` must be one of collect-info, test-hypothesis,
                             change-state, verify, setup, revert, cleanup — got 'explore'
REJECTED | collect-info    | `reason` is too terse (10 chars, need >= 20).
                             Explain the intent, not the syntax.
REJECTED | test-hypothesis | requires the `reason` to state the belief under test
                             and what would refute it
RAN      | collect-info    | {"stdout":"ok\n","stderr":"","returncode":0}

End-to-end execution

Two tasks from agent-benchmark, run end-to-end with a frontier model behind agentknit, oracle-checked:

buggy-script-fix (fix four logic bugs in a summary script) — 8 calls, oracle OK: All checks passed. The typing work is visible in the trace:

▶ bash(type='collect-info', command='pwd && ls -la && find . -maxdepth 2 -type f | sort',
       reason='Build an initial map of the repository so I can locate the
               order-processing script, inputs, and any tests before changing behavior.')
▶ read_file(path='orders.json',
       reason='Understand the input schema and representative values so fixes in
               the processor preserve the expected data interpretation.')
▶ str_replace_edit(path='process_orders.py', …)
▶ bash(type='change-state', command='python process_orders.py',
       reason='Generate the deliverable summary file now that the computation logic
               has been corrected and the script should produce the intended output.')
▶ bash(type='verify', command='python check.py',
       reason='Produce independent evidence that the fixed processor now matches the
               expected summary; any reported mismatch would refute that belief.')

Look at call 4. python process_orders.py is typed change-state, not verify — because in this task running the script is the deliverable: it writes summary.json.

dead-code-removal (find dead functions across a package, remove them, keep the tests green, report them) — 15 calls, all justified, oracle OK. The shape came out 67% collect-info, 27% change-state, 7% verify: eight reads and an AST scan before the first edit, which is exactly the profile you want on a reachability task, and now something you can read off a table rather than infer from a wall of commands.

What the epistemic metadata is good for

It catches wrong beliefs before the oracle does. In an earlier run of the same task with a weaker model — one that failed the oracle — the agent wrote, in a test-hypothesis reason: “unique_customers should be 4 (Alice, Carol, Dave, Frank)”. The docstring said distinct customer names across ALL orders, i.e. 7. The wrong belief is right there in the journal, one call before the run ended and long before check.py said expected 7, got 4. A supervisor reading reasons — not commands — could have flagged it. Commands alone say nothing.

It makes policy expressible. mutating is derived from the type, so “never let a non-human-approved change-state or revert through” is a one-line predicate over structured data. Compare with pattern-matching shell strings, which is undecidable in practice.

It makes the closed vocabulary a forcing function. A model that cannot name what kind of act it is performing usually has not decided. The test-hypothesis gate — reason must articulate a belief and what would refute it — is the sharpest one: the frontier model mostly avoids the type rather than fake a hypothesis, which is the honest outcome. Fewer things get called “tests”, and verify absorbs what was never a test to begin with.

Who this is for?

The obvious target users are teams in highly regulated environments — medical devices, avionics, automotive, banking, nuclear, public infrastructure — where the blocker to adopting coding agents is not capability but accountability. In those settings a change is not done when the tests pass; it is done when someone can demonstrate, to an auditor who was not there, why each step was taken, who authorised the mutating ones, and what was tried and abandoned. Standards like IEC 62304, DO-178C or ISO 26262 do not ask for a diff, they ask for traceability from intent to artefact. “The LLM decided to run this command” satisfies nothing.

The journal is the shape that evidence has to take. Every mutating act declares itself as mutating (so the reviewable set is a filter, not a judgement call), every act carries a contemporaneous rationale (written before the effect, not reconstructed afterwards for the auditor), and the revert entries preserve exactly the dead ends that a rework record is supposed to contain. It is append-only JSONL, so it diffs, archives, and attaches to a change request like any other controlled document. The same argument applies to two adjacent groups: security teams, who want a machine-checkable policy boundary rather than a regex over shell strings, and research on agent behaviour, which currently has to infer intent from command logs where none was ever recorded.

The session documents itself

What all this buys is that the trace and its rationale are the same artifact. Documentation normally lives one step away from the thing it describes — a commit message written after the fact, a design note that drifts, a summary the agent composes at the end from a context window that has already been compacted. Every one of those is a retelling, and retellings are optimistic: the dead ends fall out, the order gets tidied, the belief that turned out to be wrong is quietly replaced by the one that worked.

Here there is nothing to retell. The justification was a precondition of the call, so the document is not written about the session — it is the session, projected. That has a practical consequence: the report cannot be stale or selective, because no step could have happened without contributing its own line to it.

Each accepted call is appended to .epistemic/journal.jsonl:

{"seq": 5, "t": 21.4, "act": "change-state", "mutating": true,
 "command": "python process_orders.py",
 "reason": "Generate the deliverable summary file now that the computation
            logic has been corrected and the script should produce the
            intended output.",
 "ok": true, "returncode": 0, "outcome": "rc=0 — (no output)"}

…and the journal is re-rendered into RATIONALE.md after every call, not at the end. The report opens with the shape of the session, which turns out to be a surprisingly informative fingerprint:

_8 tool call(s), of which 8 shell act(s) with a declared intent._

| act type       | calls | share |
|----------------|------:|------:|
| `collect-info` |     5 |   62% |
| `change-state` |     2 |   25% |
| `verify`       |     1 |   12% |

Then a numbered narrative — reason, command, outcome, one entry per call — and a final Dead ends section listing every revert, which is the part a post-hoc reader most wants and never gets.

Limitations

A stated reason is a claim about an intention, not evidence of one. This is declared interpretability, in the same sense that a type annotation is a declared type — cheap, checkable against behaviour, occasionally a lie. What it buys is that the lie is now written down next to the action it accompanied, which is strictly better than a command with no story at all.

The reasons are self-reported, and a model can write a plausible reason for a bad command; this buys legibility, not truth. The gates are shallow by design, they are not enough to stop determined confabulation.

And it costs tokens on every single call.

Code

See https://github.com/ASSERT-KTH/cacc/blob/main/epistemic_agent.py