Interface Leakage Audit

The Colony & the Method · docs/reports/interface_leakage_audit.md @ 44baf03d5041 (gen2-runtime) — opens the published snapshot ac338733bbba

Three ways to read this page. Precise is the document itself, exactly as it is written in the repository. Plain and Clear were written for this website to help you meet that document — they are about it. They are not it, and they are not evidence.

Eighty-four pages about the colony. Each agent is an Elixir process holding a generative model and doing inference, attached to a body that logs into a Minecraft world as an ordinary player. Around that sit the broadcast suite that films them and the runbooks that keep the whole thing running. There are typed specifications for each organ of the model, plus the world and genome specs. There are also the adversarial review personas used to attack a proposed change before it ships.

It is for the reader curious how a running system is put together and how it is held to account. The accountability half is the more distinctive. There is a lab protocol governing evidence and attribution, and a claim fence that restricts the vocabulary a claim is allowed to use. There is a public gate log. And there is a standing invitation to reproduce any verdict from the commit and the seed named in its receipt.

Start with the public read, then the lab protocol, then the falsification invitation. If you want the mathematics rather than the operations, go straight to the typed organ specs.

What it is not: a description of a mind, and not all one kind of document. A large part of this corpus is design and planning — specs marked as proposed rather than applied, organs designed but not built, plans that were later superseded — and each page states which it is. A specification is not a running system, and these pages are careful about the difference; the reader should be too. Eight documents were withheld from publication because they describe private infrastructure.

PlainClearPrecise

Your browser cannot switch reading levels, so the document itself is shown.

This is the document. Rendered from the repository at the commit above, with nothing rewritten for the web. A gate re-renders it on every deploy and fails the build if a single byte differs.

Purpose

Verify Validation Invariants #1, #2, #3, #15: the learner-facing interface leaks no world state, no semantic action labels, no privileged metadata, and no reward signal.

Method

Structural audit of encoded observations (SP.Interface.Audit.audit_observation/1): must be integer_channel → finite_number, channels in 0..28.
Deep token scan (Audit.scan/1) for forbidden semantic tokens.
Engineering sensor-payload check (Audit.sensor_payload_ok?/1): no material IDs, no coordinates.
Blind leakage-probe baseline audits every observation in a live episode and attempts malformed/coordinate-smuggling actions.
In-loop debug? leak trap raises if any observation is ever non-clean.

Artifacts used

SP.InterfaceTest, SP.LeakageProbeTest, SP.Body.SensorTest, SP.InvariantsTest (#1, #2, #3, #5, #15).
scripts/evidence.exs leakage section.

Result summary

From scripts/evidence.exs (seed 7, all six baselines, debug?: true):

all baselines ran with debug? leak-trap ON, none raised: true
leakage-probe baseline detected leaks:    0 (audited 200 observations)
leakage-probe malformed actions rejected: 600
full encoded obs channels=29/29 structurally_clean=true token_scan_clean=true

Encoded observations are pure int → float; any atom/string/struct/tuple or out-of-range key would be flagged (negative tests in SP.LeakageProbeTest).
Action channels are a per-seed permutation; the same feature maps to many different channel ids across seeds (SP.InterfaceTest), so semantics cannot be hard-coded.
Absolute coordinates in action params are rejected (:absolute_coordinate_forbidden).
Sensor payloads carry no material IDs / coordinates (Audit.sensor_payload_ok?).
No :reward/:score/:return/:fitness key exists on any learner-facing path or in eval metrics (#15).

Falsifiable evidence log (third-party re-derivation)

Beyond the in-process tests, every recorded run emits a durable evidence log (runs/<run>.jsonl + .meta.json) capturing, per tick, the full world snapshot, the exact afferent observation + sensor signals, and the efferent actions. mix sp.verify <log> re-derives the no-leak verdict from the raw bytes alone (rebuilding the channel map from the seed; recomputing audit_observation, scan, per-channel organ provenance, and encode_observation equivalence). It is falsifiable, not self-attested:

$ mix sp.verify runs/seed314-morphology_seeking.jsonl
VERIFIED: 46 frames, 0 blanket violations.            # exit 0

# after tampering with one observation value:
VIOLATION: 1/46 frames leaked. First: {"frame":20,"reasons":["encode_equivalence"],"tick":21}   # exit 1

See docs/observability/evidence_log.md and the overlooker UI (docs/ui/overlooker.md). Negative tests in SP.Sim.BlanketTest inject each leak class and confirm the verifier rejects it.

Pass/Fail

PASS.

Residual risks

Scripted validation baselines intentionally use the debug Lens/reveal_*; they are not learners and must not be deployed as policies (see security_leakage_boundary.md).
Value-distribution inference across many observations is the learner's intended problem, not a leak; per-seed remap blocks cross-scenario memorisation.

sha256 caa3d3bb18e3a886 — of the original file, so what was ingested stays checkable.

Written for this website — not the document. This is a plain-language retelling, written to help you meet the document. It is not the source, and it is not evidence. It has not yet been checked by a person. Read the document itself → (or choose Precise in the reading-level control above)

This report asks a narrow question: can the learner see anything it is not supposed to see? The interface it is given should carry no world state, no meaningful action labels, no privileged extra information, and no score to chase.

The method mixes structural checks with live probing. Observations must be plain numbers keyed by number, within a fixed range. A scan looks for forbidden words. A deliberate probe runs a whole episode auditing every observation and trying to smuggle coordinates into actions.

The results report no leaks found across the audited observations, every malformed action rejected, and every channel structurally clean. Action channels are shuffled differently for each seed, so their meaning cannot be memorised.

The part worth knowing is that this is not self-attested. Every recorded run writes a durable log, and a separate command re-derives the verdict from those raw bytes alone.

Plain · written 2026-08-01 by claude-opus-5 · not yet checked by a person · about the document whose sha256 is caa3d3bb18e3a886

Written for this website — not the document. This is a clearer retelling, written to help you meet the document. It is not the source, and it is not evidence. It has not yet been checked by a person. Read the document itself → (or choose Precise in the reading-level control above)

This is a focused audit report. Its question is whether the interface presented to a learner leaks anything: world state, semantic action labels, privileged metadata, or a reward signal.

The method has five parts. A structural audit requires every encoded observation to be a plain mapping from a channel number to a finite number, with channels inside a fixed range. A deep scan looks for forbidden meaningful tokens anywhere in the structure. A sensor-payload check requires that no material identities and no coordinates appear. A blind probe strategy audits every observation across a live episode and also attempts malformed actions and coordinate smuggling. And a trap can be switched on in the loop that raises immediately if any observation is ever unclean.

The results are given as raw counts. Every strategy ran with the trap on and none raised, and the probe detected no leaks across the observations it audited. Every malformed action it attempted was rejected, and all channels were structurally clean and clean under the token scan. Supporting points follow. Any wrong type or out-of-range key would be flagged, and negative tests prove that. Action channels are permuted per seed, so the same feature maps to different identifiers in different runs and semantics cannot be hard-coded. Absolute coordinates in action parameters are refused by name, and no score-like key exists anywhere on the learner-facing path.

The most important section is the one about third-party re-derivation. Beyond the in-process tests, every recorded run emits a durable log capturing, per tick, the world snapshot, the exact observation and sensor signals, and the actions that came back. A separate command re-derives the no-leak verdict from those raw bytes alone, rebuilding the channel mapping from the seed and recomputing every check. The report shows both a clean verification and, after deliberately tampering with one observation value, the resulting violation and its non-zero exit. It states that this makes the claim falsifiable rather than self-attested, and that negative tests inject each class of leak to confirm the verifier rejects them.

The verdict is a pass, followed by residual risks. Some scripted validation strategies deliberately use a debug view and are explicitly not learners, and a separate document covers that boundary. And inferring value distributions across many observations is named as the learner's intended problem rather than a leak.

Clear · written 2026-08-01 by claude-opus-5 · not yet checked by a person · about the document whose sha256 is caa3d3bb18e3a886

Interface Leakage Audit

How to read this page

Precise — the source document