Wiki · The Colony & the Method
Benchmark Integrity Report
How to read this page
Three ways to read this page. Precise is the document itself, exactly as it is written in the repository. Plain and Clear were written for this website to help you meet that document — they are about it. They are not it, and they are not evidence.
Eighty-four pages about the colony. Each agent is an Elixir process holding a generative model and doing inference, attached to a body that logs into a Minecraft world as an ordinary player. Around that sit the broadcast suite that films them and the runbooks that keep the whole thing running. There are typed specifications for each organ of the model, plus the world and genome specs. There are also the adversarial review personas used to attack a proposed change before it ships.
It is for the reader curious how a running system is put together and how it is held to account. The accountability half is the more distinctive. There is a lab protocol governing evidence and attribution, and a claim fence that restricts the vocabulary a claim is allowed to use. There is a public gate log. And there is a standing invitation to reproduce any verdict from the commit and the seed named in its receipt.
Start with the public read, then the lab protocol, then the falsification invitation. If you want the mathematics rather than the operations, go straight to the typed organ specs.
What it is not: a description of a mind, and not all one kind of document. A large part of this corpus is design and planning — specs marked as proposed rather than applied, organs designed but not built, plans that were later superseded — and each page states which it is. A specification is not a running system, and these pages are careful about the difference; the reader should be too. Eight documents were withheld from publication because they describe private infrastructure.
Your browser cannot switch reading levels, so the document itself is shown.
Precise — the source document
This is the document. Rendered from the repository at the commit above, with nothing rewritten for the web. A gate re-renders it on every deploy and fails the build if a single byte differs.
Purpose
Establish that the environment is a sound benchmark: it is not trivially solvable or impossibly hard, simple baselines do not collapse it, no shortcut/oracle exists, conservation/boundedness hold, and seeded golden traces guard against silent drift.
Method
- Survival difficulty across the reference seed batch for all baselines.
- Anti-shortcut: confirm no reward/oracle channel; confirm random can't reliably win; confirm scripted skill helps.
- Conservation/boundedness checks.
- Golden regression artifact + CI diff.
Artifacts used
scripts/evidence.exs,SP.GoldenTest,SP.SoakTest,SP.EvalTest,config/golden/reference_episode.json.
Result summary
Difficulty band (reference batch 101–112, 400-tick horizon, in-episode development):
Random mean=265.00 horizon=6/12
Homeostatic mean=289.58 horizon=8/12
ProbeFirst mean=286.83 horizon=8/12
MorphologySeeking mean=258.50 horizon=7/12
Infrastructure mean=262.25 horizon=7/12
Survival is neither trivial nor impossible: no baseline reaches the horizon on all seeds, and sense-using policies (Homeostatic/ProbeFirst) lead. With a fixed developed body, the sense-using agent beats blind random by +24.3% (see sensory_ablation_report). Simple baselines do not collapse the benchmark.
Anti-shortcut / no oracle:
- No
reward/score/return/fitnessfield anywhere on the learner path or in eval metrics (Invariant #15;SP.EvalTest,SP.InvariantsTest). - The learner cannot read world state, coordinates, materials, or hidden layers (see interface_leakage_audit).
- Morphology/sense gating means there is no action that bypasses the capability ladder.
Conservation / boundedness:
transport material-mass delta: 0.0
after 500 microsteps, region maxima: %{nut: 4.74 (cap 5), tox: 0.01 (cap 3)}
Field diffusion conserves mass exactly (property test); all fields stay within documented caps over 500–2000 microstep runs.
Regression guard:
The golden artifact pins a seeded episode (structural metrics exact, floats within
1e-6); SP.GoldenTest re-derives it and CI diffs config/golden/ to catch
unintended dynamics/interface changes.
Pass/Fail
PASS.
Residual risks
- Difficulty is seed-dependent; difficulty claims should use batches (done here).
- The golden band is a single reference episode; broaden to a multi-seed band if stricter regression coverage is desired.
sha256 4f586e3becb2abe6 — of the original file, so what was ingested stays checkable.
Plain — written for this website, not the source document
This is a short report asking whether the test world is a fair benchmark: not so easy that anything wins, not so hard that nothing does, with no hidden shortcut and no score to exploit.
It follows the same shape as its sibling reports. A purpose, a method, the artifacts it used, a summary of results, a single pass or fail, and the risks that remain.
The results say survival is neither trivial nor impossible, because no simple strategy reaches the end on every seed, and the strategies that use their senses do better than the ones that ignore them. A random strategy cannot reliably win. Material is conserved exactly in transport, quantities stay inside their stated caps over long runs, and a pinned reference episode guards against silent drift.
The verdict is a pass, and the residual risks are printed rather than omitted.
Plain · written 2026-08-01 by claude-opus-5 · not yet checked by a person · about the document whose sha256 is 4f586e3becb2abe6
Clear — written for this website, not the source document
This report asks whether the environment is sound enough to be used as a benchmark at all. Its purpose section lists what that means here: not trivially winnable and not impossible, not collapsed by simple strategies, with no shortcut or oracle available, with conservation and boundedness holding, and with a pinned trace to catch silent drift.
The method is a batch of runs across a reference set of seeds for every simple strategy, plus explicit checks that no score-like channel exists, that acting at random cannot reliably succeed, and that a scripted skill does help. Conservation and boundedness are checked separately, and a stored reference episode is re-derived and compared automatically.
The results are given as a table of mean survival per strategy with the number of seeds each reached the horizon on. The reading offered is that no strategy reaches the horizon on all seeds, and the strategies that use their senses lead, so the benchmark is neither trivial nor collapsed. A separate ablation is cited for the advantage of using senses with the body held fixed.
The anti-shortcut section states that no score, reward, return or fitness field exists anywhere on the learner's path or in the evaluation metrics. The learner cannot read world state, coordinates, materials or hidden layers, and no action bypasses the capability ladder. Conservation is reported as an exact zero change in transported mass, with field values remaining inside documented caps over long runs.
The verdict is a pass. The residual risks say plainly that difficulty depends on the seed, which is why batches were used, and that the pinned episode is a single reference rather than a band, which could be broadened if stricter coverage were wanted.
Clear · written 2026-08-01 by claude-opus-5 · not yet checked by a person · about the document whose sha256 is 4f586e3becb2abe6