Reproducibility Guide

The Colony & the Method · docs/reproducibility.md @ 44baf03d5041 (gen2-runtime) — opens the published snapshot ac338733bbba

Three ways to read this page. Precise is the document itself, exactly as it is written in the repository. Plain and Clear were written for this website to help you meet that document — they are about it. They are not it, and they are not evidence.

Eighty-four pages about the colony. Each agent is an Elixir process holding a generative model and doing inference, attached to a body that logs into a Minecraft world as an ordinary player. Around that sit the broadcast suite that films them and the runbooks that keep the whole thing running. There are typed specifications for each organ of the model, plus the world and genome specs. There are also the adversarial review personas used to attack a proposed change before it ships.

It is for the reader curious how a running system is put together and how it is held to account. The accountability half is the more distinctive. There is a lab protocol governing evidence and attribution, and a claim fence that restricts the vocabulary a claim is allowed to use. There is a public gate log. And there is a standing invitation to reproduce any verdict from the commit and the seed named in its receipt.

Start with the public read, then the lab protocol, then the falsification invitation. If you want the mathematics rather than the operations, go straight to the typed organ specs.

What it is not: a description of a mind, and not all one kind of document. A large part of this corpus is design and planning — specs marked as proposed rather than applied, organs designed but not built, plans that were later superseded — and each page states which it is. A specification is not a running system, and these pages are careful about the difference; the reader should be too. Eight documents were withheld from publication because they describe private infrastructure.

PlainClearPrecise

Your browser cannot switch reading levels, so the document itself is shown.

This is the document. Rendered from the repository at the commit above, with nothing rewritten for the web. A gate re-renders it on every deploy and fails the build if a single byte differs.

Reproducibility is a first-class invariant (#13): identical inputs reproduce identical traces, bit-for-bit, offline.

Sources of determinism

Explicit PRNG — SP.Determinism (SplitMix64) is threaded as an immutable value through every stochastic step. We never use :rand (process-local, version-dependent).
Splitting, not sharing — regions and children get split sub-generators, so adding a region/probe never perturbs another stream.
No wall-clock — time is logical (ticks/microsteps). No System.* time, no Process.sleep.
No external deps — nothing to version-drift; mix test is hermetic.
Seed-derived interface — the opaque channel map is a pure function of the scenario seed (SP.Interface.channel_map/2).

What a seed determines

(seed, regions, w, h, micro_per_decision, dev_interval, agent, scramble) fully determines: the world generation, all dynamics, the body's development (given the genome), the channel map, and therefore the entire episode trace.

Reproducing a run

# from a captured provenance block:
SP.Sim.new(seed: 314, agent: SP.Baselines.MorphologySeeking,
           max_ticks: 250, micro_per_decision: 3, dev_interval: 5,
           world_opts: [regions: 2, w: 6, h: 6])
|> SP.Sim.run()

Verification

SP.SimTest — same seed ⇒ identical points/1; different seed ⇒ differs.
SP.World.DynamicsTest — same seed ⇒ identical world after N steps.
SP.GoldenTest — a stored episode reproduces exactly (ints) / within 1e-6 (floats); CI diffs the regenerated artifact.
mix run scripts/evidence.exs prints the reproducibility checks live.

Declared tolerance

Integer/structural metrics: exact.
Float metrics: reproduced exactly within a process; the golden test allows 1e-6 to absorb only formatting/serialization rounding, not dynamics drift.

Maintenance constraint: never iterate atom-keyed maps for ordered/float work

Erlang map iteration order for atom keys is not stable across BEAM instances (it depends on atom-table indices, which differ with how many modules are loaded — e.g. mix run vs the full mix test suite). Two pitfalls follow:

Map.keys on an atom-keyed map returns a VM-dependent order. Material classes are therefore defined as a fixed literal list (@class_order), with a compile-time guard, so Determinism.choice/2 over them generates identical worlds everywhere.
Float reductions over atom-keyed maps (Material.weighted/2, mass/1) iterate in canonical class order, not map order. Tiny float-summation differences would otherwise be amplified into macroscopic divergence by the agent's threshold/argmax decisions.

Integer-keyed maps (fields, region cells, body parts) are atom-table-independent and safe. The cross-VM golden regression test (SP.GoldenTest) is what catches violations of this constraint — it compares a mix run-generated artifact against a mix test-computed run. When adding code, never let trajectory-affecting output depend on atom-keyed map iteration order.

Caveats

Cross-Erlang/Elixer-version float bit-identity is not guaranteed by the BEAM in general; our float operations are simple arithmetic and reproduce within tolerance across the supported matrix (OTP 27, Elixir 1.17/1.18). Integer/PRNG state is exact everywhere.

sha256 18d26fabdc10297c — of the original file, so what was ingested stays checkable.

Written for this website — not the document. This is a plain-language retelling, written to help you meet the document. It is not the source, and it is not evidence. It has not yet been checked by a person. Read the document itself → (or choose Precise in the reading-level control above)

This guide explains how the same inputs are made to produce the same run, and where that promise stops.

Five sources of determinism are listed. One generator, threaded through the program by hand rather than kept in a hidden place. Sub-generators split off for new parts, so adding one never disturbs another's sequence. No clock: time is counted in ticks. No outside dependencies to drift. And the mapping of the interface's channels is seed-derived, a pure function of the seed and nothing else.

The guide then says exactly which settings a run needs in order to be repeated, shows the call that repeats one, and names the tests that check it.

The most useful section is a maintenance rule learned the hard way. Iterating over a map keyed by names is not stable between runs, and tiny differences in adding up decimals would be amplified into visibly different behaviour, so anything affecting a trajectory must not depend on that order.

Plain · written 2026-08-01 by claude-opus-5 · not yet checked by a person · about the document whose sha256 is 18d26fabdc10297c

Written for this website — not the document. This is a clearer retelling, written to help you meet the document. It is not the source, and it is not evidence. It has not yet been checked by a person. Read the document itself → (or choose Precise in the reading-level control above)

A guide rather than a report. It explains why runs repeat, how to repeat one, and where the promise ends.

It opens by naming reproducibility as a first-class invariant: identical inputs reproduce identical traces, offline.

Five sources of determinism follow. All randomness comes from one generator threaded through every step as a plain value, rather than the language's built-in source, which is described as process-local and version-dependent. New regions and children receive split sub-generators rather than sharing one, so adding something never perturbs another stream. Time is logical, counted in ticks, with no clock reading and no sleeping. There are no external dependencies to drift. And the opaque mapping between channels and features is seed-derived, a pure function of the seed.

A short section states exactly which settings, taken together, determine an entire episode, and a code block shows how to reconstruct a run from a captured provenance record. Named tests follow: identical seeds produce identical traces, identical seeds produce identical worlds after many steps, and a stored episode reproduces exactly for whole numbers and within a stated tolerance for decimals.

The declared tolerance is written out rather than assumed, and the guide says what the tolerance is for, which is rounding in serialisation rather than any drift in the dynamics.

The longest and most useful section is a maintenance constraint, and it reads as something learned from a real failure. Iterating a map keyed by names is not stable between separate runs of the virtual machine, because it depends on an internal table whose contents vary with how much has been loaded. Two consequences are drawn. A list of material classes is therefore fixed in a literal order with a compile-time guard, so the same worlds are generated everywhere. And decimal reductions over such maps are made to iterate in that one fixed order, because tiny differences in summation would otherwise be amplified into visible divergence by the agent's own threshold and choice decisions. Maps keyed by numbers are said to be safe. The guide names the regression test that catches violations, because it compares an artifact produced one way against a run computed another, and it ends the section with an instruction for anyone adding code.

A final caveat limits the claim: decimal bit-identity across runtime versions is not something the platform offers in general, so reproduction is claimed within tolerance across a supported matrix, while integer and generator state are exact everywhere.

Clear · written 2026-08-01 by claude-opus-5 · not yet checked by a person · about the document whose sha256 is 18d26fabdc10297c

Reproducibility Guide

How to read this page

Precise — the source document