Wiki · Evidence & Verdicts
QA & Validation Methodology
How to read this page
Three ways to read this page. Precise is the document itself, exactly as it is written in the repository. Plain and Clear were written for this website to help you meet that document — they are about it. They are not it, and they are not evidence.
Eighty-seven dated pages: receipts, pre-registrations, handoffs, validation records and review verdicts. A receipt is written at the moment a piece of work was checked. It names what was claimed, the commit and the seed, what was actually run, and the outcome in one of a small set of controlled words. Then it names what the work did not achieve. That last part is what makes it a receipt rather than an announcement. A pre-registration is the same discipline run in advance: the conditions that would count as a pass and the conditions that would falsify the claim are written down before the run, so neither can be adjusted once the numbers arrive.
That is why so many small dated stubs are an audit trail rather than noise. No one of them is meant to be a good read. The value is in the sequence and in the dates, because you can watch a prediction be registered, then the run happen, then the verdict land — sometimes against the prediction. Pages here record a falsified result, a rejected fix, a retracted overclaim, and a green receipt that turned out not to be reproducible from the commit that carried it. A record that carried only successes would be worth a good deal less than this one.
A gentle way in is to read a pre-registration first, so the shape becomes familiar, then a result page, then one of the corrections. This section sits off the main navigation on purpose: it is the record you check the rest of the site against, not the place to begin.
What it is not: documentation, and not a summary. Nothing here has been tidied in hindsight. Every entry reads as of its date, a later entry may overturn an earlier one, and the presence of a page is not a claim that its result stood.
Your browser cannot switch reading levels, so the document itself is shown.
Precise — the source document
This is the document. Rendered from the repository at the commit above, with nothing rewritten for the web. A gate re-renders it on every deploy and fails the build if a single byte differs.
Test taxonomy (mix test, 100 tests, 0 failures)
| suite | file | covers |
|---|---|---|
| Determinism | determinism_test.exs |
PRNG reproducibility, ranges, split independence |
| Core | core/signal_test.exs, core/directive_test.exs |
schema validation, directive shapes |
| World fields | world/field_test.exs |
diffusion mass conservation (property), neighbours |
| World dynamics | world/dynamics_test.exs |
boundedness over 500 steps, determinism, seam gating |
| World graph | world_test.exs |
adjacency, seam-open invariants, transport conservation |
| Body | body_test.exs |
action gating, graph validity, metabolism/viability |
| Genome | genome_test.exs |
mutate/recombine/repair always valid (property) |
| Development | body/development_test.exs |
no impossible graphs (property), stage transitions |
| Sensors | body/sensor_test.exs |
organ-gating, hidden-layer invisibility, payload cleanliness |
| Interface | interface_test.exs |
opacity, per-seed remap, decode, leakage (property) |
| Sim | sim_test.exs |
episode determinism, reset, no world mutation from agent, fuzz |
| Eval | eval_test.exs |
senses-matter (behavioral), layer visibility, ablation suite |
| Invariants | invariants_test.exs |
the 15 mandatory invariants as a checklist |
| Leakage-probe | leakage_probe_test.exs |
leakage, malformed signals/directives, channel fuzz |
| Soak | soak_test.exs |
2000-step boundedness, repeated expansion, bounded trace |
| Golden | golden_test.exs |
benchmark regression against a stored artifact |
Property testing without external deps
SP.Prop.forall/4 (in test/support) samples N seeded cases and reports the
failing sample with its iteration index — reproducible, offline, no StreamData.
No-flake discipline
Time is purely logical (decision ticks, microsteps). There is no
Process.sleep anywhere in the suite. Determinism tests assert byte-identical
traces, so any nondeterminism surfaces immediately rather than flaking.
The 15 mandatory invariants → tests
SP.InvariantsTest maps one test per invariant (#1–#15) for at-a-glance audit;
several are also exercised in depth by the dedicated suites above.
Leakage-probe / fuzz coverage (spec §D)
- malformed signal payloads → rejected by
SP.Core.Signalschema; - malformed/garbage directives → tolerated by
SP.Sim, counted asdecoded_failures, never executed; - out-of-range/negative channels →
{:error, …}, never raise (channel fuzz property); - absolute-coordinate smuggling →
:absolute_coordinate_forbidden; - debug reveal separated from the production encode path;
- the leakage-probe baseline runs an episode auditing every observation and asserts
leaks == 0.
Soak / stability (spec §F)
2000-microstep runs assert every field stays within its documented cap; repeated
seam expansion (8×) keeps every region's law valid and the graph connected; a
1500-tick episode with keep_points: false keeps the trace bounded.
Regression bands (spec §H)
The golden artifact (config/golden/reference_episode.json) pins a seeded
episode's structural metrics exactly and its float metrics within 1e-6.
Regenerate intentionally with mix run scripts/gen_golden.exs; CI diffs it.
Running QA
mix test # everything
mix compile --warnings-as-errors --force
mix format --check-formatted
mix run scripts/evidence.exs # validation evidence numbers
sha256 7515cb729343551a — of the original file, so what was ingested stays checkable.
Plain — written for this website, not the source document
This page describes how a codebase is tested. It is written for someone trying to judge whether the testing is any good, not just count it. It lists each suite and what it covers, then the parts that matter more than the list. There is a small property tester written in the project rather than pulled in from outside, and a rule that time in tests is logical so that nothing can flake. There is a checklist mapping one test to each mandatory invariant, fuzzing aimed at leakage, long soak runs, and a stored reference episode that pins regressions.
Plain · written 2026-08-01 by claude-opus-5 · not yet checked by a person · about the document whose sha256 is 7515cb729343551a
Clear — written for this website, not the source document
A methodology page rather than a result. It says how this codebase is checked, in enough detail that a reader can decide whether to believe the checking.
It opens with a table of suites, one row per file, saying what each covers. There is determinism, the core message schemas, the world's fields and dynamics, the graph, the body, the genome, development and sensors. There is the interface between agent and world, the simulation itself, evaluation, the invariant checklist, a leakage probe, a soak run, and a comparison against a stored artifact. Several rows are property tests rather than worked examples, including mass conservation, boundedness, a genome that stays valid however it is mutated, and development that cannot produce an impossible body.
Then come the parts that are arguments rather than lists. Property testing uses a small helper written in the project rather than an outside library, and it reports the failing sample with its index so a failure can be reproduced. Time is purely logical, with no sleeping anywhere in the suite, and determinism tests assert byte-identical traces, so anything nondeterministic surfaces as a failure instead of an occasional flake. A dedicated suite maps one test to each mandatory invariant so the audit can be done at a glance.
The fuzz and leakage section lists what gets thrown at the system and what each malformed input should produce: rejected outright, tolerated and counted but never executed, or refused by name. A baseline episode audits every observation and asserts that nothing leaked. Soak runs assert that bounds hold over long runs and repeated expansion. A stored reference episode pins the structural numbers exactly and the floating-point ones within a tolerance, and is regenerated only on purpose. The page ends with the commands that run all of it.
Clear · written 2026-08-01 by claude-opus-5 · not yet checked by a person · about the document whose sha256 is 7515cb729343551a