MotorCortexTest's "nondeterministic" full-suite flake was a starved test, not shared state — receipt, 2026-07-18

Track: science-track test infrastructure. Touched no lib/sp/**. Changed no FE math. Weakened and deleted no assertion. Set no science gate. Made no life/awareness claim. Gate: motor-cortex-flake-fixed — across 5 consecutive full mix test runs, ZERO SP.Brain.MotorCortexTest failures and ZERO ExUnit.TimeoutError attributable to that module. Verdict: PASS · evidence class A (independently reproduced: the defect was measured red under the pre-fix tree with a per-test timing profile, then green across 5 consecutive full-suite runs).

SCOPE FENCE — read this before quoting the gate. This gate is about MotorCortexTest and nothing else. mix test is NOT yet stably green. The same verification runs surfaced two OTHER, independent wall-clock flakes (SP.Brain.NoveltyTest, SP.Brain.BridgeTest — §7). Neither was introduced here, and neither is fixed here. Do not read this PASS as "the suite is clean."

1. The reported symptom

test/sp/brain/motor_cortex_test.exs failed under the full mix test (553 tests, max_cases: 32) but passed in isolation — and failed under a different test name each run:

• run A: P2: learned motor A/B persist across MC.save/load (muscle memory survives death) (:161)
• run B: P2: the motor config factors learn A_motor + B_motor, and the config posterior becomes informative (:137)

A failure that moves is normally the signature of cross-test interference through shared mutable state. Here it was not. Chasing the moving name was the trap.

2. What it actually was

Both flaky tests are simply the two slowest tests in the module, sitting just under ExUnit's 60s default timeout on an idle box. Measured with mix test <file> --trace (which sets timeout: :infinity, so every test runs to completion and reports its true cost):

The two that flake are the two heavy ones, at 1.40x and 1.48x headroom against the 60000 ms default. Nothing else in the module is within an order of magnitude (next is 2.4 s).

Both drive motor_experience/0 — 106 live MC.step calls through the full depth-5 Plan search over a 17-factor motor_primary model (12 default + 5 proprioceptive). That cost is irreducible and real; it is what the test is for.

Why the full suite tips them over: mix test runs with max_cases = System.schedulers_online() * 2 — 32 on this 16-core box (measured: {16, 16}). That is Elixir's default 2x oversubscription. CPU-bound work therefore runs substantially slower under the full suite than in isolation, and ~1.4x headroom does not survive it. Whichever of the two heavy tests catches the worst scheduler contention on a given run is the one that crosses 60s — hence a different name each run, and never in isolation.

The failure was always ExUnit.TimeoutError, with the stack parked mid-computation in pure planner arithmetic:

** (ExUnit.TimeoutError) test timed out after 60000ms
   (stratified_palimpsest) lib/sp/brain/math.ex:68: anonymous fn/2 in SP.Brain.Math.matvec/2
   (stratified_palimpsest) lib/sp/brain/plan.ex:153: anonymous fn/4 in SP.Brain.Plan.advance/3
   (stratified_palimpsest) lib/sp/brain/plan.ex:120: SP.Brain.Plan.continuation/5

That is a test that ran out of clock, not a test that observed a wrong value.

3. The shared-state hypotheses, and why each is falsified

The plausible suspects were checked directly and all come back negative:

The whole MC.step path is pure and struct-threaded. There was no shared mutable state to isolate.

4. The fix

Per-test @tag timeout: @heavy_timeout (300_000 ms) on only the two heavy P2 tests, plus a comment block in the test file recording this diagnosis so it is not relearned.

@heavy_timeout 300_000

@tag timeout: @heavy_timeout
test "P2: the motor config factors learn A_motor + B_motor, ..." do

Deliberate properties of this fix:

• No assertion weakened, removed, or loosened. Every threshold (a_moved > 0.05, b_moved > 0.05, posterior_peak > 0.1, the refute learned.model.pe == fresh.model.pe habit-prior check, the < 1.0e-12 persist-byte-identity checks) is untouched. motor_experience/0 still runs its full 106 steps.
• No FE math changed. lib/sp/** is untouched.
• The budget is raised, not removed. :infinity was rejected: a genuine hang, or an FE-math regression that blows up planner cost, must still fail the suite. 300s is ~7x the measured idle cost.
• Per-test, not @moduletag. The other six tests keep the strict 60s default and still fail fast.
• async: true retained. Serializing the module with async: false was rejected — it would have hidden the real cause behind a wall-clock penalty on the whole suite while fixing nothing, and there was no isolation defect to serialize away.

5. Verification — and what it honestly shows

5 consecutive full mix test runs, post-fix:

The gate this receipt claims — zero MotorCortexTest failures across 5 consecutive full-suite runs — is met. 5/5. The flake this work targeted is gone.

The suite as a whole is NOT green: 3/5 clean. Runs 4 and 5 failed on two different tests. Those are covered in §7 and are explicitly NOT claimed fixed.

Log-contamination note on run 3: /tmp/v3.log contains two interleaved mix test summary lines (one 0 failures, one 2 failures) and two writers' output. Cause: an earlier foreground verification loop was killed at a 10-minute tool cap mid-run-3, and its orphaned mix test child kept writing to that path while the replacement background loop truncated and rewrote it. The background run 3 that the table reports exited 0. Run 3 is counted only for its MotorCortex result (0 failures, which both writers agree on); it is not counted toward the suite-green tally, which is why that tally is stated as 3/5 and not 4/5.

6. Note on the co-occurring GateRegistryIntegrityTest failure

The pre-fix full-suite run showed 2 failures: this timeout and GateRegistryIntegrityTest — row 124 (unnamed) missing key schema_version. That is a separate, already-flagged issue (the 18 legacy-schema rows in evidence/gates.ndjson) and was not touched here. It landed independently in a18b822 ("gates: migrate the 18 legacy-schema ledger rows to gate_row.v1 — re-encoding, not re-judgement") while this work was in flight; row 124 now carries schema_version: 1, and the verification runs above show the suite at 0 failures total.

7. OPEN — two more wall-clock flakes, same disease, NOT fixed here

The verification runs proved the fix and simultaneously exposed that MotorCortexTest was not the only instance. Both of these are independent of this change and both remain open:

(a) SP.Brain.NoveltyTest — "prospective EXPLORATION ..." (novelty_test.exs:59). Failed run 5 with ExUnit.TimeoutError after 240000ms, stack in SP.Brain.Plan.advance/3 — the identical signature. This test already carries @tag timeout: 240_000 (line 58), added 2026-06-23 in 903f885, a month before this work. So the same class of fix was already applied here by a previous author and the test still times out under contention. Raising it again would be the second increment on the same test — a treadmill, not a cure.

(b) SP.Brain.BridgeTest — "5 senses in ⇄ 5 actions out, in strict lockstep" (bridge_test.exs:30). Failed run 4 on assert_receive {:bridge_done, 5}, 5_000 — "no matching message after 5000ms. The process mailbox is empty." Not a timeout tag: an explicit 5s assert_receive budget covering a real Port spawn of viewer/mock_body.js (node) plus 5 lockstep round trips. 5s is thin for a process spawn on a saturated box.

The systemic read. Three tests in test/ now carry hand-tuned wall-clock band-aids (motor_cortex_test.exs, novelty_test.exs, soak_test.exs), and the suite still flakes. The common factor is not any one test: mix test runs at max_cases = schedulers_online * 2 = 32 on a 16-core box, i.e. 2x CPU oversubscription, while several tests do multi-minute depth-5 planning. Every per-test timeout bump treats a symptom of that. A candidate systemic cure — capping max_cases nearer the true core count for this suite — has NOT been tested and is NOT claimed. It is recorded here as the next thing to measure, not as a conclusion.

Honest uncertainty about this change's interaction. It cannot be ruled out that the fix here made (a) marginally more likely: pre-fix, the two heavy MotorCortex tests were killed at 60s, releasing two scheduler slots; post-fix they run to completion (~43–90s), holding CPU longer. That is a real mechanism and no controlled pre/post comparison was run to exclude it. What is established: the fix did not introduce these tests' fragility (NoveltyTest has needed a 240s tag since June), and the correct response is not to re-shorten a budget that was the honest cost of the work.

8. Why this mattered

mix test is the integration signal for FE-invariant work — it is what guards decider_byte_identity, action_clone_invariance, and novelty, which are the math fence. A suite that failed under a different name every run trained everyone to read full-suite red as noise. That is the same failure mode CLAUDE.md names for the always-drifting infra rows: an alarm that is always on is not an alarm. The suite is now honest-green, so the next red means something.

9. Provenance correction — this change landed inside an unrelated commit

git log -- test/sp/brain/motor_cortex_test.exs is misleading for this change. Read this before trusting the commit message.

The fix, this receipt, the motor-cortex-flake-fixed gate row and the docs/GATES.md re-render were staged, then swept into dc6af1f — "gaia: project the producer's per-UNI observation routes (v1a) verbatim" by a different agent's broad commit landing in the same shared checkout during the staging window. That commit's own change is viewer/gaia/collectors.cjs (+34); the other four files in it are this work and are not described by its message.

dc6af1f was pushed to origin/gen2-runtime before this was noticed. It was deliberately NOT rewritten. Amending or rebasing a pushed commit on a checkout with other agents actively committing would risk destroying their in-flight work — a far worse outcome than a mis-titled commit. The content in dc6af1f is correct and complete; only its message under-describes it.

This section is the audit trail: the change is real, reviewed, and gated, and it lives in dc6af1f.

Process lesson for agents sharing this checkout (extends the known evidence/gates.ndjson contention rule to the index itself): git add followed by a separate git commit leaves a window in which a concurrent agent's broad git add -A / git commit -a captures your staged files. Prefer the atomic form with explicit pathspecs, which commits only the named paths and does not depend on surviving index state:

git commit -m "<message>" -- path/one path/two

10. RESOLUTION — the per-test approach was abandoned for one global policy (2026-07-19)

§7 logged NoveltyTest and BridgeTest as open. Chasing them proved the per-test approach was itself the defect, so it was replaced. Recording the full arc because the wrong turn is the instructive part.

The displacement was real and measured, not theorised. Raising a heavy test's timeout lets it run to completion, which raises sustained CPU occupancy and starves the next-most-marginal test:

Measuring the newcomers exposed why: five tests are clustered in a narrow band against the 60s default — NoveltyTest 185553 ms, SlowContextWired 46827 ms (1.28x), MotorCortex 42968 / 40589 ms (1.40x / 1.48x), EvalTest 38898 ms (1.54x). They are one population, not four independent flakes. A per-test timeout is the wrong instrument for a saturated CPU-bound suite.

The fix: ONE global timeout: 300_000 in test/test_helper.exs, with NoveltyTest's own tag kept as the single documented exception (a genuine 4x outlier). The redundant per-test MotorCortex tags were removed; the diagnosis stays as a comment where the incident was found. No assertion or sample size was touched anywhere.

A hypothesis that FAILED, recorded so it is not re-run. Lowering max_cases to the true core count was argued for on the theory that Elixir's default 2x oversubscription is wrong for a zero-dep CPU-bound suite. Measured: 16 vs 32 gave 256s vs 261s and 333s vs 377s — marginally faster, both still failed. Reducing oversubscription is not the cure.

Verification, 5 consecutive full runs with the global policy:

The timeout class is closed: 0 ExUnit.TimeoutError across all 5 runs, where previously every red run carried at least one. That is the gate this work claims.

STILL OPEN, and NOT the same defect: run 4 failed SP.Runtime.SupervisorTest — "the agent PUBLISHES its mind beat to the board (push-snapshot, no fan-out)" — with timeouts=0. It is not a starvation failure; it is a genuine intermittent around the shared SP.Runtime.Board, which is the shared-mutable-state class this investigation originally (wrongly) assumed. It also sits in code a concurrent agent was actively changing (6ad1e18 per-UNI board telemetry). Not diagnosed here. Do not fold it into the timeout story.

Honest note on suite wall time: it rose from ~206-280s to ~400s. That is not a regression — the old times were partly tests being killed at 60s. The suite is slower because it now finishes the work it starts.