Overview · Written article
Maintenance
Watchdogs, boot persistence, health checks, backups, ledger repair and disaster recovery — including the four things this estate knows are not underwritten.
How to read this page
Three ways to read this page. Precise is the document itself, exactly as it is written in the repository. Plain and Clear were written for this website to help you meet that document — they are about it. They are not it, and they are not evidence.
These 13 pages are the only ones on this site written for this site. Everything else you can read here is a real file from a working repository, republished as it stands. These were written by hand for someone arriving with no context: an entry point, one article for each major part of the estate, and a set of practical guides for installing, running and maintaining things.
They carry one unusual discipline. Every citation in them points at a real file at a named commit, and the build resolves it there. If the file has moved, or a quoted range has run off the end of it, the build fails. It will not publish a reference that no longer lands. The wording of an article can drift out of date; what it points at cannot silently do so.
Start with "Start here". It is short, and it explains the one idea the rest of the estate is arranged around. After that, read whichever part you care about, or go to the quick start if you would rather run something than read about it.
What they are not: authoritative. An article is prose about documents, and the documents are the record. Where the two disagree, the document is right and the article is the thing to fix.
Your browser cannot switch reading levels, so the document itself is shown.
Precise — the source document
This is the article as written. Every citation and quoted block in it is resolved against the real file at the real commit when the site is built.
PARTLY RUNNABLE BY YOU. The health checks and the ledger tools run anywhere. The watchdogs and boot installers are Windows and PowerShell 5.1. The backup and archive timers are Linux systemd units marked as design, not deployed.
Most of this page is about a single distinction that this estate takes more seriously than most: crash-restart and reboot-survival are two different claims, and collapsing them is how a system comes to believe it is durable when it is not.
# TWO DIFFERENT CLAIMS, AND COLLAPSING THEM IS HOW A SYSTEM COMES TO BELIEVE IT IS DURABLE.
# crash-restart - the watchdog notices a dead track_server.cjs and starts it. PROVEN by killing
# the process and running `track_watchdog.ps1 -Once` (measured 2026-07-31: DOWN
# detected, restarted, back up in ~2s).
# reboot-survival - the watchdog itself comes back after a logon. THIS script is what makes that
# possible, and it is still NOT PROVEN until a real reboot happens. Installing a
# Startup entry is not evidence that it fired.Installing a startup entry is not evidence that it fired. That sentence is the whole discipline.
Health: the one call
Before anything else, and before reading any file to answer a state question:
curl -s http://127.0.0.1:8090/api/status
It answers what is up, what step the journey is on, every surface's live probe, and what to do next. The estate's own run sheet puts it bluntly: never grep the repository to answer a state question. A repository tells you what was intended; only a probe tells you what is.
Three questions in particular are deliberately not answered by any committed file, because no committed file can answer them honestly — they are facts about a run or about now, not about the tree:
| question | the command |
|---|---|
| Are the trees clean? | git status -sb |
| Does the suite pass? | mix test |
| Do the gates pass? | node viewer/gate_runner.cjs |
A governing document upstream used to answer all three in prose. One of those answers was measured at one moment and was false three minutes later — a half-life of 176 seconds — and it was committed reading as present tense. Run the commands.
Watchdogs — crash restart
Each supervises one surface, each takes the same options, and each can be run once for a single check instead of looping.
powershell -File viewer/door_watchdog.ps1
powershell -File viewer/gaia/gaia_watchdog.ps1
powershell -File viewer/track/track_watchdog.ps1
powershell -File viewer/hud/hud_watchdog.ps1
powershell -File viewer/systray_watchdog.ps1
The last one supervises the studio processes and the media server, with a toast notification and an automatic restart.
A documented conflict you should know about before hour three: two supervisors disagree while the system is on air. One stands down during a stream; the other restarts unconditionally. That is recorded rather than resolved, which is the honest state — resolving it needs a decision about which behaviour is correct under air, and that decision has not been made.
The HUD watchdog stands down when the native service is running, so it is a fallback leg rather than a duplicate.
Quiet mode — stop the media stack, keep watching
The broadcast stack is heavy and this is not a machine that exists only to broadcast. Quiet mode stops the expensive half — the mixer, the media server, the browser profiles holding the camera views, the encoders pushing to air — while the monitoring surfaces stay up, so the machine is still observable while it is quiet.
The state is a declared latch on disk, and it is written before anything is stopped. That ordering is the whole design. Every supervisor reads the latch and stands down rather than restarting what quiet mode just closed, and without it quiet mode does not survive: this system has several supervisors, one of them on a five-second timer, and any one of them will undo a quiet in seconds.
powershell -File viewer/channel_windows_watchdog.ps1
That is one of the supervisors that must consult the latch. Run it once for a single check to see what it would do rather than leaving it looping.
The failsafe direction is deliberate and it is the opposite of the healer's. An unreadable or absent latch reads as not quiet, so a corrupt file brings the stack back rather than silently holding a studio dark. Quiet and resume are both reachable from the operator's surfaces rather than only from a shell, because a mode you cannot leave without a terminal is a trap.
After a reboot, prove the state rather than assume it
A boot that looks fine is a failure this estate keeps meeting. This runs after the machine comes back and writes a verdict rather than an impression.
powershell -File viewer/hud/native/hud_boot_healthcheck.ps1
It proves the monitoring surface is actually serving rather than merely running, checks the monitors it expects to find, and asks the one thing no liveness probe can answer: did the box stay quiet. If something restarted the stack during boot, the check re-asserts quiet rather than reporting a state that has already stopped being true.
Boot persistence — reboot survival
A separate claim, installed separately, and proved separately.
powershell -File viewer/door_boot_install.ps1
powershell -File viewer/gaia/gaia_boot_install.ps1
powershell -File viewer/track/track_boot_install.ps1
powershell -File viewer/hud/hud_boot_install.ps1
The mechanism is a hidden script in the per-user startup folder, non-elevated. Each installer pairs with a prover that reports PROVEN only if the machine actually rebooted after the install marker was written:
powershell -File viewer/door_boot_proof.ps1
powershell -File viewer/gaia/gaia_boot_proof.ps1
powershell -File viewer/hud/hud_boot_proof.ps1
powershell -File viewer/hud/native/hud_native_boot_proof.ps1
That design is the point. A prover that reported success because a file exists would be certifying its own installation, not the behaviour it was installed for.
The restart drills
Two drills exercise the native HUD's recovery paths deliberately, rather than waiting to find out during a show. Both need an elevated shell.
powershell -File viewer/hud/native/_drill_crash_restart_elevated.ps1
powershell -File viewer/hud/native/_drill_service_restart_elevated.ps1
A recovery path that has never been exercised is a hypothesis.
Certificates
The LAN certificate is regenerated only if it is missing or expires within thirty days, and the bring-up calls it every time — so the renewal is a side effect of normal operation rather than a task someone has to remember.
powershell -File viewer/gen_auto_cert.ps1
The trust-store install is idempotent, needs no elevation, and runs at every bring-up:
powershell -File viewer/install_lan_cert.ps1
If overlays or web sources render black after a certificate change, this is the first thing to check.
Ledger maintenance
The scientific control plane's ledger is append-only and hash-chained. Keeping it honest is maintenance work in its own right, and there is a set of one-shot recorders for it. They are idempotent — an entry already accounted for is skipped, so re-running appends nothing.
mix run scripts/control_plane_bootstrap_prelude.exs
mix run scripts/control_plane_record_own_history.exs
mix run scripts/control_plane_backfill_phases_6_7.exs
mix run scripts/control_plane_backfill_evidence_objects.exs
mix run scripts/control_plane_record_phase9_steps.exs
mix run scripts/control_plane_record_phase9_builds.exs
mix run scripts/control_plane_correct_step_coverage.exs
mix run scripts/control_plane_correct_pre_registration_null.exs
mix run scripts/control_plane_supersede_receipt_reference.exs
These exist because the ledger's own anti-silence guard fired. Work had continued and the record had not — steps were marked done in the plan and accounted for nowhere in the ledger. The guard said so, and these scripts are the answer to it rather than a way of silencing it.
Note the two named correct_ and the one named supersede_. A correction is appended, never
edited. An append-only ledger that permits a fix in place is a ledger with no memory of having been
wrong, which is the only thing it was built to remember.
Backup, archive and heartbeat
Four scheduled units on the Linux side. All four are marked design/reference, authored and not yet run on node hardware — so treat this section as the intended shape, not as a running system.
| unit | cadence | what it does |
|---|---|---|
| colony archive | 03:30 daily | snapshots the world and each agent's memory, with a checksum manifest |
| backup | 04:00 daily, persistent | rsync to a date-stamped destination; an unset destination is a hard failure, never a silent no-op |
| heartbeat | every 60s | a read-only re-probe appended to an audit log; always exits 0 so it cannot crash-loop its own timer |
| collector | every 10 minutes | the evidence collector, rootless |
The ordering is deliberate: the archive completes at 03:30 before the backup at 04:00 sweeps the archive root. Reverse them and every backup is a day stale.
Upgrades and disaster recovery
- Before touching the colony container, capture first. The agents' minds live in the container's ephemeral filesystem and removing the container destroys them. There is a mandatory capture procedure and an evidence-hold verification that must pass first.
- One deployment script refuses to run. It is stale against the current topology and prints a refusal saying that running it blind takes the live camera down. It requires an explicit acknowledgement environment variable to override. That is the correct shape for a stale script: present, loud, and not silently functional.
- The emergency stop needs two environment values with no invented defaults. A panic button that guesses its own target is not a panic button.
- The fan-out arm does not survive a reboot, and that is intentional fail-closed design. The encrypted endpoint store persists; the in-memory unlock does not. Do not "fix" it by persisting the passphrase.
Housekeeping
node viewer/obs_cleanup.cjs
Retires stale scenes and inputs left behind by old experiments. Idempotent.
Two real housekeeping gaps, stated rather than left to be discovered:
- There is no log rotation anywhere in either repository. Logs accumulate in a directory that is ignored by version control. Nothing prunes them.
- One evidence store is explicitly never pruned — a write-once record that grows without bound by design, kept out of version control precisely because it would bloat the repository as the agents tick.
What is NOT underwritten
The four things this estate knows it cannot currently promise. They are published here rather than buried in an appendix, because a maintenance guide that lists only the parts that work is a sales document.
- No soak test exists. Memory-leak, reconnect and token-refresh behaviour over days is unknown.
- Window capture is a documented dice-roll, realised twice. Detection exists; automatic recovery does not, and the durable fix is not built.
- A four-hour run is not underwritten. The estate's own run sheet says: attend it, or do not do it.
- The emergency stop has never been fired by a human. Sixty seconds of rehearsal before a public run is cheap insurance against the one path nobody has walked.
There is a fifth, and it is the sharpest: 35 of the 36 registered gates have no row in the canonical ledger. The instruments run and their results are real, but the record that they ran is not yet written where the schema says it must be. Authoring those rows is the operator's task and no agent may do it for him — the ledger has a single writer, and that constraint is worth more than the convenience of automating it away.
Corrected 2026-08-24. This paragraph previously read "every registered gate has zero rows in the canonical ledger", and that was false. One gate does have rows. Measured by joining the registry to the ledger on each gate's row alias rather than on its identifier:
sight-blindis recorded under the namehud-sight-shows-blind, against a receipt dated 2026-07-17, and its latest verdict there is PARTIAL rather than PASS.The join key is the whole trap. Matching on the identifier returns zero and looks like a clean, checked answer, which is how a wrong number survives being checked. The same false sentence stood in four of the estate's own governing documents for a fortnight before it was caught there, and it reached this page from them — so the correction upstream did not reach the copy a reader could actually see. The original wording is kept above rather than quietly replaced, because a record that permits a fix in place has no memory of having been wrong, which is the only thing it was built to remember.
Plain — written for this website, not the source document
Keeping the system running is its own job: restarting things that crash, making them come back after a reboot, checking health, backing up, repairing the record, and recovering from disaster.
It hangs on one distinction the project takes seriously. Surviving a crash and surviving a reboot are two different claims, and collapsing them is how a system comes to believe it is durable when it is not. So each startup entry has a separate prover that reports success only if the machine actually rebooted after it was installed. Installing one is not evidence that it fired.
It also insists that questions about the current state are answered by asking the running system rather than reading files, and names three such questions no committed file can answer honestly. Corrections to the record are appended, never edited.
Two of its sections are about stopping rather than starting: a quiet mode that shuts the heavy media half while the watching stays up, and a check after a reboot that asks whether the machine actually stayed quiet.
The page ends by listing what the project knows it cannot promise, including that no long-duration test exists and that the emergency stop has never been fired by a person. One of those admissions carries a correction: it had understated how much of the record exists.
Plain · written 2026-08-24 by claude-opus-5 · not yet checked by a person · about the document whose sha256 is eca87f85d4313c9a
Clear — written for this website, not the source document
This is the operations page: watchdogs, boot persistence, health checks, certificates, repair of the append-only record, backups and disaster recovery. Most of it turns on one distinction the estate takes seriously: crash-restart and reboot-survival are two different claims, and collapsing them is how a system comes to believe it is durable when it is not. Installing a startup entry is not evidence that it fired.
Health comes first: ask the running system rather than read the repository, because a repository tells you what was intended and only a probe tells you what is. Three questions are deliberately not answered by any committed file — whether the trees are clean, whether the suite passes, whether the gates pass — because they are facts about a run, not about the tree. A governing document upstream answered all three in prose, and one answer was false three minutes after it was measured, while reading as present tense.
Watchdogs handle crash restart, one per surface. A conflict is recorded rather than resolved: two supervisors disagree while the system is on air, one standing down during a stream, the other restarting unconditionally, and nobody has decided which is correct.
Boot persistence is installed and proved separately. Each installer pairs with a prover that reports success only if the machine rebooted after the install marker was written. A prover that said yes because a file exists would certify its own installation rather than the behaviour it was installed for. Two drills exercise recovery paths deliberately, because a recovery path never exercised is a hypothesis.
Certificate renewal is a side effect of normal operation rather than a task someone must remember.
The evidence record has its own maintenance: one-shot recorders that append nothing if run again. They exist because the record's own anti-silence guard fired: work had continued and the record had not. A correction is appended, never edited. An append-only record that permits a fix in place has no memory of having been wrong, which is the only thing it was built to remember.
Backups, archiving and a heartbeat are four scheduled units, all marked as design and not yet run on node hardware — the intended shape rather than a running system. Reverse two of them and every backup is a day stale.
Upgrades come with warnings: removing the colony container destroys the agents' memories, so capture comes first. One deployment script refuses to run because it is stale against the current topology, and the fan-out arm does not survive a reboot by design. Two housekeeping gaps are stated rather than left to be found: nothing rotates the logs, and one evidence store grows without bound by design.
The closing section lists what is not underwritten — no soak test, a capture failure detected but not automatically recovered, a long run nobody has underwritten, and an emergency stop no human has fired. A fifth is named as the sharpest: all but one of the registered checks have no row in the record meant to hold them, and writing those rows is the operator's task, not an agent's. The page carries a correction to that line, which previously said none of them did, and names the join key that hid it.
Two sections concern stopping rather than starting. A quiet mode shuts the expensive media half while monitoring stays up, held by a latch written before anything stops, because the supervisors would otherwise undo it in seconds. Its failsafe leans the opposite way to the healer's: an unreadable latch reads as not quiet. A companion check after a reboot asks whether the machine actually stayed quiet.
Clear · written 2026-08-24 by claude-opus-5 · not yet checked by a person · about the document whose sha256 is eca87f85d4313c9a