Overview · Written article
Maintenance
Watchdogs, boot persistence, health checks, backups, ledger repair and disaster recovery — including the four things this estate knows are not underwritten.
How to read this page
Three ways to read this page. Precise is the document itself, exactly as it is written in the repository. Plain and Clear were written for this website to help you meet that document — they are about it. They are not it, and they are not evidence.
These 13 pages are the only ones on this site written for this site. Everything else you can read here is a real file from a working repository, republished as it stands. These were written by hand for someone arriving with no context: an entry point, one article for each major part of the estate, and a set of practical guides for installing, running and maintaining things.
They carry one unusual discipline. Every citation in them points at a real file at a named commit, and the build resolves it there. If the file has moved, or a quoted range has run off the end of it, the build fails. It will not publish a reference that no longer lands. The wording of an article can drift out of date; what it points at cannot silently do so.
Start with "Start here". It is short, and it explains the one idea the rest of the estate is arranged around. After that, read whichever part you care about, or go to the quick start if you would rather run something than read about it.
What they are not: authoritative. An article is prose about documents, and the documents are the record. Where the two disagree, the document is right and the article is the thing to fix.
Your browser cannot switch reading levels, so the document itself is shown.
Precise — the source document
This is the article as written. Every citation and quoted block in it is resolved against the real file at the real commit when the site is built.
PARTLY RUNNABLE BY YOU. The health checks and the ledger tools run anywhere. The watchdogs and boot installers are Windows and PowerShell 5.1. The backup and archive timers are Linux systemd units marked as design, not deployed.
Most of this page is about a single distinction that this estate takes more seriously than most: crash-restart and reboot-survival are two different claims, and collapsing them is how a system comes to believe it is durable when it is not.
# TWO DIFFERENT CLAIMS, AND COLLAPSING THEM IS HOW A SYSTEM COMES TO BELIEVE IT IS DURABLE.
# crash-restart - the watchdog notices a dead track_server.cjs and starts it. PROVEN by killing
# the process and running `track_watchdog.ps1 -Once` (measured 2026-07-31: DOWN
# detected, restarted, back up in ~2s).
# reboot-survival - the watchdog itself comes back after a logon. THIS script is what makes that
# possible, and it is still NOT PROVEN until a real reboot happens. Installing a
# Startup entry is not evidence that it fired.Installing a startup entry is not evidence that it fired. That sentence is the whole discipline.
Health: the one call
Before anything else, and before reading any file to answer a state question:
curl -s http://127.0.0.1:8090/api/status
It answers what is up, what step the journey is on, every surface's live probe, and what to do next. The estate's own run sheet puts it bluntly: never grep the repository to answer a state question. A repository tells you what was intended; only a probe tells you what is.
Three questions in particular are deliberately not answered by any committed file, because no committed file can answer them honestly — they are facts about a run or about now, not about the tree:
| question | the command |
|---|---|
| Are the trees clean? | git status -sb |
| Does the suite pass? | mix test |
| Do the gates pass? | node viewer/gate_runner.cjs |
A governing document upstream used to answer all three in prose. One of those answers was measured at one moment and was false three minutes later — a half-life of 176 seconds — and it was committed reading as present tense. Run the commands.
Watchdogs — crash restart
Each supervises one surface, each takes the same options, and each can be run once for a single check instead of looping.
powershell -File viewer/door_watchdog.ps1
powershell -File viewer/gaia/gaia_watchdog.ps1
powershell -File viewer/track/track_watchdog.ps1
powershell -File viewer/hud/hud_watchdog.ps1
powershell -File viewer/systray_watchdog.ps1
The last one supervises the studio processes and the media server, with a toast notification and an automatic restart.
A documented conflict you should know about before hour three: two supervisors disagree while the system is on air. One stands down during a stream; the other restarts unconditionally. That is recorded rather than resolved, which is the honest state — resolving it needs a decision about which behaviour is correct under air, and that decision has not been made.
The HUD watchdog stands down when the native service is running, so it is a fallback leg rather than a duplicate.
Boot persistence — reboot survival
A separate claim, installed separately, and proved separately.
powershell -File viewer/door_boot_install.ps1
powershell -File viewer/gaia/gaia_boot_install.ps1
powershell -File viewer/track/track_boot_install.ps1
powershell -File viewer/hud/hud_boot_install.ps1
The mechanism is a hidden script in the per-user startup folder, non-elevated. Each installer pairs with a prover that reports PROVEN only if the machine actually rebooted after the install marker was written:
powershell -File viewer/door_boot_proof.ps1
powershell -File viewer/gaia/gaia_boot_proof.ps1
powershell -File viewer/hud/hud_boot_proof.ps1
powershell -File viewer/hud/native/hud_native_boot_proof.ps1
That design is the point. A prover that reported success because a file exists would be certifying its own installation, not the behaviour it was installed for.
The restart drills
Two drills exercise the native HUD's recovery paths deliberately, rather than waiting to find out during a show. Both need an elevated shell.
powershell -File viewer/hud/native/_drill_crash_restart_elevated.ps1
powershell -File viewer/hud/native/_drill_service_restart_elevated.ps1
A recovery path that has never been exercised is a hypothesis.
Certificates
The LAN certificate is regenerated only if it is missing or expires within thirty days, and the bring-up calls it every time — so the renewal is a side effect of normal operation rather than a task someone has to remember.
powershell -File viewer/gen_auto_cert.ps1
The trust-store install is idempotent, needs no elevation, and runs at every bring-up:
powershell -File viewer/install_lan_cert.ps1
If overlays or web sources render black after a certificate change, this is the first thing to check.
Ledger maintenance
The scientific control plane's ledger is append-only and hash-chained. Keeping it honest is maintenance work in its own right, and there is a set of one-shot recorders for it. They are idempotent — an entry already accounted for is skipped, so re-running appends nothing.
mix run scripts/control_plane_bootstrap_prelude.exs
mix run scripts/control_plane_record_own_history.exs
mix run scripts/control_plane_backfill_phases_6_7.exs
mix run scripts/control_plane_backfill_evidence_objects.exs
mix run scripts/control_plane_record_phase9_steps.exs
mix run scripts/control_plane_record_phase9_builds.exs
mix run scripts/control_plane_correct_step_coverage.exs
mix run scripts/control_plane_correct_pre_registration_null.exs
mix run scripts/control_plane_supersede_receipt_reference.exs
These exist because the ledger's own anti-silence guard fired. Work had continued and the record had not — steps were marked done in the plan and accounted for nowhere in the ledger. The guard said so, and these scripts are the answer to it rather than a way of silencing it.
Note the two named correct_ and the one named supersede_. A correction is appended, never
edited. An append-only ledger that permits a fix in place is a ledger with no memory of having been
wrong, which is the only thing it was built to remember.
Backup, archive and heartbeat
Four scheduled units on the Linux side. All four are marked design/reference, authored and not yet run on node hardware — so treat this section as the intended shape, not as a running system.
| unit | cadence | what it does |
|---|---|---|
| colony archive | 03:30 daily | snapshots the world and each agent's memory, with a checksum manifest |
| backup | 04:00 daily, persistent | rsync to a date-stamped destination; an unset destination is a hard failure, never a silent no-op |
| heartbeat | every 60s | a read-only re-probe appended to an audit log; always exits 0 so it cannot crash-loop its own timer |
| collector | every 10 minutes | the evidence collector, rootless |
The ordering is deliberate: the archive completes at 03:30 before the backup at 04:00 sweeps the archive root. Reverse them and every backup is a day stale.
Upgrades and disaster recovery
- Before touching the colony container, capture first. The agents' minds live in the container's ephemeral filesystem and removing the container destroys them. There is a mandatory capture procedure and an evidence-hold verification that must pass first.
- One deployment script refuses to run. It is stale against the current topology and prints a refusal saying that running it blind takes the live camera down. It requires an explicit acknowledgement environment variable to override. That is the correct shape for a stale script: present, loud, and not silently functional.
- The emergency stop needs two environment values with no invented defaults. A panic button that guesses its own target is not a panic button.
- The fan-out arm does not survive a reboot, and that is intentional fail-closed design. The encrypted endpoint store persists; the in-memory unlock does not. Do not "fix" it by persisting the passphrase.
Housekeeping
node viewer/obs_cleanup.cjs
Retires stale scenes and inputs left behind by old experiments. Idempotent.
Two real housekeeping gaps, stated rather than left to be discovered:
- There is no log rotation anywhere in either repository. Logs accumulate in a directory that is ignored by version control. Nothing prunes them.
- One evidence store is explicitly never pruned — a write-once record that grows without bound by design, kept out of version control precisely because it would bloat the repository as the agents tick.
What is NOT underwritten
The four things this estate knows it cannot currently promise. They are published here rather than buried in an appendix, because a maintenance guide that lists only the parts that work is a sales document.
- No soak test exists. Memory-leak, reconnect and token-refresh behaviour over days is unknown.
- Window capture is a documented dice-roll, realised twice. Detection exists; automatic recovery does not, and the durable fix is not built.
- A four-hour run is not underwritten. The estate's own run sheet says: attend it, or do not do it.
- The emergency stop has never been fired by a human. Sixty seconds of rehearsal before a public run is cheap insurance against the one path nobody has walked.
There is a fifth, and it is the sharpest: every registered gate has zero rows in the canonical ledger. The instruments run and their results are real, but the record that they ran is not yet written where the schema says it must be. Authoring those rows is the operator's task and no agent may do it for him — the ledger has a single writer, and that constraint is worth more than the convenience of automating it away.
Plain — written for this website, not the source document
Keeping the system running once it exists is its own job: restarting things that crash, making them come back after a reboot, checking health, backing up, repairing the record, and recovering from disaster.
It hangs on one distinction the project takes seriously. Surviving a crash and surviving a reboot are two different claims, and collapsing them is how a system comes to believe it is durable when it is not. So each startup entry has a separate prover that reports success only if the machine actually rebooted after the entry was installed. Installing one is not evidence that it ever fired.
It also insists that questions about the current state are answered by asking the running system rather than by reading files, and it names three such questions no committed file can answer honestly. Corrections to the record are appended, never edited.
The page ends by listing what the project knows it cannot promise, including that no long-duration test exists and that the emergency stop has never been fired by a person.
Plain · written 2026-08-01 by claude-opus-5 · not yet checked by a person · about the document whose sha256 is 27075bfb77786667
Clear — written for this website, not the source document
This is the operations page: watchdogs, boot persistence, health checks, certificates, repair of the append-only record, backups and disaster recovery. Most of it turns on one distinction the estate takes more seriously than most — that crash-restart and reboot-survival are two different claims, and collapsing them is how a system comes to believe it is durable when it is not. Installing a startup entry is not evidence that it fired.
Health comes first, and the instruction is to ask the running system rather than read the repository, because a repository tells you what was intended and only a probe tells you what is. Three questions in particular are deliberately not answered by any committed file — whether the trees are clean, whether the suite passes, whether the gates pass — because they are facts about a run rather than about the tree. A governing document upstream used to answer all three in prose, and one of those answers was measured at one moment and was false three minutes later, while reading as present tense.
Watchdogs handle crash restart, one per surface. A conflict is recorded rather than resolved. Two supervisors disagree while the system is on air, one standing down during a stream and the other restarting unconditionally. Resolving it needs a decision about which behaviour is correct, and nobody has made it.
Boot persistence is installed separately and proved separately. Each installer pairs with a prover that reports success only if the machine actually rebooted after the install marker was written. A prover that said yes merely because a file exists would be certifying its own installation rather than the behaviour it was installed for. Two drills exercise recovery paths deliberately, on the grounds that a recovery path never exercised is a hypothesis.
Certificate renewal happens as a side effect of normal operation rather than as a task someone has to remember.
The evidence record has its own maintenance: a set of one-shot recorders, all of which append nothing if run again. They exist because the record's own anti-silence guard fired — work had continued and the record had not. Two of them are corrections and one supersedes an earlier reference, and the page underlines why. A correction is appended, never edited. An append-only record that permits a fix in place has no memory of having been wrong, which is the only thing it was built to remember.
Backups, archiving and a heartbeat are four scheduled units, all marked as design and not yet run on node hardware, so the section is the intended shape rather than a running system. Their ordering is deliberate: reverse two of them and every backup is a day stale.
Upgrades come with warnings. Removing the colony container destroys the agents' memories, so capture comes first. One deployment script refuses to run because it is stale against the current topology. And the fan-out arm deliberately does not survive a reboot. Two housekeeping gaps are stated rather than left to be discovered: nothing rotates the logs, and one evidence store grows without bound by design.
The closing section lists what is not underwritten — no soak test, a capture failure with detection but no automatic recovery, a long run that is not underwritten, and an emergency stop no human has fired. A fifth is named as the sharpest. Every registered check has no rows in the one record that is meant to hold them, and writing those rows is the operator's task rather than an agent's.
Clear · written 2026-08-01 by claude-opus-5 · not yet checked by a person · about the document whose sha256 is 27075bfb77786667