UNI SIGHT — real pixels: per-UNI field-of-view + producer full-feed → UNI.OS visual world model

The Colony & the Method · docs/UNI_SIGHT_PLAN.md @ 44baf03d5041 (gen2-runtime) — opens the published snapshot ac338733bbba

Three ways to read this page. Precise is the document itself, exactly as it is written in the repository. Plain and Clear were written for this website to help you meet that document — they are about it. They are not it, and they are not evidence.

Eighty-four pages about the colony. Each agent is an Elixir process holding a generative model and doing inference, attached to a body that logs into a Minecraft world as an ordinary player. Around that sit the broadcast suite that films them and the runbooks that keep the whole thing running. There are typed specifications for each organ of the model, plus the world and genome specs. There are also the adversarial review personas used to attack a proposed change before it ships.

It is for the reader curious how a running system is put together and how it is held to account. The accountability half is the more distinctive. There is a lab protocol governing evidence and attribution, and a claim fence that restricts the vocabulary a claim is allowed to use. There is a public gate log. And there is a standing invitation to reproduce any verdict from the commit and the seed named in its receipt.

Start with the public read, then the lab protocol, then the falsification invitation. If you want the mathematics rather than the operations, go straight to the typed organ specs.

What it is not: a description of a mind, and not all one kind of document. A large part of this corpus is design and planning — specs marked as proposed rather than applied, organs designed but not built, plans that were later superseded — and each page states which it is. A specification is not a running system, and these pages are careful about the difference; the reader should be too. Eight documents were withheld from publication because they describe private infrastructure.

PlainClearPrecise

Your browser cannot switch reading levels, so the document itself is shown.

This is the document. Rendered from the repository at the commit above, with nothing rewritten for the web. A gate re-renders it on every deploy and fails the build if a single byte differs.

Saved from planning (2026-05). Cross-repo: Strings (Minecraft bodies) + UNI.OS (vision engine). Status: PLAN — not yet implemented. Owner approved investigation; build pending go-ahead.

Context

The owner wants the UNIs to SEE — raw pixels, the full video feed, not the symbolic proxy — with a per-UNI field of view, and to LEARN to see/understand what they're seeing.

Two systems; pixels belong in one:

Strings (Minecraft colony) is a tabular, symbolic active-inference engine. Its σ is a deliberate proxy; the "no-leakage / no-foreign-layer" claim (gates 8/9/14/17) requires pixels NEVER enter, and a Dirichlet-categorical A cannot ingest a frame. Pixels are excluded there by design.
UNI.OS is a vision-first active-inference system that already ingests raw pixels the PURE way (no neural net): aion_vwm/discrete/patch_codec.py (frame → 8×8 patch luma → 8-bin discrete codes, decodable back), aion_vwm/discrete/markov_world.py (DiscretePatchMarkovWorld: Dirichlet HMM over patch codes, Baum–Welch EM, exact forward–backward, log-evidence = −F, frame regeneration), aion_active/pixel_model.py (action-conditioned RGB world model), online adaptive_train_frame MCP tool, ffmpeg ingest. This is exactly "see the raw feed and learn to see."

Owner decisions: frame source = BOTH (per-UNI first-person POV AND the producer's full-frame live feed); mode = LIVE real-time in the loop; process = investigate (done) → this plan.

Approach (phased; each independently falsifiable)

Phase 1 — Real field-of-view capture (Strings side; no covenant impact)

Per-UNI POV: viewer/ already depends on prismarine-viewer. Add a headless POV renderer per body (viewer/pov.js, used by body.js): render the bot's first-person view to raw RGB at low res/fps (≈64×64 @ ~4 fps) on a per-UNI channel (socket/MJPEG), OUT of the σ lockstep. Falsifiable: dump PNGs.
Producer full feed: tap the director camera (:3020, prismarine-viewer) for full-frame RGB → one producer stream. Falsifiable: saved frames match the on-stream shot.

Phase 2 — Real-time vision bridge → UNI.OS world models (the heart)

Vision service in UNI.OS (extend mcp_server or new aion_vwm/serve/realtime.py): per stream id (per-UNI + producer), encode_clip_to_patch_bins → DiscretePatchMarkovWorld, learn ONLINE (add an incremental single-forward-step + Dirichlet count update; the adaptive_train_frame pattern). One model per stream; persist under artifacts/models/. Falsifiable: per-stream FREE ENERGY DROPS + held-out next-frame regeneration MSE drops (logged proof rows, UNI.OS docs/falsifiable_claims.md).

Phase 3 — "Understand what it's seeing"

Learned hidden states cluster scenes (tree/cave/night/mob). Falsifiable: state↔context correlation; next-frame prediction beats persistence. Likely needs codec enrichment (color + edges, not luma-only).

Phase 4 (optional, covenant-safe) — let sight inform action

Discretise each UNI's visual belief into ONE small symbolic percept (visual-novelty bin / scene class) and add it as a Strings modality (like :prey/:build), so ACTIONS can be vision-informed WITHOUT raw pixels crossing the blanket. Gates 8/9/14/17 stay green (only a discrete bin crosses).

Critical files

UNI.OS: src/aion_vwm/discrete/{patch_codec,markov_world}.py, ingest/ffmpeg_clip.py, new src/aion_vwm/serve/realtime.py (+ online-update on DiscretePatchMarkovWorld), src/active_inference_suits/mcp_server/server.py, docs/falsifiable_claims.md. Strings: viewer/pov.js (new), viewer/body.js, viewer/director.js; optional Phase 4: lib/sp/brain/{genome,mc_codec,bridge}.ex + viewer/body.js.

Honest risks / scope

Research-grade and heavy: real-time per-UNI vision = N headless renderers + N world models + online EM at frame rate. Starts LOW-RES/LOW-FPS and scales; "understand all the world" is a direction.
Headless prismarine-viewer server-side needs a GL context (headless-gl/node-canvas) — feasibility- check first.
Cross-runtime: Strings (Elixir/Node) → frames → UNI.OS (Python). Three runtimes in the loop.
No covenant break: vision lives in UNI.OS; Strings stays symbolic; if sight feeds back, only a discrete bin crosses. No LLM, no RL anywhere.

Recommended first step

De-risking spike: prove ONE bot's headless POV renders to real frames AND one UNI.OS world model's free energy drops on them. If that holds, the rest follows.

Status (2026-05 — built + proven offline; live-enable pending)

The owner chose vision-primary, bridge to UNI.OS, full build. Done + committed:

Frame source (Node 25): headless-gl has no prebuilt for Node 25, so frames are captured via Playwright + system Chrome (its own GL) screenshotting the Director :3020 / a per-UNI POV. Proven: real Minecraft renders captured server-side, no native build.
Visual cortex (UNI.OS): aion_vwm/serve/realtime.py — per-stream DiscretePatchMarkovWorld, batch warm-start → online refine (observe_frame_online), persistence, TCP server. Fixed a latent symmetric-init collapse (every state identical ⇒ one useless scene-state); seeded asymmetric init now yields discriminative scenes (day vs night → disjoint states). Proven: free energy DROPS (batch + online) on real frames; held-out generalises. Audited neural-net-free (tests/aion_vwm/test_vision_bridge_nn_free.py).
Bridge: viewer/vision_forward.cjs (replay + live modes) → service; percepts back. Proven end-to-end.
Per-UNI POV: viewer/body.js serves an opt-in first-person view (UNI_POV_PORT).
Vision-primary brain (E): opt-in :scene factor (genome :sight_cortex organ; vision_primary/0), codec outcome(:scene,…), body σ 15th channel (gated by UNI_PERCEPT_DIR), bridge parse. The categorical brain ingests the discrete scene-state (never pixels). Default UNIs unchanged (12 factors). mix sp.uni.prove §8 + test/sp/brain/vision_test.exs green; full suite green.
Covenant: the blanket is extended, not weakened — only a discrete scene-state crosses (like :prey/:build); pixels stay in the external pure-FEP cortex. Gates 8/9/17/18 stay green; FALSIFICATION §8.

Live-enable (pending, deliberate): relaunch with UNI.OS vision service up, per-UNI POV + vision_forward running, bodies spawned with UNI_PERCEPT_DIR + a Genome.vision_primary() lineage. Vision-primary brains are a fresh lineage (13-factor) — a new pixel-seeing colony, distinct from the saved 12-factor brains.

sha256 bcddc7745a437f8e — of the original file, so what was ingested stays checkable.

Written for this website — not the document. This is a plain-language retelling, written to help you meet the document. It is not the source, and it is not evidence. It has not yet been checked by a person. Read the document itself → (or choose Precise in the reading-level control above)

This is a plan for letting agents see real pixels, with a status section added later saying what was built and what is still switched off.

The plan's central move is a separation. One system is symbolic by design, and its no-leakage claim requires that pixels never enter it. The other already ingests raw frames without any neural network. So pixels live in the second system, and if sight ever informs action, only one small discrete value crosses back.

Four phases follow, each with a stated way to show it false: capture real first-person frames, learn a world model from each stream, check whether the learned states track scenes, and optionally feed one discrete percept back.

The later status section records real work: a capture route found after a build failed, a latent collapse fixed, free energy dropping on real frames, and an audit for the absence of neural networks. Switching it on live is deliberately still pending.

Plain · written 2026-08-01 by claude-opus-5 · not yet checked by a person · about the document whose sha256 is bcddc7745a437f8e

Written for this website — not the document. This is a clearer retelling, written to help you meet the document. It is not the source, and it is not evidence. It has not yet been checked by a person. Read the document itself → (or choose Precise in the reading-level control above)

This document began as a plan and later gained a status section, so it reads as both. Its header says plainly that it was a plan, not yet implemented, with the build pending a go-ahead.

The context sets up a separation that the whole design rests on. One system is a tabular, symbolic engine whose input is a deliberate proxy, and whose no-leakage and no-foreign-layer claims require that pixels never enter it, since its categorical machinery cannot ingest a frame at all. The other system already takes raw frames in a way that uses no neural network, converting a frame into discrete patch codes and learning over them with exact inference, so that free energy can be read as evidence. The plan's conclusion is that pixels belong in the second system.

Four phases follow, and each names how it could be shown false. The first captures real first-person views per body at low resolution and frame rate, outside the lockstep of the symbolic senses, plus a full-frame feed from the camera; the refuting check is simply dumping the images and comparing. The second builds a real-time bridge so each stream trains its own world model online, with the refuting check being that free energy must drop and that regeneration of unseen frames must improve. The third asks whether the learned hidden states cluster into recognisable scenes, checked by correlating states with context and by beating a trivial baseline at prediction. The fourth is optional and deliberately covenant-safe: reduce a visual belief to one small discrete value and add it as an ordinary channel, so actions can be informed by sight without any pixel crossing.

A risks section is candid. This is research-grade and heavy, since real-time vision per agent means many renderers and many models learning at frame rate; understanding the world is called a direction rather than a destination. A specific technical feasibility question is flagged, and so is the awkwardness of three runtimes in one loop. It repeats that the covenant is not broken because the vision stays outside and only a discrete value would cross.

A recommended first step is a de-risking spike: show one body's view rendering to real frames and one model's free energy dropping on them.

The status section added later records what was built. A capture route was chosen after a native component had no build available for the current runtime, and real renders were captured. The visual side gained a real-time service with a warm start and online refinement. A latent collapse was found and fixed, where a symmetric start made every state identical and therefore useless. An asymmetric start now yields states that discriminate day from night. Free energy drops on real frames and material kept back from training generalises, and an audit checks the absence of neural networks. A bridge and a per-body view were built, and an opt-in factor lets the categorical side ingest the discrete scene value while default agents stay unchanged. The covenant is described as extended rather than weakened.

The last paragraph is the honest one: switching this on live is still pending and deliberate, and vision-primary agents would be a fresh lineage distinct from the saved ones.

Clear · written 2026-08-01 by claude-opus-5 · not yet checked by a person · about the document whose sha256 is bcddc7745a437f8e