A machine follows its channel, and comes back from a revision that cannot start

Context

ADR-0013 said how a machine keeps itself current: channels a maintainer signs, a cache that holds what they point at, an update check that decides when, and an immune system that undoes a generation that does not come up. Until now the update check did the first three things and then stopped, on purpose — building is the daemon’s and activation is the stem’s, and a timer that could reach activation would be a timer with more authority than the owner. What was missing was the path from “now is the moment” to a generation, and the proof that a bad one is abandoned with nobody asked. ADR-0037 had just made the daemon’s build and the stem’s switch real on an installed machine; a channel was the next thing to ask of them.

Decision

A channel revision is a typed change like any other, proposed by the update check as the system, judged by the keeper, built by the daemon from what the cache holds, and left to the bootloader to keep or abandon.

A manifest carries, beside the revision and the built systems, the genome’s inputs as Nix locked them — tacidos, tacidos/nixpkgs, tacidos/impermanence, each with the hash of its tree. tacid-channel publish writes and signs it; tacid-channel cache-sign signs the cache the way Nix reads signatures, held in a test to a signature Nix itself made.
ChangeItem::ChannelAdvance { channel, revision, inputs } moves the genome’s lock file: the locked object of each named input becomes what the channel locked, and original — where the flake declares the input comes from — is not the channel’s to change. Nix follows a lock whose locked differs in kind from its original without a word and without a fetch, as long as the hash can be had; and a hash the cache answers for is had from the cache. So a machine moves to a revision through the same door its binaries come through, and the origin is never asked.
The pin is state of the agent’s region of the genome (mind/agent.state.json), rendered into the lock rather than the module. An undo reverts the commit, lock and all, and the revision before is built from a lock that names sources still in the store.
The update check proposes over the daemon’s socket with tx.propose — a method that takes a transaction and no model — and the daemon takes the principal from the socket, not the document: its own uid is the system, and nothing a person does. The keeper classifies the advance as reversible and boot-class: reversible because the generation before stays bootable, boot-class because a revision carries the keeper’s own binary, and a switch that restarts the keeper under its own feet is a switch nobody can gate.
Boot-class means boot counting. Every machine’s bootloader counts: a new entry gets three tries, a boot that reaches boot-complete.target is blessed, and that target requires the health gate — tacid-health, after the nervous system has had its chance to come up. A boot whose daemon does not start is not blessed; three of them, and the bootloader picks the generation before.
The channel’s cache is every machine’s first substituter, and its key is trusted by name, declared in tacid.core.update beside the manifest host and the keyring.

Findings

NixOS ships the blessing and never starts it. systemd-bless-boot.service and boot-complete.target are in every system’s unit directory, and nixpkgs’ boot counting writes the counters; but upstream starts the blessing from basic.target with a static link that NixOS’s generated unit directory does not carry. Counting alone would have abandoned every generation after three boots, however well it ran. The core module wants the service from basic.target, as upstream does. The packaged unit had also lost upstream’s condition on the bootloader’s counting variable, so a boot that was not counted — a blessed entry booted again, or a switch in the middle of one — ran the blessing and failed it, a failed unit on every second boot; the condition is declared beside the want. The first boot of an installed machine now reads Marked boot as 'good'. (Boot attempt counter is at 1.), after the health gate ran and passed.
A lock’s original and locked may disagree in kind. Pinning a github: input to a path: in the store leaves original as the flake declared it and locked as the pin, and nix flake metadata and nix build follow the lock without complaint or fetch. This is what makes a channel possible without a network to GitHub: the lock says github, the hash says which tree, and the cache has the tree.
Substitution comes before fetching, for every input scheme. A locked input with a hash is looked for in the store and the substituters before its fetcher runs, so a path: into a store that does not have it, and a github: the machine cannot reach, both arrive from the cache the same way. The test relies on it; so will every machine behind a firewall.
The update check’s revision lives in the genome. It had read a state file of its own; the revision a machine follows is a fact about its declaration, and the genome’s state is where it is now read from. One truth, in the repository the owner can read.
The bootloader’s entries have no names a generation can predict. NixOS names a boot entry by the hash of its content, and the counters ride on the name — nixos-<hash>+3.conf new, +2-1 after an unblessed boot, nixos-<hash>.conf once blessed — and its loader.conf prefers the default generation’s entry by that name over a glob of them all. The stem’s set-boot-default, bootctl set-default nixos-generation-N.conf, named an entry that had never existed; and a default set in the firmware’s variables would have been the wrong kind besides, since default ignores boot assessment and a machine told to boot a generation would have kept booting it after three failures. The operation is gone. What the keeper needs after a switch whose health gate fails is the profile back at the generation that was running and the system switched to it, and that is one operation now, restore: nix-env --switch-generation N and that generation’s own switch-to-configuration switch, which writes the preferred entry as part of what it does. Reading that path against the stem’s checks found a second thing: the keeper had been passing a profile link where the stem accepts only a store path, so the immune system’s own rollback would have been refused by the process it trusted to carry it out. The channel test reads the entries by the generation each names inside, and asserts the counters themselves: +3, +2-1, +1-2, +0-3, and the boot after that is the generation before.
The proxy refused the channel’s address for want of a port the scheme already names. The first full run of the test ended at the follower’s first boot: the health gate found the egress proxy not running, and the proxy had refused the test’s route to the channel, http://channel, because an upstream override had to carry a port. The scheme names one. An override is now a scheme, a host, and a port or the scheme’s own — with an IPv6 address’s own colons told apart from a port’s by its brackets — and the error names what it wants. Found by the immune system doing its job: a proxy that does not start is a machine whose health gate fails and whose boot is never blessed, which is what the run reported.
A directory bound in from /persist hides what NixOS writes there at activation. The same boot showed sshd failing every few seconds: /etc/ssh/sshd_config: No such file or directory. The service allow-list had declared /etc/ssh as what the SSH service keeps, and the bind mount of /persist/etc/ssh over it hid the sshd_config NixOS had just written into the root’s own /etc/ssh. The installation test’s machine had been failing the same way at every boot without the test looking. What sshd keeps is its host key, and it now lives in /persist/etc/ssh by its own path, made by sshd on the first boot; /etc/ssh itself is not persisted. The rule for the allow-list follows: a service persists the state it writes, never a directory NixOS writes into.
A store path written into a module’s text is a name, not a reference. The second full run reached the update check, which could not read its keyring: the test had pointed the follower’s tacid.core.update.keys at a path in the host’s store, interpolated into the text of the body module the machine carries. Evaluated from that text, on the host or on the machine, the path is a string with no store reference behind it, so nothing put the file into the follower’s closure, and the follower — with nothing in its store but that closure — had no such file. The keyring is what the option’s default says it is, a file in the genome; the machine carries the test’s copy as text, and the test writes it there as the owner, beside the body module, in the same commit. The rule generalises: what a genome declares must be reproducible from the genome’s text and the sources it pins, and a store path of another machine’s is neither.
A copy of a tree is not the tree a flake fetches. The third full run reached the daemon’s build, which could not find the revision’s source: path '/nix/store/…-source' does not exist, with the cache beside it holding that very tree. The test had made each revision with builtins.path, which scans a tree for store paths and keeps what it finds as references — and a real store path quoted in a test of the cache signer, inside this very tree, gave the copy a reference. A flake input is fetched without references, and Nix names a locked input’s store path from its hash alone; so the follower computed one name from the hash in its lock, asked the cache for it, and the cache had the tree under another. The revisions are now fetched the way a flake fetches them, builtins.fetchTree by hash, so the name in the lock, the name in the cache, and the name the follower computes are one; and the signer’s test writes its store hashes in halves, so no copy of this tree carries a reference. It is the lesson before it from the other side: a store path is a name for content, and two names for one content are a machine that cannot find what it has.
A change that did not take left the lock moved. Reading the daemon’s path for a failed build, while the fourth full run was under way, found that it put the declaration back and the lock not: saving a declaration only ever moves a lock forward, to what the declaration follows, so the genome of a machine whose new revision would not build said “I follow nothing” beside a lock that named the revision — and its next build, of anything, would have been of that revision. The daemon now keeps the lock as it stood and puts it back with the declaration, whether the build failed or the keeper rolled the activation back, and a test holds it to that: after a revision that will not evaluate, the lock, the declaration and the tree are as they were.
Persistence had followed the service list, and the genome went with it. The fourth and fifth full runs failed the follower’s own evaluation of its genome with “the fileSystems option does not specify your root file system”, while the same genome, lock and procedure built the revision on the host; the fifth run printed the genome the machine had, and it was not the installer’s: a daemon’s template, “written by a daemon that found no genome”, with no body/disks.nix and the minimal profile. The installer had written the genome under /persist; the machine had never bound it in. environment.persistence was switched on by a non-empty service list — a service with a directory to keep — and finding 7 had emptied that list on a headless machine, taking the genome, the state, the logs, the uid records and the machine id with it: every boot began on a root made again from its blank, with a daemon writing a template where its genome had been. Persistence now follows the root’s own declaration, tacid.core.root.ephemeral, and the test asserts the genome is the installer’s and bound in before the owner touches it. Two lessons: a guarantee that holds only while something unrelated is true is not one; and a machine that evaluates a genome should be asked to show it before anyone reads the error it produced.
An entry written before the machine had an id is renamed by the first boot that has one. The sixth full run reached the end of the update: the follower evaluated and built the revision from its own genome once the genome was where a boot keeps it, staged it, booted it and blessed it — and the first generation’s entry, blessed since the first boot, was counted again: +3. NixOS names an entry by the hash of its contents, and the contents carry the machine id; the installer had written the entries from inside the new root before the machine had one, systemd made one at the first boot, and the next bootloader installation — the revision’s — wrote the first generation’s entry with the id in it, under a new name, with a new counter, and removed the old. The machine’s id is now minted at birth by the keeper, carried in the installation the stem carries out, and written into the new root and where the machine keeps what survives a boot before the bootloader is installed, so the entries the installer writes are the entries the machine will bless.

Consequences

A machine that comes back on the generation before the one it staged says so in the journal at the keeper’s start and moves the profile back to what runs, through the stem’s restore operation, so that the default the bootloader reads is the generation the machine is. The keeper knows what it staged because it wrote it down beside the journal with the boot id of the boot that staged it, and knows a boot has happened because the id changed: a keeper restarted in the same boot leaves a staged revision staged, a staged revision that booted is left to boot counting, and a profile moved by hand has nothing to reconcile. Found by reading, while the sixth run was under way: boot counting had brought the follower back and left its profile naming the revision it would not boot.
tacid-channel is the maintainer’s side of an update, and the test’s: keys, keyrings, a signed manifest, a signed cache. The release pipeline will call it; the channel test does.
The update test is the first with three nodes and a network: a channel, the live installer, and a follower the installer puts on an encrypted disk with nothing in its store but its own closure, which ends the test running a revision it fetched from the channel — after a revision that could not start was tried, counted, and abandoned. It is installed rather than imaged because the image builder needs hardware virtualisation to install a bootloader and the test runs where there is none; the twenty-five minutes it costs prove the task as it was set, a fresh install that updates itself.
Every Tacid machine’s boot is assessed. A machine whose health gate fails three boots in a row falls back to the generation before; a machine with one generation keeps booting it. What the gate checks at boot is what it checks after a switch: the nervous system’s units.
The stem has one operation fewer and one that is true: restore moves the profile back to a generation it has and switches to it, and nothing in Tacid names a bootloader entry.
The test’s revisions are made from the part of this tree a machine evaluates — the flake, the modules, the profiles, the crates — and not from the tests, the documents or the hosts, so a revision, and every system and cache made from it, is the same tree until something a machine evaluates changes; an edit to the test does not rebuild what the test is about to boot, which it had been doing, at three compilations of the package a time.
Still to come: a real channel, with a real cache behind a CDN, and the revision naming that CI gives it. The tool that publishes it exists; the pipeline that runs it does not yet.

Alternatives rejected

Let the update check build and activate. The shortest path, and a second authority. The proposal path is longer by one socket and gives the keeper’s classifier, the journal, the approval path and the undo to updates, which no timer should be without.
Move the flake’s declaration to the channel’s URL instead of the lock. A flake.nix that changes at every update is a genome whose history is noise, and a declaration that stops being portable the day it names a store path.
Switch to a channel revision live and gate it with the keeper. The keeper would be restarted by the switch it was gating. Boot counting is the gate that survives the change it judges.
Health-gate boots with something lighter than the nervous system’s units. A boot whose daemon is dead is a machine with no agency, which is the one thing this project is for.