63. The keeper does not hold itself while the stem is working

Status: accepted. Fixes the defect recorded in ADR-0062’s consequences, and makes good on a property ADR-0004 has always claimed.

Context

The process model says the daemon requests, the keeper authorises, the stem executes. That split was real in space — three users, three sockets, three sets of permissions — and it was not real in time. Keeper::collect took the keeper’s lock, asked the stem to collect the store, and waited there with the lock in its hand.

Every part of that wait is unyielding. SocketStem::call is a blocking UnixStream with no timeout. KeeperService::dispatch is a synchronous function called from inside an asynchronous serve_connection. The keeper’s runtime is #[tokio::main(flavor = "current_thread")]. So for the length of a collection the keeper held three things at once: its own lock, its only runtime thread, and therefore its socket’s entire attention. It accepted no connection, read no request, and answered nobody — not the daemon, not tacid approve, not a person at a console.

Two things make this worse than an ordinary bug.

Nothing has to fail for it to happen. tacid-collect.service asks for a collection on the owner’s own schedule, weekly by default. On a machine that has been running for months a store collection is minutes. That is minutes a week of a nervous system that is up, healthy, correct, and unreachable — with systemd rightly reporting the unit active the whole time, because it is.

It is the shape ADR-0062 closed at one end and named as open at the other. That record made “started” mean “can be reached” and bounded every unbounded wait on the boot path, and then said plainly that a daemon which binds, says READY=1 and then wedges is invisible to the manager exactly as before. This was found by reading for that case. It is not a wedge — the keeper comes back — but for as long as it lasts it is indistinguishable from one to everything that depends on it.

Decision

1. Blocking work goes on the blocking pool. dispatch is synchronous and does not pretend otherwise; serve_connection now hands it to tokio::task::spawn_blocking instead of running it on the runtime’s thread. The accept loop keeps accepting, every other connection keeps reading and writing, and the cost is cloning four Arcs per request.

2. A dispatch that panics is answered rather than dropped. It became visible only because spawn_blocking returns a JoinError where a direct call would have unwound the connection task in silence. A client left waiting for a reply that is never coming is worse off than one told the keeper could not manage this request.

3. The long operation is split into deciding, doing and writing down, and only the first and the last hold the keeper. Keeper::plan_collection reads the retention policy and returns a Collection; Keeper::stem hands out the client; the call happens with nothing locked; Keeper::record_collection writes down what came back. What the stem is asked for is still the keeper’s word — Collection::asking builds the operation — so the caller in the middle carries a decision rather than making one. That is the line worth keeping: the keeper decides and remembers, and the gap between those is not a decision.

4. The collection only, for now. It is the one whose middle is long enough to matter. activate and the two snapshot operations have the same shape and are shorter, and they are worth the same treatment when there is a reason beyond symmetry.

Consequences

• The keeper answers other clients while the stem works. There is a test that says so, and it fails without the change rather than hanging: the clients in it are blocking sockets with read timeouts, on threads of their own, because a test whose regression mode is a hung suite is a test nobody thanks you for.
• The lock is dropped and retaken around the stem call, so the keeper’s state can change in between. For the collection that is harmless — what was decided is a list of generation numbers, and the stem checks its own preconditions — but it is a real property and the next operation split this way needs the same question asked of it.
• The record is written against the moment the collection was decided rather than the moment the stem finished. That is deliberate: what is being written down is a decision, and the journal is a record of decisions.
• Keeper::stem is public now. Nothing is given away by it: the client carries this boot’s token and the stem refuses every caller but the keeper’s own uid, so a copy of it elsewhere is a copy of nothing.

Alternatives rejected

A multi-threaded runtime. It moves the number of clients one long call can freeze from all of them to all but a handful, and running blocking work on the runtime is what the blocking pool exists to avoid whatever the flavour. It also would not have touched the lock, which is the half that matters.

A read timeout on the stem client. The operations differ by orders of magnitude: a seal is milliseconds, an activation a minute, a collection an hour. One number is wrong for all of them and a number per operation is a table that goes stale. Worse, a timeout on a collection that is working is a collection abandoned in the middle, which is not an improvement on waiting.

An asynchronous stem client. The right long answer, and a much larger change: the trait has three stand-ins used throughout the keeper’s tests, and every Keeper method is synchronous. The blocking pool buys the same property today without making every caller asynchronous, and it does not stand in the way of doing this later.