0070. A daemon that bound and then wedged can only be caught from outside

Status: accepted. Closes the case ADR-0062 left open, and is only safe because of ADR-0063.

Context

ADR-0062 fixed “started” meaning “spawned”: the daemons are Type=notify and say READY=1 after they bind, so an After= on one waits for a door that is open rather than a process that exists. It named what that leaves: a daemon that binds, says READY=1, and then wedges.

The health gate calls such a daemon running, and is right to. It reads the kernel’s own accounting — a unit is running when its cgroup holds a process — which is the correct answer to the question it can ask. It replaced a worse one: /run/systemd/units keeps an invocation symlink for a unit that failed, so a list read from that directory called a dead daemon running. Moving to cgroups fixed “has a file” being mistaken for “is alive”. It cannot fix “has a process” being mistaken for “is answering”.

And nothing inside this machine can. Keeper::gate runs in two places: tacid-health.service at boot, and inside the keeper after an activation. In the second, the keeper is the process running the gate — so a keeper probing its own socket is either a deadlock or a tautology, since a probe that succeeds proves the runtime is free, which it demonstrably is not. A liveness probe means something only when the prober is not the probed. The keeper can probe the stem and the proxy. Nothing in the nervous system can probe the keeper.

The thing outside it is the service manager.

Decision

WatchdogSec = 180 on tacid-keeper and tacid-egress, pinged every ninety seconds by a task on each daemon’s own runtime.

The runtime is the point. What a ping proves is not that the process exists — the manager can see that, and so can the gate — but that the loop which answers connections is still turning. A ping sent from a thread that could not notice the process being stuck says nothing, expensively.

Three minutes, and generously. This machine emulates its processor; a working daemon under load must never be killed for being slow. The case being caught is a permanent wedge, not a slow minute. tokio::time::interval with MissedTickBehavior::Skip, because a burst of datagrams after a stall says the same thing one does and the manager has already decided by then.

Restart = "on-failure" beside it. A watchdog with nothing after it is a kill switch. The socket is inherited again on restart, so a keeper that comes back is a machine that recovers by itself rather than one waiting for somebody to notice.

Two daemons, and the record says why not the others.

• tacid-keeper is safe because of ADR-0063, and would not have been before it. A store collection used to hold the keeper’s only runtime thread, so a ping task could not run and the manager would have killed a keeper that was working. The watchdog now also catches a regression of that bug, which is the argument for putting it there rather than only on the proxy.
• tacid-egress is async throughout.
• tacid-stem does not get one. It serves inline and holds its runtime for the length of an activation — bounded, and long. A watchdog there would fire on a machine doing exactly what it was asked.
• tacidd does not get one either, and the reason is weaker and stated as such: nobody has audited its loop for a call that blocks it. Adding a watchdog to a loop nobody has read is how a working daemon gets killed on a timer.

WATCHDOG_USEC is read for the period rather than the number being written twice, so the unit is the single source of truth. WATCHDOG_PID is honoured: a manager watching only a unit’s main process says so that way, and a process that is not the one being watched says nothing.

Consequences

• The interval is derived by interval_from, which takes the two values rather than reading them, for the reason tell takes an address rather than reading one: this workspace forbids unsafe, tests included, and setting an environment variable in a process is exactly that. Splitting the reading from the deciding costs three lines and buys a test of every way the deciding can be wrong — including the whitespace case, since a value arriving with a newline on it silently turning the watchdog off is the failure that looks exactly like success.
• tell sends a message rather than a constant, and there is a test for each word. Sending the wrong bytes for WATCHDOG=1 would be a working daemon killed and restarted for ever on a timer, which is a worse machine than one with no watchdog at all.
• A keeper that exits because its journal does not verify will now be restarted rather than left dead. It fails again, and systemd’s start limit stops it after a few tries, leaving a failed unit — which is the same end state, reached noisily instead of silently.
• The blind spot is narrowed and not closed. A daemon whose runtime turns while its work is stuck — a task looping without progress — still pings. Catching that means the manager asking a question about the work rather than about the loop, which is a health check and not a watchdog, and it is not built.

Alternatives rejected

Have the health gate probe each socket. The obvious fix, and it cannot work for the keeper for the reason in the context above. It would work for the stem and the proxy, and buys nothing there that the manager does not already give — and it would put a connection attempt in the path of an activation’s health gate, where a timeout is a good generation rolled back.

A shorter period, so a wedge is caught sooner. Every second shaved is a second closer to killing a working daemon on a loaded machine, and the thing being caught does not resolve itself with time. Three minutes of a wedged keeper is a bad three minutes; three minutes of a slow one killed at ninety seconds is a rollback.

A watchdog on every daemon. Two of the four have loops nobody has established are free to run a timer. Uniformity here would be a decision made for the shape of the table rather than for the machines.