TacidOS
DevelopersGitHub

Decision 0027

VM tests, stub model, cassettes, eval set, injection corpus, fuzzing, hardware lab

Architecture decision record 0027: VM tests, stub model, cassettes, eval set, injection corpus, fuzzing, hardware lab

Status
Accepted
Date
2026-09-03

Context

An agent loop that can only be tested with a live model cannot be tested in CI. Security claims without adversarial tests are claims.

The three plans

  • Grok: a hardware-less integration harness; a VM that boots, starts the agent against a stub model and drives the TUI; a fixed eval set; published metrics.
  • Fable: NixOS VM tests for the loop and one per allow-listed service; the three GPU reference machines as the physical matrix.
  • Codex: a VM test matrix; a physical CI lab; an embedded bench with a USB relay; quality gates per release; fuzzing implied by “auditable by security researchers”.

Decision

NixOS VM tests for every module and for the full loop; a scripted deterministic stub model; record/replay cassettes of real providers; a tool-use eval set scored per model (install a service, edit a file, explain a journal, diagnose a unit, flash a board); a prompt-injection corpus as a CI gate with zero side effects required at level 2; fuzzing of the stem protocol and the scrubber; permission regression tests; an OVMF + swtpm harness for Secure Boot and TPM paths; budgets measured in CI; a physical hardware lab and an embedded bench later. Supply chain: signed tags, SLSA provenance, SBOMs, reproducibility checks on the core.

Consequences

  • The classifier and the injection corpus exist before the agent loop is trusted (Phase 1).
  • A release without the gates is not a release.

Alternatives rejected

  • Manual testing on the maintainer’s laptop: how regressions ship.