TacidOS
DevelopersGitHub

Decision 0016

Sandboxed local engines behind traits; bring-your-own-key providers; no default

Architecture decision record 0016: Sandboxed local engines behind traits; bring-your-own-key providers; no default

Status
Accepted
Date
2026-09-03

Context

The mind is portable. Local models must be good enough that the owner does not immediately export an API key, and cloud models must be reachable without the agent ever holding the key.

The three plans

  • Grok: offline by construction; llama.cpp, whisper.cpp, Piper; profiles tiny/core/lab; a small model fine-tuned on Tacid’s tools beats a big generic one; cloud off by default with a NET LLM badge.
  • Fable: local for STT, classification and small edits; frontier cloud for planning and code; Anthropic, OpenAI and xAI via a first-run picker, no silent default; CUDA/ROCm/Vulkan.
  • Codex: a runtime abstraction with llama.cpp-class inference as the floor; native provider adapters, not a lowest common denominator; bring your own key; privacy profiles; cost visibility.

Decision

Local engines (llama-server, whisper-server, a Kokoro TTS server; mistral.rs later) run as sandboxed subprocesses behind Rust Backend, Stt, Tts and Embed traits: read-only model files, seccomp, no network, embedded chat templates ignored (they parse hostile files; review). Backends: CUDA on NVIDIA; Vulkan by default on AMD with ROCm opt-in (the ROCm closure is large and often broken; review); Vulkan on Intel; CPU everywhere. Constrained decoding (JSON Schema → GBNF) so small models call tools reliably. A signed registry (models/registry.toml) with license, source, hashes, sizes, RAM/VRAM, backends and tasks; profiles tiny/core/lab sized from detected hardware; weights never in the netinstall ISO.

Cloud: bring your own key, no default provider, keys in the keeper and injected by the egress proxy. Two adapters first, the Anthropic Messages API and the OpenAI-compatible family (OpenAI, xAI, Google’s compatibility endpoint, OpenRouter, llama-server); native adapters follow when a provider’s capabilities diverge enough to matter; the trait stays rich. Privacy class derives from provenance and path policy, never from a classifier. Badge and ledger on every request; cost shown and never claimed authoritative. Offline is a state, not a failure.

Consequences

  • Five native adapters on day one would be five APIs for one maintainer; two cover almost all.
  • Resource governance: eviction on memory pressure; VRAM yielded to owner workloads.
  • Model choice is a registry entry, never an architectural constant.

Alternatives rejected

  • Ollama as the architecture: a product dependency; it can be a registry source.
  • A Tacid-hosted inference gateway: a business, an account, a liability.
  • Linking inference in-process: hostile parsers inside the agent’s address space.