First-hand information for everyone
This paper describes the Unfireable Safety Kernel, a separate safety module that runs outside an AI agent so the agent cannot bypass it. The authors argue that putting controls inside an agent’s own process—system prompts, output filters, and guardrail libraries—leaves those controls reachable and changeable by the agent. Their kernel is meant to be an architectural control: a last line of defense that checks and authorizes consequential actions as they happen.
The researchers first set out four properties an authorization layer must have to be truly out of reach of a controlled system. These are process separation (the safety code runs in a different process), pre-action enforcement on a structurally-only path (all actions must go through the kernel before they happen), fail-closed at both the request and system levels (if something goes wrong, the default is to refuse the action), and externalized signed evidence verifiable outside the controlled system’s trust boundary (audit records are cryptographically signed so others can check them). They call this layer execution-time AI alignment, distinct from training-time alignment methods such as Reinforcement Learning from Human Feedback (RLHF) or Constitutional AI, and from inference-time tools like prompts and filters.
They implemented a reference kernel in Rust. The design includes a four-seam defense-in-depth architecture and a “no-kernel, no-agent” lifecycle rule so the agent will not start without a reachable kernel. The system records decisions in an append-only transparency log signed with Ed25519 (a cryptographic signature scheme) under an operator key the kernel does not hold. The authors also applied formal checks: a mathematical check called an SMT theorem using the Z3 solver, and an exhaustive bounded-model check of the production decision function using the Kani tool (four verification harnesses, all passed). A disciplined migration from Python to Rust was validated by exact byte-equivalence on 1,000 deterministic fixtures and identical reject behavior for 17 adversarial test classes. The reference code is available under an Apache-2.0 license on GitHub.
To test the kernel they ran it governing a live, escapable AI system: a deterministic, self-improving world model with a purpose-built escape-seeking adversary. Over 1,000 attempted self-modifications, all 704 attempts that targeted the safety-critical core were refused and did not succeed. An additional 300 modification attempts made while an operator kill switch was in effect were also refused. In a separate campaign of 6,240 authorization round-trips, 6,038 were served by the compiled Rust kernel over HTTP and the authors recorded no successful bypasses. They also compared the architecture to three contemporary systems that claim control-plane functions (GalileoAgentControl, Microsoft Agent Governance Toolkit, and Microsoft Authorization Fabric) and note a key difference: in those comparators the agent invokes the control; with the Unfireable Safety Kernel the agent does not have that choice.
Why this matters: if an agent can change or disable its own safety code, human supervisors can lose control over consequential actions. The kernel provides an architectural way to preserve human authority even when an agent is misaligned, compromised, or deceptively cooperative. It aims to make authorization decisions reliably unreachable from the agent itself and to produce auditable, cryptographically verifiable records of those decisions.
The paper also lists important caveats. The architecture assumes the operator’s deployment infrastructure (the host, the network seams, and systems holding operator keys) is trusted in the usual operational sense. The kernel does not prevent prompt injection upstream of the kernel, does not attempt to filter model text outputs, and known mitigations for side-channel leaks are not yet implemented. A denial-of-service attack against the kernel would cause an outage; the fail-closed behavior turns such attacks into safe refusals but does not remove the operational impact. Finally, misuse of the operator key by an insider is detectable but not prevented by the current design; multi‑party key schemes are planned but not yet implemented.