AXIS Documentation / Operations / Failure & Recovery

Failure & Recovery

VerifiedFail-closed by design

AXIS is designed to fail closed: when evidence, policy or the database is not trustworthy, protected work does not proceed. Recovery is deliberate and evidence-visible.

Fail-closed behaviors

Situation	Behavior
Audit WAL cannot be written	Protected writes do not execute; `audit_wal_failure`; health non-ok
WAL corruption detected at startup	Startup blocked; corruption surfaced with `audit_corruption_detected`; never silently repaired
Manifest missing / policy hash mismatch	No permissive fallback; healthy startup requires the full chain
Parser failure or unsupported shape	Controlled rejection; never a silent read assumption
Database timeout after dispatch	`execution_state: unknown`; no automatic retry
Extended-protocol / COPY / batch attempts	Fail-closed rejection (wire path and HTTP batch)

Restart procedure

Stop accepting new traffic.

Verify evidence before restart: GET /evidence/verify; resolve any corruption reports first.

Start AXIS; startup re-validates WAL continuity, manifest, policy hash and the dry-run corpus.

Confirm /health is ok and /policy/status shows the expected version.

Re-run /evidence/verify and spot-check recent audit events.

Recovering from execution_state: unknown

Identify the request via error_id in the audit trail and application logs.
Check execution_state and result evidence for the event; check the database for the row state.
If the write applied: do not re-run; record the resolution in operations notes.
If the write did not apply: re-submit with a new request; the retry is a new audited event.
Never auto-retry protected writes.

Corruption response

Detection is surfaced, never silently repaired: audit_corruption_detected events, verification reports.
Stop protected traffic until the evidence store is restored from a verified backup.
After restore, run /evidence/verify; compare bundle first/last hashes against out-of-band copies if available.

Session and approval state across restarts

Prepared-statement sessions are in-memory: after restart, EXECUTE without a session fails safe as unresolved.
Approvals persist in the local store (AXIS_APPROVAL_DB_PATH); expired approvals do not execute.
Approvals do not expire on policy rollback; a retry re-evaluates under the then-active policy.

Disaster recovery

Restore order: database → AXIS evidence (WAL + JSONL + approvals) → policies → Control Plane.
Evidence bundles are the out-of-band verification artifact; keep at least one copy off-box.
Validate restored state end-to-end before reopening protected writes.

Practice the recovery drill in shadow mode first: restart, verify, restore. The steps are identical in enforce mode but the stakes are higher.

Related: Transactions · Troubleshooting · Operations