AXIS Documentation / Operations / Troubleshooting

Troubleshooting

VerifiedSymptom → Cause → Verify → Resolve

Each entry follows the same structure: Symptom, Cause, Verify, Resolution. Start with GET /health, GET /runtime/stats and GET /logs, then use the audit trail for decision-level evidence.

Startup and health

Symptom AXIS container restarts or /health does not return status: ok.

CauseThe startup chain failed: WAL continuity, manifest validation, policy hash mismatch, policy validation, or activation dry-run corpus failure.

VerifyContainer logs; /policy/status; /evidence/verify; check manifest SHA-256 against the policy file bytes.

ResolveFix the failing component, then restart. There is no permissive fallback: the gate stays closed until the chain passes.

Symptom audit_corruption_detected or verification reports corruption.

CauseTampering, partial writes, or manual edits to the WAL/JSONL; hash chain broken.

VerifyGET /evidence/verify; compare event hashes and first/last bundle hashes against out-of-band copies.

ResolveStop protected traffic; restore evidence from a verified backup; re-verify. Never let AXIS silently repair corruption.

Policy

Symptom POST /query returns policy_not_loaded (503).

CausePolicy failed to load or manifest invalid; AXIS_POLICY_DIR/AXIS_POLICY_MANIFEST misconfigured.

VerifyGET /policy/status; container logs; manifest path and active policy file existence.

ResolveFix path or policy file; restart. Startup will not open traffic until the chain passes.

Symptom policy_activation_failed (409) on activate.

Causeexpected_hash does not match the candidate's stored file hash or the manifest, or re-validation failed.

VerifyPOST /policy/diff; recompute SHA-256 of the candidate file; compare with the manifest.

ResolveCorrect the hash, re-create the candidate, retry activation. On failure the previous active policy remains.

Symptom Dry-run says BLOCK for SQL that should be allowed.

CauseClassification is stricter than intended (e.g. SELECT INTO, data-modifying CTE, volatile function), or the rule match misses.

VerifyPOST /policy/dry-run response details: QueryType, risk signals, rule trace; POST /policy/validate warnings.

ResolveAdjust the rule (never weaken classification); validate, diff, create candidate, activate.

Approvals

Symptom Retry with approval_id returns approval_expired (410) or approval_rejected (409).

CauseTTL elapsed before retry, or the approval was resolved as reject.

VerifyGET /approvals/<approval_id> shows status and resolved_at.

ResolveSubmit a new request to create a new approval; resolution is immutable.

Symptom Resolve returns unauthorized_operator (401/403).

CauseMissing or wrong AXIS_OPERATOR_TOKEN; header/format mismatch.

VerifyGET /policy/status → operator_auth_enabled; check the token header name (X-AXIS-Operator-Token or Authorization: Bearer).

ResolveSet/rotate the token; retry with the correct header.

Database

Symptom db_connection_failed (502) or db_unavailable (503).

CausePostgreSQL unreachable; credentials wrong; axis_executor role missing; network segment blocked.

Verifydocker compose ps; check DB logs; test with psql using the same URL.

ResolveFix connectivity/credentials; restart AXIS after the DB is healthy.

Symptom db_query_timeout (504) and execution_state: unknown.

CauseQuery exceeded AXIS_DB_QUERY_TIMEOUT_MS; slow lock, large scan, or contention.

VerifyAudit event for the request; database pg_stat_activity for the executor session.

ResolveCheck whether the write applied; do not auto-retry. Re-submit explicitly only after confirming state.

Rate limits and size limits

Symptom rate_limit_exceeded (429) under bursty load.

CauseAXIS_RATE_LIMIT_REQUESTS_PER_MINUTE (120) or AXIS_RATE_LIMIT_BURST (30) exceeded.

VerifyGET /runtime/stats; log events with rate_limit_hit.

ResolveBack off client-side; if legitimate, raise limits deliberately (monitored).

Symptom oversized_request (413) / oversized_sql (413).

CauseBody > 1 MB or SQL > 256 KB (AXIS_MAX_BODY_BYTES, AXIS_MAX_SQL_BYTES).

VerifyRequest size in logs; limit values in configuration.

ResolveChunk the work into smaller statements; adjust limits only with justification.

Control Plane

Symptom Control Plane shows data but AXIS is unreachable or shows stale data.

CauseMock mode enabled (demo-only), or AXIS_BACKEND_URL wrong / unreachable from the Control Plane container.

VerifyControl Plane logs; curl the backend URL from inside the control-plane container; check the mock flag.

ResolveDisable mock mode for real data; fix AXIS_BACKEND_URL. Mock mode is never production evidence.

Prepared statements

Symptom EXECUTE returns session_not_found (404) or prepared_statement_not_found.

CauseMissing session_id, statement prepared in a different session, or sessions lost after restart (in-memory store).

VerifyCheck the session_id in request and audit events; check AXIS uptime.

ResolveRe-issue PREPARE in the same session; treat restart as session loss (fail-closed by design).

Wire path (lab)

Symptom psql/driver cannot connect to the pgwire listener.

CauseListener disabled (AXIS_PGWIRE_ENABLED=false default), wrong port, or lab mode off.

VerifyEnvironment: AXIS_PGWIRE_ENABLED=true, AXIS_PGWIRE_LISTEN_ADDR (default 0.0.0.0:6544), AXIS_PGWIRE_LAB_MODE=true.

ResolveEnable the lab listener explicitly; remember it is experimental and disabled by default.

Symptom Prepared statements fail on the wire path.

CauseExtended Query (P/B/E) is planned, not implemented; clients fail closed.

VerifyClient error; RFC-032/EQ roadmap status in the changelog.

ResolveUse Simple Query or the HTTP path; do not use the wire POC for production traffic.

Related: Error Codes · Failure & Recovery · Operations