← Developers

04.2 — Playbook: incident response

Spec matching SPEC-DX-01

The Invariant

Diagnose at the boundary where the error is emitted — never upstream by guessing. Every failure in Axiom-Trace has a stable code and a layer that owns it. The playbook is: read the code, identify the layer (auth / FSM / lock / channel), fix the cause — never retry-blast a 4xx.

The Contract

The error-code map — the whole vocabulary of failure:

HTTP Code Layer Meaning
400 ACTION_FAILED, BAD_REQUEST manifest constraint/validation — fix the call
401 AUTH_REQUIRED, AUTH_FAILED, TENANT_AMBIGUOUS auth credential problem — fix the credential
403 ACTION_FAILED (RBAC), EDGE_CAP_EXCEEDED, SCOPE_INSUFFICIENT, EDGE_SIGNATURE_REQUIRED, LICENSE_SUSPENDED manifest / license / CORS structural — won’t heal itself
409 ACTION_FAILED (FSM), PIN_AMBIGUOUS, IDEMPOTENCY_IN_PROGRESS FSM / auth / idempotency state or ambiguity — refresh and retry
413 BATCH_TOO_LARGE sync >1000 events — split
429 RATE_LIMITED rate limit honor Retry-After
503 DRAINING, NO_ENGINE, STRIPE_NOT_CONFIGURED lifecycle / config wait or configure

The Flow

Triage order — cheapest check first:
 1. read `code` in the response body        (never just the status)
 2. auth?      → session TTL / role / scope
 3. contract?  → FSM state, constraint rules, required payload fields
 4. channel?   → outbox depth edge-side, ingest acks, last_poll_at
 5. physical?  → container logs, disk, postgres health

Fail-closed walkthrough — the most common “stuck” report:

ACCEPT_LOT → 400 ACTION_FAILED (peso_coerente)
  → the weight delta is outside ±0.5% — the constraint is doing its job
  → REJECT_LOT the lot (role MAGAZZINIERE) or fix the payload — never bypass

Payload Example

# Check the edge's heartbeat + sync health from the cloud
curl -s https://cp.axiom-trace.eu/api/v1/tenant/usage -H "Authorization: Bearer $PS" \
  | jq '{edges: [.edges[] | {node_id, last_poll_at}]}'

# Edge-side: is the outbox draining?
curl -s http://edge:8291/api/v1/outbox -H "Authorization: Bearer $TOKEN" | jq .
docker logs axiom-edge --tail 50

Known Failures

Report Most likely cause → first action
“Dashboard shows nothing” Edge offline or outbox backed up → last_poll_at, container logs
“Facts duplicated” Idempotency key regenerated per retry → fix client; dedup already saved you on event id
“Action rejected that should pass” A pre_commit_rule or FSM transition — read the error message, it names the rule
“Notifications silent” AXIOM_CP_MAIL_MODE=log or outbox backed up → GET /admin/notifications, worker logs
“Everything 401 at once” Key rotation after re-claim → re-provision, don’t retry the dead key