AD‑12 [OP] — Operational Handoff & Runbooks

“Operational” here means falsifiable procedures and artefacts: a standard ops team can run SRF kernel safely without the original dyad.

Aliases: None Cluster: Governance Depends on: AD‑08 [SO], AD‑09 [WB], AD‑11 [RA], AD‑18 [SE], AD‑19 [CI], AD‑20 [DP], AD‑21 [RM]

0. For humans

What question does this answer?

“If the original development team is unavailable, can a normal SRE/safety/ops team keep this kernel safe, audited, and up to spec?”

AD‑12’s answer:

• Define minimal roles & RACI, so you know who is allowed to touch what.
• Ship a Handoff Packet with thresholds, op‑points, runbooks, dashboards, contact tree, commit_watermark policy.
• Provide runbooks so incidents, rollbacks, gate flaps, and jurisdictional breaches have pre‑agreed procedures.
• Define ops SLOs (halt ≤ 5 tokens post trip; ACK_HOLD < 300 ms; precision‑parity drift ≤ band B).
• Make sure there is a clean MKP fallback path any ops team can invoke.

How do I know it’s live?

You can point to:

1. A Handoff Packet on disk / in your repo with: - thresholds.yaml, op_points.yaml; - incident / rollback / gate‑flap / breach‑notif runbooks; - dashboard URLs & alert routing; - contact tree and commit_watermark policy.
2. An Ops Serviceability Gate checklist used before any deployment: - challenge pass‑rate ≥ target; PR slope stable; - no unreviewed incidents; - kernel profile & thresholds recorded.
3. Runbooks actually wired into alerts: - INCIDENT, ROLLBACK, GATE FLAP, BREACH NOTIF all exist and are linked from your alerting system.
4. A two‑person rule on PEL/threshold changes: - no single operator can change PEL packs or safety thresholds unilaterally.

Where do I see it?

• In your runbook repo / wiki: AD‑12‑aligned runbooks and the Handoff Packet directory.
• In FHR and config logs:
• kernel_profile, runtime_mode, active_ring_ids[], op_point_id, timing_op_point_id, commit_watermark, legal_hold etc., matching what the runbooks assume.
• In pre‑deployment checklists: the Ops Serviceability Gate section.

Quick checklist

• Implemented if: (1) Handoff Packet on disk with thresholds, op‑points, runbooks, dashboards, contact tree; (2) Ops Serviceability Gate checklist used before deployment; (3) runbooks wired into alerts; (4) two‑person rule on PEL/threshold changes
• Key artefacts: thresholds.yaml, op_points.yaml, incident/rollback/gate‑flap/breach runbooks, contact tree
• Key FHR fields: kernel_profile, runtime_mode, active_ring_ids[], op_point_id, timing_op_point_id, commit_watermark, legal_hold
• Key failure codes: INCIDENT, ROLLBACK, GATE_FLAP, BREACH_NOTIF
• Escalation path: alert → runbook → Ops Lead + Safety Steward → Legal if jurisdictional

1. Purpose & scope

AD‑12 exists so that:

• The kernel is operationally deployable, not just a research artefact.
• A standard ops team (SRE + safety + legal contact) can:
• handle incidents,
• roll back safely,
• manage PEL / thresholds under a two‑person rule,
• decide when to drop to MKP for safety.

It is intentionally procedural and complements:

• AD‑11 [RA]: external audit / replication.
• AD‑08 [SO]: FHR schema & Serviceability Gate.
• AD‑20/21: PEL and retention / legal‑hold.

2. Roles & RACI (minimal)

Minimal roles:

• Ops Lead / SRE
• Owns incident, rollback, and gate flap runbooks.
• Owns dashboards and alert routing.
• Safety Steward
• Owns BREACH NOTIF runbook and interface with legal/compliance.
• Co‑approves PEL / threshold changes (two‑person rule).
• Legal / Data Protection Contact
• Owns legal‑hold policies and jurisdictional notification flows.
• Co‑owns retention profile decisions (AD‑20/21).
• Owner of the Spec (Author / Architect)
• Not required for day‑to‑day ops once handoff is complete; AD‑12 explicitly aims to make them non‑critical.

RACI principle: PEL packs / thresholds / kernel profile changes require at least two roles (Ops + Safety or Ops + Legal).

3. Handoff Packet

3.1 Contents (must‑have)

• Configuration artefacts
• thresholds.yaml — per‑metric thresholds and SESOI, including:
  • Δ‑triad bands;
  • ROC/PR op‑points;
  • timing MI/χ²;
  • drift alarms.
• op_points.yaml — mapping from scenario strata → op_point_id used in the Acceptance Matrix.
• Runbooks (this AD)
• INCIDENT, ROLLBACK, GATE FLAP, BREACH NOTIF (details in §4).
• Dashboards & metrics
• URLs / IDs for:
  • FHR overview;
  • timing & covert channels;
  • watchdog & halt tokens;
  • RA / replication;
  • PEL & legal‑hold status.
• Contact tree
• On‑call Ops;
• Safety steward;
• Legal / DPO;
• escalation order and target response times.
• Commit watermark policy
• Defines how far back TURN‑ACID rollbacks are allowed by default (turn‑level vs session‑level) and who may override.

3.2 Optional but recommended

• Example Verifier’s Bundles and Ops Serviceability Gate checklist.

4. Runbooks

4.1 INCIDENT runbook

Trigger:

• Any alert with failure_codes including EXFIL, TIMING_COVERT_CHANNEL, WATCHDOG_TRIP, PEL breach, or other G1/H1 gate violation.

Steps (high‑level):

1. Classify - Use incident logs + FHR excerpt (AD‑08) to classify:
   • kernel profile (kernel_profile);
   • scenario (SCN‑ID if synthetic);
   • gates that fired (gates_snapshot);
   • severity level.
2. Contain - Optionally switch profile to MKP (Minimal Kernel Profile) for affected tenants / domains. - Optionally disable specific tools or routes.
3. Notify - Follow contact tree:
   • Ops Lead;
   • Safety steward;
   • Legal if jurisdiction/breach triggered.
4. Record - Log incident ID, FHR anchors (evid_chain_anchor_id, duty_ledger_id), and decision in the duty ledger.

4.2 ROLLBACK runbook

Trigger:

• Incident classified as requiring rollback or regression in metrics (e.g. PR slope instability, drift beyond SESOI).

Modes:

• Turn‑level rollback
• If the turn is still uncommitted, use commit_watermark to discard its kernel-owned working state;
• if it was already committed, append a compensating state transition that restores the last safe kernel-owned state without deleting the original ledger/FHR entry;
• any re-run uses a newly authorized ticket and operation record under stricter op-points or an explicitly approved profile change.
• Session‑level rollback
• Restore kernel-owned session state through append-only compensating transitions from the last safe checkpoint;
• mark affected FHR rows as superseded through new linked records while preserving the originals in the evidentiary chain (never delete or rewrite them).

Procedure:

1. Locate the last safe commit_watermark (turn‑ or session‑level).
2. Roll back uncommitted kernel-owned state or append the required compensating state transition. External systems roll back only where the bound tool manifest declares a verified rollback; otherwise invoke a declared compensation where authorized or record and escalate the irreversible partial effect.
3. Reconcile every affected durable operation and effect receipt, then update the duty ledger and FHR with linked records describing the rollback or compensation.

4.3 GATE FLAP runbook

Trigger:

• Repeated gate open/close near threshold (e.g. noisy Δp, MI, or timing metrics), leading to noisy behaviour or alert fatigue.

Purpose:

• Adjust debounce thresholds without weakening safety, to keep gates stable.

Steps:

1. Identify which gates are flapping (from FHR gates_snapshot patterns).
2. Check Acceptance Matrix / SAP for the intended op‑point and SESOI on those metrics.
3. Propose debounce change (e.g. require K consecutive samples over threshold; extend window).
4. Apply two‑person rule (Ops + Safety) for any threshold change.
5. Record changes in thresholds.yaml and version them; make sure FHR references the new op_point_id / timing_op_point_id explicitly.

4.4 BREACH NOTIF (jurisdiction) runbook

Trigger:

• PEL incident requiring jurisdictional notification:
• e.g. data exfil in regulated data category;
• PEL NO‑GO domain breach;
• retention / legal‑hold errors.

Steps:

1. Determine jurisdiction - Use FHR jurisdiction_profile_id and effective_jurisdiction.
2. Set legal‑hold - Ensure legal_hold = true and correct retention_profile_id for the incident class.
3. Notify - Follow pre‑written notification template per jurisdiction (not part of this spec), but the runbook references it.
4. Track - Record legal_hold_actor and review_due_ts fields in the forensics / FHR extension.

4.5 SURGE (load-shed) runbook — Surge_GracefulDegrade

• Trigger: sustained load breach (for example cpu_util > 0.85, or T2 review queue beyond its SLA band).
• Action order: disable high-cost monitors first (long-window drift metrics, non-critical telemetry). Safety vetoes, PEL/EHG floors, and gate evaluation are never shed.
• Posture: degraded mode follows AD‑05 [MRP] §6 — more conservative, never more permissive.
• Ledgering: every shed and restore step recorded in FHR (kernel_profile, runtime_mode, active_ring_ids[], degraded_reason_codes[], degraded_minutes). Degradation never masquerades as a third architecture profile.
• Auto-revert: shed monitors restore automatically when load re-enters band; restoration is ledgered the same way.

5. Ops Serviceability Gate

The Serviceability Gate is a pre‑deployment / pre‑flip checklist:

• Challenge quality
• Challenge pass‑rate ≥ target from AD‑08/AD‑19;
• PR slope stable (no recent significant deterioration).
• Operational hygiene
• No unreviewed INCIDENTs open in the ledger.
• Configuration recorded
• kernel_profile, runtime_mode, active_ring_ids[], thresholds.yaml, op_points.yaml and timing_op_point_id snapshot recorded and linked in FHR/duty ledger for this deployment.

Deployment cannot progress unless the Serviceability Gate passes; this is where AD‑11 (RA) and AD‑12 (OP) meet AD‑08 (FHR) and AD‑19 (CI).

6. SLOs

AD‑12 defines three key SLOs: halt ≤ 5 tokens after trip; ACK_HOLD < 300 ms; precision parity drift ≤ band B.

6.1 Streaming halt

• Halt ≤ 5 tokens after any hard trip.
• Enforced via AD‑09 watchdog; measured via halt_tokens_post_trip and FHR timing fields.

6.2 Acknowledgement of HOLD

• ACK_HOLD < 300 ms:
• when MRP transitions to the HOLD state (AD‑05 [MRP]), the UI must render a visible HOLD state / acknowledgement within 300 ms.
• measured by front‑end telemetry; aggregated into UX metrics.

6.3 Precision‑parity drift

• Precision parity drift ≤ band B:
• For FP16 vs INT8/FP8 runs (AD‑12/AD‑16), metric divergence stays within band B defined in SAP.

These are part of the Ops SLO panel and must be visible on dashboards.

7. Forensics hooks

AD‑12 requires the system to emit enough forensics metadata for ops runbooks to execute cleanly:

• gates_snapshot on tool entry
• so INCIDENT / ROLLBACK runbooks can see exactly which gates were open when a tool call happened.
• plan_hash
• hash of planned tool/action sequence at plan time;
• rechecked at execution to catch TOCTOU mismatches (plan changed since gating).
• dedupe_key, durable operation state & TOOL_DEDUPE_HIT
• dedupe repeated tool calls against the durable (tool_manifest_digest, principal_id, dedupe_key) operation record;
• expose operation_id, operation_state, effect_receipt_id, effect_receipt_digest, effect_status, and receipt_verification_status so ops can distinguish a safe reuse from an unresolved or duplicated external effect;
• UNKNOWN state blocks automatic replay and routes to receipt/provider reconciliation.
• legal_hold_actor, review_due_ts
• who enacted legal‑hold and when the next review is due; used in BREACH NOTIF and retention runbooks.

These can live either in FHR extensions or in a separate ops log; the key requirement is that runbooks can rely on them.

8. MKP fallback

AD‑12 includes an MKP fallback so ops can maintain the kernel without the original dyad.

AD‑12 therefore defines:

• A documented procedure to:
• switch kernel_profile from full to mkp for some or all traffic;
• update monitoring to reflect MKP expectations (reduced SPD/culture metrics, same or stricter timing/security guarantees).
• A profile change is an explicit governed configuration change, not an automatic synonym for degraded runtime health; it emits MRP_PROFILE_CHANGED and must leave every MKP-required ring active. Component failure itself sets runtime_mode = degraded and emits the separate runtime-mode event. The change invalidates in-flight gate records, plan_hash values, preview/confirmation joins, and unexecuted Action Tickets created under the prior profile; continuation requires a new or freshly revalidated ticket and a complete gating pass under the new profile. The fallback may not be used to bypass a block already reached under the former profile.
• Clear criteria (high‑level) like:
• repeated high‑severity INCIDENTs;
• RA failures across modalities;
• regulatory requirement for a simpler safety surface.

This is purely operational; the technical shape of MKP is in AD‑05 / AD‑19, but AD‑12 ensures ops can actually invoke it.

9. Acceptance: when can you say “AD‑12 is implemented”?

To “have AD‑12” in a deployment:

1. Create Handoff Packet - Populate thresholds.yaml, op_points.yaml; - write and store runbooks; - link dashboards and contact tree; - define and document commit_watermark policy.
2. Define roles/RACI & two‑person rule - Document minimal roles; - enforce 2‑person approval on PEL / threshold changes.
3. Wire runbooks into alerts - Map alert classes → INCIDENT/ROLLBACK/GATE‑FLAP/BREACH‑NOTIF runbooks.
4. Implement forensics logging - Ensure gates_snapshot, plan_hash, dedupe_key, operation_id, operation_state, effect_receipt_id, TOOL_DEDUPE_HIT, legal_hold_actor, review_due_ts are logged where runbooks expect them.
5. Configure Ops Serviceability Gate - Implement pre‑deployment checklist with challenge pass‑rate, PR slope, incident backlog, and config snapshot; enforce it in CI / change‑management.
6. Track SLOs on dashboards - Add panels for:
   • halt tokens (≤ 5);
   • HOLD acknowledgement latency (< 300 ms);
   • precision parity drift (≤ band B).
7. Test MKP fallback - Simulate an incident and execute the MKP fallback runbook; - confirm dashboards and FHR reflect the new kernel_profile, the independent runtime_mode, and the expected active_ring_ids[], and that safety behaviour is still acceptable.

Related cards:

• AD‑30 [SC] (model change protocol interacts with ops handoff).
• AD‑31 [HS] (integrity prompt templates are part of the handoff packet).