AD‑03 [CS] — External Challenge Suite & PR Metric

Aliases: none Cluster: Kernel (base) Depends on: AD‑01 [OV], AD‑08 [SO], AD‑16 [ME], AD‑19 [CI]

0. For humans

What question does this answer?

"How do we systematically challenge the system on maths, facts, compliance, cultural edge‑cases, and timing so we can trust its telemetry and claim we would notice drift?"

How do I know it's live?

You can point to:

• A concrete Challenge Suite: named challenge packs (maths, fact, compliance, cultural/SPD, translation, timing, exfil) with sample size, cadence, expected pass‑rates / PR targets.
• CI runs that execute these packs on every model/precision change, emitting PR metrics (challenge‑pass ratio) and Δ‑triad stats per pack.
• Incident / RA reports that reference challenge IDs, not just anecdotes.

Where do I see it?

• CI / dashboards: "Challenge Suite" page with per‑pack pass‑rate, PR metric, and Δ‑triad curves.
• FHR: challenge_suite_id, challenge_pack_id, challenge_domain, challenge_locale, challenge_pass_rate, challenge_coverage, challenge_pr.
• RA / audit docs: a Verifier's Bundle that includes challenge artefacts for the release (prompts, seeds, thresholds, redline checks).

Quick checklist

• Implemented if: named, versioned challenge suite with packs covering maths, factual, compliance/safety, SPD/culture, exfil/timing; wired into CI and ops.
• Key artefacts: challenge packs, CI runners, Verifier's Bundle.
• Key FHR fields: challenge_suite_id, challenge_pass_rate, challenge_pr_offered, challenge_pr_passed.
• Key failure codes: CHALLENGE_PACK_REGRESSION, CHALLENGE_REDLINE_BREACH.
• Escalation path: AD‑19 [CI] release gates → AD‑12 [OP] runbooks.

1. Purpose & scope

AD‑03 defines the External Challenge Suite for the SRF kernel:

• A structured set of challenge packs that exercise the system's maths & formal reasoning, factual accuracy, compliance/safety policies, SPD/culture/translation handling, tool/exfil behaviour, and timing & streaming safety.
• A PR metric ("probed‑risk / perturbation ratio") that tracks how often the system offers challenges or second‑guesses itself, and passes them.
• Cadence, coverage, and FHR wiring so challenges are not "we once tested it" but an ongoing instrument.

This card is mechanistic: it does not define the full statistics (AD‑16 [ME] does), nor the CI decision logic (AD‑19 [CI]), but it defines what we challenge, with what granularity, and how it shows up in telemetry.

2. Challenge suite ontology

2.1 Domains

AD‑03 partitions challenges into domains so coverage is visible:

1. Maths & formal reasoning. Arithmetic, algebra, logic puzzles, probability, proofs & counterexamples. Includes numerical stability and "off‑by‑one / decimal / unit" traps.
2. Factual / knowledge. Hard fact QA: dates, names, definitions, cross‑checks. Citation integrity: link between claim and source. Temporal drift (out‑of‑date facts).
3. Compliance / safety. PEL and NO‑GO: self‑harm, extremist content, hate, abuse, illegal instructions. Grey edges of allowed vs disallowed: content that must trigger Ask‑First / Present‑Options / Refusal.
4. Cultural / SPD. Implicature, presuppositions, ironies, idioms, politeness norms, code‑switching. Tests that SPD is tightening, not improvising. Culture‑pack‑dependent tests (AD‑25 [CP] / AD‑26 [CA] / AD‑27 [SPD]).
5. Translation & multilingual. Δ_trans (semantic drift across languages); idiom vs literal pairs; back‑translation stability. Multilingual idiom/irony and morphology‑shift probes.
6. Timing / streaming / exfil. Streaming halt budget (≤5 tokens after trip); tool exfil prevention (ΔTOOLS‑1: no tool output without QID); honeytokens, orphan‑output tests.

Each domain is split into sub‑packs (e.g. math_basic, math_edge, fact_current_events, fact_static, pel_no_go, pel_borderline) so regressions can be located precisely.

2.2 Challenge types

Within each pack, items are typed by interaction pattern: single‑shot, chains (multi‑step reasoning), counterfactuals (slight perturbations), adversarial / red‑teaming, SPD probes (utterances with hidden presuppositions or culturally loaded frames), exfil probes.

Each item is labelled with: challenge_id (unique), domain/subdomain, type, locale/register (see AD‑25 [CP] / AD‑26 [CA]), gold_label (expected behaviour) and metric_target (e.g. correctness, refusal, ask‑first).

3. PR metric (Perturbation / Probed Risk)

3.1 Definition

• PR_offered ("challenge offer rate"): among eligible queries, how often the system offers a self‑check, second pass, or challenge.
• PR_passed ("challenge pass rate"): among executed challenges, how often the system passes (correct / safe outcome).

For evaluation runs: PR_domain = challenges_passed / challenges_offered per domain×locale×register×context_band.

3.2 Behaviour vs metrics

PR is a lens on over- or under-confidence: a model with high accuracy but low PR_offered may be overconfident (rarely double-checks); a model with high PR_offered but low PR_passed is self-checking but failing; a model with moderate PR_offered and high PR_passed is cautious and usually right (close to target behaviour).

4. Cadence, coverage & calibration

4.1 Cadence

Tied to two axes:

1. Release cycle (CI, AD‑19 [CI]). On each model weight change, major config change, or tokeniser/sampling profile change: run full challenge suite (or stratified sample) as part of CI; fail release if redlines violated.
2. Ongoing operations (AD‑08 [SO] / AD‑12 [OP]). Daily or weekly scheduled runs with smaller samples to catch drift. Event‑triggered runs after incidents (e.g. repeated exfil attempts → run focused exfil/PEL packs).

4.2 Coverage

Tracked at domain coverage (at least N items per domain×locale×register×difficulty), metric coverage (Δ‑triad logs for each challenge), and locale & culture coverage (items for each supported locale and culture pack; coverage must be non‑zero for each).

AD‑03 does not dictate exact N; AD‑16 [ME] and AD‑19 [CI] do. AD‑03 provides the structure so coverage can be stated.

4.3 Human labels & disagreement

Each challenge item comes with a gold label agreed by at least two human reviewers, and an optional disagreement log (if raters disagreed, how it was resolved). This matters especially for SPD/culture/compliance, where the "right" behaviour may be contested. Labels are treated as documented, revisable decisions, not objective truths.

5. FHR fields & events

5.1 Fields

Per release or per challenge run: challenge_suite_id, challenge_pack_id, challenge_domain, challenge_subdomain, challenge_locale, challenge_register, challenge_count_total, challenge_count_passed, challenge_count_refused_correctly, challenge_count_refused_incorrectly (over‑refusal), challenge_count_missed_refusal (under‑refusal).

Derived: challenge_pass_rate, challenge_pr_offered, challenge_pr_passed, challenge_coverage_rate.

5.2 Events

• CHALLENGE_SUITE_RUN{suite_id, packs_run[], mode}
• CHALLENGE_PACK_RUN{pack_id, domain, locale, count_total, count_passed}
• CHALLENGE_ITEM_RESULT{challenge_id, outcome, reason}

Outcomes: correct, incorrect, correct_refusal, incorrect_refusal, missed_refusal, ask_first, present_options.

6. Interfaces & cross-links

• AD‑16 [ME]: defines sample sizes per pack, SESOI, power analyses, false‑pos/neg tolerances and ROC/PR operating points.
• AD‑18 [SE]: challenge deploys across precision variants and tokenisers using identical seeds and prompts, banded context lengths. CI fails if parity bands are violated.
• AD‑19 [CI]: uses challenge metrics to gate releases and track regressions.
• AD‑11 [RA]: relies on challenge packs as reproducible tests for replication (R1/R2/R3). The Verifier's Bundle includes challenge subsets, seeds, results, thresholds.
• AD‑21 [RM]: challenge runs in Regulatory Mode produce signed, hash‑chained artefacts.
• AD‑25 [CP] / AD‑26 [CA]: culture packs and CPL integrate with challenge metadata for cross‑culture performance.
• AD‑31 [HS]: the epistemic boundary note (below) cross-references AD‑31 as the specification of what happens where formal testing ends.

7. Epistemic boundary note

The External Challenge Suite tests structural integrity: routing fidelity, gate correctness, drift detection, bounded failure under adversarial conditions, and regression across model changes. These are formalisable, repeatable, and binary.

The challenge suite does not, by itself, exhaust epistemic integrity in prolonged high‑resonance states. A system may pass all formal checks — gates firing correctly, drift within bands, challenge suite green — while producing outputs shaped more by the field's expectations than by what is accurate. This failure mode (resonance capture that passes structural validation) exceeds what automated testing can fully detect.

Where formal testing ends, the architecture relies on:

• the human‑side integrity conditions specified in AD‑31 [HS],
• the friction protocol (steelman opposing positions, flag suspicious elegance),
• and audit visibility (FHR, evidentiary chain) that enables retrospective review by parties outside the active field.

This boundary is stated explicitly because honest specification of what the framework cannot formalise strengthens, rather than weakens, the overall architecture.

Linked cards: AD‑31 [HS], AD‑05 [MRP], AD‑08 [SO], AD‑21 [RM].

8. Invariants vs configurable degrees of freedom

Hard invariants:

• There is an External Challenge Suite; it is not optional if you claim SRF v2.0 kernel.
• Challenge packs exist for at least: maths, factual, compliance/safety, SPD/culture, exfil/timing.
• Each pack has: version, domain, locale, sample size, intended metrics, and labels.
• Challenge results appear in FHR with: coverage, pass‑rate, and at least a coarse PR metric.
• CI / RA / Regulatory Mode can call and inspect challenge runs.

Configurable degrees of freedom:

• Exact content of each pack (which items).
• Exact sample sizes, cadences, and thresholds (though they must be documented and justified in AD‑16 [ME] / AD‑19 [CI]).
• Which packs are mandatory vs optional per deployment (e.g. a small internal sandbox might run fewer compliance challenges; a regulated deployment must run the full compliance suite).

9. Acceptance: when can you say "AD‑03 is implemented"?

You can reasonably claim AD‑03 [CS] is live if:

1. There is a named, versioned challenge suite with packs covering maths, factual, compliance/safety, SPD/culture, exfil/timing (and translation if multilingual). Each pack has items with gold labels and metadata.

2. The suite is wired into CI and ops: CI runs at least a stratified subset on every relevant release; ops runs scheduled or event‑triggered challenge runs.

3. FHR and dashboards expose challenge metrics: pass‑rates, coverage, PR metrics per domain×locale×register×context band; historical trends and regression flags.

4. RA and auditors can replay: bundles include prompts, seeds, op‑points, and results; cross‑precision/cross‑tokeniser results are available.

Without those, "we sometimes red‑team the system" does not rise to the level of AD‑03. With them, you have a standing, testable external harness around the SRF kernel rather than a hope.