AD‑16 [ME] — Measurement & Evaluation (SAP)

Aliases: None Cluster: Kernel (base) Depends on: AD‑08 [SO], AD‑11 [RA], AD‑17 [AM], AD‑19 [CI]

0. For humans

What question does this answer?

“When we say a change ‘works’ or is ‘Adopted’, what exactly was measured, how, and under what statistical discipline?”

AD‑16’s job:

• Turn “measurement” into a first‑class subsystem with pre‑registration, power, multiplicity control, control charts, and invariance checks, not one‑off plots.
• Define an explicit SAP (statistical analysis plan) with primary/secondary endpoints, SESOI, A/A, neg/pos controls, placebo toggles, and alpha‑spending.
• Define the Acceptance Matrix that governs promotion / demotion of mechanisms.
• Add reporting standards (denominators, missingness codes, uncertainty intervals, error bars, calibration curves).

How do I know it’s live?

You can point to:

• A written SAP for each release with:
• named primary and secondary endpoints;
• SESOI per endpoint;
• planned controls, multiplicity plan, alpha‑spending boundaries;
• prereg ID / registry.
• A published Acceptance Matrix per release with rows/columns as below.
• FHR schema actually containing:
• denominators, missingness codes, uncertainty intervals;
• time‑series fields for entropy‑slope/PR;
• operator variance fields.
• Public or internal reports that include:
• estimation‑first plots, not just p‑values;
• calibration curves / Brier scores for thresholded metrics;
• variance decompositions and inter‑rater κ for double‑coded labels.

Where do I see it?

• In docs: the SAP document, Acceptance Matrix appendix, and measurement section of the main report.
• In FHR: the canonical AD‑08 §4.7a families metric_denominators{}, metric_missingness_codes{}, metric_ci_lo{} / metric_ci_hi{} / metric_ci_level{}, time‑series fields, and operator/rater variance entries.

Quick checklist

• Implemented if: (1) Written SAP with preregistered endpoints, SESOIs, controls, multiplicity, alpha‑spending; (2) Acceptance Matrix published per release; (3) FHR carries the AD‑08 §4.7a denominator, missingness, interval, time‑series, and operator/rater fields; (4) Control suite and invariance tests wired; (5) Estimation‑first graphics and calibration curves in reports; (6) Time‑series drift analysis live
• Key artefacts: SAP document, prereg ID, Acceptance Matrix template, control capsules
• Key FHR fields: metric_denominators{}, metric_missingness_codes{} (M1–M4), metric_ci_lo{}, metric_ci_hi{}, metric_ci_level{}, entropy_slope, pr_score, operator_variance_between, operator_variance_within, rater_kappa, rater_bias_est
• Key failure codes: DRIFT_ALARM, INVARIANCE_FAIL, POWER_INSUFFICIENT
• Escalation path: drift alarm → AD‑19 [CI] gate → AD‑12 [OP] runbook

1. Purpose & scope

AD‑16 defines the measurement backbone for SRF v2.x:

• SAP & endpoints: what we measure, in what order of importance, with which SESOIs.
• Error discipline: power, multiplicity control (Holm‑Bonferroni / hierarchical), alpha‑spending and sequential boundaries.
• Control suite: A/A, neg/pos controls, placebo toggles.
• Drift & time series: CUSUM / EWMA on PR and entropy‑slope; time‑series models with diagnostics and half‑life curves.
• Invariance & variance: temporal/cultural invariance, operator variance, rater bias.
• Reporting & missingness: denominators, missingness codes/tags, IPW/MI, sensitivity bounds.
• Acceptance Matrix: the final promotion/demotion table per release.

Synthetic pack and generator definitions live with the challenge suite ontology in AD‑03 [CS]; CI runners live in AD‑19 [CI]; AD‑11 [RA] hosts the prereg & replication harnesses; AD‑17 [AM] carries the Acceptance Matrix that promotion decisions key on.

2. SAP: endpoints, SESOI, multiplicity, registry

2.1 Endpoints

Primary endpoints (preregistered):

1. Δ steps‑to‑decision - turns/time to resolution under matched tasks.
2. RI stability - post‑intervention variance reduction over K turns.
3. PR (Perturbation Ratio) pass‑rate - on the challenge suite.
4. Safety correctness under gates - false‑allow / false‑block rates with CI.

Secondary endpoints:

• Δ band stability.
• Entropy‑slope trends.
• MRP Mean Time‑to‑Contain (MTTC).
• UX comprehension metrics.
• Energy cost E.

2.1a DMI and RI measurement semantics

Decision‑Making Integration (DMI). DMI is a categorical, per‑target judgement recorded as dmi_state ∈ {unknown, provisional, adopted, contradicted}. adopted requires behavioural evidence that the user applies or restates the relevant distinction in decision‑relevant reasoning; assent, praise, or repetition alone is insufficient. Every judgement carries dmi_target_id and dmi_evidence_refs[] so it can be audited.

Resonance Index (RI). RI retains the published expansion Resonance Index and the runtime field resonance_index. ri_stability_flag = true only when RI remains within a preregistered band for the active locale/register/context band over ri_window_turns, against ri_baseline_id. Stability is not inferred merely because the same source cluster recurs.

AD‑28 [AF] defines how DMI and RI are consumed by the awe guard; AD‑08 [SO] registers their fields. AD‑17 [AM] reports release‑level validation of these measures but is not the source of per‑turn DMI or RI state.

2.2 SESOI per endpoint

For every endpoint we define a Smallest Effect Size of Interest (SESOI):

• SAP includes a SESOI table for:
• Δ steps‑to‑decision
• Δ RI variance
• PR pass‑rate
• MTTC

• correctness.

• Promotion to “Adopted” requires:

• CI outside null and
• estimated effect meeting SESOI
• or TOST pass for “no‑worse‑than” cases.

2.3 Multiplicity & sequential audits

Given many metrics, false positives are inevitable; therefore:

• Multiplicity
• Use Holm–Bonferroni or hierarchical testing with alpha‑spending across sequential audits.
• Sequential boundaries
• Name the alpha‑spending function (e.g. O’Brien–Fleming vs Pocock).
• List decision boundaries up front in the SAP.

2.4 Registry & artefact IDs

Every major measurement release gets:

• A preregistration ID (internal or OSF‑style).
• Data‑card IDs for each dataset/split.
• All linked from FHR for traceability.

3. Control suite: A/A, negatives, positives, placebo

Components:

• A/A tests
• To detect placebo pipeline gains or metric drift.
• Negative controls
• Style‑only perturbations; Δ should not improve task outcomes.
• Positive controls
• Meaning‑change perturbations; outcomes should degrade.
• Placebo toggle
• Non‑functional UI toggle to detect expectancy bias in human ratings.

Placement:

• Control capsules + placebo wiring live under AD‑11 [RA] and AD‑15 [PA].

4. Drift, invariance, variance

4.1 Temporal & cultural invariance

• Drift detection: CUSUM / Page–Hinkley on PR and entropy‑slope.
• Measurement invariance across languages/cohorts via multi‑group checks.

Placement: AD‑16 [ME] + AD‑11 [RA] (invariance capsule).

4.2 Operator variance & leakage

• Strict train/dev/eval disjointness for Δ micro‑benchmarks.
• Cluster‑robust SEs to account for human operator variance; publish σ²_between vs σ²_within.
• HARK‑guard: any threshold change after peeking is logged in the Decision/Data Ledger with reason codes.

Placement: AD‑11 [RA] for leakage & variance, AD‑08 [SO] for DDL link, AD‑16 for operator variance reporting.

4.3 Rater bias audit (labels like Awe)

For double‑coded labels (e.g. Awe):

• Report κ plus bias/frequency indices.
• Refuse promotion if κ < 0.70 or bias skew > pre‑set threshold.

5. Missingness, coding, and FHR schema

Codes & protocols

• Codes:
• M1 sensor missing
• M2 dropout
• M3 privacy redaction
• M4 outage.
• Use IPW (Inverse Probability Weighting) or MI (Multiple Imputation) where appropriate; report bounds.
• FHR includes metric_denominators{} for all rates and metric_missingness_codes{} for affected metrics; mechanism hypotheses use metric_missingness_mechanisms{}.

Missingness mechanism tags

For each metric, add hypotheses: MCAR, MAR, MNAR and run sensitivity bounds accordingly; keep IPW/MI explicit.

Placement: AD‑16 [ME] + AD‑08 [SO] for schema expansions.

6. Acceptance Matrix

Deliverable

The Acceptance Matrix is published per release.

Structure

• Rows:
• domains (facts, policy, idiom, group).
• Columns:
• primary endpoints,
• neg/pos controls,
• CI widths,
• power achieved,
• invariance verdict,
• drift alarms,
• energy trade‑off E.

Promotion rule

• Promotion to “Adopted” is conjunctive across strata (i.e. domains): all required cells must meet SAP criteria.
• Decays / drift trigger demotion with half‑life reported.

Placement: AD‑16 [ME] template, AD‑08 [SO] link to FHR.

7. Time‑series modelling & control charts

Control charts

• CUSUM / EWMA on entropy‑slope and PR with tolerance bands (to stop flapping).

Time‑series models

Treat weekly entropy‑slopes and PR as time‑series:

• Use ARIMA or Bayesian trend models; publish parameters and residual diagnostics.
• Include residual ACF/PACF plots and Ljung–Box, or Bayesian posterior predictive checks, so drift alarms aren’t artefacts.

Reporting

• Report half‑life curves where relevant (how quickly gains decay).

Placement: AD‑16 [ME] + AD‑08 [SO] for time‑series fields in FHR.

8. Reporting standards & exemplars

Estimation‑first graphics

• Every exemplar / figure includes:
• median difference + 95% CI (bootstrap / BCa),
• distribution overlays,
• sample sizes.
• Non‑normal distributions use Cliff’s delta.

Gate calibration curves

• For any thresholded metric (Δ bands, PR alarms), publish calibration curves and Brier scores.

Field health reporting

• Public report items include:
• SAP overview, endpoints and power,
• multiplicity and alpha‑spending plan,
• Acceptance Matrix,
• control‑chart flags and time‑series stats.

9. Synthetic benchmark cross‑link

AD‑16 assumes a Synthetic Benchmark Pack:

• Generators: templated item grammars + controlled perturbations for Δ triads, anchor checks, challenge PR tasks, entropy‑slope probes.
• Strata: domain, language, register, user expertise.
• Seeds: fixed; train/dev/eval disjoint.
• Artifacts: prompt sets, expected invariants, acceptance matrix template, CI runners.

Limits:

• Synthetic tasks validate mechanisms, not product claims.

Placement: AD‑03 [CS] (pack and generator definitions, within the challenge suite ontology), AD‑19 [CI] (CI runners, seed panels, bench harness), AD‑11 [RA] (replication harnesses and Verifier's Bundle).

10. Invariants vs configurable degrees of freedom

Hard invariants (for SRF‑conformant measurement)

• SAP exists with preregistered primary & secondary endpoints and analysis plan; no ad‑hoc fishing.
• Every primary/secondary endpoint has a SESOI; promotion requires CI outside null and SESOI met (or TOST).
• Multiplicity handled with Holm–Bonferroni / hierarchical procedures and alpha‑spending; sequential boundaries named in advance.
• Control suite includes A/A, negative/positive controls, and at least one placebo toggle.
• FHR holds the AD‑08 §4.7a denominator, missingness, uncertainty-interval, and mechanism maps per metric.
• Operator variance and rater reliability are reported (σ²_between vs σ²_within, κ + bias).
• Acceptance Matrix published per release; promotion conjunctive across strata; decay triggers demotion with half‑life.
• Entropy‑slope and PR treated as time‑series with residual diagnostics; not just single thresholds.
• Registry & artefact IDs exist and are linked in FHR.

Degrees of freedom (configurable)

Within those invariants, deployments can tune:

• Exact SESOI values per endpoint (but they must exist and be preregistered).
• Choice of alpha‑spending function (e.g. O’Brien–Fleming vs Pocock) and detailed sequential boundaries.
• IPW vs MI details, provided missingness codes & mechanism tags are reported and sensitivity bounds run.
• Exact forms of CUSUM/EWMA and time‑series models, as long as residual diagnostics are included.
• Domain‑specific thresholds for invariance, rater κ, bias bounds, etc., as long as they are set before inspection.

11. Acceptance: when can you say “AD‑16 is implemented”?

Treat AD‑16 as “implemented” when:

1. You have a written SAP with endpoints, SESOIs, controls, multiplicity, alpha‑spending, and registry IDs.
2. You publish an Acceptance Matrix per release with the specified rows/columns and promotion/demotion rules.
3. FHR populates the AD‑08 §4.7a denominator, missingness, CI, sensitivity/time‑series, and operator/rater fields.
4. Control suite & invariance tests are wired and logged as part of AD‑11 / AD‑08 capsules.
5. Reports use estimation‑first graphics and calibration curves for gates, not bare p‑values.
6. Time‑series drift analysis is live, with documented models and diagnostics.