# Verification-fixture lessons — three retracted measurements **Filed:** Batch 59, 2026-08-03. **Status:** post-mortem; no code action, but the rule below should inform every future verification. Three numbers were reported as "verified" during Batches 52 and 53 and later had to be retracted because the query that produced them was itself defective — or the fixture the query ran against was convenient rather than adversarial. This document records all three and the rule they imply. --- ## Retraction 1 — "0 of 72 staff qualified" (Batch 53 §1) **What was reported:** the ABA-eligibility gate rejected 72 of 72 active staff, motivating the addition of an exact `"ABA Staff"` role signal. **Why it was wrong:** the candidate pool passed into the eligibility scoring loop was fetched by [`ProgramAssignmentController::searchStaff`](../../app/Http/Controllers/ProgramAssignmentController.php) with an unordered `->limit(50)` at what is now line ~127. The live DB had 189 undeleted staff. MySQL returned 50 rows in an unspecified order (in practice, low ids first). Every staff row above id ~652 was excluded from the pool without ever being scored. The "72" was itself a truncated count against that pool. The real denominator was 134 (active) out of 189 (undeleted). **Fix:** Batch 57 raised the pool ceiling to 5000 with a `Log::warning` if it ever binds, and added the exact-role signal into the closure's `use` list (a separate scope bug from Batch 53 that would have shipped broken regardless of pool size). **Retracted number:** *0 of 72*. Do not cite. **Actual number after the fix:** *5 aba_qualified of 134 active checked* — verified against the demo therapist fixture in Batch 57 §3. --- ## Retraction 2 — "0→3 activeTargets, 0→1 studentsWithActive" (Batch 52) **What was reported:** the Batch 52 introduction of `DataScopeHelper::scopeStudentsViaProgramOwnership` unioned the legacy `programs.studentId` linkage with `program_student_assignments`. Verification against a tutor fixture showed `activeTargets` jumping 0→3 and `studentsWithActive` 0→1. **Why the numbers were misleading:** the helper's WHERE-clause shape was correct (legacy OR psa). But the *allowed-student set* it fed into that clause came from `getTutorStudentIds`, which is classroom-only (`assign_student_classes`). Batch 52 changed the query shape and held the fixture-tutor constant. That tutor happened to have classroom membership for the same students they were also linked to via psa, so the classroom-derived allowed set was non-empty and the union clause fired correctly against it. The 0→3 and 0→1 numbers were real for that fixture. What the verification didn't test was the case where the tutor's classroom set is empty — a psa-only technician (Demo Therapist, added in Batch 57) surfaced immediately: the union clause became `WHERE studentId IN [] OR EXISTS(psa WHERE psa.studentId IN [])`, matching nothing. Five call sites of the helper — every one — had this failure mode hidden. **Fix:** Batch 58 §E3 changed one line inside [`scopeStudentsViaProgramOwnership`](../../app/Helpers/DataScopeHelper.php#L290): `getTutorStudentIds` → `getTutorStudentIdsWithPsa`. Every helper caller inherited the fix. **Retracted numbers:** *0→3 and 0→1*. Do not cite. **Actual numbers under the demo therapist fixture, post-fix:** | Site | Actual after Batch 58 §E3 | |------|--------------------------| | `libraryMetrics.activeTargets` | 3 | | `libraryMetrics.masteredTargets` | 2 | | `libraryMetrics.programsByFolder` | 1 program returned to tree | | `libraryMetrics` per-program status breakdown | `{mastered:2, acquisition:3, probe:1, waiting:12}` | | `kpis.targetsMastered` | 2 | Full before/after table in the Batch 58 §E3 report. --- ## Retraction 3 — "six call sites" of `scopeStudentsViaProgramOwnership` (Batch 52) **What was reported:** `docs/post-demo/aba-staff-designation.md` line 93 says *"Batch 52 audit — six psa-union sites all now go through DataScopeHelper::scopeStudentsViaProgramOwnership."* **Why it was wrong:** there are five call sites of that helper, not six. Four are in `ProgramController::libraryMetrics`; one is in `SessionDataController::kpis`. Whatever was being counted as the sixth was likely a direct-union site in `ProgramController::index` that duplicates the pattern in-place — those exist (Batch 58 §E3 found three) but they are not helper calls. **Fix:** none in code — the number was descriptive, not load-bearing. Correcting the record here. **Retracted number:** *six*. Correct number: **five helper call sites, plus three in-place duplicated patterns** — total eight sites carrying the same defect class, all fixed in Batch 58 §E3. --- ## The rule **A verification is only as strong as its fixture. The fixture must be the emptiest possible input, not the convenient one.** Every one of the three retractions above shares a shape: the verification measured a real change on a real fixture, and the number was accurate for that fixture. What each verification did NOT do was test the *degenerate case* — the input on which the query's defect would manifest as zero: - Retraction 1: a pool with more than 50 rows. - Retraction 2: a tutor with zero classroom classes. - Retraction 3: not a measurement, but same pattern — the "audit" looked at what was known to have been changed, not at every callsite that could carry the same shape. ## Checklist for future verification runs Before reporting a number as verified, answer these: 1. **What is the input's ceiling?** If the query has a `LIMIT`, does the test input exceed it? If yes, does the fixture cross the ceiling? If no, the number is only valid up to that ceiling. 2. **What is the input's minimum?** If the query filters on membership in some set, run the verification against a fixture where that set is empty. If the query returns non-zero on an empty set, the filter is broken. 3. **What is the input's union?** If the query fixed a "missing linkage" bug, does the fixture only have the added linkage (not the pre-existing one)? If the fixture has both, you can't tell whether the fix works against the added linkage alone. 4. **Callsite completeness:** if a fix is described as "all N sites updated," grep for both the helper call AND the pattern the helper replaced. Sites that inline the pattern rather than calling the helper are the exact places the fix will not apply. None of the three verifications above passed all four checks. Applying them costs a handful of extra tinker calls; the alternative is what happened tonight — three retractions, one blocker found during a pre-demo sweep, and a helper bug shipped for weeks. ## Related - Batch 57 §5 first flagged retractions 1 as the "0 of 72" retraction. - Batch 58 §E3 §5 explained why Batch 52 passed with the wrong fixture. - `docs/post-demo/aba-staff-designation.md` — carries the "six sites" descriptive error; leave the doc as-is (it's historical) but point future readers here for the correct count.