Efficacy & incrementality

The interval

Can this study see the thing you are hoping for?

Start at the end, because the arithmetic below decides whether the rest of the post is worth running. Three controls — how many classrooms you have, how alike the children in one room are, and whether your standard error knows they share a teacher.

That last control is not a modelling nicety. A naive standard error on clustered data is the single most common way an education efficacy claim gets published wrong, and it always errs toward a narrower interval — which means toward a more exciting finding.

The problem

Completion makes your best lesson the one the strongest students finished

Follow one child. She finishes the lesson, scores eight out of ten on the exit items, and spends nine minutes on task. Another child opens it, stalls on the second block, and closes the tab. Your dashboard now reports that students who complete the lesson score 80%, and a slide gets made about mastery. Nothing in that sentence is false and nothing in it is evidence, because the strongest predictor of finishing a lesson is being the kind of student who finishes lessons.

• Engagement metrics describe a self-selected population. Completion, time on task and exit-item accuracy are all conditioned on the child having engaged, which is exactly the variable that also predicts the outcome.
• Before-and-after gains are contaminated by everything else. Children learn between September and December whether or not you exist. A pre/post gain measures your lesson plus the teacher plus eleven weeks of ordinary schooling, and attributes all of it to the lesson.
• There is no counterfactual in the telemetry. Nowhere in your interaction log is a comparable child who did not get the lesson, so no amount of modelling recovers the difference.
• The feedback loop is self-sealing. The unit with the highest completion gets recommended more, gets used by more motivated classes, reports even higher completion — and the ranking is now measuring teacher enthusiasm.

Be fair to engagement data. It is genuinely useful for diagnosis: which block children abandon, which item has an implausible score and is probably mis-keyed, whether a lesson is too long for a forty-minute period. Those are real jobs, and post 3’s rubric gets better because of them. It is just not evidence that your content taught anything, and it should never be the number in a sentence containing the word “improves”.

The design

Randomise where the intervention actually lands

A holdout in education is not a row filter. Children sit in classrooms, are taught by one teacher, and talk to each other — so assigning individual students to arms inside one room contaminates both arms and violates the independence the arithmetic assumes. The unit of randomisation is the section.

• Randomise sections, analyse students, adjust for the clustering. Anything else understates your uncertainty, often by a factor of two or more, and the direction of that error is always toward false confidence.
• Deterministic assignment beats random-and-stored. A hash of the section and experiment ids is stable across runs, restarts and a database migration, and an analyst in March can re-derive it from two strings without having been in the room in September.
• The publish engine subtracts the control arm, not the lesson definition. Post 5’s diff runs after assignment, so a control section that was published to before the experiment started gets an explicit removal rather than being quietly left with treatment content.
• Pair or stratify before you randomise. With few clusters, simple randomisation frequently hands you arms that differ at baseline. Stratify on prior-year performance and school; the balance you get is worth more than the elegance you lose.

import { createHash } from "node:crypto";

export interface Experiment {
  experimentId: string;         // "exp_fractions_pilot_q1"
  districtId: string;
  unitId: string;               // "unit_fractions_intro"
  arms: ["treatment", "control"];
  controlCondition: "existing_unit" | "no_supplement";   // what the control actually receives
  stratifyBy: ("school" | "prior_performance_quartile")[];
  startsAt: string;
  plannedEndAt: string;         // fixed before the first publish — see Power
  primaryOutcome: "posttest_score";
  mdeStandardised: number;      // 0.20 = detect a 0.2 SD effect
  assumedIcc: number;           // 0.15 — declared up front, because it drives everything
  owner: string;
}

/** Stable, storage-free assignment at the SECTION level. Students inherit their section's arm. */
export function armFor(sectionId: string, stratum: string, exp: Experiment): "treatment" | "control" {
  const digest = createHash("sha256").update(`${stratum}:${sectionId}:${exp.experimentId}`).digest();
  return digest.readUInt32BE(0) % 2 === 0 ? "treatment" : "control";
}

import { armFor, type Experiment } from "../measurement/assignment.js";

const sections = await sectionsForCourse(scope, courseId);
const exp = await runningExperimentFor(scope, unitId);          // zero or one

const treated = exp
  ? sections.filter((s) => armFor(s.sectionId, stratumOf(s), exp) === "treatment")
  : sections;

// Diff happens AFTER the subtraction, so a control section published to before the
// experiment started is emitted as a removal rather than silently keeping treatment content.
const deltas = diffAgainstSnapshot(elementsFor(treated), lastSnapshot);
await lms.publish({ publishId, deltas });
await audit.write({ ...counts, experimentId: exp?.experimentId ?? null,
                    holdoutSections: sections.length - treated.length });

The overlap trap deserves its own paragraph, because it is silent and it is worse here than in most domains. A district running your pilot is also, that term, running a new intervention programme, a schedule change, and a different vendor’s pilot in the same grade — and neither experiment knows about the other. If the overlap is uneven between your arms, one study reads the other’s effect as its own. The fix is boring: a registry of running experiments per district, a check at creation that flags an intersection above a threshold, and — the part that requires a phone call rather than code — asking the district what else is happening in those rooms this term.

Power

Decide the sample size before you look, and put the clustering in it

The uncomfortable arithmetic goes first, not last. Children in the same classroom resemble each other, and that similarity inflates the sample you need by a design effect — at twenty-five children a room and a typical correlation, a factor of about 4.6. Detecting a 0.20 SD effect needs roughly 73 classrooms per arm. A typical pilot runs six.

• The target effect size is a design input, not a result. Compute it from the classrooms you actually have. Six per arm resolves an effect of roughly 0.69 SD — larger than almost anything the education literature has credibly reported, which means the pilot is guaranteed to find nothing and the finding will be read as “it does not work”.
• Peeking is not significance. Checking weekly until the number goes green is a different test than the one you designed, and its real false-positive rate is several times the one you think you are running at.
• Fix the end date and the outcome up front. Both live in the experiment record. An analysis that cannot choose its own window or its own outcome has no room to drift.
• Underpowered is worse than not running. A null gets read as proof of failure and a false positive gets read as proof of success. Both move a district’s budget, and only one of them is reversible.

from math import ceil, sqrt

Z_ALPHA = 1.959964   # two-sided 0.05
Z_BETA = 0.841621    # 80% power


def design_effect(students_per_section: int, icc: float) -> float:
    """Children in one room are not independent observations. This is the price."""
    return 1.0 + (students_per_section - 1) * icc


def sections_per_arm(effect_size: float, students_per_section: int, icc: float) -> int:
    """Clusters required PER ARM to detect `effect_size` (in SD units) at 80% power."""
    per_arm_individual = 2 * (Z_ALPHA + Z_BETA) ** 2 / effect_size**2
    inflated = per_arm_individual * design_effect(students_per_section, icc)
    return ceil(inflated / students_per_section)


def min_detectable_effect(n_sections_per_arm: int, students_per_section: int, icc: float) -> float:
    """Inverted: given the classrooms you actually have, what is the smallest effect you can see?"""
    effective_n = (n_sections_per_arm * students_per_section) / design_effect(students_per_section, icc)
    return sqrt(2 * (Z_ALPHA + Z_BETA) ** 2 / effective_n)


# sections_per_arm(0.20, 25, 0.15)        -> 73   classrooms per arm
# min_detectable_effect(6, 25, 0.15)      -> 0.69 SD  <- what a six-classroom pilot can see
# min_detectable_effect(73, 25, 0.15)     -> 0.20 SD

The readout

Incremental learning, cost per point, and an interval

The query is unglamorous, which is the point. Assign every section to an arm with the same hash the publish engine used, join outcomes in the window, and compare arm means with the clustering respected. No engagement metric appears anywhere.

SELECT arm,
       count(DISTINCT section_id)                 AS sections,
       count(*)                                   AS students,
       avg(posttest_score)                        AS mean_score,
       stddev_samp(posttest_score)                AS sd_score,
       -- Between-section variance is what the design effect is made of. Report it; do not assume it.
       stddev_samp(section_mean)                  AS between_section_sd
  FROM (
    SELECT s.section_id,
           arm_for(s.section_id, s.stratum, $1)   AS arm,
           o.posttest_score,
           avg(o.posttest_score) OVER (PARTITION BY s.section_id) AS section_mean
      FROM sections AS s
      JOIN outcomes AS o
        ON  o.section_id = s.section_id
       AND o.assessed_at BETWEEN $2 AND $3
     WHERE s.district_id = $4 AND s.unit_id = $5
  ) AS t
 GROUP BY arm;

export interface EfficacyReadout {
  question: "superiority" | "non_inferiority";
  effectSize: number;            // standardised mean difference, treatment − control
  ciLow: number; ciHigh: number; // 95%, computed with the cluster-robust SE, never the naive one
  observedIcc: number;           // report what you measured, not what you assumed
  mdeAtObservedN: number;        // the honest ceiling on what this study could have seen
  underpowered: boolean;
  nonInferiorityMargin: number | null;
  verdict: "detected" | "no_detectable_effect" | "non_inferior" | "inconclusive";
}

Three things to say out loud. A result whose interval includes zero is a result — it says the effect, if any, is smaller than this study could see, and it should be reported in exactly those words rather than quietly not reported. The observed clustering belongs in the readout beside the assumed one, because if you planned at 0.15 and measured 0.28 your study was less powered than you thought and every future design should use the new number. And the interval must come from a cluster-robust estimator — which is the control you just drove at the top of this post.

The question

It is usually non-inferiority, not superiority

Here is the part almost nobody designs for. If a generated unit costs a fraction of an authored one, you do not need it to teach better. You need to know it does not teach appreciably worse. That is a different test with a different hypothesis, and running a superiority test and finding nothing is not the same as running a non-inferiority test and finding equivalence.

• The margin is a curriculum decision, not a statistical one. How much learning is a district willing to trade for coverage it could not otherwise afford? Someone with a name has to answer that, in writing, before the study.
• Non-inferiority is not the cheap option. Tight margins are expensive, as the table says, and pretending otherwise is how the answer gets fudged.
• Report the margin in every sentence that reports the verdict. “Non-inferior” alone is meaningless; “non-inferior at a 0.10 SD margin” is a claim someone can check.
• A superiority null is not evidence of equivalence. If you designed for superiority, that is the only question you get to answer, and the honest write-up says so.

The ethics

The control arm is a room full of real children

A control section is a class you deliberately did not give the thing you believe is better, for a term, to learn something. That is a decision with subjects, an owner and a date, and it belongs in the same append-only audit log as every publish and every deletion — the experiment id and the held-back section count are already fields on post 4’s audit record.

Three practical consequences. The control condition should be business as usual, not nothing — you are comparing your unit against the district’s existing materials, which is both more ethical and the comparison a district actually wants. The district decides, with informed consent at the level its own policy requires; you propose a design, you do not assign children. And there is a stopping rule: if an interim look shows real harm, the study ends, and the rule for that is written down before the first publish rather than argued about in the moment.

git clone https://github.com/xaviramirezcom/open-lesson-pipeline
cd open-lesson-pipeline
git checkout v6-efficacy
pnpm install && uv sync
pnpm lesson up
pnpm lesson seed --classrooms 12 --students-per-section 25 --icc 0.15

pnpm lesson experiment create exp_fractions_pilot_q1 \
  --district riverbend --unit unit_fractions_intro \
  --control existing_unit --stratify school,prior_quartile --mde 0.20
pnpm lesson publish --unit unit_fractions_intro --weeks 8

pnpm lesson report engagement --unit unit_fractions_intro
pnpm lesson report efficacy   --unit unit_fractions_intro --exp exp_fractions_pilot_q1
pnpm lesson report efficacy   --unit unit_fractions_intro --exp exp_fractions_pilot_q1 --naive-se
pnpm lesson power --sections-per-arm 6 --students 25 --icc 0.15

What you have now

1. A request that is a contract. Scope, framework version and grade band pinned at the edge, with generation behind a log — so a provider outage is consumer lag rather than a broken first period.
2. Alignment claims you can retract. A graph of typed nodes and classed edges with provenance, so a bad claim is a removable observation rather than permanent damage, and coverage is reported per class or not at all.
3. Appropriateness that is measured, not asserted. Four properties over immutable blocks, one rubric loaded by two runtimes, and an evaluation whose labels come from teachers rather than a model with the generator's blind spots.
4. A guardrail the editor cannot route around. Publishability derived from an inner join to gate decisions, review receipts and asset rights, so a pasted paragraph and an uploaded photograph are gated exactly like generated content.
5. Publishes that send deltas and report their own fidelity. Diffs rather than re-imports, idempotent elements with stable external ids, dead-lettered rejects, and a freshness target that catches the expired token before a term does.
6. A number that survives “compared to what?”. Randomisation at the section level, a power calculation with the clustering in it, and a readout that knows whether it is answering superiority or non-inferiority.

Explain it back