Appropriateness as a computed property

Properties

“Age-appropriate” is four measurements wearing one word

Post 1 left you with immutable, content-addressed blocks. The appropriateness layer does not store a verdict about a block; it derives properties from it, records them with the rubric version that produced them, and lets the gate decide. Drive it below — the same block, two districts.

• The four things are independent and fail independently. Readability (can she decode the sentences), vocabulary (does she know these words yet), conceptual load (does this assume a prerequisite she has not met), and sensitivity (is the subject matter appropriate for this age). A block can pass three and fail one, and which one it failed decides whether a human rewrites a sentence or the whole block goes back.
• A verdict has no history. “Was this block appropriate under the rubric we shipped in March” is unanswerable once you have overwritten the flag — which, as the evaluation section shows, makes honest measurement impossible.
• A boolean cannot be tuned per district. Same content, same rubric, different thresholds — and the thresholds belong to the district while the measurements belong to the block.
• Recomputation is the fix for everything. Derived properties can be recomputed for every block ever generated when the rubric changes. A stamped boolean cannot be un-stamped.

-- Append-only, keyed by (block, rubric version). A block is immutable, so a property
-- computed under a given rubric version is true forever and never needs updating.
CREATE TABLE block_properties (
  block_hash        text        NOT NULL,
  rubric_version    text        NOT NULL,   -- "g3.2026-08"
  -- readability
  fk_grade_level    real        NOT NULL,   -- a proxy for decoding load, and known as one
  sentence_len_p90  real        NOT NULL,
  -- vocabulary
  tier2_ratio       real        NOT NULL,   -- academic vocabulary as a share of content words
  offlist_terms     text[]      NOT NULL,   -- words outside the grade-band list, for a human to see
  -- conceptual load
  prereq_depth      int         NOT NULL,   -- from post 2's graph, not from the text
  unexplained_terms text[]      NOT NULL,   -- terms used before they are introduced in this lesson
  -- sensitivity
  topic_flags       text[]      NOT NULL,   -- classifier labels, verbatim, never rewritten
  classifier_version text       NOT NULL,
  computed_at       timestamptz NOT NULL DEFAULT now(),
  PRIMARY KEY (block_hash, rubric_version)
);

The rubric

One definition, two runtimes

The gate runs in the publish path. The evaluation harness runs in CI and in research notebooks, in a different language. Those are genuinely different runtimes — and the moment you write the rubric twice, they start to drift. Someone tightens a threshold in the gate, or fixes a null-handling bug in the harness, and only one side gets it. Nothing errors. The evaluation just quietly stops describing production.

• The rubric is data, not code. A versioned document with thresholds and a decision table. Both runtimes load the same file.
• Neither runtime may hard-code a threshold. If a number appears in a source file, it is a bug — and it is the kind of bug a linter can find.
• Parity is a conformance suite, not a convention. A directory of blocks with expected decisions, run by both runtimes in CI. Adding a threshold without adding a fixture fails the build.
• The rubric version is stamped on every decision. A held block records which rubric held it, which is what makes a rubric change a migration you can reason about rather than a silent reclassification of your entire corpus.

version: g3.2026-08
grade_band: "3"
supersedes: g3.2026-02
thresholds:
  fk_grade_level:     { max: 4.5 }        # a proxy for decoding load, not a measure of quality
  sentence_len_p90:   { max: 16 }
  tier2_ratio:        { min: 0.04, max: 0.14 }
  prereq_depth:       { max: 2 }          # relative to the objective, from the content graph
  offlist_terms:      { max_count: 3 }
  unexplained_terms:  { max_count: 0 }
decisions:
  # First match wins. Every rule names the human action it implies.
  - when: { topic_flags: { any_of: [violence, self_harm, sexual, hate] } }
    result: hold
    action: escalate_to_safety_reviewer
  - when: { unexplained_terms: { count_gt: 0 } }
    result: hold
    action: regenerate_block
  - when: { fk_grade_level: { gt: 4.5 } }
    result: hold
    action: simplify_language
  - when: { offlist_terms: { count_gt: 3 } }
    result: flag                          # visible to the teacher, not blocking
    action: review_vocabulary
  - else: pass
district_overrides:
  riverbend: { fk_grade_level: { max: 3.2 } }    # reads roughly two years below band

import { loadRubric } from "./load.js";       // reads rubrics/*.yaml — the ONLY source of numbers
import type { BlockProperties, GateDecision } from "./types.js";

export function evaluate(
  props: BlockProperties, rubricVersion: string, districtId: string,
): GateDecision {
  const rubric = loadRubric(rubricVersion, districtId);   // applies district_overrides
  for (const rule of rubric.decisions) {
    if (rule.else || matches(rule.when, props)) {
      return {
        result: rule.result ?? "pass",
        action: rule.action ?? null,
        rubricVersion,          // stamped, always — this is what makes a rubric change auditable
        failedOn: rule.else ? null : Object.keys(rule.when),
      };
    }
  }
  throw new Error(`rubric ${rubricVersion} has no terminal rule`);   // never default to pass
}

"""The Python side loads the same YAML. No thresholds live in this file, by design."""
from lessonpipe.rubric import load_rubric, matches
from lessonpipe.contracts import BlockProperties, GateDecision


def evaluate(props: BlockProperties, rubric_version: str, district_id: str) -> GateDecision:
    rubric = load_rubric(rubric_version, district_id)
    for rule in rubric.decisions:
        if rule.is_else or matches(rule.when, props):
            return GateDecision(
                result=rule.result or "pass",
                action=rule.action,
                rubric_version=rubric_version,
                failed_on=None if rule.is_else else sorted(rule.when),
            )
    raise ValueError(f"rubric {rubric_version} has no terminal rule")

Each fixture is a block plus the decision BOTH runtimes must produce.
CI runs the gate and the evaluation harness over the same directory and diffs.

  fixtures/
    g3_passing_explanation.json
    g3_offlist_vocabulary.json          -> flag / review_vocabulary
    g3_unexplained_term.json            -> hold / regenerate_block
    g3_sensitive_topic.json             -> hold / escalate_to_safety_reviewer
    g3_riverbend_override.json          -> hold under riverbend, pass under oakhurst

That conformance directory is the entire discipline in one folder. It fails loudly the day someone edits one runtime, which is the only moment the divergence is cheap to fix.

Evaluation

The offline number that means nothing

You want to know whether your generator produces good third-grade content. You take a thousand generated blocks, ask a frontier model to score them against a quality rubric, and get 94%. Everyone is delighted. In production, teachers rewrite something in six lessons out of ten. Here is the same corpus, graded three ways.

Three separate mechanisms are hiding in that gap. The judge shares the generator’s blind spots — score content from one model family with a judge from the same family and you have measured self-consistency. The rubric was probably derived from the outputs, which fits the exam to the student: the criteria that would have failed it never got written. And the eval set is made of survivors — build a benchmark from lessons teachers published and your negatives are near-misses rather than the content that was held, abandoned or never generated because the request failed.

• The label has to come from outside the model’s world. Teacher edits are the label. They are free, continuous, and made by the only people whose judgement the product is actually claiming to augment.
• Judge with a different family, or with humans, or both. And measure the judge against the human sample periodically, because a judge is a model and it drifts too.
• Sample the population you deploy on, not the one you kept. Held blocks, abandoned drafts and failed requests all belong in the denominator.
• Join evaluation on the rubric version in force at the time. Always. It is one predicate, and skipping it lets a rubric change silently rewrite your history.

"""Build an eval set that cannot flatter itself.

Three rules, each undoing one of the failure modes above:
  1. Population is every block generated in the window — held and abandoned included.
  2. Properties are read at the rubric version in force when the block was generated.
  3. The label is the teacher's edit distance, not a model's opinion.
"""
QUERY = """
SELECT b.block_hash,
       b.kind,
       p.fk_grade_level, p.tier2_ratio, p.prereq_depth, p.topic_flags,
       g.result                                    AS gate_result,
       -- The label: did a human change this block, and how much?
       coalesce(e.edit_distance, 0)                AS edit_distance,
       coalesce(e.was_deleted, false)              AS was_deleted
  FROM blocks AS b
  -- INNER JOIN on the rubric version in force at generation time. Never "the current rubric".
  JOIN rubric_windows AS w
    ON  b.created_at >= w.effective_from
    AND b.created_at <  coalesce(w.effective_to, 'infinity'::timestamptz)
    AND w.district_id = b.district_id
  JOIN block_properties AS p
    ON  p.block_hash = b.block_hash
    AND p.rubric_version = w.rubric_version
  JOIN gate_decisions AS g
    ON  g.block_hash = b.block_hash
    AND g.rubric_version = w.rubric_version
  -- LEFT, not INNER: blocks nobody ever reviewed are the negatives you most need.
  LEFT JOIN teacher_edits AS e
    ON  e.block_hash = b.block_hash
 WHERE b.district_id = %(district_id)s
   AND b.created_at BETWEEN %(start)s AND %(end)s
"""

The outer join on teacher edits is the whole survivorship fix in one keyword. Make it an inner join — which is what you write if you are thinking “I need blocks with labels” — and your dataset silently becomes blocks a teacher opened, which is a biased sample of blocks that passed the gate, which is a biased sample of blocks the generator was good at. The population is defined by the window and the district, and nothing else is allowed to filter it.

Feedback

The teacher’s edit is the only label you did not have to pay for

A teacher rewriting a hook is annotating your training data. A teacher deleting an item is telling you the alignment was wrong. A teacher swapping an image is telling you something about a rights record, a reading level or a classroom you do not know about. All of it is free and continuous, and almost every product in this space throws it away by storing edits as an overwrite.

• Store the edit, not just the result. Which block was replaced, by what, by whom, and — where the interface can cheaply ask — why. A one-click reason chip on an edit is the highest-value component in the product.
• Edit distance is a graded signal, not a boolean. A punctuation fix and a full rewrite are both “edited”, and treating them the same throws away most of the information.
• Deletions are the strongest signal you have. They are the only unambiguous statement that the block should not have existed, and they are rare enough to review individually.
• Segment by teacher before you believe anything. One teacher who rewrites everything is a style preference; forty teachers who all rewrite the same block is a defect.

The trap is closing the loop too tightly. Fine-tuning directly on teacher edits sounds obviously right and drifts your generator toward whatever a small, self-selected group of early adopters prefers — usually more text, more scaffolding, and their own voice. Use edits as evaluation first and as training data second, deliberately, with a held-out set of teachers whose edits never enter the training corpus so you retain an uncontaminated measure. And note the governance constraint that arrives in the next post: teacher edits are staff data and manageable, but the moment student work enters this loop you are training on children, and that is a decision with a different shape entirely.

git clone https://github.com/xaviramirezcom/open-lesson-pipeline
cd open-lesson-pipeline
git checkout v3-rubric
pnpm install && uv sync
pnpm lesson up
pnpm lesson seed --with-teacher-edits    # synthetic edits from 30 simulated teachers

pnpm lesson gate --district riverbend --rubric g3.2026-08
pnpm lesson eval --judge same-family                      # the flattering number
pnpm lesson eval --judge teacher-edits                    # the honest one
pnpm lesson eval --judge teacher-edits --survivors-only   # the honest one, biased on purpose
pnpm lesson rubric diff g3.2026-02 g3.2026-08 --recompute

Explain it back