Publishing & fidelity

The diff

Send what changed, not what is

Post 4’s compiler gives you the publishable set for a lesson version. What a platform needs is the delta between that and what it already has. Those are different objects, and the gap between them is where all four of the full-republish failures live — one of which is much worse than the others.

• A republish orphans student work. Delete and recreate an assignment and the submissions attached to the old object are stranded, the gradebook column is empty, and a teacher spends her Sunday re-entering marks. This is not a data problem; it is the single fastest way to lose a district.
• Quota is a shared, finite budget. Every course in the district draws from the same allowance. A full re-import of forty courses at two in the morning is quota you cannot spend on the urgent correction at twenty to ten on the morning somebody notices an error.
• Republish is proportional to the course; diff is proportional to the change. A stable unit changes a handful of blocks a week. One of those numbers grows with your success and one does not.
• Republish destroys the why. After a wholesale rewrite, nothing entered and nothing left — the course simply is. The moment a teacher asks why her lesson looks different today, you have nothing to answer with.

-- One row per element per run. Append-only: snapshots are history, not state.
CREATE TABLE publish_snapshot (
  district_id    text        NOT NULL,
  env            text        NOT NULL,
  course_id      text        NOT NULL,       -- "crs_88214"
  lesson_id      text        NOT NULL,       -- "lsn_4c19"
  lesson_version int         NOT NULL,       -- the version compiled
  snapshot_id    text        NOT NULL,       -- "snap_2026_08_25"
  element_key    text        NOT NULL,       -- stable across runs; becomes the external id
  element_kind   text        NOT NULL,       -- 'page' | 'assignment' | 'quiz_item' | 'file'
  content_hash   text        NOT NULL,       -- post 1's block hash, or a hash of the rendered element
  first_published_at timestamptz NOT NULL,   -- carried forward across runs, never recomputed
  captured_at    timestamptz NOT NULL DEFAULT now(),
  PRIMARY KEY (district_id, env, course_id, snapshot_id, element_key)
);

export type Op = "add" | "update" | "remove";

export interface ElementDelta {
  elementKey: string;      // stable identity — this is what makes the publish idempotent
  op: Op;
  kind: string;
  contentHash: string | null;
}

export function diff(current: SnapshotRow[], previous: SnapshotRow[]): ElementDelta[] {
  const prev = new Map(previous.map((r) => [r.elementKey, r.contentHash]));
  const out: ElementDelta[] = [];
  for (const row of current) {
    const before = prev.get(row.elementKey);
    if (before === undefined) out.push({ ...row, op: "add" });
    else if (before !== row.contentHash) out.push({ ...row, op: "update" });  // update, never delete+create
    prev.delete(row.elementKey);
  }
  for (const [elementKey] of prev) out.push({ elementKey, op: "remove", kind: "", contentHash: null });
  return out;
}

Updating rather than deleting-and-creating is the line that protects student work, and it only exists because the element key is stable. That key is carried into the platform as an external identifier, so a re-publish finds and updates the existing object instead of creating a second one beside it. Get this wrong and the first symptom is not an error; it is a teacher with two copies of every assignment.

The adapter

Every platform has its own shape, and none of it is your engine's business

The wrong shape here is a class per platform with the packaging baked in, each with its own loop, its own retry, its own quiet divergence. Six months later one of them has a bug the others do not. Instead, every adapter declares a capability descriptor and one generic engine reads it.

export interface LmsCapabilities {
  id: string;
  transport: "rest" | "cartridge_package" | "deep_link";
  supportedKinds: ElementKind[];         // what survives; everything else degrades or drops
  degradations: Partial<Record<ElementKind, ElementKind | "drop">>;
  maxBatchSize: number;
  maxPackageBytes: number | null;        // package imports have a size ceiling; REST does not
  supportsUpdateInPlace: boolean;        // false => removal must become unpublish, not delete
  supportsDeleteWithSubmissions: boolean;
  rateLimitPerMinute: number;
  externalIdField: string | null;        // null => no idempotency; see below
}

// The numbers below are placeholders. Read the real ones off each platform's current
// published limits and your own instance's configuration — never hard-code a guess.
export const MAJOR_LMS: LmsCapabilities = {
  id: "canvas",
  transport: "rest",
  supportedKinds: ["page", "assignment", "quiz_item", "file"],
  degradations: { interactive: "page", drag_and_drop: "page", hotspot: "drop" },
  maxBatchSize: 50,
  maxPackageBytes: null,
  supportsUpdateInPlace: true,
  supportsDeleteWithSubmissions: false,   // unpublish instead — the engine handles this generically
  rateLimitPerMinute: 180,
  externalIdField: "integration_id",
};

• Capabilities are data, not code. Adding a platform is a descriptor plus a transport function; the batching, pacing, checkpointing, degradation and dead-lettering are inherited.
• Degradations are an explicit, auditable policy. An interactive block that becomes a static page is a known loss, declared up front and counted, not a surprise a teacher discovers. A kind mapped to drop must never be silently dropped — it is dead-lettered with a reason.
• A platform that cannot delete with submissions gets removals routed to unpublish. Generically, once, in the engine. A special case is where somebody eventually deletes an assignment thirty children had already submitted to.
• No external id field means you cannot be idempotent. That is a fact about the platform, and the honest response is a descriptor that says so and an engine that refuses to auto-publish to it — rather than a retry that quietly duplicates a course.

import type { LmsCapabilities } from "./capabilities.js";
import { lastCheckpoint, saveCheckpoint } from "./checkpoint.js";
import { deadLetter } from "./deadletter.js";
import { unpublish } from "./unpublish.js";

export async function publishBatches(
  caps: LmsCapabilities,
  publishId: string,
  deltas: ElementDelta[],
  send: (batch: ElementDelta[]) => Promise<BatchResult>,
): Promise<void> {
  const [removals, rest] = partition(deltas, (d) => d.op === "remove");
  if (removals.length && !caps.supportsDeleteWithSubmissions) await unpublish(publishId, removals);

  const resume = (await lastCheckpoint(publishId))?.batchIndex ?? -1;
  for (let i = 0; i * caps.maxBatchSize < rest.length; i++) {
    if (i <= resume) continue;                                  // already durably accepted
    const batch = rest.slice(i * caps.maxBatchSize, (i + 1) * caps.maxBatchSize);
    const res = await send(batch);                              // paced by rateLimitPerMinute
    for (const r of res.rejected) await deadLetter(publishId, r.elementKey, r.reason);
    await saveCheckpoint({ publishId, batchIndex: i, accepted: res.accepted,
                           rejected: res.rejected.length, at: new Date().toISOString() });
  }
}

The checkpoint is written after a batch is accepted, so a crash between batch 7 and 8 resumes at 8 rather than replaying an entire district. Idempotency is the external id’s job at the element level and the checkpoint’s job at the batch level; you need both.

Partial failure is the normal case, not the exception. A fifty-element batch comes back accepted with three rejected — a maths fragment the importer’s sanitiser stripped, a file type the instance disallows, a validation rule you will never see documented. Failing the publish throws away forty-seven good elements over three bad ones. Dropping the three makes the dashboard green and the lesson wrong. You dead-letter each element with the importer’s verbatim reason and surface the dead-letter rate beside the publish’s success flag.

export interface DeadLetterRow {
  publishId: string;       // "pub_7f31"
  districtId: string;      // "riverbend"
  courseId: string;        // "crs_88214"
  lessonId: string;        // "lsn_4c19"
  elementKey: string;
  elementKind: string;
  reason: string;          // importer-reported, verbatim, never rewritten or prettified
  batchIndex: number;
  at: string;
}

A publish that reports success while dropping elements is worse than one that fails, because it does not wake anyone up. It wakes a nine-year-old up, in a lesson with a hole in it.

Fidelity

The one number that tells you how much of the lesson survived

Fidelity rate is elements submitted versus elements that render correctly in the student view. The denominator matters, and it is not “elements authored” — a block the gate held and a block type the transport cannot carry are both real losses, but they are losses you caused before the request left your building, and folding them into one rate makes every cause look like the importer’s fault.

• The last row cannot be measured from a response code. An importer returning success tells you it accepted the element, not that a child can use it. Rendering fidelity needs a real check — a headless fetch of the published page against an expected-elements manifest, run on a sample.
• Instrument per lesson, per course, per publish. A district-level average hides the one course that fell off a cliff.
• Alert on a relative drop against that binding’s own trailing baseline. A text-heavy lesson round-trips near-perfectly and an interactive one does not; an absolute floor either screams constantly or never fires.
• Report degradation separately from rejection. Losing two elements to a transport that cannot carry them is a product decision to revisit in the editor’s block palette. Losing one to a sanitiser is a packaging bug. One alert for both tells you nothing.

import { db } from "../db/client.js";

export async function checkFidelity(
  districtId: string, courseId: string, lessonId: string, observed: number,
): Promise<{ baseline: number; dropPct: number; alert: boolean }> {
  const { rows } = await db.query(
    `SELECT percentile_disc(0.5) WITHIN GROUP (ORDER BY fidelity) AS baseline
       FROM publish_audit
      WHERE district_id = $1 AND course_id = $2 AND lesson_id = $3
        AND ts >= now() - interval '30 days'`,
    [districtId, courseId, lessonId],
  );
  const baseline = rows[0]?.baseline ?? observed;
  const dropPct = baseline > 0 ? (baseline - observed) / baseline : 0;
  return { baseline, dropPct, alert: dropPct >= 0.1 };     // relative, never absolute
}

The silent killer

A course that stopped updating three weeks ago

The district’s access token expires on a Tuesday. The publish is rejected. The error is caught, logged at warning level, and the job exits successfully — because it did run, it just did not accomplish anything. The nightly dashboard is green. The course sits frozen at whatever it was on the fourth, teachers keep assigning from it, and the corrections your team has shipped since — including the one that fixed a wrong answer key — have reached nobody. Somebody notices in a month, usually a parent.

The fix is not a better try-catch. It is structural: every course-and-lesson binding carries a freshness target and a last-successful-publish timestamp, and the alert fires on the absence of a success rather than the presence of an error. A caught exception can be swallowed. A timestamp that fails to advance cannot be.

interface Binding {
  districtId: string; courseId: string; lessonId: string; lms: string;
  freshnessSlaMinutes: number;              // e.g. 1440 for a nightly-synced course
  lastSuccessfulPublishAt: string | null;   // advanced ONLY on a fully completed publish
}

export async function staleBindings(now = Date.now()): Promise<Binding[]> {
  const bindings = await db.query<Binding>("SELECT * FROM publish_bindings WHERE active");
  return bindings.rows.filter((b) => {
    // Never-published is stale, not absent. A null must page someone.
    const last = b.lastSuccessfulPublishAt ? Date.parse(b.lastSuccessfulPublishAt) : 0;
    return now - last > b.freshnessSlaMinutes * 60_000;
  });
}

• The timestamp advances on completion only. Not on start, not on “the job ran”. A publish with dead-lettered elements does not advance it.
• Never-published is stale, not exempt. The most common form of this bug is a binding that never worked once and was therefore never compared to anything.
• The freshness checker is a separate process. A checker that runs inside the publish job dies with the publish job.
• The alert names the binding, not the job. “This district, this course, this lesson, twenty-one days stale” is actionable; “publish-engine warnings elevated” is not.

Monitor for the thing that should have happened, not just the thing that went wrong. The same table gives you the publish ledger — every lesson, every course, every publish id, appended as batches are accepted. It costs one table and a little write volume, and it is what makes two earlier posts work: post 2’s alignment retraction needs to know which courses hold a claim you have revoked, and post 4’s deletion fan-out needs to know where content went. Without it, both stop at your database.

git clone https://github.com/xaviramirezcom/open-lesson-pipeline
cd open-lesson-pipeline
git checkout v5-publish
pnpm install && uv sync
pnpm lesson up
pnpm lesson seed --with-submissions      # the mock course already has student work in it

# The naive nightly publish: export the course, import the course.
pnpm lesson publish --course crs_88214 --mode republish

# The same course, same night, diffed against the previous snapshot.
pnpm lesson publish --course crs_88214 --mode diff

# Where the elements went.
pnpm lesson fidelity --course crs_88214 --lesson lsn_4c19

Explain it back