Measurement & incrementality

The interval

Can this test see the thing you are hoping for?

Start at the end, because the arithmetic below decides whether the rest of the post is worth running. Three controls: how big the audience is, how much of it you are willing not to market to, and how big the effect really is.

A large share of the tests people actually run cannot detect the effect they are hoping for, and instead produce a number, with a decimal point, that is noise. You can know that before you spend a cent — which is the entire reason to compute it first.

The problem

Last click makes your best channel the one that would have happened anyway

Follow one shopper. She has a cart from Tuesday, an email in her inbox, and every intention of buying. On Thursday she types the brand name into a search box, clicks the ad at the top because it is at the top, and completes the order. Last click hands that channel the entire purchase. Nothing about the ad changed her mind; it was standing between her and a checkout she was already walking toward.

• Attribution divides credit, never causation. Every model in the family takes the set of conversions that happened and splits them among the touchpoints that preceded them. The denominator is fixed before the model runs.
• Multi-touch does not fix the selection. First-touch, linear, time-decay and data-driven models redistribute credit among touchpoints that all share the same problem — the people who saw them were the people most likely to convert.
• There is no counterfactual in the data. These models are fit on observational logs. Nowhere in that log is a comparable person who was not exposed, so no amount of modelling can recover the difference.
• The feedback loop is self-sealing. Brand search reports nine times return, gets more budget, and reports nine times on the new budget too — because the extra spend bought more clicks from people who were already going to buy.

Be fair to attribution. It is genuinely useful for diagnosis and pacing: which creative is getting clicks today, whether spend is tracking to plan, where a sudden drop in traffic came from. Those are real jobs. It is just not evidence that money caused revenue, and it should never be the number in a sentence containing the word “drove”.

The holdout

Hold out in your platform, not in theirs

A holdout is the counterfactual you buy by not spending. The mechanism is one line of arithmetic: hash the profile id together with the experiment id, take it modulo ten thousand, and compare against the holdout size. Nothing is stored, so nothing can be lost, and the assignment can be re-derived from first principles by an analyst months later who was not in the room.

• Deterministic beats random-and-stored. A hash of two strings is stable across runs, across restarts, and across a warehouse migration. A stored random assignment is a table someone can truncate.
• Salting by experiment keeps tests independent. The same profile in the holdout for one test is not systematically in the holdout for the next, because the hash input changed.
• The sync engine subtracts it, not the audience definition. Membership is compiled, then the holdout is removed on the way out. Someone editing the audience SQL cannot re-include the control group, and a second overlapping audience cannot leak treatment in through a side door.
• The experiment record is the contract. Audience, destination, holdout size, start, planned end, target effect size and owner are written once, before the first sync, and the analysis is read against them.

import { createHash } from "node:crypto";

export interface Experiment {
  experimentId: string;        // "exp_prospecting_q3"
  brandId: string;             // "northwind"
  audienceId: string;          // "aud_high_intent_30d"
  destination: "meta" | "google" | "klaviyo" | "webhook";
  holdoutBps: number;          // 1000 = 10% held out
  startsAt: string;            // ISO 8601
  plannedEndAt: string;        // fixed before the first sync — see Power
  primaryMetric: "revenue_per_profile";
  mdeRelative: number;         // 0.05 = detect a 5% relative lift
  owner: string;               // "growth@northwind"
}

/** Stable, storage-free assignment. Re-derivable months later from these two strings. */
export function isHeldOut(profileId: string, exp: Experiment): boolean {
  const digest = createHash("sha256").update(`${profileId}:${exp.experimentId}`).digest();
  return digest.readUInt32BE(0) % 10_000 < exp.holdoutBps;
}

import type { MemberRow } from "./types.js";
import { isHeldOut, type Experiment } from "../measurement/holdout.js";

const compiled: MemberRow[] = await compileAudience(scope, audience);       // post 5
const experiments: Experiment[] = await experimentsFor(scope, audience.id); // zero or one

const treated: MemberRow[] = compiled.filter(
  (p) => !experiments.some((exp) => isHeldOut(p.profileId, exp)),
);

// Diff happens AFTER the subtraction, so a held-out profile that was synced
// before the experiment started is emitted as a removal, not silently left behind.
const { adds, removes } = diffAgainstSnapshot(treated, lastSnapshot);
await destination.send({ syncId, adds, removes });
await audit.write({ ...counts, holdoutBps: experiments[0]?.holdoutBps ?? 0,
                    experimentId: experiments[0]?.experimentId });

The overlap trap deserves its own paragraph, because it is silent. Two experiments run on populations that intersect — a prospecting test on one audience and an email test on a lifecycle audience sharing forty percent of its members — and neither knows about the other. Each treats the other’s treatment as noise, which inflates variance and hides a real effect. Worse, if the overlap is uneven between arms, one test reads the other’s lift as its own. The fix is boring: a registry of running experiments per brand, and a check at creation that refuses or flags an intersection above a threshold.

Power

Decide the sample size before you look

The uncomfortable arithmetic goes first, not last. At a three percent baseline conversion rate, detecting a five percent relative lift — three percent against 3.15 percent — needs roughly two hundred thousand profiles per arm. Most audiences on most brands are not that big, which is exactly what the interactive at the top of this post is showing you.

• The target effect size is a design input, not a result. Compute it from the audience you have, and if the answer is “we could only detect a thirty percent lift”, say so before spending.
• Peeking is not significance. Checking daily until the number goes green is a different test than the one you designed, and its real false-positive rate is several times the one you think you are running at.
• Fix the end date and the metric up front. Both live in the experiment record. An analysis that cannot choose its own window or its own metric has no room to drift.
• Underpowered is worse than not running. A null from an underpowered test gets read as “it does not work” and a false positive gets read as proof. Both move budget.

/** Two-sided z-test for a difference in proportions. Returns profiles required PER ARM. */
export function sampleSizePerArm(
  baselineRate: number,      // 0.03
  relativeLift: number,      // 0.05
  alpha = 0.05,
  power = 0.8,
): number {
  const p1 = baselineRate;
  const p2 = baselineRate * (1 + relativeLift);
  const pBar = (p1 + p2) / 2;
  const zA = 1.959964;                                   // two-sided alpha = 0.05
  const zB = 0.841621;                                   // power = 0.80
  const numerator =
    zA * Math.sqrt(2 * pBar * (1 - pBar)) + zB * Math.sqrt(p1 * (1 - p1) + p2 * (1 - p2));
  return Math.ceil((numerator ** 2) / ((p2 - p1) ** 2));  // ~196k at 3% baseline, 5% lift
}

/** Inverted: given the audience you actually have, what is the smallest lift you can see? */
export function minDetectableLift(baselineRate: number, perArm: number): number {
  let lo = 0.001, hi = 2.0;
  for (let i = 0; i < 40; i++) {
    const mid = (lo + hi) / 2;
    if (sampleSizePerArm(baselineRate, mid) > perArm) lo = mid; else hi = mid;
  }
  return hi;
}

The readout

Incremental revenue, incremental cost per acquisition, and an interval

The query is unglamorous, which is the point. Assign every profile in the compiled audience to an arm with the same hash the sync engine used, join conversions in the window, and compare per-profile means. No attribution model appears anywhere.

SELECT
  arm,
  count()                                         AS profiles,
  countIf(orders > 0)                             AS converters,
  countIf(orders > 0) / count()                   AS cvr,
  sum(revenue) / count()                          AS revenue_per_profile,
  stddevSamp(revenue)                             AS revenue_sd
FROM (
  SELECT
    p.profile_id                                                        AS profile_id,
    if(holdout_bucket(p.profile_id, {exp:String}) < {bps:UInt16},
       'holdout', 'treated')                                            AS arm,
    countIf(o.order_id != '')                                           AS orders,
    sum(o.revenue)                                                      AS revenue
  FROM audience_snapshot AS p
  LEFT JOIN orders AS o
    ON o.brand_id = p.brand_id
   AND o.profile_id = p.profile_id
   AND o.ts BETWEEN {start:DateTime} AND {end:DateTime}
  WHERE p.brand_id = {brand:String} AND p.audience_id = {aud:String}
  GROUP BY profile_id, arm
)
GROUP BY arm;

export interface LiftReadout {
  incrementalRevenue: number;        // (rpp_treated - rpp_holdout) * treatedProfiles
  incrementalCac: number;            // spend / incrementalConversions
  ciLow: number; ciHigh: number;     // 95% on incremental revenue
  includesZero: boolean;
  underpowered: boolean;             // observed n < the design's requirement
}

Two things to say out loud. A result whose interval includes zero is a result — it says the effect, if any, is smaller than this test could see, and it should be reported in exactly those words rather than quietly not reported. And a negative point estimate on a retargeting audience is the single most common honest finding in this whole discipline; it usually means you were paying to reach people already on their way to checkout, and the ads mostly bought attention you already had.

The governance tie-back is short. A holdout is a group of real people you deliberately chose not to market to, for a quarter, to learn something. That is a decision with a subject, an owner and a date, and it belongs in the same append-only audit log as every sync and every deletion — the holdout size and experiment id are already fields on post 4’s audit record.

git clone https://github.com/xaviramirezcom/open-activation-platform
cd open-activation-platform
git checkout v6-measurement
pnpm install
pnpm activate up            # streaming + warehouse + config store, via docker compose
pnpm activate seed          # synthetic multi-brand traffic, orders and spend

pnpm activate experiment create exp_prospecting_q3 \
  --brand northwind --audience aud_high_intent_30d --dest meta --holdout-bps 1000
pnpm activate sync --audience aud_high_intent_30d --dest meta --days 28

pnpm activate report attribution --audience aud_high_intent_30d --model last_click
pnpm activate report lift        --audience aud_high_intent_30d --exp exp_prospecting_q3

What you have now

1. Collection you control at the edge. A first-party server-side collector, server-stamped time, and events keyed so a device's history stays ordered — not a pixel whose behaviour a browser vendor decides for you.
2. Identities that can be un-merged. Profiles derived from a link graph with provenance on every link, so the merge that was wrong is a removable observation rather than permanent damage.
3. Features that do not leak the future. One feature definition, two paths, computed as of the start of the label window — the difference between an offline number you brag about and an online lift that exists.
4. Consent enforced by structure, deletion that fans out. Consent as a join key with no code path around it, and a tombstone that reaches your warehouse, your graph, your audiences and every destination you ever synced to.
5. Syncs that send deltas and report their own match rate. Diffs rather than full replaces, idempotent batches with checkpoints, dead-lettered rejects, and a freshness target that catches the expired token before the month does.
6. A number that survives “compared to what?”. A deterministic holdout the sync engine enforces, a power calculation done first, and a readout with an interval on it.

Explain it back