Experiments

Endpoints

The configuration

POST /v1/experiments
{
  "slug": "block-freemail-q4",
  "hypothesis": "Reviewing consumer mailboxes cuts support load without cutting paid conversion.",
  "policy": "default",
  "variants": [
    { "name": "control",   "version": 14, "weight": 80 },
    { "name": "treatment", "version": 15, "weight": 20 }
  ],
  "mode": "enforce",
  "metric": { "event": "subscribed", "window_days": 14 },
  "guardrails": [ { "event": "support_ticket", "max_rate_delta": 0.02 } ],
  "min_sample": 400
}

Enforce and shadow

• Assignment is per person, not per request. A request’s bucket is a hash of the experiment’s own salt and the hashed address, mod 100, cut by the weights. Someone who retries your signup form lands in the same variant every time.
• enforce returns the assigned variant’s decision. The experiment block on the decide response names the slug, the variant and the mode.
• shadow evaluates every variant and records each decision, but the active version answers. The response’s experiment.shadow_decision is what the assigned variant would have done. A one-variant shadow experiment is how you try a new version with zero risk to signups before enforcing it.
• One running experiment per policy at a time. A shadow variant that reads enrichment buys it, so starting one can change cost even though it cannot change a decision.

GET /v1/experiments/:slug/report

How it is analysed

• Each variant’s metric rate is a Beta posterior over its events and exposures, starting from a uniform Beta(1, 1) prior.
• The probability that treatment beats control comes from 10,000 Monte Carlo draws from the two posteriors. A verdict is called at 90%: treatment_better at 0.9 or above, control_better at 0.1 or below, otherwise inconclusive.
• The draws run in the nightly job, never on a request. Before the first nightly run, the report shows exposures, decision mix and per-variant rates with the probability left null and a sentence saying it is computed nightly.
• Below min_sample in any variant the verdict is insufficient_data and no rate or probability is shown. A three-exposure experiment at 100% is the most common way a dashboard lies.
• A shadow experiment’s verdict is always inconclusive: every request received the same decision, so the metric cannot separate the variants. Compare the decision mix, then run it in enforce mode to measure the outcome.
• Metrics come only from events — ones you send to /v1/events or ones Stripe sends. Sortance’s own decision counts are never the metric.

Errors