Change a signup rule, then find out what it cost you

Shadow first: zero risk to signups

A shadow experiment evaluates a new policy version on every request and records what it would have decided, while your current version keeps answering. A week later you can see, on real traffic, how many signups it would have sent to review or denied, and which rule did it — before a single real person was affected.

What shadow cannot tell you is whether the change helps. Everyone got the same decision, so their outcomes cannot separate the variants. The report says exactly that, and shows the two decision mixes side by side instead.

Then enforce, on a share of traffic

• You choose the split, e.g. 80% control and 20% treatment, and the event that counts as success — usually subscribed.
• Assignment is deterministic per person: a hash of the experiment’s salt and the hashed address. Someone who retries the form gets the same experience, and the analysis is per person, not per request.
• Guardrails watch a second event — support tickets, say — and flag a variant that moves it more than you allowed.
• One running experiment per policy, and starting one is a separate, explicit call from creating it.

Scored on outcomes, never on our own counts

“We denied 20% more” is not an outcome. “Paid conversion held at 4.1% while denials rose” is. Experiments are scored only on events: the ones you send to /v1/events, and the ones Stripe sends when you connect it. Without events there is no result, and the report says so.

The report refuses to guess

• Each variant’s rate is a Beta posterior over its events and exposures, with a 90% credible interval.
• The headline is the probability that the treatment is better, computed from 10,000 draws in a nightly job — and the expected cost of being wrong either way. No p-values: “94% likely better” is a sentence a founder can act on.
• Below the minimum sample, nothing is shown. No rate, no interval, no probability — only how far there is to go. A three-exposure experiment at 100% is the most common way a dashboard lies.

What the report’s one-line summary reads like, at each stage:

• “Not enough data yet — 212 of 400 minimum exposures in treatment.”
• “Treatment is better with 94% probability; expected cost of being wrong is 0.2 percentage points.”

Those are the report’s own sentence templates, with example numbers.

Where experiments come from

You can write one in English (“try the new policy on 20% of traffic for two weeks and watch paid conversion”) and the compiler drafts the configuration for you to confirm. And once outcomes are flowing, segment reports surface the groups that convert worst, each with a proposed rule and a ready-made experiment to try it as. The proposal is always review, never deny; tightening it is your call.