Most companies that say they run experiments actually run tests. The distinction sounds pedantic and is not: a test is one comparison, and a program is a system that turns a stream of comparisons into things the organisation now believes. Plenty of teams do the first and get almost nothing from it — a folder of dashboards, a handful of contested wins, and no accumulated understanding of what moves their product.
The gap is rarely statistical. It is that hypotheses were vague enough that a null result taught nobody anything, that success metrics were chosen after the data arrived, that guardrails were not agreed, and that the one experiment that did work has been quoted for two years without anyone checking whether it still holds.
This guide covers the experiment lifecycle and the decisions inside each stage, what separates a hypothesis from a plan, how to choose guardrails, why most experiments should be expected to fail, the honest version of sample size, what to do at low traffic, and when experimenting is simply the wrong tool.
Key Takeaways
- A hypothesis needs a mechanism. Name why you think it will work, not just what you will change — the mechanism is what you actually learn, and it is what makes a failure informative.
- Most experiments will not win. Large published programs consistently report that only a minority of tests produce a clear positive. Design the program around that instead of being surprised by it.
- Guardrails before results. Most local wins are borrowed from somewhere else. Agree what must not get worse while nobody is invested in the outcome.
- Decide the stopping rule in advance. Peeking until a result looks good is the most common way teams produce confident nonsense.
- Low traffic is a reason to change method, not to fake rigour. Five usability sessions beat an underpowered test that cannot reach significance.
- Ask what a flat result would change. If the answer is nothing, skip the experiment and ship.
What Product Experimentation Actually Is
Product experimentation, in one paragraph
Product experimentation is the practice of learning what genuinely changes user behaviour by shipping controlled variations and measuring the difference, rather than settling it by argument or seniority. It is broader than A/B testing, which is one technique inside it: a program also covers how ideas get prioritised for testing, what counts as a testable hypothesis, which metrics are protected, how decisions are recorded, and — most importantly — what the team does with the large majority of experiments that do not produce a win.
The organisational value is not the individual wins. It is that a functioning program slowly replaces a set of shared assumptions with a set of shared findings, and that changes which arguments are possible. A team that has tested six variations of its signup flow has a different conversation about the seventh than a team that has tested none.
The stages that get skipped are the first and the last — and they are the two that make a program compound rather than accumulate.
The Hypothesis Is the Whole Experiment
Almost every weak experiment can be traced to a weak hypothesis. The tell is that you cannot say, before running it, what you would learn from a null result.
Not a hypothesis
"Redesign the onboarding flow to improve activation."
This is a plan. If activation does not move, you know only that this particular redesign did not work — not why, and not what to try next. Every possible result leads to a shrug.
A hypothesis
"Adding a progress indicator to setup will raise setup completion by at least 3 points, because users abandon when they cannot tell how much is left."
A flat result now means something: uncertainty about remaining effort was probably not the reason people abandoned, so look elsewhere.
The four parts worth writing down every time: the change, the primary metric and the size of effect that would matter, the mechanism you believe explains it, and what you will conclude if nothing happens. The mechanism is the one teams omit, and it is the one that turns an experiment into knowledge rather than a coin flip with extra steps.
Specifying the minimum effect that would matter also does quiet, useful work: it forces the question of whether the change is worth the cycle time at all. A great many experiments are proposed for effects too small to justify a two-week slot, and this is where that becomes obvious.
Primary Metrics and Guardrails
One primary metric per experiment. Not three — one. Multiple primary metrics means you will find something that moved and declare victory, which is not measurement, it is fishing.
The primary metric should be as close as possible to the value you actually care about while still being sensitive enough to move. That is a real tension. Clicks move quickly and mean little; retention means a great deal and moves too slowly to steer by. For most onboarding and adoption work, the sweet spot is an activation event — a first genuine moment of value rather than a click on the thing you changed.
The rule that saves programs: if your primary metric is the immediate interaction with the thing you changed, you will win almost every experiment and improve almost nothing. A bigger button gets more clicks. Measure one step past the change.
Guardrails are the metrics you agree in advance must not get worse. They exist because most local wins are borrowed from somewhere else in the system, and without guardrails those loans are invisible.
| If the experiment targets… | The likely hidden cost | Guardrail to set |
|---|---|---|
| Signup conversion | Lower-intent users who never activate | Week-one activation rate of the signup cohort |
| In-app prompt engagement | Fatigue and opt-outs across all messages | Dismissal and notification opt-out rates |
| Trial-to-paid conversion | Customers converted before they saw value | Day-30 and day-90 retention of converts |
| Onboarding speed | Skipped setup that resurfaces as tickets | Support contacts per new account |
| Feature adoption | Attention taken from a more valuable feature | Usage of the adjacent core workflow |
| Any UI change | Performance and accessibility regressions | Error rate, load time, keyboard and screen-reader paths |
A guardrail in practice: after an experiment lifts adoption of one feature, check that the adjacent core workflow did not quietly pay for it.
Set the thresholds before launch. This is not a formality — after results arrive, every threshold becomes negotiable, and the negotiation is always won by whoever wants to ship. The leading versus lagging indicator distinction is useful here: your primary metric is usually leading, and at least one guardrail should be lagging, precisely because that is where the borrowed wins show up.
Most Experiments Fail, and the Program Depends on That Being Fine
Published accounts from large experimentation programs consistently report that only a minority of tests produce a clear positive result, with a substantial share landing flat. Teams encountering this for the first time usually conclude that something is wrong with their process. Nothing is wrong. That is what testing ideas looks like when you are testing genuinely uncertain ones.
This has three practical consequences, and a program that ignores them stalls within two quarters.
If the expected win rate is low, cost per experiment is the variable that decides whether the program is worth running. Long build times kill it.
They only are if the hypothesis had a mechanism. Otherwise most of your program produces nothing at all, which is why the mechanism is non-negotiable.
If a flat quarter reads as a wasted quarter, teams will start testing only safe changes — and safe changes have small effects.
A high reported win rate is worth interrogating rather than celebrating. It usually means one of three things: the team is only testing changes it already knows will work, results are being called early when they look good, or the primary metric sits too close to the change to be meaningful.
Sample Size and Duration, Honestly
The uncomfortable arithmetic of experimentation is that the smaller the effect you want to detect, the more traffic you need — and the relationship is unforgiving. Halving the effect size you want to catch roughly quadruples the sample you need. This is why so many product teams run experiments that could never have detected the thing they were looking for.
Three rules cover most of the practical damage:
- Compute the required sample before launching, from your baseline rate and the minimum effect worth acting on. If the answer is longer than a month, redesign the experiment rather than running it hopefully.
- Fix the stopping rule in advance and do not look for significance early. Repeatedly checking a running test and stopping when it crosses a threshold produces false positives at a much higher rate than the stated one — this is the single most common way honest teams produce wrong answers.
- Run at least one full weekly cycle, and preferably two. Weekday and weekend users behave differently, and a test that ran Tuesday to Friday has measured a population, not your population.
There is also a subtler trap in onboarding experiments specifically: the outcome you care about often happens days after the exposure. If your activation window is seven days, a test that has been running seven days contains almost nobody who has had the chance to activate. Our guide to A/B testing onboarding goes into the mechanics of that lag and the tests worth running in that surface specifically.
What to Do When You Do Not Have the Traffic
Most B2B products cannot run classical experiments on most of their surfaces, and pretending otherwise produces the worst outcome available: confident conclusions from underpowered tests. If a surface sees a few hundred users a month, the honest answer is that you will not detect a three-point change in it, ever.
The alternatives are real methods, not consolation prizes:
Only large effects are detectable at low volume, so test substantially different approaches rather than variations of one. A restructured flow, not a reworded button.
Five to eight usability sessions will locate a broken step faster and more definitively than a test that cannot reach significance.
Session replay on the specific step where people drop tells you what a percentage never will — which control they were hunting for.
Hold a small cohort back from a change and compare over a longer window. Slower, weaker than a clean split, and far better than nothing.
Before-and-after on one obvious change, with seasonality acknowledged out loud. Weak evidence honestly labelled beats strong evidence invented.
Run the test across every entry point into a flow rather than one page, to get the volume up to something detectable.
When Not to Experiment
Experimentation has a cost — cycle time, instrumentation, coordination, and the delay before a known improvement reaches users. Treating it as a universal requirement is its own failure mode, and it is the one that makes engineers hate the program.
Do not test a fix for something broken
Data loss, a dead link, an accessibility defect, a confusing error message. You are not going to keep the broken version if the test comes back flat, so the test is pure delay.
Do not test what you would ship regardless
If the decision is already made — for strategic, legal or brand reasons — running an experiment is theatre that costs two weeks and buys nothing.
Do not test effects too small to matter
If the best plausible outcome is a fraction of a point on a metric nobody reports, the slot is better spent on something with a chance of mattering.
Do not test values questions
Whether to use a dark pattern that would lift conversion is not an empirical question. It will win the test and cost you trust, and the test result will be used to end the argument.
The single filter that catches most of these: what would we do differently if this came back flat? If the answer is "nothing", skip the experiment and ship the change.
Turning Tests Into a Program
The difference between a team that runs experiments and a company that has an experimentation program is almost entirely in the two stages nobody enjoys: what happens before the test is chosen, and what happens after the result is in.
1. A backlog that is prioritised, not a queue
Experiment ideas should be ranked by expected value over cost — the size of the effect if it works, times the confidence that it might, divided by the effort and the traffic it consumes. Traffic is a genuinely scarce resource: two experiments running on the same surface interfere, and a queue that runs first-in-first-out will spend your scarcest asset on whatever was suggested most recently. The same prioritisation frameworks used for features work here with little adaptation.
2. A decision log everyone can search
One entry per experiment: hypothesis, mechanism, what you did, what happened, what you decided, and the date. This is the artefact that makes the program compound, because it is what stops the same idea being re-proposed every eight months and stops a single two-year-old win being cited as permanent truth. Include the failures — they are the majority of the value, and they are what a searchable log gives you that a dashboard does not.
3. Instrumentation that exists before the test
Experiments fail most often for the dullest reason: the event needed to measure the outcome was not being collected, or was collected inconsistently between variants. A stable event taxonomy is a precondition for the program, not a nice-to-have alongside it.
4. A short cycle time on the surfaces you test most
Because the win rate is low, experiment throughput matters more than the quality of any single test — and throughput is usually limited by engineering scheduling rather than by ideas. Surfaces where a variant can be built and shipped without a release get tested an order of magnitude more often, which is why in-app guidance, onboarding flows and messaging tend to be the best-understood parts of a mature product.
5. Re-tested wins
Effects decay. A prompt that lifted adoption two years ago is competing with a different product, a different user base and a different set of habits. Periodically re-running a foundational win — or simply removing it for a holdout cohort — is the only way to find out whether your beliefs are still true.
Test the onboarding surface without waiting for a release
Experiment throughput is usually limited by how long a variant takes to ship. Kompassify lets you build and target tour, checklist, tooltip and in-app message variants yourself, segment who sees which, and read step-level completion for each — so the surfaces closest to activation can be tested in days instead of sprints. Free up to 100 monthly active users, plans from $129/month, GDPR-compliant and EU-hosted.
Start for free →The One-Sentence Version
A real experimentation program is not a stack of tests — it is hypotheses with mechanisms so failures teach you something, guardrails agreed before anyone is invested, a stopping rule fixed in advance, and a searchable log of what you now believe.
Frequently Asked Questions
What is product experimentation?
Product experimentation is the practice of learning what actually changes user behaviour by shipping controlled variations and measuring the difference, rather than by argument or seniority. It is broader than A/B testing, which is one technique within it: a program also covers how ideas are prioritised for testing, what counts as a testable hypothesis, which metrics are protected as guardrails, how decisions are recorded, and what the team does with the large majority of experiments that do not produce a win.
What makes a good experiment hypothesis?
A good hypothesis names the change, the expected effect on one specific metric, the mechanism you believe explains it, and what you will conclude if it fails. The mechanism is the part teams skip and the part that makes the experiment worth running, because it is what you actually learn. "Adding a progress indicator to setup will raise setup completion, because users abandon when they cannot tell how much is left" teaches you something either way. "Redesign onboarding to improve activation" is a plan, not a hypothesis, and a null result from it tells you nothing.
What are guardrail metrics?
Guardrail metrics are the measures you agree in advance must not get worse, even if the primary metric improves. They exist because most local wins are available by borrowing from somewhere else: a more aggressive prompt lifts clicks and raises opt-outs, a shorter signup lifts conversion and lowers activation, an upsell lifts trial revenue and raises churn. Typical guardrails are downstream activation or retention, support contact rate, opt-out and unsubscribe rates, error rates and page performance. Set the thresholds before the experiment starts, because afterwards every threshold becomes negotiable.
How many product experiments actually succeed?
Far fewer than teams expect, and that is the normal state rather than a sign of a broken process. Published accounts from large experimentation programs consistently report that only a minority of tests produce a clear positive result, with many landing flat. The practical implication is that a program has to be designed around failure being the common case: cheap tests, fast cycles, a decision log that makes null results useful, and leadership that does not treat a flat result as a wasted quarter. A team that reports a high win rate is usually testing only safe changes, or stopping tests when they look good.
What should you do if you do not have enough traffic to run experiments?
Accept it and change methods rather than running underpowered tests that produce confident nonsense. Below meaningful volume, the useful alternatives are qualitative: five to eight usability sessions will find a broken flow far faster than a test that cannot reach significance. You can also test bigger changes rather than small ones, because only large effects are detectable at low volume, use before-and-after comparisons on a single obvious change while acknowledging their weaknesses, or run a holdout cohort over a longer window. The mistake is not the absence of experiments — it is pretending an underpowered one was conclusive.
When should you not run an experiment?
When the answer is already known, when you would ship regardless of the result, when the change is a fix for something plainly broken, when the effect is too small to be worth the cycle time, or when the decision is a matter of strategy or values rather than of measurement. Testing whether to fix an accessibility defect or a data-loss bug is not rigour, it is delay. A useful filter before every experiment is the question: what would we do differently if this came back flat? If the answer is nothing, skip the test and ship.