How to Run Growth Experiments

Most teams describe what they do as testing, but very little of it qualifies as an actual experiment. A button color gets changed. A subject line gets swapped. The result gets glanced at for a few days, and whichever version has a slightly higher number gets called the winner. This is testing in name only. Without a real hypothesis, a defined success metric, and enough data to trust the result, it is closer to guessing with extra steps.

Growth experiments that actually move revenue look different at every stage of the process, from what gets tested first to how a result gets declared valid.

Start With a Hypothesis, Not a Tactic

A tactic says: let's try adding urgency to the subject line. A hypothesis says: subscribers are opening emails but not clicking through, which suggests the subject line is doing its job but the body copy isn't giving them a reason to act, so testing a clearer call to action in the first two lines should move click-through rate.

The difference is not just phrasing. A hypothesis is falsifiable and tied to a specific piece of data showing where the current process is underperforming. A tactic is copied from somewhere else and applied without knowing whether it addresses the actual problem. Hypotheses come from looking at where the numbers drop off. Tactics come from lists of generic tips.

Prioritize Before You Test, Not After

Most businesses can think of more experiment ideas than they have time or traffic to test. Without a prioritization method, the loudest idea in the room wins, regardless of whether it is the highest-impact one. A simple framework like ICE, scoring each idea on Impact, Confidence, and Ease on a 1 to 10 scale, forces a comparison instead of a debate. An idea that could meaningfully lift conversion but is hard to build might still outrank an easy idea that barely moves the needle.

The goal of prioritization is not certainty about which test will win. It is making sure the tests that get run are the ones most likely to matter, given limited time and traffic.

Define the Metric Before the Test Starts, Not After

A test that doesn't specify its success metric in advance is vulnerable to a specific bias: after the results come in, whichever metric looks best gets presented as the outcome. Open rate went up, so the test is called a win, even though click-through and revenue per email both declined. Deciding the primary metric before launch, and treating everything else as secondary context, keeps the result honest.

Respect Sample Size

A test run on two hundred people that shows a fifteen percent lift can look exciting and mean almost nothing. Small sample sizes produce noisy results that swing widely from natural variation alone, not from the change being tested. Before declaring a winner, the sample needs to be large enough that the difference is unlikely to be random chance, and the test needs to run long enough to capture a full cycle of customer behavior, not just whichever days happened to fall within the test window.

This is one of the most common failure points in growth experimentation: ending a test the moment one version pulls ahead, without accounting for the fact that a few days of data rarely represents the full picture.

Watch for Vanity Metrics Masking a Flat Result

Open rate, click rate, impressions, and follower counts are easy to track and easy to move, which makes them tempting to optimize for their own sake. But a test that raises open rate while leaving revenue per recipient unchanged has not actually produced growth. It has produced a better-looking dashboard.

The fix is straightforward but often skipped: tie every experiment back to a revenue or retention metric, even if a secondary engagement metric is also tracked. An experiment that moves engagement without moving revenue is worth understanding, since it might reveal something about the audience, but it should not be reported as a growth win.

Build a Backlog, Not a One-Off Test

A single successful test rarely changes a business by itself. What changes a business is a running backlog of hypotheses, prioritized, tested in sequence, with results documented well enough that six months from now the pattern of what worked and what didn't is still visible. Without that documentation, the same failed idea gets re-tested by someone new who wasn't around for the first attempt, and the same successful pattern doesn't get applied to the next campaign because nobody remembers it worked.

Where to Start

For a business running its first real growth experiments, the highest-leverage starting point is usually the part of the funnel with the biggest, most measurable drop-off: cart abandonment, email open-to-click ratio, or first-purchase-to-second-purchase rate. These stages already have a baseline number to beat, a clear metric to track, and enough volume to reach a valid sample size relatively quickly.

Once that first disciplined test is complete, win or lose, the real value isn't the single result. It's a repeatable process, hypothesis, prioritization, defined metric, valid sample, that keeps producing evidence instead of opinions about what should work next.