Building a Growth Experimentation Program

Ask a growth team why they only shipped three tests last quarter and you’ll get the same answer everywhere: we ran out of time. You will almost never hear that they ran out of ideas. The backlog is full. The reason nothing moves is that intake, specification, build, analysis and decision-making are all happening ad hoc, in Slack, at whatever pace the busiest person can manage.

That makes experimentation an operations problem. Fix the operations and the same team ships three times as much without working longer.

Why experiment throughput stalls, and what it is not

The bottleneck is specification and decision-making, not creativity. Ideas arrive as one-line Slack messages, someone half-builds one, results get eyeballed in a dashboard, and the decision gets deferred because nobody agreed in advance what would count as success.

Watch where time actually goes across a four-week window. In most SaaS growth marketing teams, build takes maybe 20% of the elapsed calendar time. The rest is waiting: waiting for a spec, waiting for a design, waiting for someone to decide whether a 4% lift with a wide confidence interval means ship.

The intake format that makes an idea testable

One form, one place, five fields. Refuse anything that arrives another way, including from the CEO, because the exception is what kills the system.

• Hypothesis: If we change X for audience Y, metric Z will move because of reason R. The “because” is not optional. Without it you learn nothing from a loss.
• Primary metric and guardrail metric: One metric you expect to move, one you refuse to damage. Trial signups up, trial-to-paid not down.
• Expected effect size: A number, even a guessed one. It forces a sample size conversation immediately.
• Build estimate: In days, from whoever would build it.
• Prior art check: Has a similar test run before? Link the archive entry or state that you searched and found nothing.

That last field does more work than the other four combined. It is the only thing standing between you and re-running a test from 2024 that already lost.

The growth experiment brief template covers the full version, including the pre-registered decision rule. Keep the intake form shorter than the brief: intake decides whether the idea is worth specifying, the brief is what you write once it clears prioritisation.

Prioritisation with a hard capacity limit

Score each idea as impact times confidence divided by effort, on a one to five scale each. That produces a number between 0.2 and 25 and lets you rank. Fine. Any reasonable scoring rule works.

The part everyone skips is the capacity limit. Write down the maximum number of tests that may be live at one time, and stop starting new ones when you hit it.

Team shape	Tests live at once	Tests shipped per month	What usually breaks first
One growth marketer, borrowed engineering	2	2 to 4	Analysis waits on a shared data analyst
Two marketers plus a half-time engineer	3	4 to 6	Design queue for variant assets
Pod of three to four, dedicated engineering	4 to 5	8 to 12	Decision meetings slip and tests over-run
Multiple pods, platform team	10 plus	20 plus	Interaction effects between concurrent tests

Notice what the table does not say. It does not promise twenty tests a month for a small team, because that number gets quoted in conference talks by companies running consumer traffic at a scale no B2B SaaS product has. Booking.com runs thousands of concurrent tests. You are not Booking.com and pretending otherwise produces a program that ships nothing and feels bad about it.

If you want to compare against real distributions rather than one table, the experiment velocity benchmarks break throughput out by team size and traffic band.

The weekly cadence

Three touchpoints. Same days, same length, no negotiation. The rhythm matters more than the specific agenda.

A sample Friday agenda: five minutes on throughput numbers, then ten minutes per test with a fixed structure. Hypothesis restated, result shown, decision called, archive entry written live on screen. Three tests fills 45 minutes. If you have six to review, you launched too many at once.

The rule that keeps this honest: no peeking at results before the end condition, and no discussing a running test’s outcome in the health check. Ron Kohavi’s work on trustworthy experiments is blunt about what repeated peeking does to false positive rates, and the temptation is strongest exactly when the early numbers look good.

Decision rules written before launch

The most valuable fifteen minutes in the entire program happen before a test goes live. Write down three numbers and one date.

Set the runtime cap at four weeks for most B2B funnels. Longer than that and seasonality, pricing changes and product releases contaminate the comparison anyway.

If you genuinely cannot get enough traffic, stop split testing small changes and test bigger swings instead: a different pricing structure, a removed signup step, a completely different onboarding path. Large effects are detectable with small samples. Two percent lifts are not.

The archive that stops repeats

Every test gets an entry, win or loss. Same fields every time: hypothesis, variant description with a screenshot, dates, sample size, primary and guardrail results, the decision, and one sentence on what it taught.

Losses matter more than wins here. A win gets shipped and becomes the product, so it documents itself. A loss disappears into nobody’s memory and comes back as a bright idea eighteen months later, usually from someone who joined after it ran.

Make the archive searchable by surface and by metric, so someone planning a pricing page test can pull every pricing page test you have run. Whatever tool you use, keep it in one place. The A/B testing tools for SaaS comparison covers which platforms store this well and which expect you to keep your own record, which most of them do.

A team that reports only its wins is not experimenting. It’s doing marketing with extra steps and a dashboard. The honest program shows the quarterly slide with 11 tests, 3 wins, 2 kills, 6 inconclusive, and the learning from each, because that is what tells a board the process is real.

What to test when the funnel is small

Prioritise surfaces by traffic concentration, not by how interesting the idea is. Pricing pages, signup flows and the first-run experience usually carry the highest conversion density in a SaaS funnel, which makes them the only places a mid-size product can detect modest effects.

Everything below that traffic threshold gets tested differently. Sequential before-and-after with a matched comparison period. Qualitative research on five customers. Painted-door tests to measure demand before building. These are weaker evidence, and you should say so when you report them, but weak evidence gathered honestly beats a split test that never reached power and got called a win anyway.

This is also where compounding mechanics matter more than incremental tests. If the underlying growth loop is broken, no sequence of button experiments will fix it, and the time is better spent on the loop. Model it first in a SaaS growth model template so you can see which input actually moves the output before spending six weeks testing one that does not.

What the program costs

Real numbers. A dedicated growth pod of three, loaded cost, runs roughly $450K to $600K a year in most US markets. Testing tooling adds $500 to $4,000 a month depending on traffic and whether you use PostHog, VWO, Optimizely or a homegrown flag system. Analyst time is the hidden cost and it’s usually underestimated by half.

Against that, the honest tradeoff. A program running at eight tests a month with a 25% win rate produces roughly two wins monthly, and most wins are small. The value compounds over quarters, not weeks, and the first two months produce almost nothing because you are building the process rather than running it.

Anyone promising a transformed funnel in six weeks is selling something. What you get instead is a team that stops arguing about opinions, because the archive settles arguments that used to run for an hour.

Where this fits in a growth strategy

Experimentation optimises a motion that already works. It does not find the motion. A seed-stage company with 40 customers should be doing customer interviews and channel exploration, not split testing headline copy, and the sequencing by stage is laid out in B2B SaaS growth strategy by stage.

Once a channel is working, experimentation is how you compound it. That includes the mechanics around it: a referral program ROI calculator will tell you whether a referral test is worth the engineering time before you queue it, which is a faster answer than running the test.

Start next Monday

Pick the three pieces that cost nothing: the intake form, the capacity limit, and the pre-registered decision rule. Run them for four weeks exactly as written, even when it feels bureaucratic, and count how many tests reach a decision compared with the previous month.

Then add the archive. Then, only once the cadence holds for a full quarter, worry about which testing platform you are on.