Incrementality Testing for B2B SaaS

Every guide to incrementality testing was written for a company doing 40,000 conversions a month. Yours does 120 opportunities a quarter. The standard advice, run a two-week geo holdout and read the lift, produces a number with a confidence interval so wide you could drive a budget through it.

This playbook is the B2B version: which test designs survive at low volume, the sample size arithmetic worked out, the three tests to run first, and how to write up a result that finance will accept.

Why does standard incrementality advice break in B2B SaaS?

Because the statistics need volume and B2B does not have it. The arithmetic is unforgiving and it is worth seeing directly.

To detect a 20 percent lift with reasonable confidence, you need something in the region of a few hundred conversions per arm. A company closing 40 deals a quarter cannot get there in a year, let alone a test window. So the standard design fails, and teams conclude incrementality testing is for consumer companies.

The way out is to move up the funnel. You may have 40 closed-won deals but 120 opportunities, 600 MQLs and 4,000 form fills. Measuring on opportunities gives you three times the volume, and measuring on leads gives you fifteen times. You lose some validity, because lead lift does not automatically become revenue lift, but you gain the ability to run a test at all.

Outcome measured	Quarterly volume (example)	Detectable effect	Validity concern
Closed-won revenue	40	Not detectable	None, but unusable
Opportunities	120	About 30%	Low, close to revenue
MQLs	600	About 15%	Medium, quality can shift
Form fills	4,000	About 6%	High, easily gamed by channel

Pick opportunities if you can tolerate only detecting large effects, and MQLs if you need more sensitivity. Never test on form fills alone, because a channel can inflate form volume with unqualified traffic and post a positive result while destroying pipeline quality. If you do test on leads, report opportunity rate alongside as a guardrail.

Which four test designs work, and what does each need?

Four designs, with genuinely different volume requirements. Two of them are viable for almost any B2B SaaS company.

Geo holdout. Split markets into test and control, turn the channel off in control, compare. Needs enough comparable geos, which in practice means at least 20 metros with similar baseline conversion rates. Works well for the US market, poorly for companies whose pipeline is concentrated in three cities.

Audience holdout. Randomly exclude a percentage of your target audience from seeing the campaign, then compare conversion rates between excluded and exposed. LinkedIn supports this natively through audience exclusion, which makes it the easiest test to actually execute. The catch is that exposure is not random within the served group, so you are comparing eligible-and-served against eligible-and-withheld, which is close enough for most decisions.

PSA ghost ads. The control group is served an unrelated public service ad through the same auction and targeting, so both groups cleared the same selection process. It is the cleanest design available and support is uneven: strong in programmatic display, limited on LinkedIn.

Switchback. Alternate the channel on and off in time blocks, typically weekly, and compare. Cheap to run and it needs no audience infrastructure, but it is badly confounded by B2B sales cycles. If your cycle is 60 days, a week-on week-off pattern smears effects across blocks and the result is uninterpretable. Use it only for short-cycle, self-serve motions.

Design	Minimum volume	Setup difficulty	Sales cycle tolerance	Verdict for B2B SaaS
Geo holdout	High, 20+ comparable metros	Medium	Good	Use for brand and paid social
Audience holdout	Medium	Low	Good	Start here
PSA ghost ads	Medium to high	High	Good	Best design, limited availability
Switchback	Low	Low	Poor above 30 day cycles	Self-serve motions only

How do you work out sample size at 120 opportunities a quarter?

Do the arithmetic before you design the test, because it tells you whether the test is worth running at all. Here is the worked case.

Assume a company generating 120 opportunities a quarter, evenly split across a channel mix, with paid social contributing roughly 30 of those. You want to test whether paid social is incremental.

Split 50/50 gives you about 15 opportunities per arm per quarter. That is nowhere near enough. So you extend the window to two quarters, giving 30 per arm, and you widen the holdout to 50 percent of audience rather than a small slice, which maximises contrast.

With 30 events per arm, the detectable effect at conventional confidence is roughly a 50 percent difference. That sounds useless, and for fine-tuning it is. For the question “is this channel doing anything at all”, a 50 percent detection threshold is genuinely informative, because a channel producing less than half the conversions attribution claims is a channel you would reallocate.

Which three tests should you run first?

Branded search, retargeting, and a LinkedIn audience holdout. In that order, because that is the order of likely budget released per hour of effort.

Branded search pause. Turn off paid search on your own brand terms in half your geos, or entirely for four weeks if geos are not workable, and measure total branded conversions including organic clicks. What teams typically find is that a large share of paid branded clicks were cannibalising organic clicks that were free. The finding is rarely 100 percent non-incremental, because competitor bidding on your brand is real and there are navigational queries where you genuinely need the top slot, but the incremental share is usually far below what the platform reports.

This is the highest-value test in B2B SaaS for one reason: branded search is often among the largest single line items in the paid budget and it is almost never questioned, because its reported ROAS looks spectacular. It looks spectacular because it is measuring people who already decided to buy.

Retargeting holdout. Exclude a random half of your retargeting audience. Retargeting reports beautifully for the same reason branded search does, and the incremental share is usually a fraction of the attributed share. This test is easy because retargeting audiences are already defined in the platform.

LinkedIn audience holdout. Exclude 50 percent of your matched account list from the campaign for a full quarter, then compare opportunity creation between the two halves. This is the test that tells you whether your ABM paid layer is doing anything, and it interacts directly with what your paid media attribution is claiming.

How do you read a null result without lying to yourself?

Report the confidence interval, not the point estimate, and state explicitly what effect sizes the test could and could not rule out.

A null result at low volume means one of two things: the effect is small or zero, or the test lacked power. Those are different conclusions and the interval tells you which. If your interval on lift runs from minus 10 percent to plus 55 percent, you have learned almost nothing. If it runs from minus 5 percent to plus 15 percent and your decision threshold was 40 percent, you have learned enough to cut.

The dishonest moves to avoid, all common. Extending the test because the result was not what you wanted. Switching the outcome metric after seeing the data. Excluding a geo that “had an anomaly”. Splitting the result by segment until one segment shows significance. Each of these turns a test into a search for a comfortable number, and the people who run them usually believe they are being rigorous.

Incrementality also has a real limitation worth admitting. It measures the marginal effect of the spend level you tested, in the period you tested it, for the audience you tested on. It says nothing about long-term brand effects that accumulate over years, which is where marketing mix modeling and brand lift measurement do work that experiments cannot. Anyone who tells you incrementality is the complete answer is selling something.

How do you document a result finance will accept?

One page, written before the test, filled in after. Finance rejects incrementality findings when they arrive as a slide with a lift percentage and no method.

Two details make the difference with a CFO. First, defining the outcome metric by CRM field rather than by name, because “opportunity” means four different things inside most companies. Second, stating the confounds you know about rather than waiting to be asked, which converts the conversation from interrogation to review.

Keep the records in one place. A folder of six of these over two years is the most persuasive marketing measurement asset a team can own, and it outlives any dashboard. If you are still deciding between measurement approaches, the comparison of multi-touch attribution vs incrementality testing lays out where each is honest, and the marketing tracking plan template covers the event definitions the tests depend on.

What to do next

Pick the branded search test and size it this week. Count your branded paid conversions over the last two quarters, write the decision rule, and check whether four weeks off in half your geos gives you enough events.

If it does not, the test to run instead is the retargeting holdout, which usually has more volume and almost always has a surprising answer. Put the result into the SaaS marketing dashboard template alongside the attributed numbers so the gap between the two is visible every month, and read incrementality testing for B2B SaaS for the conceptual background. The wider SaaS metrics and analytics hub covers how this fits with the rest of the measurement stack, including measuring lifecycle email in SaaS, which has the same low-volume problem and a similar solution.