Measuring Lifecycle Email in SaaS

Most lifecycle email reports open with a 44 percent open rate and close with a revenue figure lifted straight out of last touch in the CRM. Both numbers are wrong, and they are wrong in opposite directions. Opens are inflated by machines fetching tracking pixels on behalf of people who never read the message, while last touch hands email the credit for upgrades that were going to happen anyway. You do not fix that with a better attribution model. You fix it by deliberately withholding email from some people and watching what they do.

Why open rate stopped being a measurement input in September 2021

Apple shipped Mail Privacy Protection alongside iOS 15 in September 2021. When a recipient turns it on, Apple’s proxy downloads remote content in advance, including your tracking pixel, whether or not a human ever looks at the message. The open fires. Nobody read anything.

That single change broke four things at once, and most SaaS teams only rebuilt one of them. Subject line tests judged on opens became coin flips. Re-engagement segments defined as did not open in 90 days started excluding the most engaged Apple users and including people who had already forgotten you exist. Send time optimisation trained itself on proxy timestamps. And deliverability monitoring lost its cheapest proxy for inbox placement.

In the B2B lists we look at, Apple Mail typically accounts for somewhere between a third and a half of recorded opens, which means a reported 44 percent open rate might represent 25 percent real attention or 40 percent, and nothing in your platform can tell you which. Treat any open based figure in a vendor case study from 2022 onward as decorative.

What replaces open rate, and what each replacement actually tells you

Four metrics survive. Click to delivered, reply rate, spam complaint rate, and downstream conversion measured against a holdout. Each answers a different question and none of them answers all three.

Click to delivered is your creative metric. Use delivered rather than sent as the denominator, because bounces are a list problem and mixing them in makes a good email look weak. It tells you whether the message earned an action, and it is the number to judge copy, offer and send time on. Click to open ratio is dead for the same reason opens are.

Reply rate matters more in sales assisted motions than most teams admit. A plain text activation email from a named human that pulls 3 percent replies is worth more than one pulling 9 percent clicks into a docs page, because a reply creates a conversation an account executive can work. It also reads as a strong engagement signal to mailbox providers.

Spam complaint rate is now a hard operating constraint rather than a vanity concern. Google and Yahoo’s bulk sender requirements, which took effect in February 2024, set a ceiling of 0.3 percent, and in practice you want to sit under 0.1 percent. Watch it per sequence, not per account, because one aggressive winback campaign can poison delivery for your onboarding mail.

Downstream conversion is the only one that carries revenue, and it is only trustworthy against a control group. That is the rest of this page. If you want the rates other teams are hitting on the first three, our SaaS email benchmarks split them by sequence type and ACV band rather than averaging a newsletter together with a password reset.

How to design a holdout when you only get 800 trials a month

Start from the effect you need to detect, not from the split you find comfortable. A reasonable approximation for a two proportion test at 80 percent power and 95 percent confidence is sixteen times the baseline variance divided by the squared difference you want to catch. With a baseline trial to paid rate of 18 percent, the variance term is 0.18 multiplied by 0.82, which is 0.1476.

Work an example. A product takes 800 trials a month, converts 18 percent of them, and the team wants to know whether the six email onboarding sequence does anything at all.

Read the last row twice. An unbalanced 90/10 design needs nearly three times the total population to answer the same question, because the small arm governs the standard error. That is why the split you use to test is not the split you use to monitor.

So the rule at low volume is this. Run the initial test at 50/50, accept that you can only detect large effects, and be honest that a sequence producing a two point lift is invisible to you and always will be at this volume. Once the sequence has proven itself, drop to a permanent 5 to 10 percent holdout as a regression alarm rather than a measurement instrument.

When a permanent holdout is worth the revenue it costs

Once a sequence is live and proven, keep 5 to 10 percent of eligible accounts out of it permanently. That is our position, and it costs money, so here is the arithmetic.

Say your dunning sequence recovers 38 percent of failed payments and failed payments represent $50,000 of MRR a year. A permanent 5 percent holdout forgoes roughly 5 percent of the recovery, about $950 a year. In exchange you get a live control that tells you the day a deliverability problem, a broken merge tag or a platform migration quietly halves your recovery rate. Most teams discover those failures six weeks late through a revenue dip nobody can explain.

The holdout is worth it when three conditions hold. The sequence runs continuously rather than as a campaign. The outcome it drives is measurable inside 30 days. And the cost of a silent failure is larger than the cost of the withheld sends. Dunning, onboarding and trial expiry clear that bar comfortably. A quarterly product newsletter does not, and forcing a permanent holdout onto it is theatre.

Incrementality and switchback tests for always on sequences

User level randomisation fails in two specific SaaS situations. When several people share a workspace and forward things to each other, and when the sequence is tied to a product surface you cannot hold back, such as an in app banner that fires off the same event.

Switchback testing solves both by randomising time instead of people. Turn the sequence on for a week, off the next, and repeat. Then compare the outcome for cohorts that entered during on weeks against cohorts that entered during off weeks. You need enough blocks for the comparison to mean anything, and sixteen weekly blocks is the practical floor. Fewer than that and a single bad week of seasonality dominates the result.

Two things ruin switchbacks. Carryover, where someone entering during an off week still gets touched because they were already in a related sequence, and seasonality, where your on weeks happen to land on the first week of each month. Randomise the block order rather than alternating strictly, and check that on and off weeks are balanced across the month.

Per sequence attribution rules that stop email double counting with sales

Set the rules before you need them, in writing, and get the sales leader to agree in the same meeting. The argument you are avoiding is the one where marketing claims 60 percent of pipeline, sales claims 70 percent, and finance stops reading either deck.

Three rules cover most cases. First, a click must precede the conversion inside the attribution window. An open never qualifies, for the reasons above. Second, if an account executive logged a call, meeting or manual email inside that same window, the deal is sales sourced and email influenced, and those go in separate columns. Third, the window varies by sequence, because a trial expiry email that works does so within 72 hours, while a winback might land three weeks later.

Notice what is missing. The newsletter gets no revenue line. Teams hate this, and it is the right call, because a newsletter that a customer reads for eight months before renewing is doing something real that no window based rule can capture without inventing a number. Measure it on subscriber retention and reply quality, and defend it as a brand investment rather than pretending it closed deals. The same discipline applies on the ads side, and we walk through it in paid media attribution for SaaS.

The four numbers to put in front of a board

Boards do not want seven email metrics. They want to know whether the program pays for itself and what would break if it stopped. Four numbers do that job.

Incremental MRR. The holdout derived difference in revenue per enrolled account, multiplied by the treated population, stated monthly. One number, one method, restated every quarter with the confidence interval attached. If the interval crosses zero, say so.

Activation lift. The percentage point difference in accounts reaching your activation milestone between treated and held out groups. This is where onboarding email earns its keep, and it usually shows a clearer effect than revenue does because it happens sooner and closer to the message.

Involuntary churn recovered. Dollars of failed payments rescued by dunning, net of the holdout baseline recovery that would have happened through in product retries anyway. Most teams report gross recovery here and overstate it by 30 to 50 percent, since some customers fix their own cards without being asked.

Expansion MRR influenced. Clearly labelled as influenced. This is the number a CFO will push on hardest, so give it the weakest claim and the clearest definition.

Model those four against cost before you present them. Our SaaS email revenue calculator takes list size, click to delivered, conversion and ACV and returns incremental revenue and program payback, which is a more useful shape for a board than a funnel chart.

What this costs, and the three ways it fails

The honest cost is time, not tooling. Designing a clean test, tagging holdouts across your platform and CRM, suppressing parallel campaigns and writing the analysis takes a competent marketing operations person roughly two days per sequence up front and half a day a month after that. At a loaded rate that is a few thousand dollars a year, against a program usually worth six figures.

Three failure modes account for most of the disappointments we see.

The first is underpowered tests read as wins. A team at 300 trials a month runs a four week test, sees 19.4 percent against 18.1 percent, and ships the sequence. That difference is noise at that sample size, and the team now believes something false. If your volume cannot support the test, say the sequence is unmeasured rather than dressing up a coin flip.

This second is tooling that cannot hold a group out cleanly. List based platforms often struggle to keep a stable random assignment across sequence entries and exits, so the holdout drifts and the comparison stops being random. Event based platforms handle it natively through a random bucket attribute set at account creation, which is one of several reasons we favour them in our comparison of lifecycle email platforms, and a specific point of difference in HubSpot versus Customer.io. If you run Mautic, you will be assigning the bucket yourself in the database, which works but has to be built.

The third is organisational. Someone senior asks why 8 percent of trials are not getting onboarding email, and the holdout quietly gets switched off. Pre-empt it. Write down the annual cost of the holdout in dollars, put it next to the cost of the last silent sequence failure, and get the decision made once at the right level.

Start here in the next two weeks

Pick the sequence with the most obvious revenue outcome, which is almost always trial expiry or dunning, and put a holdout on it before you touch a single word of copy. Then get the four board numbers defined and agreed with sales while the test runs, because that conversation takes longer than the test does.

If you are earlier than that and still building the first sequences rather than measuring them, start with the lifecycle email fundamentals and then ship your first sequence with the holdout built in from day one. Retrofitting a control group onto a sequence that has been running for a year is possible, but you lose the clean baseline and spend the first month arguing about what changed.