# When retention reporting drops failed onboarding > The retained population begins only after unsuccessful customers have been removed. Diagnose the cause, choose a bounded correction and verify retention definitions with visible cohort entry and exclusions. Source: https://saas-marketing.net/guides/retention-cohort-excludes-failed-onboarding/ Topic: SaaS Growth Marketing Type: guide Published: 2026-09-17 Last updated: 2026-09-17 Publisher: SaaS Marketing (saas-marketing.net) License: CC BY 4.0. Quote or republish with attribution and a link to https://saas-marketing.net/guides/retention-cohort-excludes-failed-onboarding/ ## Short answer The retained population begins only after unsuccessful customers have been removed. Start with this check: Review the cohort entry rule and show how many original accounts never reached it. The corrective action is to report acquisition-to-retention and post-activation retention as distinct views. ## Key takeaways - Review the cohort entry rule and show how many original accounts never reached it. - Report acquisition-to-retention and post-activation retention as distinct views. - Neither view should be presented as the other without its exclusions. - Review retention definitions with visible cohort entry and exclusions. --- The retained population begins only after unsuccessful customers have been removed. The useful response is a diagnosis that changes a decision, not another report describing the symptom. Use this play with the experiment owner and the analyst responsible for design integrity. The working evidence should include hypothesis, assignment rules, metric definition and decision record, with private or sensitive details removed from any shared example. ## Confirm the problem in the actual workflow Review the cohort entry rule and show how many original accounts never reached it. Start with one representative case and follow it from the original action to the reported outcome. Identify where the observed behavior first differs from the intended process. A screenshot of a final dashboard can be useful, but it may hide the source record, a delayed update or a decision made elsewhere. Keep the unit of analysis explicit: the prespecified eligible user or account cohort. The same label can conceal different populations or stages. Before comparing two results, check that they describe the same kind of work and have had a comparable chance to complete it. ## Separate the visible symptom from the cause Check the design before interpreting a result. Assignment, exclusions, outcome timing and stopping rules can change the meaning of an apparently precise statistic. Separate practical effect from statistical evidence and keep guardrails beside the primary outcome. The symptom in this case is specific: the retained population begins only after unsuccessful customers have been removed. Ask which piece of evidence would distinguish an operating failure from a measurement failure or a mismatch in the original plan. If the evidence is unavailable, record the missing source and its owner instead of treating the preferred explanation as established fact. ## A situation to work through A high post-activation retention rate can coexist with poor overall outcomes if many purchased accounts never implement successfully. This is an illustrative situation, not a reported client case. Record the equivalent evidence and assumptions for your own workflow. ## Choose the smallest useful correction Report acquisition-to-retention and post-activation retention as distinct views. Keep the change narrow enough that the responsible people can implement and inspect it. If a correction changes several things at once, describe it as a combined operating change; do not later claim that one small element caused the whole result. Assign the correction to the experiment owner and the analyst responsible for design integrity. Agree which artifact will show that the work is complete. An owner without an observable acceptance condition can close a task while leaving the original problem unresolved. A detailed checklist without an owner creates the opposite problem: the evidence requirement exists, but nobody is accountable for producing it. ## Preserve the important limitation Neither view should be presented as the other without its exclusions. This condition belongs beside the recommendation because it can change the decision. It should not disappear when the plan becomes a short presentation or a status update. A higher signup rate is not automatically a better activation path if the removed step helped users reach a useful workflow. Review the complete sequence and the relevant customer outcome. A bundled product change can be evaluated as a bundle without claiming to isolate every component. ## Verification worksheet | Review item | What to record for this issue | Owner | Evidence | | --- | --- | --- | --- | | Observed symptom | The retained population begins only after unsuccessful customers have been removed. | | | | Diagnostic test | Review the cohort entry rule and show how many original accounts never reached it. | | | | Proposed correction | Report acquisition-to-retention and post-activation retention as distinct views. | | | | Guardrail | Neither view should be presented as the other without its exclusions. | | | | Review measure | Retention definitions with visible cohort entry and exclusions | | | Download a working copy and follow the [worksheet instructions](/resources/#using-worksheets). Keep unknown facts visible rather than filling gaps with guesses. ## Decide whether to keep, revise or stop the change Review retention definitions with visible cohort entry and exclusions after the agreed observation period. Keep the correction when the intended behavior is verified and the guardrail remains acceptable. Revise it when the diagnosis was useful but the intervention did not resolve the cause. Stop and reassess when new evidence shows that the original problem was framed incorrectly. Record what changed in hypothesis, assignment rules, metric definition and decision record. This gives the next review a stable starting point and prevents a definition change from being mistaken for a performance improvement. ## Related methods and next steps - [SaaS onboarding audit checklist](/checklists/onboarding-audit/) - [Appcues vs Userpilot: test the onboarding workflow](/comparisons/appcues-vs-userpilot/) - [SaaS onboarding examples: diagnose the blocked task](/examples/saas-onboarding-teardowns/) - [Activation rate: definition and SaaS example](/glossary/activation-rate/) Return to the [saas growth topic guide](/saas-growth/), browse its [complete resource collection](/topics/saas-growth/), or use the [working resource library](/resources/). The [primary reference](https://docs.statsig.com/) provides relevant platform or methodological context; the diagnosis and example here are original editorial guidance. ## Frequently asked questions ### What is the first diagnostic check? Review the cohort entry rule and show how many original accounts never reached it. Inspect the actual working record or customer path rather than relying only on a summary report. ### What should change after the diagnosis? Report acquisition-to-retention and post-activation retention as distinct views. Record the owner and the evidence needed to verify the correction. ### What limit should the team keep visible? Neither view should be presented as the other without its exclusions. A local improvement does not establish a universal benchmark or guarantee a commercial result. ### How should the correction be evaluated? Review retention definitions with visible cohort entry and exclusions using a consistent unit and observation window. Keep the original evidence and record any measurement changes.