Decide from the evidence and preserve the learning

This lesson is part of the growth experimentation course. Interpret the result against the original question, then choose a bounded next action.

Prepare your working case

Choose one product and one customer segment. Gather the relevant product information, customer evidence and operating records before writing the answer. If you are learning without access to a business, use a clearly labelled practice case and state the assumptions you make.

Keep the scope small enough that you can explain the decision to another person. The purpose is to learn how to reason through the work, not to produce a longer document than the situation needs.

Work through the exercise

Step 1

Check implementation and data quality before reviewing the effect.

Record your answer, the evidence behind it and any unresolved dependency before moving to the next step.

Step 2

Compare the primary outcome with its uncertainty and guardrails.

Record your answer, the evidence behind it and any unresolved dependency before moving to the next step.

Step 3

Investigate segment differences without treating every exploratory result as a new winner.

Record your answer, the evidence behind it and any unresolved dependency before moving to the next step.

Step 4

Document ship, revise or stop and the conditions for a later retest.

Record your answer, the evidence behind it and any unresolved dependency before moving to the next step.

Worked situation

A signup increase accompanied by worse retained activation may fail the experiment’s business purpose even when the headline metric rises.

Treat this as an illustration of the decision, not a measured result or a claim that the same approach fits every company. In your own case, identify which condition would make the conclusion different. That condition belongs beside the recommendation.

Produce the deliverable

A decision memo with observed effect, limitations, guardrails and the next owner.

Use the working template to organize the result, then run the review checklist. Add the project name, date, owner and a link to supporting evidence. If a key input is unknown, describe how you would learn it rather than filling the gap with a confident guess.

A useful deliverable can be read without a live presentation. Another person should be able to identify the problem, understand the recommended choice and see what happens next. Remove any section that does not help them make or execute that decision.

Review your work

Ask a colleague to challenge the weakest assumption. Write down the strongest alternative explanation and the evidence that would distinguish it from your current view. This makes the result easier to revise when new information arrives.

Continue the course

Return to the course outline or use the lesson navigation below. For the wider topic, visit saas growth. The resource library and calculators provide additional working tools.

Apply decide from the evidence and preserve the learning in a working review

Use the lesson to produce a small working artifact. State the starting problem, apply the method to one case and ask another person to inspect the result. Compare their interpretation with the explanation you intended. Revise the artifact before repeating the exercise at a larger scope; reading the material alone does not establish that the method can be applied.

For this topic, involve the experiment owner and the analyst responsible for design integrity and work from hypothesis, assignment rules, metric definition and decision record. The relevant unit is the prespecified eligible user or account cohort. State the question the review should resolve before choosing a chart, an asset or a tool. If participants disagree about the unit or scope, resolve that disagreement before combining their evidence.

Evidence to prepare

Check the design before interpreting a result. Assignment, exclusions, outcome timing and stopping rules can change the meaning of an apparently precise statistic. Separate practical effect from statistical evidence and keep guardrails beside the primary outcome.

Two situations that can change the interpretation

When fewer onboarding steps produce weaker activation

A setup question can be useful if it routes the user to the right workflow, but wasteful if the answer is never used.

Use this check: Inspect what information or commitment the removed step supplied to the later workflow. More steps are not inherently better; evaluate the complete customer task.

The focused diagnostic guide provides the correction process and a working evidence sheet.

When experiment groups have unexpected sizes

A tracking failure affecting one variant can create an apparent conversion lift even when the user experience did not improve.

Use this check: Check assignment, eligibility, logging and exclusions before interpreting outcome differences. Do not repair the result by silently dropping inconvenient observations.

The focused diagnostic guide provides the correction process and a working evidence sheet.

Record the decision and the limit

A higher signup rate is not automatically a better activation path if the removed step helped users reach a useful workflow. Review the complete sequence and the relevant customer outcome. A bundled product change can be evaluated as a bundle without claiming to isolate every component.

Keep the conclusion beside the evidence that supports it. Record what the team will do, who owns the next action and which event or date will trigger a review. If the underlying definition, audience or product behavior changes, revisit the conclusion rather than assuming the old result still applies. A clear limit is useful information; it tells the next reader where additional investigation is required.

Use the complete topic collection for related methods and the category field guides when the product’s buying situation or implementation requirements change how the method should be applied.