Design a test that can answer the question

This lesson is part of the growth experimentation course. Define assignment, outcome, sample requirements and guardrails before changing the customer experience.

Prepare your working case

Choose one product and one customer segment. Gather the relevant product information, customer evidence and operating records before writing the answer. If you are learning without access to a business, use a clearly labelled practice case and state the assumptions you make.

Keep the scope small enough that you can explain the decision to another person. The purpose is to learn how to reason through the work, not to produce a longer document than the situation needs.

Work through the exercise

Step 1

Write a hypothesis connecting the intervention to behavior.

Record your answer, the evidence behind it and any unresolved dependency before moving to the next step.

Step 2

Choose the unit of assignment and prevent cross-group contamination.

Record your answer, the evidence behind it and any unresolved dependency before moving to the next step.

Step 3

Select one primary outcome and meaningful guardrails.

Record your answer, the evidence behind it and any unresolved dependency before moving to the next step.

Step 4

Estimate the observation window and predefine the decision procedure.

Record your answer, the evidence behind it and any unresolved dependency before moving to the next step.

Worked situation

Randomizing users inside a shared workspace can contaminate a test if both groups experience the same account-level setup. Account assignment may fit better.

Treat this as an illustration of the decision, not a measured result or a claim that the same approach fits every company. In your own case, identify which condition would make the conclusion different. That condition belongs beside the recommendation.

Produce the deliverable

An experiment brief with assignment logic, success measure and validity checks.

Use the working template to organize the result, then run the review checklist. Add the project name, date, owner and a link to supporting evidence. If a key input is unknown, describe how you would learn it rather than filling the gap with a confident guess.

A useful deliverable can be read without a live presentation. Another person should be able to identify the problem, understand the recommended choice and see what happens next. Remove any section that does not help them make or execute that decision.

Review your work

Ask a colleague to challenge the weakest assumption. Write down the strongest alternative explanation and the evidence that would distinguish it from your current view. This makes the result easier to revise when new information arrives.

Continue the course

Return to the course outline or use the lesson navigation below. For the wider topic, visit saas growth. The resource library and calculators provide additional working tools.

Apply design a test that can answer the question in a working review

Use the lesson to produce a small working artifact. State the starting problem, apply the method to one case and ask another person to inspect the result. Compare their interpretation with the explanation you intended. Revise the artifact before repeating the exercise at a larger scope; reading the material alone does not establish that the method can be applied.

For this topic, involve the experiment owner and the analyst responsible for design integrity and work from hypothesis, assignment rules, metric definition and decision record. The relevant unit is the prespecified eligible user or account cohort. State the question the review should resolve before choosing a chart, an asset or a tool. If participants disagree about the unit or scope, resolve that disagreement before combining their evidence.

Evidence to prepare

Check the design before interpreting a result. Assignment, exclusions, outcome timing and stopping rules can change the meaning of an apparently precise statistic. Separate practical effect from statistical evidence and keep guardrails beside the primary outcome.

Two situations that can change the interpretation

When fewer onboarding steps produce weaker activation

A setup question can be useful if it routes the user to the right workflow, but wasteful if the answer is never used.

Use this check: Inspect what information or commitment the removed step supplied to the later workflow. More steps are not inherently better; evaluate the complete customer task.

The focused diagnostic guide provides the correction process and a working evidence sheet.

When experiment groups have unexpected sizes

A tracking failure affecting one variant can create an apparent conversion lift even when the user experience did not improve.

Use this check: Check assignment, eligibility, logging and exclusions before interpreting outcome differences. Do not repair the result by silently dropping inconvenient observations.

The focused diagnostic guide provides the correction process and a working evidence sheet.

Record the decision and the limit

A higher signup rate is not automatically a better activation path if the removed step helped users reach a useful workflow. Review the complete sequence and the relevant customer outcome. A bundled product change can be evaluated as a bundle without claiming to isolate every component.

Keep the conclusion beside the evidence that supports it. Record what the team will do, who owns the next action and which event or date will trigger a review. If the underlying definition, audience or product behavior changes, revisit the conclusion rather than assuming the old result still applies. A clear limit is useful information; it tells the next reader where additional investigation is required.

Use the complete topic collection for related methods and the category field guides when the product’s buying situation or implementation requirements change how the method should be applied.