Signup headline test plan
Do we have enough traffic?
Yes, but only just. At 8,000 visitors a week, a 15% relative lift is detectable in about 5 weeks.
| Input | Value |
|---|---|
| Baseline conversion | 4.0% |
| Target (15% relative lift) | 4.6% (+0.6 points absolute) |
| Significance / power | 95% (two-sided) / 80% |
| Sample size | about 18,000 visitors per variant, about 36,000 total |
This uses the standard two-proportion formula. Check it against Evan Miller's calculator before launch. Expect small differences of a few hundred visitors.
How long to run it
- 36,000 visitors at 8,000 a week is 4.5 weeks. Run it for 5 full weeks with a 50/50 split, which is about 40,000 visitors.
- Use whole weeks so every weekday and weekend is represented equally.
- Don't stop early, even if it looks significant at week 2. Peeking inflates false positives.
- Don't extend it either if it's flat at week 5. Call it inconclusive.
Sensitivity to the lift you target:
| Detectable lift | Per variant | Duration |
|---|---|---|
| 20% relative (4.8%) | about 10,300 | about 3 weeks |
| 15% relative (4.6%) | about 18,000 | about 5 weeks |
| 10% relative (4.4%) | about 39,500 | about 10 weeks |
Headline tests often produce lifts under 15%. If the true effect is 10%, this test will probably miss it, so you'd see "no significant difference" even though the headline helps. Only run it if a 15% lift is the smallest result worth acting on. If you want to detect smaller effects, test a bolder change than a wording tweak, such as a different value proposition rather than a rephrase.
Constraints:
- Two variants only. An A/B/C test would need about 7 weeks.
- Counting visitors. The 8,000 must be unique visitors who see the headline, not sessions or page views. If it's sessions, the test will take longer.
What to write down before starting
- Hypothesis, in the skill's format: Because [evidence, e.g. session recordings, survey responses, or ad-to-page message mismatch], we believe [new headline] will cause [higher signup conversion] for [new visitors to the signup page]. We'll know this is true when conversion rises from 4.0% to at least 4.6% at 95% confidence. Without real evidence in the "because" clause, you're guessing.
- Variants. Screenshots of control and variant, with the exact copy. Change only the headline.
- Primary metric. Unique visitor to completed signup, with a precise definition of "completed" (submitted form or verified email). Decide this now.
- Secondary metrics. Signup-form start rate, scroll depth, and time on page. These help explain why it worked or didn't.
- Guardrail metrics. Activation or week-1 retention of signups, and bounce rate. A headline that over-promises can lift signups and attract low-quality users. Define the stop rule: halt if a guardrail is significantly negative.
- Test parameters.
- Sample size per variant (about 18,000), power, alpha, and the MDE (15% relative).
- Fixed end date, about 5 weeks out.
- 50/50 split with persistent assignment, so returning visitors see the same variant.
- Population, including whether to exclude internal and bot traffic.
- Decision rules.
- Ship if the variant is significant and positive with no guardrail harm.
- Keep control if it's significant and negative.
- If it's flat, treat it as inconclusive, not as evidence of no effect.
- Note which segments you'll look at (device, new vs. returning, traffic source). Fix these now to avoid cherry-picking later.
- Implementation notes.
- Server-side or client-side. Client-side can flicker on a headline, so prefer server-side or an anti-flicker setup.
- Tracking verified on both variants.
- A sample ratio mismatch (SRM) check planned. The split should land close to 50/50.
- Risks and external factors. Planned campaigns, launches, seasonality, or traffic-source changes during the 5 weeks. Log anything that happens. Avoid adding new traffic sources mid-test.
- Pre-launch QA. Check both variants on mobile and desktop, and confirm events fire.
When it finishes, record the outcome in the playbook format: result with CI and p-value, guardrails, segment deltas, the reusable pattern, and status.
If you give me the actual control headline and candidate variant, I can draft the hypothesis and a few evidence-based headline options.
Real run in an isolated folder with only this skill installed. Only the Skill and Read tools were enabled, so nothing was fetched from the web and no file was written. The example is fictional. The traffic numbers are invented.
What it does
Helps design experiments you can trust. It covers a hypothesis template ("Because X, we believe Y will cause Z for audience W, and we will know when metric M moves"), test types (A/B, A/B/n, multivariate, split URL), a sample-size quick table by baseline and expected lift, primary, secondary and guardrail metrics, what to vary, traffic allocation, client-side and server-side implementation, a pre-launch checklist, the peeking problem, and an analysis checklist with how to read each outcome. A second part turns single tests into a programme: an experiment loop, ICE prioritisation, velocity targets, a playbook entry template and a weekly, bi-weekly, monthly and quarterly cadence. References hold test templates and a sample-size guide.
How it works
- It asks for the baseline rate, traffic, the change and the smallest lift worth detecting.
- It checks whether you have enough traffic and estimates duration.
- It writes the hypothesis, metrics and what to document before launch.
Good for
Anyone deciding whether a change is better, and teams building a regular experimentation habit.
Low risk: pure instructions with no scripts, no network access and no file writes. The sample-size figures and programme targets (for example 4-8 experiments a month, a 20-30% win rate) are rules of thumb, so use a proper calculator for real decisions; the skill links to two public ones. It does not run tests or read your analytics, so it cannot tell you the outcome of a real test. Some links point to sibling skills and tool guides not included in this package. Tried once on invented traffic numbers.