Signup headline test plan

Do we have enough traffic?

Yes, but only just. At 8,000 visitors a week, a 15% relative lift is detectable in about 5 weeks.

This uses the standard two-proportion formula. Check it against Evan Miller's calculator before launch. Expect small differences of a few hundred visitors.

How long to run it

• 36,000 visitors at 8,000 a week is 4.5 weeks. Run it for 5 full weeks with a 50/50 split, which is about 40,000 visitors.
• Use whole weeks so every weekday and weekend is represented equally.
• Don't stop early, even if it looks significant at week 2. Peeking inflates false positives.
• Don't extend it either if it's flat at week 5. Call it inconclusive.

Sensitivity to the lift you target:

Headline tests often produce lifts under 15%. If the true effect is 10%, this test will probably miss it, so you'd see "no significant difference" even though the headline helps. Only run it if a 15% lift is the smallest result worth acting on. If you want to detect smaller effects, test a bolder change than a wording tweak, such as a different value proposition rather than a rephrase.

Constraints:

• Two variants only. An A/B/C test would need about 7 weeks.
• Counting visitors. The 8,000 must be unique visitors who see the headline, not sessions or page views. If it's sessions, the test will take longer.

What to write down before starting

1. Hypothesis, in the skill's format: Because [evidence, e.g. session recordings, survey responses, or ad-to-page message mismatch], we believe [new headline] will cause [higher signup conversion] for [new visitors to the signup page]. We'll know this is true when conversion rises from 4.0% to at least 4.6% at 95% confidence. Without real evidence in the "because" clause, you're guessing.
2. Variants. Screenshots of control and variant, with the exact copy. Change only the headline.
3. Primary metric. Unique visitor to completed signup, with a precise definition of "completed" (submitted form or verified email). Decide this now.
4. Secondary metrics. Signup-form start rate, scroll depth, and time on page. These help explain why it worked or didn't.
5. Guardrail metrics. Activation or week-1 retention of signups, and bounce rate. A headline that over-promises can lift signups and attract low-quality users. Define the stop rule: halt if a guardrail is significantly negative.
6. Test parameters.
   • Sample size per variant (about 18,000), power, alpha, and the MDE (15% relative).
   • Fixed end date, about 5 weeks out.
   • 50/50 split with persistent assignment, so returning visitors see the same variant.
   • Population, including whether to exclude internal and bot traffic.
7. Decision rules.
   • Ship if the variant is significant and positive with no guardrail harm.
   • Keep control if it's significant and negative.
   • If it's flat, treat it as inconclusive, not as evidence of no effect.
   • Note which segments you'll look at (device, new vs. returning, traffic source). Fix these now to avoid cherry-picking later.
8. Implementation notes.
   • Server-side or client-side. Client-side can flicker on a headline, so prefer server-side or an anti-flicker setup.
   • Tracking verified on both variants.
   • A sample ratio mismatch (SRM) check planned. The split should land close to 50/50.
9. Risks and external factors. Planned campaigns, launches, seasonality, or traffic-source changes during the 5 weeks. Log anything that happens. Avoid adding new traffic sources mid-test.
10. Pre-launch QA. Check both variants on mobile and desktop, and confirm events fire.

When it finishes, record the outcome in the playbook format: result with CI and p-value, guardrails, segment deltas, the reusable pattern, and status.

If you give me the actual control headline and candidate variant, I can draft the hypothesis and a few evidence-based headline options.