The gold standard for "does it work?"

An A/B test randomly splits users into a control group (A) and a treatment group (B), so any difference in outcome is caused by the change — not by chance or confounders.

Running a sound experiment

1. One clear hypothesis and a single primary metric.
2. Randomize assignment to remove bias.
3. Power & sample size — enough users to detect a real effect.
4. Significance — is the difference beyond noise (p-value / CI)?
5. Guardrail metrics — make sure you didn't break something else.

Common pitfalls

• Peeking early and stopping when it "looks" significant.
• Testing too many things at once.
• Ignoring practical vs. statistical significance.

Takeaway: experiments turn opinions into evidence. Test before you scale.