By the DynoWeb Team
Why test at all?
CRO changes feel obvious in hindsight, but plenty of “obvious” improvements do nothing — or backfire. A/B testing protects you from shipping changes that look good but don’t move revenue. The catch: testing only works if you test the right things and read the results honestly.
Step 1: Test the right thing
The biggest mistake is testing random ideas. Start from observed friction. Use heatmaps and session replays to find where shoppers actually struggle, then form a specific hypothesis: “Raising the add-to-cart above the fold will increase mobile add-to-cart rate.” A hypothesis you can prove or disprove beats a vague tweak.
Step 2: Change one thing
Test a single variable so you can attribute the result to it. If you change the image order, the CTA, and the copy all at once and conversion improves, you won’t know which change did the work — or whether one of them hurt.
Step 3: Size the test honestly
How long you run depends on your traffic and baseline conversion rate. Low-traffic stores need longer to reach significance. Run for full weeks to avoid day-of-week bias, and decide your sample size before you start so you’re not tempted to stop early on a lucky streak.
Step 4: Read results without fooling yourself
- Reach significance. A 10% lift on 30 conversions is noise. Wait for enough data.
- Don’t peek and stop. Checking constantly and stopping at the first good-looking moment inflates false positives.
- Segment by device. A change can win on desktop and lose on mobile — look at both.
- Watch revenue, not just clicks. A higher add-to-cart rate that lowers checkout completion isn’t a win.
Step 5: Ship and keep the loop running
Ship the winner, then confirm the real-world impact with revenue attribution. Testing isn’t a one-off — it’s the verification step inside the broader CRO loop: diagnose with behavior, hypothesise, test, ship, measure, repeat.
What if you don’t have the traffic to test?
Lower-traffic stores can’t reach significance quickly, so lean harder on qualitative evidence — replays and heatmaps — and ship higher-confidence fixes directly, measuring the before/after with attribution. Testing is a tool, not a religion; match the rigor to your traffic.