What Is A/B Testing?
A/B testing (also called split testing) is a method of comparing two versions of a webpage, email, ad, or app against each other to determine which performs better. Version A (the control) is shown to half of your traffic; Version B (the variant) is shown to the other half. After collecting enough data, you determine which version produces more conversions.
Done correctly, A/B testing removes opinion from optimization. Instead of debating which button color is better in a meeting, you let your actual users tell you — with their behavior and their wallets.
What Can You A/B Test?
Almost anything on your website can be tested. The most impactful tests typically involve:
- Headlines and hero copy — your value proposition is the single highest-leverage element on most pages
- CTA button text, color, and placement — small changes here can produce 20–40% conversion lifts
- Form length and fields — removing a single unnecessary field often increases form completions by 10–25%
- Page layout and visual hierarchy — where elements appear on the page dramatically affects attention and action
- Pricing presentation — how you frame price (monthly vs. annual, with vs. without anchoring) changes conversion rates significantly
- Social proof placement — reviews shown above the fold convert better than reviews buried at the bottom
- Images and video — product images, lifestyle photos, and explainer videos each perform differently for different audiences
How to Run an A/B Test: Step by Step
Step 1: Start With Data, Not Opinions
Every A/B test should be motivated by a specific problem identified in your data. Heatmaps showing visitors ignoring your CTA, session recordings showing checkout abandonment at a specific step, or analytics showing a 75% exit rate on a high-traffic page — these are the inputs that produce meaningful test hypotheses.
Step 2: Write a Clear Hypothesis
A good hypothesis follows this format: “If we [make this change], we expect [this metric] to improve by [X%] because [evidence/reasoning].”
Example: “If we change the hero headline from ‘Welcome to Our Agency’ to ‘Double Your Conversion Rate in 90 Days’, we expect form submissions to increase by 20% because the current headline communicates nothing about our value proposition.”
Step 3: Calculate Your Required Sample Size
This is where most beginners fail. You need to run your test long enough to reach statistical significance — typically 95% confidence that the result isn’t due to chance. Use a sample size calculator (Optimizely and VWO offer free ones) before launching any test. For low-traffic pages, you may need to test for 4–6 weeks. For high-traffic pages, 1–2 weeks is often sufficient.
Step 4: Run the Test Without Peeking
One of the most common A/B testing mistakes is stopping a test early because one variant appears to be winning. This leads to false positives — you declare a winner before the data is reliable. Set your test duration in advance and commit to it.
Step 5: Analyze Beyond the Primary Metric
A variant might increase form submissions but decrease the quality of leads. It might improve add-to-cart rate but decrease checkout completion. Always analyze downstream metrics and secondary indicators before declaring a winner and deploying.
Common A/B Testing Mistakes
- Testing too many elements at once — if you change 5 things, you won’t know which one caused the difference
- Running tests for too short a time — less than a full business cycle (usually 2 weeks minimum) means your data doesn’t account for day-of-week variation
- Ignoring statistical significance — a 60% win rate with 40 conversions is not reliable data
- Testing low-traffic pages — you need volume to get reliable results; focus testing resources on pages that receive significant traffic
- Not documenting results — every test result, win or loss, should be documented with context for future reference
A/B Testing Tools
Popular A/B testing platforms include Google Optimize (now sunset), VWO, Optimizely, Convert, and AB Tasty. For WordPress sites, you can also use native plugins. The right tool depends on your traffic volume, technical resources, and budget.
At CRO PRO, we manage A/B testing as part of our CRO retainer engagements — running 2 tests per month for Growth clients, with full setup, monitoring, analysis, and deployment handled by our team. Talk to us about adding structured testing to your optimization program.
