Landing page A/B testing
Last updated . Figures are approximate and change with the market, so check current prices before you decide.
An A/B test is a fair comparison. Two versions run at the same time, visitors are split at random, and you measure one outcome such as form submissions. Running them at the same time matters, because comparing this month with last month mixes in everything else that changed. This guide covers what to test, how much traffic you need, how long to run, which tools exist and how to read the result.
What to test first
Test the parts that decide whether a visitor stays. In rough order of impact:
- The headline and first screen. Does it repeat what the visitor clicked on, and say what you offer and who it is for?
- The offer. A free quote versus a free consultation, a different guarantee, a different price display.
- The call to action. The words on the button, and whether there is one clear action or three.
- The form. Fewer fields, or a WhatsApp button instead of a form.
- Proof. Real customer quotes, logos or numbers placed near the button. Only use proof that is true and that you can show.
- Page length and order. Moving the pricing or delivery information higher up.
Change one thing per test, or at least one idea. If you change the headline, the image and the button at once and the page wins, you will not know why.
Skip the cosmetic tweaks for now. Button colour and small wording changes rarely move results enough to detect on a small site.
Sample size: why small sites must test big changes
The smaller the change you want to detect, the more visitors you need. Evan Miller's sample size calculator takes four inputs: your baseline conversion rate, the smallest effect you want to detect, statistical power and the significance level. It returns the visitors needed per variation.
His rule of thumb is n = 16σ² / δ², where δ is the minimum effect you want to detect and σ² is p(1 − p) for a conversion rate. We applied it to a page converting at 3% (our own worked example, so treat the figures as an approximation):
| Change you want to detect | Visitors per version | Total visitors |
|---|---|---|
| 3.0% to 4.5% (a 50% relative lift) | about 2,070 | about 4,140 |
| 3.0% to 3.6% (a 20% relative lift) | about 12,930 | about 25,870 |
At 5,000 visits a month, the first test takes under a month. The second takes around five. That is the whole argument for testing big changes on small sites: a subtle tweak needs traffic you do not have, and a bold change might show a clear result in weeks.
Below a few thousand visits a month, you usually learn more from fixing obvious problems directly, such as a slow page, a broken form or an unclear headline, and from asking customers why they hesitated. Our own testing service states a minimum of roughly 5,000 sessions a month on the tested pages, and recommends an audit alone below that.
How long to run a test
Run it until you reach the sample size you set before you started, and for at least one full business cycle, usually one to two weeks, so weekdays and weekends both appear. Do not stop the day the dashboard turns green.
Evan Miller explains why. Checking a running test and stopping when it first looks significant, which he calls peeking, inflates false positives, because the significance level assumes the sample size was fixed in advance. In his example, a test stopped this way found an effect more than a quarter of the time when nothing had changed. Choose the sample size, then wait.
Tools
Google Optimize, the free tool many small sites used, was discontinued. Google's help page states it was no longer available from 30 September 2023. Options now include:
- Landing page builders with testing built in. Unbounce lists A/B testing on its Experiment plan at $149 a month billed monthly, and above. Starter at $29 and Build at $99 do not include it.
- Experimentation platforms. GrowthBook lists a free cloud Starter plan with unlimited experiments and traffic for up to 3 users, plus a free open source self-hosted version. It suits teams with a developer.
- Enterprise testing suites. VWO's pricing page, now under Wingify, lists plans without prices and asks you to book a demo. These are usually aimed at larger sites.
- A service. Someone sets up tracking, runs the tests and reports. See our conversion testing service.
Whatever you use, check that your analytics counts the same action on both versions. A test without correct tracking tells you nothing.
How to read the results
- Check the sample first. Did you reach the number you planned?
- Look at conversions, not just percentages. Two conversions against four is a 100% lift and means nothing.
- Expect most tests to be inconclusive or to lose. That is normal and still useful, because it stops you shipping a worse page.
- Compare the same outcome you chose in advance. Do not go hunting through segments afterwards for a win.
- Keep the winner and write down what you learned. The next test starts from there.
Common mistakes
- Stopping early because the early numbers look good.
- Testing with too little traffic and treating noise as a result.
- Changing several things at once.
- Running the test during a sale, a holiday or a traffic spike that changes who visits.
- Sending different traffic sources to each version.
- Testing before the basics are fixed. A slow page or a broken form does not need a test.
- Never recording what you tried.
For more on setup and costs, see how much a landing page costs.
FAQ
What is landing page A/B testing?
It is showing two versions of a page to different visitors at the same time and comparing how many complete the goal, such as submitting a form. Random split and one measured outcome make the comparison fair.
How much traffic do I need?
It depends on your conversion rate and the change you want to detect. By Evan Miller's rule of thumb, a page at 3% needs about 2,070 visitors per version to detect a jump to 4.5%, and about 12,930 per version to detect a jump to 3.6%.
How long should a test run?
Until it reaches the sample size you set in advance, and for at least one to two weeks so a full business cycle is covered. Stopping when the result first looks good gives false winners.
Can I A/B test with low traffic?
You can, but only big changes will show up. With low traffic, fixing obvious problems and gathering customer feedback usually beats testing.
What replaced Google Optimize?
Google ended Optimize on 30 September 2023. Alternatives include builders with built-in testing, experimentation platforms and enterprise suites. Check pricing and limits on each vendor's page.
How OnePDA can help
Our landing page design is a one-off price from €490, with copy, design, build, a form and tracking, and two rounds of revisions. Prices exclude VAT where it applies.
If you already have a page and traffic, CRO services start with a conversion audit at €390. It reviews your analytics, key pages and the path to an enquiry, then ranks what to fix. A/B tests cost €490 a month and need roughly 5,000 sessions a month on the pages we test. We do not promise an uplift, and we report losing tests as well as wins. See the price list for the rest.