A/B testing your website, how to measure what really works

Published

Contents

Action points

  • Before your first test, write down a hypothesis with one goal chosen in advance.
  • Work out how many visitors your test needs before you start.
  • Send a test enquiry yourself through version A and through version B.
  • Put a rel=canonical pointing to the original on every variant with its own URL.
Reading type0 min

Prefer to outsource?

Conversion optimisation →

As a business owner, you want to make choices on your site based on evidence, rather than taste or the mood of the day. In this guide you can read how A/B testing works, what to test, how many visitors you need and how long a test runs.

What A/B testing is and what it proves

A/B testing, also called a split test, means showing two versions of a page or element to different visitors at the same time. Version A is your current page, version B the adjusted one. You then measure which version performs better on one goal you choose in advance, such as a submitted quote form.

Running them at the same time is what makes the difference. If you run the old version in May and the new one in June, you also measure the weather, the holidays and your competitors. When both versions run in the same weeks, those influences hit A and B equally hard.

Every visitor is randomly shown one version and keeps seeing it. That keeps the comparison fair.

What does it prove? That this one specific change makes the difference, for your visitors on your site. After that, a tip from a trade blog or a colleague's taste carries less weight than your own result.

Sound familiar? Everyone on your team has an opinion about the colour of the enquiry button. A test turns that debate into a measurement.

Business owner from Brabant at the meeting table holding up two coloured buttons to test them

Taste counts for less once you have proof.

What to test in an A/B test on your website

On an SME website, a few standard elements lend themselves well to a first test.

  • You test the headline and the promise, for example "Skilled painting work" versus "Your exterior painted within one week".
  • You test the button text, such as "Send" versus "Get your quote".
  • You test the form, with eight fields versus four fields.
  • You test the order of blocks, with your reviews above or below your working method.
  • You test proof close to the button, such as three short reviews right next to the enquiry button.

Change exactly one thing per test. If you adjust the headline, the button and the photo at the same time, all you see is that version B wins. Which of the three changes did it remains a guess.

Owner of a café in Brabant sprinkles salt, pepper and paprika into the soup all at once and tastes it with a frown

So choose the element closest to your goal. Want more enquiries? Then start with the form or the text on the button. You will find ideas for structuring a page like that in our steps for a landing page that brings in customers.

Write down a hypothesis first

A hypothesis is a short prediction you record before the start. That way you know exactly afterwards what you are judging the test on.

Furniture maker from Brabant first marks out a line with a pencil and try square before she starts sawing

A fixed sentence helps here, for example this one. "Because many visitors abandon our eight-field form halfway through, we expect four fields to produce more submitted enquiries."

Alongside that sentence, record four things.

  1. You note what you are changing, in one sentence.
  2. You note why you expect it to help.
  3. You choose one goal, such as the submitted form.
  4. You work out how many visitors the test needs.

That way you avoid picking whichever figure happens to look good afterwards. Does version B win on clicks but lose on enquiries? Then the goal you wrote down beforehand is what counts.

Write it down first, then test.

Sample size and statistical significance

The sample size is the number of visitors your test needs for a reliable result. According to the NIST statistical handbook, the US standards institute, that number depends on three things.

  • The risk of a false alarm, meaning you see a winner that is really down to chance.
  • The risk of overlooking a real difference.
  • The spread, or how widely your figures vary.

The NIST table works with a 5% risk of a false alarm and a 10% risk of missing a real effect. A difference half the size then requires 43 observations instead of 11. This is a general statistical principle that applies just as well to your test.

Imagine you want to demonstrate a small difference between two button texts. If you halve the difference you are looking for, you need almost four times as many visitors.

On the edge of the bath, a short row of rubber ducks stands on the left and a much longer row on the right

the researchers behind ExP, the online experimentation platform that started at Microsoft, differences of a fraction of 1% in key metrics can already have major business consequences. Differences that small call for a very sensitive measurement with a lot of data. So an SME site is better off looking for a substantial difference.

What does statistically significant mean? The difference between A and B is so large that chance becomes an unlikely explanation, within the false alarm risk you chose in advance. So significance says something about reliability. It says little about the size of your gain.

Low traffic? Three smart routes

Not sure whether your site gets enough visitors for a test? Then you have three routes, each with its own pace.

Test a bigger change

A bigger difference requires fewer observations, as the NIST table showed. A new promise at the top or a form half as long is more likely to produce a big difference than a different shade for your button.

Painter from Brabant holds two almost identical shades of grey side by side and sees no difference

Measure higher up the funnel

A funnel is the route from first visit to enquiry. More happens higher up that route, such as a click on "See how we work" or opening your form. If you measure at a step like that, you collect enough observations sooner.

Then check whether those extra clicks carry through to real enquiries. A click is an intermediate step, the enquiry remains your goal.

Do some research first

Sometimes research delivers results faster than a test. Call a few recent customers and ask why they chose you. Look at where visitors drop off on your page. A conversion scan lists those drop-off points for you.

You can read more ideas for a small site in the article on leads from low traffic.

With low traffic, test big or measure earlier in the route.

How long an A/B test runs

Google does not name a fixed minimum duration anywhere. As Google Search Central explains, the time needed depends on factors such as your conversion rate and your traffic. Your conversion rate is the share of your visitors who take the desired action.

A good testing tool, as the same Google page notes, tells you when you have collected enough data. Work out that number in advance and stick to it.

Also let a test run for whole weeks. Does your customer search for the same thing on a Tuesday morning as on a Sunday evening? With whole weeks, weekdays and weekends count equally for A and B.

Peek along the way, decide at the end

Do you look every day and stop at the first nice figure? Then you are deciding at a moment your calculation never allowed for. The NIST calculation assumes a fixed number of observations, with a risk of false alarm and missed effect chosen in advance. Stop earlier and that balance falls apart.

Checking in between is fine, for example to see whether the measurement works. You only make the decision once the planned number is in.

Wrap up a test afterwards too. Google warns that an experiment that runs unnecessarily long can come across as an attempt to mislead search engines.

Five A/B testing pitfalls

These five mistakes make a result unreliable.

  1. You stop too early, before the planned number of visitors is in.
  2. You test too many variants at once, because with A, B, C and D each version gets a quarter of your traffic.
  3. You choose a different goal afterwards, because B loses on enquiries and wins on clicks.
  4. You test during a sale, around the holidays or during a radio advert and mainly measure that exception.
  5. You make an SEO mistake, such as a wrong redirect that can cost you rankings.
Man from Brabant opens the oven too early and watches his cake sink

So plan a test in ordinary weeks. If a promotion does fall within your test period, note it in your log and weigh the result with that in mind.

Choose the goal in advance and stick to it until the end.

A/B testing and Google, keeping your SEO safe

A test also affects your SEO, the work of ranking higher in Google's regular search results. Google Search Central has its own page on testing with a few clear ground rules.

Show Google what your visitor sees

Googlebot is the robot that reads your pages. If you show Googlebot different URLs from your visitors, that is called cloaking. According to Google, that is against its spam policies, even during a test. A violation like that can push your site lower in Google or even make it disappear.

Use rel=canonical on separate URLs

If each variant has its own URL, put a rel=canonical on every variant pointing to the original URL. That is a line in the code that indicates which page is the original.

Google prefers that to noindex, because it better matches what you mean. Noindex is an instruction to keep a page out of the search results.

Choose a temporary 302 redirect

Are you redirecting visitors from the original URL to a variant? Then use a temporary 302 redirect instead of a permanent 301.

Clean up once you are done

Once you have a winner, make it permanent on your page and remove the variants and redirects. That way your test ends when it should.

Google Optimize stopped in 2023

Did you use Google's own testing service in the past? Google Optimize and Optimize 360 shut down on 30 September 2023. Experiments still running on that date were ended at that point. The ground rules above apply to every way of testing.

Unsure about canonicals or redirects on your site? We set up that technical side properly as part of technical SEO.

Measure a key event, per device too

A test is only as reliable as the measurement underneath it. Choose a key event in your analytics as the goal, such as a submitted quote form, a tap on your phone number or a completed order. Our separate guide to Google Analytics 4 covers how to set that up.

Before the start, send an enquiry yourself through version A and through version B. If your measurement counts both, the test can begin.

Then also look at the result per device. According to Statistics Netherlands (CBS), 95.5% of people aged 12 and over used a smartphone for online activities in 2025, compared with 83.9% in 2020. Over those years the laptop went from 68% to 73.3%. The PC or desktop fell from 38.8% to 35.1%.

Buying itself has also become normal in the Netherlands. In the first half of 2024, 81% of people aged 12 and over said they had bought something online, compared with 53.1% in 2014.

A variant that looks sleek on a laptop can push the button far down on a phone. So open each version on your own phone first before the test goes live.

Check every variant on a phone first.

From one-off test to fixed routine

One test teaches one lesson. Knowledge builds up when you let your tests build on each other.

Bricklayer from Brabant lays the next brick on a wall that grows course by course

So keep a simple log. For each test, note the hypothesis, the period, the number of visitors and the result. A test that ends in a draw also teaches you something, because then you know what leaves your visitors cold.

After a few rounds you see a pattern. Perhaps your customers respond mainly to speed, or rather to certainty and proof. You can read how to grow with the visitors you already attract in the article on growth from your visitors.

Prefer to outsource? We run A/B tests as one of the methods within conversion optimisation. In a call with Dominic, you discuss which page comes first and which goal goes with it.

Frequently asked questions about A/B testing (FAQ)

What is A/B testing?

A/B testing, or a split test, means showing two versions of a page or element to different visitors at the same time. You measure which version performs better on one goal you choose in advance, such as a submitted enquiry or an order.

How many visitors do I need for an A/B test?

That depends on the difference you are looking for, the risk of a false alarm, the risk of missing a real effect and the spread in your figures. According to NIST, a difference half the size requires almost four times as many observations. So with low traffic, test big changes.

How long should an A/B test run?

Google does not name a fixed duration anywhere. The time depends on your conversion rate and your traffic. Let the test run for whole weeks until the number of visitors you calculated in advance is in. Then wrap it up, because Google may see an unnecessarily long test as misleading.

What does statistically significant mean?

The difference between A and B is so large that chance becomes an unlikely explanation, within the false alarm risk you chose in advance. It says something about the reliability of your result. It says little about the size of the gain.

Is a split test bad for my SEO?

Follow Google's ground rules and you limit the impact on your SEO. Show Googlebot the same URLs as your visitors, put rel=canonical on variants pointing to the original, use a temporary 302 redirect and wrap up the test once you are done.

What can I test with an A/B test on my website?

Start with elements close to your goal. Think of the headline and the promise, the button text, the number of fields in your form, the order of blocks and reviews close to the button. Change exactly one thing per test.

Conclusion

A/B testing shows which version of your page delivers more, measured on one goal you choose in advance. Change one thing per test and write down a hypothesis first. Small differences require lots of visitors, so with low traffic test big steps or measure higher up the funnel. Let a test run for whole weeks, stop at the planned moment and follow Google's ground rules.

The next step

How we tackle this together

Discover your biggest growth opportunities with a free opportunity analysis (worth €416) — a tailor-made report, free of charge and with no obligation.

1Analysis
2Action plan
3Proposal
Schedule a call
Dominic Pietersz

About the author

Dominic Pietersz

Your dedicated point of contact

Dominic helps SMEs grow through Google and other channels. With his international experience, he turns challenges into concrete growth.

Follow Dominic on LinkedIn

Inspiration · I put two identical platters of meatballs on the buffet at the street barbecue, one with a note saying "spicy" and the other with "homemade". After an hour I could see which platter was empty first.