HomeEngagementContentDiscoveryOther resources
Guides

A/B testing

Suggest Edits

This guide introduces the Bloomreach Discovery A/B testing feature.

What's A/B testing

Bloomreach A/B testing allows you to test the effectiveness of your rule’s variants. A variant is a change or version of something on your site, like a redirect or a ranking rule.

A/B testing lets you create different rule variants, split the incoming traffic to these variants, and run a test to evaluate which variant drives your business objectives.

Why is A/B testing valuable

A/B testing helps you understand the effect of changes on your site and then make data-driven decisions. It challenges and validates what you know about your shoppers.

For example, can you predict the outcome of burying “office chairs” for bedroom furniture searches?

It makes sense: if someone searches for bedroom furniture, it's reasonable to assume that they aren't interested in an “office chair.”

But what if the shopper:

  • likes having an office alcove in their bedroom,
  • lives in a studio apartment,
  • is a college student, or
  • is a dedicated business professional who starts every morning by rolling out of bed and answering emails?

Sometimes, what sounds logical doesn't reflect shoppers' realities. A/B testing shows the effect of your proposed changes on key performance indicators (KPIs). You can use the test data to make informed decisions.

📘

Note

To compare Bloomreach Discovery A/B testing with split testing and before-after comparisons, visit the FAQs.

A/B testing capabilities overview

1. What you can test

Bloomreach currently supports A/B testing for:

Note: If you A/B test a broad match ranking rule, the test will track all variations of the query that the broad match rule uses.

We don't support A/B testing for:

2. Number of test experiences

A/B testing can involve more than two buckets. Typically, it’s easier to find statistically significant results when you limit the number of buckets to two. What constitutes a manageable number is up to you, but we support a maximum of 4 buckets per A/B test.

3. Technical requirements

Two requirements must be met before A/B testing functionality can be leveraged:

  • No site caching: Bloomreach's A/B testing will not work on your site if you use any type of caching.

  • br_uid_2 parameter: A/B testing traffic split will only work if you correctly integrate the _br_uid_2 parameter. Read the Integration Guide for details on integrating this cookie. Note that the _br_uid_2 parameter values in the pixel and API must match.

4. Reporting method

Bloomreach A/B testing reports the test results using a classical statistical (that is, frequentist) approach.

How A/B testing works

Here is an overview of how Discovery A/B testing works:

1. Create test variants

Start by creating different variants of the rule you want to test.

2. Set up and activate the test

Set up the test using the A/B testing application and activate it to start running.

3. Traffic split

  • Random assignment: When users first interact with the test, they are randomly assigned to distinct groups called buckets. Each bucket represents a different variant being tested. To maintain consistency, users stay in their assigned bucket throughout the test duration.

  • _br_uid_2 parameter: Here’s where the _br_uid_2 parameter helps track and assign users to specific buckets.

4. Monitoring and analysis

Monitor the results while the test is running. Assess the results to see if the test meets your business objectives.

5. Complete the test

  • End test: When you are ready to conclude the test, you can end it and choose to apply one of the test buckets as the new default for all traffic.

  • No changes: If the results aren't satisfactory, you can end the test without changing the site experience.

📘

Note

For multi-query rules in the same active test, check the frequently asked questions page.

Related articles


Updated about 1 month ago