4. Retrieve and evaluate A/B test results

While the test runs, you can return to the A/B tests page anytime to check the results by clicking the Open button next to the test.

When are the results visible

Results will be available 24 hours after starting the test and will continue to show updated results. The update schedule can vary for different customer groups. The data might reflect updates from a few hours ago or, at most, 24 hours prior.

Roles to access test reports

You must have at least one of these IAM roles to view the A/B test reports:

• Categories merchandising editor
• Global merchandising editor
• Search merchandising editor
• SEO thematic pages editor
• SEO widgets editor

Steps to interpret A/B test results

Following are the steps to interpret the A/B test results:

1. Review metrics

Check key metrics that are relevant to your business goals. Compare the metrics between the test variant and control group.

Each metric displayed in the dashboard has an informational tooltip that shows the metric’s definition or what it calculates.

The available metrics are listed below:

• Visitors - The number of unique shoppers that visit and interact with a test bucket during the experiment.
• Visits - The total number of times a test bucket was visited during the experiment. One unique visitor can have multiple visits.
• ATC - The number of times a product was added to the shopping cart in a test bucket.
• Conversion - The number of orders (identified by unique order IDs) placed in a test bucket.
• Revenue - The total revenue generated from the sales in a test bucket.
• AOV (Revenue | Conversions) - The average amount spent per conversion.
• ATC rate (ATC | Visits) - Percentage of visits that resulted in adding items to the cart.
• CR (Conversions | Visits) - Percentage of visits that resulted in a conversion.
• RPV (Revenue | Visits) - Revenue generated per visit.

2. Examine lift

You can find the Lift values with the absolute value for the performance measures.

Look at the lift value to see how much the test variant improved over the control. The lift displayed in the table is averaged over the report period. Lift can be positive or negative.

3. Check confidence level

The confidence level in the results panel indicates the statistical certainty of the lift. A higher percentage (more than 90%) suggests the results are reliable. The confidence in the table is of the latest day in the report period.

4. Analyze graphs

Use graphs to see trends over time and spot any patterns.

5. Make decisions

• If the lift is positive and the confidence level is high, consider implementing the changes from the test variant on your site.
• If results are inconclusive, consider ending the test and starting a new one.

You can download the test results as a CSV file using the download button next to the date selector.

Troubleshooting

1. Why do I see a different traffic split?

You may see a different or uneven traffic split due to the following causes:

1. Caching issues: Your current caching strategy might interfere with the proper traffic allocation between the test groups.

2. Empty or skewed _br_uid_2 parameter: Sending traffic with empty or skewed _br_uid_2 parameters can result in an inaccurate split of the test groups.

3. Low-volume query: Low traffic can make it hard to see the impact of the test and properly bucket the traffic.
   We suggest that you investigate these issues. For further help, contact Bloomreach Support.

2. Why do I see 0% values?

You may see a change of 0% in certain metrics, which likely means there is no statistically significant difference between the control and test buckets for those specific metrics. This could be due to:

1. Small effect size: The actual change (test applicability or traffic impacted) is too small to move the needle or show statistical significance.

2. Confidence intervals: The confidence interval might include 0, suggesting no clear difference.

3. Lack of sensitivity in the metric: The selected metric may not be sensitive enough to capture changes caused by the given test parameters.

A/B test metadata in API response

The Product and Category Search API response includes metadata of the active A/B tests that the particular search request participates in. This can be used to analyze which A/B test experiments and variant audiences a user was part of.

The response data includes the experiment and variant names. The _br_uid parameter (cookie) for that API request helps identify the user.

Here is an example of the response metadata with A/B test data:

{
  "metadata": {
    "query": {
      "query_tags": {
        "untagged_tokens": [
          "watch"
        ]
      },
      "precision": {
        "configured": {
          "value": ""
        },
        "applied": {
          "value": "text_match_precision"
        }
      }
    },
    "experiments": [
      {
        "ab_test_experiment_name": "Search\+ vs Keyword",
        "ab_test_variant_name": "Variant A"
      },
      {
        "ab_test_experiment_name": "Category Ranking Rule - Global",
        "ab_test_variant_name": "Control"
      }
    ]
  }
}

The experiments field in the response shows A/B test metadata and has the following properties:

• ab_test_experiment_name - The name of the A/B test experiment as set in the dashboard.
• ab_test_variant_name - The name of the assigned variant, as set in the dashboard.
• It is only available if a valid _br_uid parameter is passed in the API call, and the user falls into an active A/B test experiment.

This compatibility with the API enables you to monitor or log A/B test performance outside the Bloomreach dashboard. You can pass this metadata to your preferred or custom-built Analytics tools.

Best practices to assess the validity of test results (Optional reading)

Note that while these are general best practices, they can differ based on factors like your business goals and business size.

Assessing KPIs, lift, and confidence

• Guiding metric: Your guiding metric depends on what you're trying to test and your business's goals. For example, if you're trying to improve relevance, you might focus more on the ATC (Add to Cart) rate. If you’re focusing on increasing revenue with merchandising, you might look at Conversion rate and RPV.

• Confidence: Use confidence level as the metric to determine when a test is over and there is a clear winner. A confidence level of over 90% is a good benchmark.

• Assessing lift: The lift threshold can vary based on your business size. A small or mid-sized business may see a 5% lift, while a large enterprise business may see a .05% lift. This seemingly small percent lift can still translate into millions of dollars in incremental lift for the business. It’s crucial that the lift is statistically significant. Significance can be confirmed when the confidence is over 90%.

📘

Note

Statistical significance and confidence level

Statistical significance is the probability that the results are not caused by a random chance. This is represented by a p-value. A p-value of less than 0.10 means there's less than a 10% chance the results are random, making them statistically significant.

Confidence level indicates the percent confidence in the results. Suppose you estimate that your ATC rate will improve between 3% and 4%. A 90% confidence means that if you repeat the experiment 100 times, the result will fall within this range 90 times.

The A/B test results panel shows the confidence level. Confidence level and statistical significance are distinct yet related concepts. A confidence level of 90% corresponds to a p-value of less than 0.10. Therefore, you can use confidence level to deduce the significance of the results.

Factors affecting confidence

• Scale of changes: A test may not reach confidence if there isn't much variation between the two experiences. For example, if you just lock 2 products and then A/B test a category page, the experiences are still very similar and will likely not have a conclusive winner.

• Traffic: It will take longer to reach confidence if you're testing a low-traffic query/category or if you're only A/B testing a small portion of your traffic. Bloomreach recommends testing with 100% traffic for most tests where you feel fairly confident. Only run the test with smaller traffic if the change is significant and you are hesitant to expose it to half of your traffic.

• Testing merchandising strategy: If your goal is to test an overarching merchandising strategy, like boosting sales, newness, low stock levels, or optimizing for a specific signal like Conversion or Revenue, we recommend testing at a global level to reach confidence faster.