A/B test basic evaluation

When you run an A/B test in Bloomreach, the platform automatically splits your audience and tracks each variant's performance. This guide shows you how to build a report that measures the performance across six key metrics — conversion rate, average order value, and revenue — and how to determine whether your results are statistically significant enough to act on.

The guide uses a weblayer campaign as an example, but the same approach applies to experiments, emails, and recommendations.

Prerequisites

Before you start, make sure you have:

• Basic navigation skills in the platform.
• Purchase event tracking set up in your project.
• The right campaign event tracked — show for weblayers and experiments, campaign > delivered for emails.

Evaluation metrics

How the evaluation works

This guide uses a View count weblayer campaign as an example. The report measures purchase behavior within 24 hours of a customer seeing the weblayer and compares it to the control group over the same period.

The evaluation has 3 parts, each built in sequence:

• Segmentation: Divides customers into three groups: Variant A, Control group, and Mixed.
• Expression: Calculates the time between a purchase and the last time the customer saw the weblayer.
• Report: Combines the segmentation and expression to display the six metrics.

Create segmentations

The segmentation splits customers into 3 groups for comparison.

• Variant A: Customers who were shown the weblayer.
• Control group: Customers who didn't see anything.
• Mixed: Customers that saw the weblayer but also fell into the control group. See the FAQ to understand why this happens.

To create a segmentation:

1. Go to Analyses > Segmentations and create a new segmentation.
2. Create the first segment and name it Mixed. Set the conditions to include customers who match both the variant and the control group funnel conditions.
3. Copy the Mixed segment twice.
4. In one copy, delete the control group funnel condition. Name it Variant A.
5. In the other copy, delete the variant funnel condition. Name it Control group.

Find banner ID

Use the banner_id in your segment conditions — not the banner_name. If multiple campaigns share the same name, the analysis breaks. IDs are always unique. To find the banner_id, open your weblayer and copy the ID from the URL.

Create expressions

This expression calculates the time difference, in seconds, between each purchase and the last time the customer saw the weblayer. This is what makes the attribution window work in the report.

1. Go to Data & Assets > Data manager > Definitions and create a new expression.
2. Set the expression to apply to the purchase event.
3. Add the timestamp attribute.
4. Add the - arithmetic operator.
5. Click + Add attribute and create a running aggregate. Set it to return the timestamp of the last time the weblayer was shown.
6. Save the expression.

Create reports

The report pulls everything together— your segmentation, expression, and six metrics— into a single view.

Report structure

Process

1. Go to Analyses and create a new report.

2. In Rows, add your segmentation.

3. Create the Visitors metric. Count the first occurrences of the banner event and rename it Visitors.

4. In the Event filter, specify the banner event for your weblayer. This filter applies to all metrics in the report.

5. Create the Customers metric. Count unique customers with a purchase event. To save time, copy the Visitors metric and edit it.

   

6. In the Event filter, add a filter for the purchase event. Use your expression and set the condition to less than 86400 — that's 24 hours in seconds. This limits attribution to purchases made within 24 hours of seeing the weblayer.

   

7. In the Customer filter, exclude the Mixed segment.

   

8. Create the Conversion metric. Copy the Customers metric and apply this formula: (event_purchase_first / event_banner_first) × 100. Format it as a percentage.

9. Create the Revenue metric.

10. Create the AOV metric: Revenue ÷ Customers.

    

11. Copy Revenue to create the RPV metric. Set the formula to: sum_event_purchase_total_price / event_banner_first.

12. Click Preview to see the report. Drag and drop metrics using the six-dot icon to reorder columns.

Interpret your results

When you preview the report, compare Variant A against the Control group across all six metrics. A few patterns are worth knowing before you draw conclusions.

Conversion rate and RPV can point in different directions. A variant can show a higher conversion rate but lower revenue per visitor — meaning it converted more people but for smaller purchases. In that case, RPV is the more reliable indicator of business impact. Base your decision on RPV rather than conversion rate alone.

A small absolute difference can still matter. Even a fraction of a percentage point in conversion rate translates to meaningful revenue at scale. Use the A/B test significance calculator to check whether the difference is large enough to be reliable given your audience size.

Wait for sufficient sample size before deciding. If your visitor counts are low, a result that looks conclusive may be noise. A confidence level of 95% or above is generally the threshold for acting on a result.

Evaluate statistical significance

The report shows raw numbers — visitors, conversions, and revenue per variant. Before acting on the results, check whether the difference between your variants is statistically significant.

Use the A/B test significance calculator to determine whether your results are reliable. Enter the audience size and conversion count for each variant. The calculator shows the confidence level and which variant performed better.

For in-platform analysis, use the Compare conversions tool in Funnels. It runs a Chi-Square test on your funnel data and shows statistical confidence between variants. See Compare conversions for details.

FAQ

Why does the Mixed segment exist?

When a customer first sees your campaign, the app assigns them to either the variant or the control group. That assignment sticks. If they come back on the same device, they're in the same group.

The issue arises when a customer switches devices or uses an incognito window. The app sees a new cookie and treats them as a new customer, potentially assigning them to a different group. When they log in — say, during checkout — the app recognizes them and merges the 2 profiles.

If those 2 sessions landed them in different groups, you can't reliably attribute their behavior to either one. They go into the Mixed segment and get excluded from the evaluation.

Why don't I see my expression or aggregate in the attribute list?

You need to create the expression or aggregate in Data & Assets > Data manager > Definitions first, then come back to the report to select it. If you built it elsewhere in the app, it won't show up in the list.