A/B test basic evaluation

An A/B test basic evaluation shows how your campaign performs against a control group or an alternative variant. Use it to measure conversion rate, average order value, and revenue—and confirm whether your campaign generates real uplift.

This guide uses a weblayer campaign as an example, but the same logic applies to experiments, emails, and recommendations.

Prerequisites

Before you start, make sure you have:

• Basic navigation skills in the platform.
• Purchase event tracking set up in your project.
• The right campaign event tracked — show for weblayers and experiments, campaign > delivered for emails.

Evaluation metrics

How the evaluation works

In this guide, we will show you how to create a report for a campaign showing a so-called "View count weblayer". We will look at the purchase behavior of visitors within 24 hours after they saw the weblayer, and compare it to the control group within the same time period.

The evaluation has 3 parts, each built in sequence:

• Segmentation: Divides customers into three groups: Variant A, Control group, and Mixed.
• Expression: Calculates the time between a purchase and the last time the customer saw the weblayer.
• Report: Combines the segmentation and expression to display the six metrics.

Create segmentations

The segmentation splits customers into 3 groups for comparison.

• Variant A: Customers who were shown the weblayer.
• Control group: Customers who didn't see anything.
• Mixed: Customers that saw the weblayer but also fell into the control group. See the FAQ to understand why this happens.

To create a segmentation:

1. Go to Analyses > Segmentations and create a new segmentation.
2. Create the first segment and name it Mixed. Set the conditions to include customers who match both the variant and the control group funnel conditions.
3. Copy the Mixed segment twice.
4. In one copy, delete the control group funnel condition. Name it Variant A.
5. In the other copy, delete the variant funnel condition. Name it Control group.

🚧

Important

Keep the Mixed segment first (on the left). The segmentation logic depends on this order.

Find banner ID

Use the banner_id in your segment conditions — not the banner_name. If multiple campaigns share the same name, the analysis breaks. IDs are always unique. To find the banner_id, open your weblayer and copy the ID from the URL.

Creating expressions

This expression calculates the time difference, in seconds, between each purchase and the last time the customer saw the weblayer. This is what makes the attribution window work in the report.

1. Go to Data & Assets > Data manager > Definitions and create a new expression.
2. Set the expression to apply to the purchase event.
3. Add the timestamp attribute.
4. Add the - arithmetic operator.
5. Click + Add attribute and create a running aggregate. Set it to return the timestamp of the last time the weblayer was shown.
6. Save the expression.

Create reports

The report pulls everything together— your segmentation, expression, and six metrics— into a single view.

Report structure

Process

1. Go to Analyses and create a new report.

2. In Rows, add your segmentation.

3. Create the Visitors metric. Count the first occurrences of the banner event and rename it Visitors.

4. In the Event filter, specify the banner event for your weblayer. This filter applies to all metrics in the report.

5. Create the Customers metric. Count unique customers with a purchase event. To save time, copy the Visitors metric and edit it.

   

6. In the Event filter, add a filter for the purchase event. Use your expression and set the condition to less than 86400 — that's 24 hours in seconds. This limits attribution to purchases made within 24 hours of seeing the weblayer.

   

7. In the Customer filter, exclude the Mixed segment.

   

8. Create the Conversion metric. Copy the Customers metric and apply this formula: (event_purchase_first / event_banner_first) × 100. Format it as a percentage.

9. Create the Revenue metric.

10. Create the AOV metric: Revenue ÷ Customers.

    

11. Copy Revenue to create the RPV metric. Set the formula to: sum_event_purchase_total_price / event_banner_first.

    

12. Click Preview to see the report. Drag and drop metrics using the six-dot icon to reorder columns.

Note that every time you create a new metric, it is easier to copy the previous metric and then just edit it

📘

Advanced tip

To calculate absolute uplift —the actual revenue your campaign generated — add a metric with this formula:
(RPV Variant A − RPV Control group) × Visitors Variant A.

FAQ

Why does the Mixed segment exist?

When a customer first sees your campaign, the app assigns them to either the variant or the control group. That assignment sticks. If they come back on the same device, they're in the same group.

The issue arises when a customer switches devices or uses an incognito window. The app sees a new cookie and treats them as a new customer, potentially assigning them to a different group. When they log in — say, during checkout — the app recognizes them and merges the 2 profiles.

If those 2 sessions landed them in different groups, you can't reliably attribute their behavior to either one. They go into the Mixed segment and get excluded from the evaluation.

Why don't I see my expression or aggregate in the attribute list?

You need to create the expression or aggregate in Data & Assets > Data manager > Definitions first, then come back to the report to select it. If you built it elsewhere in the app, it won't show up in the list.