A/B Test Basic Evaluation
The A/B Test Basic Evaluation presents a high-level overview of the results of your A/B test. It lets you compare the performance of your campaign compared to a control group or a different version and see whether there are any uplifts in the key metrics.
This is how your analysis will look:
You will get the following insights from this report:
| Visitors | The number of people who have been targeted by the campaign. |
| Customers | The number of people who have made a purchase after having been targeted, and within a predefined time frame. |
| Conversion | = Customers/Visitors |
| Average Order Value (AOV) | = Revenue/Customers |
| Revenue per Visitor (RPV) | = Revenue/Visitors |
| Revenue | The amount of revenue generated by the campaign. This metric uses a predefined time frame that starts at the moment the customer has been targeted. |
How to create the A/B test evaluation in Bloomreach Engagement
In this guide, we will show you how to create a report for a campaign showing a so-called "View count weblayer". We will look at the purchase behavior of visitors within 24 hours after they saw the weblayer, and compare it to the control group within the same time period.
Note that you can also use this logic to create evaluations for any campaign: experiments, emails, recommendations, etc.
This evaluation has 3 parts that will be explained in this guide:
- A basic segmentation of customers
- An expression
- The final report (which makes use of the segmentation and expression)
| Requirements | |
|---|---|
| Bloomreach Engagement skills | Basic - we will show you step-by-step how to create every part of this evaluation. We assume you can navigate and do basic operations in Bloomreach Engagement. |
| Data and tracking | Tracking of the purchase eventTracking the appropriate event for the campaign you want to evaluate (these are tracked automatically) show for weblayers and experimentscampaign > delivered for evaluating emails |
Creating the Segmentation
We need this segmentation to divide the customers into 3 groups:
- Mix - customers that saw the weblayer but also fell into the control group. See the FAQ to understand why this happens.
- Variant A - customers who were shown the weblayer
- Control Group - customers who didn't see anything
1. Go to Analyses > Segmentations and create a new one.
2. Create the first segment - Mixed, as shown in the animation below.
Getting the banner_id
You can get this value when you go to your weblayer and copy the following value from the URL. You could also use the banner_name here. However, if you have more weblayer campaigns under this name, your analysis will be invalidated. The ID is always unique, that's why it is a good practice to use it.
3. Now copy the Mix segment twice and create the Variant and Control Group segments by deleting one of the funnel conditions in each new segment. You will delete the Variant condition from Control Group segment and vice versa.
It is important to have the "Mixed" segment as the first one on the left side. Read the documentation about segmentations for explanation.
Good job!
Save your work. You have now created the customer segmentation with 3 segments: Variant A, Control Group, and Mixed. This means you can now easily filter your customers depending on which segment they belong to and use this in your other analyses.
Creating the Expression
This expression will calculate (using a running aggregate) for each event purchase the difference in seconds between that event and the last time the buyer saw our weblayer. This will enable us to include time attribution in the evaluation.
1. Create a new expression in Data&Assets > Data manager > Definitions > New.
2. Follow the animation below. Set the expression for event purchase.
3. Add the timestamp attribute.
4. Add the - arithmetic operator using your keyboard.
5. Following the animation below, click on the+ add attribute button and create a new running aggregate. This will look at the timestamp of when our weblayer was shown for the last time.
Good job!
Save your work. You can now move on to the last step - creating the final report using both the segmentation and the expression.
Creating the Report
This will be the overall structure of your report. The next steps will guide you how to create each part of it.
| ROWS | Viewcount Groups (that's the segmentation you created before) |
| COLUMNS | n/a |
| METRICS | You will define 6 metrics: Visitors Customers Conversion AOV RPV Revenue |
| CUSTOMER FILTER | A global filter that will exclude the "Mixed" group from the evaluation. |
| EVENT FILTER | A global filter that will narrow down the metrics so they only relate to our weblayer campaign. It will also define the time attribution window. |
1. Create a new report under Analyses.
2. Add the segmentation you created before to ROWS.
3. Now let the first METRIC be the number of visitors. We’ll count them by counting the first occurrences of a banner event. Rename the metric.
4. Now you need to define what kind of banner event you mean. In order to do this for the report globally, you won’t change it in the metric itself, but in the event filter of the report. Note that the event filter will only allow you to define an event that you are actually using in the analysis. Hence, in this case, it is for now the banner event.
Metric - Visitors
Event filter specifying the banner
5. Having defined the banner, now create the Customers metric, counting the number of people who purchased as a result of seeing the campaign (banner). Follow the specifications in the screenshot below.
Note that every time you create a new metric, it is easier to copy the previous metric and then just edit it.
Metric - Customers
6. Now, before continuing with the next metric, let's go back to the global filter. For the purpose of this report, we want to take into account only purchases that were made within 24 hours (86400 seconds) after seeing the weblayer. This is why we created the expression.
7. In the event filter, specify the filter for the purchase event. Using the same logic as in step 4, click on + add an event attribute and find your expression in the list. Then define it as "less than 86400".
Now the global event filter is complete.
8. While at filters, look at the global customer filter and exclude the people who have seen both the Variant and the Control Group, defined as "Mixed" in the segmentation you created. We can’t really assign them to any of the groups.
9. This is what the filters will look like at the end:
10. Go back to the metrics and create the Conversion metric. You can start by copying the "Customers" metric as it is a part of the conversion calculation.
11. This metric needs to be calculated as a formula based on the previous metrics you created in steps 5 and 3: (event_purchase_first / event_banner_first)x100 and format as a percentage.
12. You have already created Visitors, Customers and Conversion. Now create the Revenue and AOV metric as shown below.
13. Now go ahead and copy the Revenue metric to create the basis for the Revenue per Visitor (RPV) metric.
14. Create a formula sum_event_purchase_total_price / event_banner_first as outlined in the animation below.
15. Click on Preview to see your report. Scroll down. If you want to rearrange the columns, drag and drop the metrics with the six dots icon.
Amazing!
You have now created a basic A/B test evaluation. This is one of the fundamental reports used heavily by our consultants and we are sure you will use it a lot as well.
Advanced tip
If you want to get the absolute uplift (how much money the campaign made), create a metric using this formula: (RPV Variant A - RPV Control Group) x Visitors Variant A.
Troubleshooting / FAQ
See our troubleshooting guide to find answers to the following questions and more:
- Why do I have the "mixed" group in my customer segments?
- I don't see the expression/segmentation/aggregate I created in the list of events/attributes
Updated about 1 year ago