The A/B Test Basic Evaluation presents a high-level overview of the results of your A/B test. It lets you compare the performance of your campaign compared to a control group or a different version and see whether there are any uplifts in the key metrics.
|Visitors||The number of people who have been targeted by the campaign.|
|Customers||The number of people who have made a purchase after having been targeted, and within a predefined time frame.|
|Average Order Value (AOV)||= Revenue/Customers|
|Revenue per Visitor (RPV)||= Revenue/Visitors|
|Revenue||The amount of revenue generated by the campaign. This metric uses a predefined time frame that starts at the moment the customer has been targeted.|
In this guide, we will show you how to create a report for a campaign showing a so-called "View count weblayer". We will look at the purchase behavior of visitors within 24 hours after they saw the weblayer, and compare it to the control group within the same time period.
Note that you can also use this logic to create evaluations for any campaign: experiments, emails, recommendations, etc.
This evaluation has 3 parts that will be explained in this guide:
- A basic segmentation of customers
- An expression
- The final report (which makes use of the segmentation and expression)
|Bloomreach Engagement skills||Basic - we will show you step-by-step how to create every part of this evaluation. We assume you can navigate and do basic operations in Bloomreach Engagement.|
|Data and tracking|| Tracking of the |
Tracking the appropriate event for the campaign you want to evaluate (these are tracked automatically)
We need this segmentation to divide the customers into 3 groups:
- Mix - customers that saw the weblayer but also fell into the control group. See the FAQ to understand why this happens.
- Variant A - customers who were shown the weblayer
- Control Group - customers who didn't see anything
1. Go to
Segmentations and create a new one.
2. Create the first segment - Mixed, as shown in the animation below.
You can get this value when you go to your weblayer and copy the following value from the URL. You could also use the
banner_name here. However, if you have more weblayer campaigns under this name, your analysis will be invalidated. The ID is always unique, that's why it is a good practice to use it.
3. Now copy the Mix segment twice and create the Variant and Control Group segments by deleting one of the funnel conditions in each new segment. You will delete the Variant condition from Control Group segment and vice versa.
It is important to have the "Mixed" segment as the first one on the left side. Read the documentation about segmentations for explanation.
Save your work. You have now created the customer segmentation with 3 segments: Variant A, Control Group, and Mixed. This means you can now easily filter your customers depending on which segment they belong to and use this in your other analyses.
This expression will calculate (using a running aggregate) for each event
purchase the difference in seconds between that event and the last time the buyer saw our weblayer. This will enable us to include time attribution in the evaluation.
1. Create a new expression in
Data manager >
2. Follow the animation below. Set the expression for event
3. Add the
4. Add the
- arithmetic operator using your keyboard.
5. Following the animation below, click on the
+ add attribute button and create a new running aggregate. This will look at the timestamp of when our weblayer was shown for the last time.
Save your work. You can now move on to the last step - creating the final report using both the segmentation and the expression.
This will be the overall structure of your report. The next steps will guide you how to create each part of it.
|ROWS||Viewcount Groups (that's the segmentation you created before)|
|METRICS||You will define 6 metrics:|
|CUSTOMER FILTER||A global filter that will exclude the "Mixed" group from the evaluation.|
|EVENT FILTER||A global filter that will narrow down the metrics so they only relate to our weblayer campaign. It will also define the time attribution window.|
1. Create a new report under
2. Add the segmentation you created before to
3. Now let the first
METRIC be the number of visitors. We’ll count them by counting the first occurrences of a banner event. Rename the metric.
4. Now you need to define what kind of banner event you mean. In order to do this for the report globally, you won’t change it in the metric itself, but in the event filter of the report. Note that the event filter will only allow you to define an event that you are actually using in the analysis. Hence, in this case, it is for now the
5. Having defined the banner, now create the Customers metric, counting the number of people who purchased as a result of seeing the campaign (banner). Follow the specifications in the screenshot below.
Note that every time you create a new metric, it is easier to copy the previous metric and then just edit it.
6. Now, before continuing with the next metric, let's go back to the global filter. For the purpose of this report, we want to take into account only purchases that were made within 24 hours (86400 seconds) after seeing the weblayer. This is why we created the expression.
7. In the event filter, specify the filter for the
purchase event. Using the same logic as in step 4, click on
+ add an event attribute and find your expression in the list. Then define it as "less than 86400".
8. While at filters, look at the global customer filter and exclude the people who have seen both the Variant and the Control Group, defined as "Mixed" in the segmentation you created. We can’t really assign them to any of the groups.
9. This is what the filters will look like at the end:
10. Go back to the metrics and create the Conversion metric. You can start by copying the "Customers" metric as it is a part of the conversion calculation.
11. This metric needs to be calculated as a formula based on the previous metrics you created in steps 5 and 3:
(event_purchase_first / event_banner_first)x100 and format as a percentage.
12. You have already created Visitors, Customers and Conversion. Now create the Revenue and AOV metric as shown below.
13. Now go ahead and copy the Revenue metric to create the basis for the Revenue per Visitor (RPV) metric.
14. Create a formula
sum_event_purchase_total_price / event_banner_first as outlined in the animation below.
15. Click on Preview to see your report. Scroll down. If you want to rearrange the columns, drag and drop the metrics with the six dots icon.
You have now created a basic A/B test evaluation. This is one of the fundamental reports used heavily by our consultants and we are sure you will use it a lot as well.
If you want to get the absolute uplift (how much money the campaign made), create a metric using this formula: (RPV Variant A - RPV Control Group) x Visitors Variant A.
See our troubleshooting guide to find answers to the following questions and more:
- Why do I have the "mixed" group in my customer segments?
- I don't see the expression/segmentation/aggregate I created in the list of events/attributes
Updated 4 months ago