Use Case Evaluation
The action-based Use Cases (such as weblayers or omnichannel orchestrations), where the success of the Use Case depends on the actions of customers and varies from industry to industry or from website to website, have AB tests built-in.
Some Use Cases are in “best practice” form regardless of the business. For these Use Cases, there will be generic performance reporting.
We strongly recommend to focus on the evaluation of the Use Case from day 1 to maximize the value. Our delivery team is happy to guide you through the prebuilt process of evaluation.
Evaluation Dictionary
The dictionary below is helpful for understanding metrics in the evaluation dashboard and their calculation.
Benefit/Revenue calculations
| Metric | Definition | Formula |
|---|---|---|
| Impressions | Sum of all actions that translate into a customer being impacted by a marketing campaign, e.g., weblayer showed/clicked, email opened/clicked, SMS delivered, push notifications delivered/clicked. | / |
| Visitors | Sum of all customers impacted by the marketing campaign (weblayers = show/click, emails = open/click). | / |
| Frequency | An average number of impressions (e.g., open email, show banner, etc.) per visitor. | = Impressions / Visitors |
| Revenue | The total value of all customer purchases impacted by the campaign (e.g., opened/clicked on the email, showed/clicked on the weblayer, etc.) occurred within the attribution window. | / |
| Purchases | Sum of customer purchases impacted by the campaign (e.g., opened/clicked on the email, showed/clicked on the weblayer, etc.) that occurred within the attribution window. | / |
| Buyers | Sum of customers impacted by the campaign (e.g., opened/clicked on the email, showed/clicked on the weblayer, etc.) who purchased within the attribution window. | / |
| Conversion rate (CR) | Percentage of impressions that were converted into purchases within the attribution window. | = count all purchases / count of all campaign impressions |
| Unique Conversion rate (UCR) | The proportion of customers who have seen the campaign and were converted into a purchase within the attribution window. | = all buyers / unique customers with impressions |
| Average Order Value (AOV) | Average revenue from a purchase/order. | = total revenue / total number of purchases |
| Revenue Per Visitor (RPV) | The average revenue per customer that has an impression (e.g., open email, show banner, etc.). | = total revenue / all visitors |
| Revenue Per Recipient (RPR) | Average revenue per customer that received the email campaign (tracked status ‘delivered’). | = total revenue / all customers with campaign delivered |
| Revenue Per Buyer (RPB) | Average revenue per customer that has at least 1 purchase within the attribution window. | = total revenue / all buyers |
| Attribution window for Campaign Performance (in hours) | Time between email opened or clicked and purchase. We recommend to set it up to 24h, 48h or 72h. | = (timestamp - last time campaign qualified) / 3600 |
Uplift calculations
Uplift represents the difference in performance between Variant A and the Control Group. If the uplift value is positive, Variant A is the winner, and the use case should be maintained. If the uplift is negative, the Control Group performs better than the Variant, and the use case hypothesis should be adjusted.
Uplift results should be taken into consideration together with the statistical significance. The results are significant if they reach more than 98%. The significance value can be found as part of the Evaluation dashboard, more specifically Conversion funnel > Confidence.
| Metric | Definition | Formula |
|---|---|---|
| Revenue Uplift | Extra revenue brought by Variant A compared to the Control Group. Uplift determines the absolute financial outcome of your Bloomreach Engagement campaigns. | = [ RPV(Variant A) - RPV(Control Group) ] x Visitors(Variant A) |
| Revenue Uplift Potential | Potential Uplift determines the theoretical financial outcome of your Bloomreach Engagement campaign if Variant A would be deployed to all customers (Variant A and Control Group). This outcome is an extrapolation of known data, not a guaranteed number. | = [ RPV(Variant A) - RPV(Control Group) ] x Visitors(Variant A + Control Group) |
| UCR Uplift | The difference between UCR of Variant A and UCR of Control Group in percentage points. | = [ UCR(Variant A) - UCR(Control Group) ] x 100 |
| UCR Uplift % | Percentage difference between UCR (Variant A) and UCR (Control Group). | = [ UCR(Variant A) - UCR(Control Group) ] / UCR(Control Group) x 100 |
| AOV Uplift | The difference between AOV of Variant A and AOV of Control Group in raw numbers. | = AOV(Variant A) - AOV(Control Group) |
| AOV Uplift % | Percentage difference between AOV (Variant A) and AOV (Control Group). | = [ AOV(Variant A) - AOV(Control Group) ] / AOV(Control Group) x 100 |
| RPV Uplift | The difference between RPV of Variant A and RPV of Control Group in raw numbers. | = RPV(Variant A) - RPV(Control Group) |
| RPV Uplift % | Percentage difference between RPV (Variant A) and RPV (Control Group). | = [ RPV(Variant A) - RPV(Control Group) ] / RPV(Control Group) x 100 |
| RPB Uplift | The difference between RPB of Variant A and RPB of Control Group in raw numbers. | = RPB(Variant A) - RPB(Control Group) |
| RPB Uplift % | Percentage difference between RPB (Variant A) and RPB (Control Group). | = [ RPB(Variant A) - RPB(Control Group) ] / RPB(Control Group) x 100 |
| Attribution Window for Uplift since AB Split (in hours) | Time between AB test split and purchase - used only in uplift calculations. We recommend to set it up to 24h, 48h or 72h. | = (timestamp - last time split qualified) / 3600 |
Campaign metrics
There are two types of metrics in the evaluation: unique and non-unique ones. The unique ones count the number of customers that have taken action, and the non-unique ones count the number of events that have occurred. For example, one customer will open the email three times. Non unique open metrics = 3, unique open metric = 1.
Ecommerce Benchmarks
These benchmarks are specific to each campaign, but achieving them for each campaign typically results in meeting the overall monthly benchmarks.
Ecommerce Benchmark for emailing metrics
- Unique Delivery rate - 99% and above
- Unique Open rate - 20% and above
- Unique Click rate from opened -15% and above
Ecommerce Benchmark for ‘negative’ emailing metrics
- Unique Hard Bounce rate - less than 1%; in the long run, aim at less than 0.5%
- Unique Soft Bounce rate - less than 2%
- Unique Unsubscribe rate - less than 0.5%
- Unique Complained ‘Spam’ rate - less than 0.1%
- Unique Pre Blocked rate - less than 0.01%
- Unique Clicked_honeypot rate - less than 1%
Ecommerce Benchmark for weblayer metrics
- Unique Click rate from the show - 1.5% - 4% and above, namely for banners with vouchers
Updated 9 months ago