Use Case Evaluation

The action-based Use Cases (such as weblayers or omnichannel orchestrations), where the success of the Use Case depends on the actions of customers and varies from industry to industry or from website to website, have AB tests built-in.

Some Use Cases are in “best practice” form regardless of the business. For these Use Cases, there will be generic performance reporting.
We strongly recommend to focus on the evaluation of the Use Case from day 1 to maximize the value. Our delivery team is happy to guide you through the prebuilt process of evaluation.

Evaluation Dictionary

The dictionary below is helpful for understanding metrics in the evaluation dashboard and their calculation.

Benefit/Revenue calculations

MetricDefinitionFormula
ImpressionsSum of all actions that translate into a customer being impacted by a marketing campaign, e.g., weblayer showed/clicked, email opened/clicked, SMS delivered, push notifications delivered/clicked./
VisitorsSum of all customers impacted by the marketing campaign (weblayers = show/click, emails = open/click)./
FrequencyAn average number of impressions (e.g., open email, show banner, etc.) per visitor.= Impressions / Visitors
RevenueThe total value of all customer purchases impacted by the campaign (e.g., opened/clicked on the email, showed/clicked on the weblayer, etc.) occurred within the attribution window./
PurchasesSum of customer purchases impacted by the campaign (e.g., opened/clicked on the email, showed/clicked on the weblayer, etc.) that occurred within the attribution window./
BuyersSum of customers impacted by the campaign (e.g., opened/clicked on the email, showed/clicked on the weblayer, etc.) who purchased within the attribution window./
Conversion rate (CR)Percentage of impressions that were converted into purchases within the attribution window.= count all purchases / count of all campaign impressions
Unique Conversion rate (UCR)The proportion of customers who have seen the campaign and were converted into a purchase within the attribution window.= all buyers / unique customers with impressions
Average Order Value (AOV)Average revenue from a purchase/order.= total revenue / total number of purchases
Revenue Per Visitor (RPV)The average revenue per customer that has an impression (e.g., open email, show banner, etc.).= total revenue / all visitors
Revenue Per Recipient (RPR)Average revenue per customer that received the email campaign (tracked status ‘delivered’).= total revenue / all customers with campaign delivered
Revenue Per Buyer (RPB)Average revenue per customer that has at least 1 purchase within the attribution window.= total revenue / all buyers
Attribution window for Campaign Performance (in hours) Time between email opened or clicked and purchase. We recommend to set it up to 24h, 48h or 72h.= (timestamp - last time campaign qualified) / 3600

Uplift calculations

Uplift represents the difference in performance between Variant A and the Control Group. If the uplift value is positive, Variant A is the winner, and the use case should be maintained. If the uplift is negative, the Control Group performs better than the Variant, and the use case hypothesis should be adjusted.

Uplift results should be taken into consideration together with the statistical significance. The results are significant if they reach more than 98%. The significance value can be found as part of the Evaluation dashboard, more specifically Conversion funnel > Confidence.

MetricDefinitionFormula
Revenue UpliftExtra revenue brought by Variant A compared to the Control Group.

Uplift determines the absolute financial outcome of your Bloomreach Engagement campaigns.
= [ RPV(Variant A) - RPV(Control Group) ] x Visitors(Variant A)
Revenue Uplift PotentialPotential Uplift determines the theoretical financial outcome of your Bloomreach Engagement campaign if Variant A would be deployed to all
customers (Variant A and Control Group). This outcome is an extrapolation of known data, not a guaranteed number.
= [ RPV(Variant A) - RPV(Control Group) ] x Visitors(Variant A + Control Group)
UCR UpliftThe difference between UCR of Variant A and UCR of Control Group in percentage points.= [ UCR(Variant A) - UCR(Control Group) ] x 100
UCR Uplift %Percentage difference between UCR (Variant A) and UCR (Control Group).= [ UCR(Variant A) - UCR(Control Group) ] / UCR(Control Group) x 100
AOV UpliftThe difference between AOV of Variant A and AOV of Control Group in raw numbers.= AOV(Variant A) - AOV(Control Group)
AOV Uplift %Percentage difference between AOV (Variant A) and AOV (Control Group).= [ AOV(Variant A) - AOV(Control Group) ] / AOV(Control Group) x 100
RPV UpliftThe difference between RPV of Variant A and RPV of Control Group in raw numbers.= RPV(Variant A) - RPV(Control Group)
RPV Uplift %Percentage difference between RPV (Variant A) and RPV (Control Group).= [ RPV(Variant A) - RPV(Control Group) ] / RPV(Control Group) x 100
RPB Uplift The difference between RPB of Variant A and RPB of Control Group in raw numbers.= RPB(Variant A) - RPB(Control Group)
RPB Uplift % Percentage difference between RPB (Variant A) and RPB (Control Group).= [ RPB(Variant A) - RPB(Control Group) ] / RPB(Control Group) x 100
Attribution Window for Uplift since AB Split (in hours)Time between AB test split and purchase - used only in uplift calculations. We recommend to set it up to 24h, 48h or 72h.= (timestamp - last time split qualified) / 3600

Campaign metrics

There are two types of metrics in the evaluation: unique and non-unique ones. The unique ones count the number of customers that have taken action, and the non-unique ones count the number of events that have occurred. For example, one customer will open the email three times. Non unique open metrics = 3, unique open metric = 1.

Ecommerce Benchmarks

These benchmarks are specific to each campaign, but achieving them for each campaign typically results in meeting the overall monthly benchmarks.

Ecommerce Benchmark for emailing metrics

  • Unique Delivery rate - 99% and above
  • Unique Open rate - 20% and above
  • Unique Click rate from opened -15% and above

Ecommerce Benchmark for ‘negative’ emailing metrics

  • Unique Hard Bounce rate - less than 1%; in the long run, aim at less than 0.5%
  • Unique Soft Bounce rate - less than 2%
  • Unique Unsubscribe rate - less than 0.5%
  • Unique Complained ‘Spam’ rate - less than 0.1%
  • Unique Pre Blocked rate - less than 0.01%
  • Unique Clicked_honeypot rate - less than 1%

Ecommerce Benchmark for weblayer metrics

  • Unique Click rate from the show - 1.5% - 4% and above, namely for banners with vouchers