Use Case Evaluation

The action-based Use Cases (such as weblayers or omnichannel orchestrations), where the success of the Use Case depends on the actions of customers and varies from industry to industry or from website to website, have AB tests built-in.

Some Use Cases are in “best practice” form regardless of the business. For these Use Cases, there will be generic performance reporting.
We strongly recommend to focus on the evaluation of the Use Case from day 1 to maximize the value. Our delivery team is happy to guide you through the prebuilt process of evaluation.

Evaluation Dictionary

The dictionary below is helpful for understanding metrics in the evaluation dashboard and their calculation.

Benefit/Revenue calculations

Metric

Definition

Formula

Impressions

Sum of all actions that translate into a customer being impacted by a marketing campaign, e.g., weblayer showed/clicked, email opened/clicked, SMS delivered, push notifications delivered/clicked.

/

Visitors

Sum of all customers impacted by the marketing campaign (weblayers = show/click, emails = open/click).

/

Frequency

An average number of impressions (e.g., open email, show banner, etc.) per visitor.

= Impressions / Visitors

Revenue

The total value of all customer purchases impacted by the campaign (e.g., opened/clicked on the email, showed/clicked on the weblayer, etc.) occurred within the attribution window.

/

Purchases

Sum of customer purchases impacted by the campaign (e.g., opened/clicked on the email, showed/clicked on the weblayer, etc.) that occurred within the attribution window.

/

Buyers

Sum of customers impacted by the campaign (e.g., opened/clicked on the email, showed/clicked on the weblayer, etc.) who purchased within the attribution window.

/

Conversion rate (CR)

Percentage of impressions that were converted into purchases within the attribution window.

= count all purchases / count of all campaign impressions

Unique Conversion rate (UCR)

The proportion of customers who have seen the campaign and were converted into a purchase within the attribution window.

= count of all purchases / unique customers with impressions

Average Order Value (AOV)

Average revenue from a purchase/order.

= total revenue / total number of purchases

Revenue Per Visitor (RPV)

The average revenue per customer that has an impression (e.g., open email, show banner, etc.).

= total revenue / all visitors

Revenue Per Recipient (RPR)

Average revenue per customer that received the email campaign (tracked status ‘delivered’).

= total revenue / all customers with campaign delivered

Revenue Per Buyer (RPB)

Average revenue per customer that has at least 1 purchase within the attribution window.

= total revenue / all buyers

Uplift calculations

Uplift represents the difference in performance between Variant A and the Control Group. If the uplift value is positive, Variant A is the winner, and the use case should be maintained. If the uplift is negative, the Control Group performs better than the Variant, and the use case hypothesis should be adjusted.

Uplift results should be taken into consideration together with the statistical significance. The results are significant if they reach more than 98%. The significance value can be found as part of the Evaluation dashboard, more specifically Conversion funnel > Confidence.

Metric

Definition

Formula

Revenue Uplift

Extra revenue brought by Variant A compared to the Control Group.

Uplift determines the absolute financial outcome of your Bloomreach Engagement campaigns.

= [ RPV(Variant A) - RPV(Control Group) ] x Visitors(Variant A)

Revenue Uplift Potential

Potential Uplift determines the theoretical financial outcome of your Bloomreach Engagement campaign if Variant A would be deployed to all
customers (Variant A and Control Group). This outcome is an extrapolation of known data, not a guaranteed number.

= [ RPV(Variant A) - RPV(Control Group) ] x Visitors(Variant A + Control Group)

UCR Uplift %

Percentage difference between UCR (Variant A) and UCR (Control Group).

= [ UCR(Variant A) - UCR(Control Group) ] / UCR(Control Group) x 100

AOV Uplift %

Percentage difference between AOV (Variant A) and AOV (Control Group).

= [ AOV(Variant A) - AOV(Control Group) ] / AOV(Control Group) x 100

RPV Uplift %

Percentage difference between RPV (Variant A) and RPV (Control Group).

= [ RPV(Variant A) - RPV(Control Group) ] / RPV(Control Group) x 100

Campaign metrics

There are two types of metrics in the evaluation: unique and non-unique ones. The unique ones count the number of customers that have taken action, and the non-unique ones count the number of events that have occurred. For example, one customer will open the email three times. Non unique open metrics = 3, unique open metric = 1.

Ecommerce Benchmarks

Ecommerce Benchmark for emailing metrics

  • Unique Delivery rate - 99% and above
  • Unique Open rate - 20% and above
  • Unique Click rate from opened -15% and above

Ecommerce Benchmark for ‘negative’ emailing metrics

  • Unique Hard Bounce rate - less than 1%; in the long run, aim at less than 0.5%
  • Unique Soft Bounce rate - less than 2%
  • Unique Unsubscribe rate - less than 0.5%
  • Unique Complained ‘Spam’ rate - less than 0.1%
  • Unique Pre Blocked rate - less than 0.01%
  • Unique Clicked_honeypot rate - less than 1%

Ecommerce Benchmark for weblayer metrics

  • Unique Click rate from the show - 1.5% - 4% and above, namely for banners with vouchers

Did this page help you?