2. Configure the A/B test

Once you start the A/B test, the New A/B test page opens. Configure the below components:

2.1 Name the test

Name your test. The test name can only contain alphanumeric characters, spaces, underscores, and hyphens. Note that the default name is automatically filled in when starting a test from a rule (method 2 or 3).

Best practice
Be specific when naming your test. The A/B tests page lists all the tests running on your site. If your test has a clear name, it's easy to find in the list. Avoid repetitive names like test1, test2, my test, my other test, or test1 for Bob.

2.2 Select success metric

Use the dropdown to select a relevant success metric to gather data. Note that this step is only to help you set a goal; the test itself doesn't use this metric. You still have to decide the test winner on your own.

1. AOV

2. ATC Rate

3. CR

4. RPV

   

2.3 Allocate test traffic

Drag the slider under Traffic allocation to set the percentage of traffic to include in the test. We suggest setting a high traffic allocation to quickly gather data and reach statistical significance.

2.4 Set up the test buckets

(Skip this step if you used method 2 or 3 to start the test).

1. Click Select rule variants to test.
2. Now, select the rule to test. You can choose the rule types below.
   • Ranking rule
   • Redirect rule
   • Ranking algorithm rule
   • Recall algorithm rule
   • Personalization rule

You can sort the rules table or click the filter names (in grey bubbles) to see specific rule types. Then, click the Select rule button next to the desired rule.

📘

Note

What isn't available for selection
Rules for a specific audience, with a specific duration, with a running test, or with an influence other than the current site context can’t be selected and are, therefore, not listed.

3. Now, select the variant(s) to include in your test. Select 2 to 4 rule variants to include in the test. To include a variant, click the toggle next to that variant. The active variant is included by default as the Control bucket. You can optionally add notes for each variant.

4. Once you've selected the variants, click the yellow Select rule variants button.

   

5. This returns you to the New A/B test page, and the selected variants will be visible as different test buckets.

2.4.1 Configure API parameters (Optional step)

❗️

Warning

This functionality is not Generally Available yet. Contact your Bloomreach representative if you want to use or know more about A/B testing with API parameters.

🚧

Warning

The API parameter configurations set in the A/B test will only apply for the duration of the test. When the test ends, the winning bucket is chosen, but the API parameters will no longer be applied.

The functionality to save and apply the winning API configuration directly will be available soon.

Under the API params field, you can additionally configure the values of different parameters of the Search API for any test bucket.

This gives the bucket more flexibility and allows you to test how your rule performs with or without other Search settings applied, even those from other rule types.

1. Click + Add for a test bucket for which you want to change the API configuration.

   

2. In the API parameters modal, click + Add parameter. For each entry, you can either type the name and desired value of a Search API parameter or choose them from the dropdown list suggestions.

3. Click the OK button. You can still edit the parameters before the test starts. If a user doesn't have permission to edit API parameters or if the test is running, you can only view the configuration.

   

4. Check the Preview button and validate the products before starting the test.

📘

Note

In case of conflicting API parameters, the parameter values are applied in the following order of precedence:

1. Parameters passed directly in the API
2. Parameter customizations from the A/B test
3. Parameters set in the dashboard

2.5 Preview test buckets

Before you start a test (and even while a test is running), you can preview the different variants that are set up. To see the test buckets, click the Preview button.

You can preview one variant or choose a side-by-side preview to look at the differences between the two (or more) variants. Check the Show Changes checkbox in the preview to annotate the products with additional information about how the changes affect their ranking.