A/B testing FAQs
This guide lists the FAQs for the A/B testing application.
Overall functionality
1. A/B testing vs. split testing or bucket testing?
You might hear A/B testing called split testing or bucket testing. All of these terms refer to essentially the same testing technique. For simplicity, Bloomreach only uses the term A/B testing.
2. Is an A/B test the same as a before-and-after comparison?
No, check the table below to see how A/B testing and before-after changes differ.
| Criteria | A/B testing | Before-and-after comparison |
| How it works | A/B testing calculates the KPIs for the control and the variant of an experience over the same timeframe. | Before-and-after comparison measures the KPIs of an experience for a set number of days before and after an action is taken. |
| Decision-making | Data-driven since A/B testing lets you test conflicting rules simultaneously. You can commit to changes if the data shows that a particular group is performing well. | Risk-based since you first have to commit to your changes based on assumptions. |
| Effectiveness of changes | An A/B test isolates the effects of your changes by randomly exposing your change to a subset of your site visitors.This isolation helps you validate whether your change is indeed effective. | Before-and-after comparisons don't allow you to pinpoint the effects of your change. They can be positively or negatively impacted by external factors in the before or after experience. |
To be sure that your changes drive business to your site, we recommend A/B testing rather than comparing site data before and after changes.
A/B tests with multi-query rules in the same active test
1. How are the A/B tests handled when more than 1 term is included in the same active test?
A user is randomly assigned to a test or control when the user performs the first activity relevant to the A/B test. The user then remains in that bucket for the entire test period for all the qualifiable activities made by the user. This is true for all tests.
2. Are the results from both queries included in the test, or does the A/B test only use the first term for the test?
Results metrics will include all the activities relevant to the test, so all queries are included.
3. Is the traffic split at 50/50 per query in the test, or is the traffic split just between test bucket A and bucket B regardless of the queries?
The random splitting (which is assumed to be 50-50) of users happens at the time of the first activity inside the test, which could be either a query in a bigger test or just one in a single-query test.
4. What would happen if there were far more queries in one of the test buckets?
This situation would not arise because of the inherent randomness explained in point 1 above.
5. Does adding more queries to the test help increase traffic on test buckets? Would this help reach significance quicker for tail queries?
Generally, adding more queries increases traffic to all buckets proportionally. Significance can increase with an increase in lift or increase in traffic, but adding more tail queries does not affect overall test volume or impact it significantly. It would make no difference.
Updated about 8 hours ago