How Experiments work behind the scenes

This page describes how the Experiments feature works behind the scenes, what data is used to store them, and how the modifications set up in the Experiment are applied on the webpage.

❗️

Advanced topic

If you are not a front-end developer and you want to use our Experiments for basic changes, experimentation, or web personalization, you do not need to know how experiments work behind the scenes. This is an advanced topic for advanced users.

Data model

Let's start with the data model. Each experiment contains a configuration of AB-test variants. There is always at least one variant (meaning you cannot have an experiment with no variants).

Each experiment variant contains an ordered list of modifications. The order is important and depends on how you set up the modifications in the Experiments editor. Modifications get applied in the order they are defined.

A modification is an object with a few properties. The most important and common for every modification are type and element. The type defines the type of modification, as you select it in the Experiments editor. You can see the list of possible types with their descriptions in the Experiments documentation. The element is also important – depending on the type of the modification, this is the CSS selector of the element to modify (or remove, or add content near).

Different types of modifications have other attributes as well. For example, the Run script modification contains the text of the script, as well as the selected option when to run the script.

Fetching experiments

Depending on your integration setup (see Integrating and using Experiments for options), this is either done by the Javascript SDK or the Non-flickering Experiments's Modifications script. When the user visits your website, the SDK or the Modifications script requests the list of active experiments for the user from our back-end servers. The back end performs filtering, renders any dynamic content in the experiments (Jinja), and performs AB-testing to choose a variant for each experiment. The resulting list of experiment variants is returned to your website.

There is one additional difference in how the SDK and the Modifications script handle the HTTP requests to fetch the experiments. The SDK performs one asynchronous request to fetch the list of experiment IDs (which gets filtered), and then fetches the experiments in the second request. This is all done asynchronously and takes some time, which results in the Flickering Effect.

The Modifications script does things differently – when the script itself is requested by the website (through its <script src="..."> tag), the file is already returned with a list of static modifications for each experiment. Static modifications are those that do not contain any Jinja. Once those are loaded, the Modifications script performs an asynchronous request to fetch the dynamic personalized modifications, which are then applied asynchronously. Note that because the experiment modifications are always applied in the order they are defined, you should place your static modifications before any personalized ones. The Modifications script is loaded only with static modifications up to the first dynamic one, and must load the remaining modifications asynchronously (including any static modifications after the dynamic one).

Applying modifications

Again, how the experiment modifications are applied depends on your integration setup (see Integrating and using Experiments for options).

Basic integration

In this integration, the Modifications script is disabled and everything is performed by the Javascript SDK. This integration is basic and asynchronous. When the SDK receives the experiment modifications (after two HTTP requests), it starts executing them one by one in the order they are defined. See the section below for the behaviour that is done for each modification type.

When the SDK is querying elements based on a modification's CSS selector, it only does it once at the time it gets to apply the modification. If the element(s) exist(s), the modification gets applied. If it does not, the SDK does not wait for anything and just skips the modification.

Asynchronous / synchronous integration

If you are using the asynchronous or the synchronous solution, experiment modifications are applied by the Modifications script. The Modifications script receives the list of modifications (first static, then dynamic – read in the section above) and stores them in an internal list of modifications on the current page.

When the Modifications script loads, it sets up a MutationObserver on the whole webpage document. This means that as the page loads, or if any page content changes any time during the page view session, the Modifications script is notified and receives the content that has been added or changed. It looks into its internal list of active modifications, checks whether any of them match the new content, and if they do, it applies the modifications to the new/changed content before the user gets to see it. This is how we can modify even a dynamic webpage content without any flickering effects and without the end-user knowing about it.

Modification types

This section roughly describes what changes are done to elements in different modification types.

Change

Most visual changes made in this modification are done by either generating the equivalent CSS in a <style> (Modifications script), or by changing the inline style of the matched element (JS SDK).

The text changes are done by setting innerText of the matched element, and the HTML changes (done in the Code tab) are done by setting innerHTML of the element.

🚧

A word of caution when changing the content of elements

When you are changing either the text or the HTML content of an element, you should pay attention to selecting the deepest child node possible for the content that you want to edit. For example, if you want to change a few texts in your image carousel, try selecting the smallest text elements in the carousel and editing those instead of selecting the whole carousel element and changing its HTML.

The reason for this is that when you change the carousel's HTML, the SDK simply sets the element's innerHTML, which will probably break the Javascript running on your webpage behind the scenes and drive the behavior of the carousel (event listeners, timers, references to DOM elements, etc.).

Insert

Inserting new content is done using document.createDocumentFragment to create elements for the HTML, and then element.insertBefore to insert the content into the webpage.

Move

Similar to inserting a new content, moving existing content is done using element.insertBefore with the existing elements on the webpage. Note that both the container CSS selector and the element CSS selector must return the same number of elements.

Remove

Depending on the selected mode, the matched element is either hidden using CSS styling, or removed from the DOM using element.remove.

Run script

This is a powerful modification that allows you to run any custom Javascript. Depending on the selected mode, the modification is treated differently:

• immediately, before the page content is loaded – the modification is executed as soon as possible and pretty much ignores its element CSS selector. It is executed regardless of whether the selector matches any element on the page.
• on document ready event with all matched elements – the SDK or the Modifications script waits until the document is loaded (the DOMContentLoaded event is fired), then it selects elements using the modification's CSS selector and runs the script. You can use this.elements in the script to access the list of matched elements. The modification is executed regardless of whether the selector matches any element on the page (in that case, this.elements === []).
• once for each matched element – this is a useful mode, which ensures that your script runs exactly once for each matched element. You can access this.element to get a reference to the matched element and do any modifications to it. Additionally, the Modifications script ensures that your script gets executed for any dynamic content on the webpage (because it has set up the MutationObserver), so you can also use this to modify any elements that appear later on the webpage. This approach is also non-flickering – your script runs before the element is actually displayed to the user, so you can do any visual changes to it without flickering.

If the experiment is configured with a data layer trigger, the following five modes are available:

• immediately, when the triggering event occurs (first trigger only) - the script is executed as soon as possible the first time the data layer trigger event occurs.
• on document ready when the triggering event occurs (first trigger only) - the first time the data layer trigger occurs, the SDK or the Modifications script waits until the document is loaded, then executes the script.
• once for each matched element (first trigger only) - the first time the data layer trigger event occurs, the script is executed exactly once for each matched element.
• whenever a triggering event occurs (repeatedly) - in addition to the first time the trigger event occurs, the modification will also be executed each subsequent time the trigger event occurs and the experiment's conditions are met.
  Note: this option requires the Display setting for the experiment to be Always.
• repeatedly for each matched element - in addition to the first time the trigger event occurs, the modification will also be executed for each matched element each subsequent time the trigger event occurs, and the experiment's conditions are met.
  Note: this option requires the Display setting for the experiment to be Always.

Reverting modifications

For each applied modification, the SDK / Modifications script stores an inverse of the modification in a so-called "revert queue". This queue is used whenever the experiment had to be reverted. For example, when the Experiments editor needs to refresh the preview (it needs to revert the previous experiment preview), or when you change pages in a single page application (if the SDK is configured to do that using the Single-page application options).

👍

Custom run script revert function

Because the "Run script" modification is entirely your own code, we do not know how to revert it! You have to return an object with a remove function from the script for everything to function correctly. Read more here.

Compression

Nearly all web personalization endpoints use compression except when decided that a payload is too small, then it’s not compressed. Furthermore, old experiments are also excluded from compression, specifically the “/campaigns/experiments/show” route. However, it’s important to highlight that we are in the process of phasing out these older experiments. New experiments do use compression (endpoint: /webxp/script/).