Customize Ranking Algorithm with Ranking Studio

🌟

This is a Premium tier feature powered by Loomi AI.

Prerequisites

Knowledge
We recommend that you review the Ranking Studio guide to get a feature introduction.

Permissions
Ensure your IAM Admin assigns you the Algocontrol user permissions to access the Ranking Studio feature.

Dashboard overview

Ranking Studio provides a dashboard interface to customize and test the newly created ranking algorithm(s). You may use a mix of ranking strategies:

1. Machine learning (ML-powered) algorithm

• In this strategy, you choose an ML algorithm as the source algorithm to clone ranking signals, namely Performance and Text relevance signals.
• We add the custom signals you provide in the feed on top of the cloned signals.
• When you start ML training, the algorithm learns the signal weights with the LTR training framework.
• Upon training completion, the algorithm results can be viewed, tested, and applied to all or select queries (or categories).

2. Custom algorithm

• This strategy allows fine-grained control of the ranking signal weights. Here, you clone the signals from a source ML algorithm and choose the signal weights.
• We sync the customized weights, and the algorithm can be viewed, tested, and applied to all or select queries (or categories).
• This strategy is ideal when you have a dedicated in-house data science team that can provide comprehensive data insights to optimize the signal weights.

Both these strategies are covered in the workflow below.

Create and customize ranking algorithm

The below workflow illustrates customizing the core Bloomreach Search ranking algorithm for both Search and Category pages.

1. Navigation
2. View ranking studio homepage
3. Create new algorithm
4. Customize algorithm
5. View and edit algorithm

1. Navigation

1. Go to AI Studio > Ranking studio > Site search ranking algorithm to view the existing Search ranking algorithms.

2. Alternatively, go to AI Studio > Ranking studio > Category ranking algorithm to view the existing Category ranking algorithms.

   
   

2. View ranking studio homepage

You can now see the ranking studio homepage with the following elements:

2.1 Algorithms and Rules tabs

Algorithms tab
This tab displays the default Search or Category ranking algorithms and all newly created or existing ML algorithm(s). The following algorithms are visible:

1. Classic Search/Category ranking: This is the existing default ranking algorithm for Search and Category experiences.

2. Bloomreach-Optimized Search/Category ranking: This is a new ML-based algorithm that computes the weights of each signal by analyzing the data in your catalog(s). This is based on the LTR training framework. The below visual illustrates how you can start the algorithm training.

   

   Once the training is complete and the algorithm status is “Available,” you can view the signal distribution and create new algorithms using this as the source algorithm.

Rules tab
This tab displays the default algorithm for all Search/Category queries. Here, you can apply the ranking algorithm to all queries.

2.2 Catalog selection dropdown

Use this dropdown to choose the catalog to use for algorithm training. New algorithm(s) will be created at the selected catalog level. All the sites associated with this catalog appear next to the Catalog selection dropdown.

2.3 Algorithms’ details

This section displays details of the default Search/Category ranking algorithms and the new algorithms. These details include the algorithm name, status, the user who changed the algorithm, the date of the change, and algorithm actions.

3. Create new algorithm

1. Click + New Algorithm button. This opens the algorithm setup modal.

2. Add a name and description for the new algorithm.

3. Choose the Source algorithm from which you want us to clone signals. Since the new algorithm uses cloned signals, the signals are not synced if the source algorithm is updated. You can choose the Bloomreach-Optimized search algorithm or any other created ML algorithm as the source algorithm.

4. To view the ranking score distribution, click Show signals.

   
   

4. Customize algorithm: add signals or change signal weights

You may either want to overwrite algorithm signal weights or add new signals to customize the algorithm. Follow the steps for the chosen case:

4.1 Overwrite signal weights

1. Click Continue. This opens a detailed visual breakdown of all the signal weights.
2. Click Overwrite signal weights to edit the weights. Now, set the signal weights in percentage numbers that add up to 100. Note that if any of the weights of the ML algorithm are updated manually, it will no longer be an ML algorithm. It becomes a non-ML custom algorithm.
3. Click Apply to apply the weights.

4. Click Preview to see changes in search results.

5. Now, Save the changes to sync the new weights. The algorithm status shows Syncing. The syncing process typically takes 20 minutes.

4.2 Add new signals

1. Select the custom data signal(s). Click + new signal to add more data signals to the signal mix. The selection list populates only the attributes configured as Numeric and Facetable (Dynamic or Manual) at the Product (PID) level. Check the best practice guidelines.
2. Click the Continue button to confirm and begin the training process. You can define the weights of these new signals only when training is complete.
3. Now, the newly created algorithm’s status shows In Training. The training typically takes 24-48 hours to complete. You can train one algorithm at a time.

4. Cancel algorithm training (if needed)
   If the algorithm training is in progress and you wish to cancel it:

   1. Go to AI Studio > Ranking studio > Site search ranking algorithm (or AI Studio > Ranking studio > Category ranking algorithm).

   2. Locate the algorithm for which you would like to cancel the training.

   3. Click the Action dropdown v and select Cancel algorithm training.

      

   4. Confirm the cancellation by clicking Cancel and delete.

      

      You will see a message that the algorithm has been deleted.

      

📘

Notes

1. Please note that you cannot edit the Source algorithm directly. If you want to modify the Bloomreach-Optimized search algorithm, you must first choose it as a Source algorithm and then apply the customizations.
2. Training cancellation is not allowed for the Bloomreach-Optimized search algorithm that is undergoing LTR training.

5. View and edit the algorithm after syncing/training

1. When the syncing/training is complete, the algorithm shows the Available status.

2. Click the View action to see the algorithm details and a breakdown of the ranking signals’ weights.

   
   

3. You may want to edit the weights by clicking Overwrite signal weights. Repeat the steps as provided in the 4.1 case above.

   
   

4. Click Preview to see changes in search results.

5. Now Save the changes.

After you have made all the desired customizations and the algorithm is in the Available state, you can test the new algorithm by creating algorithm rules and applying them to all queries or selected queries.

Manage existing algorithms

• To delete an existing algorithm, click the Action dropdown to select the Delete option.

  

• Five customized algorithms are allowed per catalog. Before creating a new algorithm, consider editing or deleting an existing one.

👍

Click the dropdown next to View action. This shows the option to View ranking algorithm ID option. The algorithm ID available here can be copied and used to configure an AB test via API.

Configure algorithm rules and AB tests

Go to the Rules tab to create, manage algorithm rules, and configure AB tests.

1. Create new algorithm rules

1.1 Apply new rule for all queries

• Click the + New rule button. This opens the modal to create a new ranking algorithm rule. Select the rule scope as All queries.

• Choose the new algorithm from the dropdown. This algorithm applies to all queries.

• Save the rule.

  

  1.2 Apply new rule to select queries

• Click the + New rule button. This opens the modal to create a new ranking algorithm rule. Select the rule scope as Select queries.

• Enter the queries that you want to target with this rule. Choose the desired algorithm from the dropdown for these selected queries.

• Save the rule.

  

2. Manage existing algorithm rules

2.1 Edit Rule

• To edit an existing ranking algorithm rule for all queries/select queries, click the Edit button next to the rule.

  
  

• Choose a different algorithm from the dropdown.

• Save the rule.

2.2 Delete Rule

Click the Action dropdown to select the Delete rule option.

3. Setup AB test for algorithm rule

Example Test Goal

Validate the new algorithm's performance compared to the default ranking algorithm for a select set of queries.

1. Go to an existing algorithm rule and click the Edit button next to the rule.

2. Make the query/algorithm changes to the rule (as mentioned in sections 1.1 and 1.2 above).

3. Now, click ​​​​Save as test variant. This opens a Rule Variants modal to include the new variant in the test.

   
   

4. ​​Click Set up new test.

Notes:

1. You can see all test variants associated with a query rule by clicking the Variants icon or the downward arrow next to the Edit option.

   

2. Dashboard Preview is not supported for AB tests. We will add this capability in subsequent feature iterations. You can instead follow the below steps to compare the variants: Suppose we want to compare the results of Algo Relevancy Boost 50% vs. Algo Views Boost 50%.

• To do so, click the View option next to any of the test algorithms.

  

• Then, click Preview and go to the Side-by-Side Preview tab. On the right side, you can see the results for the selected test algorithm. To preview and compare results with the other algorithm on the left side, open the Ranking Preview dropdown and select the desired algorithm.

  
  

3. If you want to AB test an algorithm with an “All queries” scope, a similar test setup applies as above.

Conflict resolution

1. If two or more algorithms overlap, which one applies? For instance, two algorithms are applied for all queries.

If multiple algorithms are applied for all queries, then we choose the one that was most recently applied.

2. What happens when there are existing overrides on the API?

The values passed in API will decide the ranking behavior. Note that the algorithm sent on the API takes precedence over the selection in the Dashboard.

• If you have disabled relevance attributes (br_ranking.relevance = off), the algorithm's performance and custom signals will be used, but the text relevance scoring will be ignored and set to zero.
• If you have disabled performance attributes (br_ranking.performance = off), then the text relevance portions of the algorithm will be used, but the performance and custom signals scores will be ignored and set to zero.
• If you have disabled both performance and ranking attributes (br_ranking = off), the ranking algorithm will not be used. Instead, it will be entirely based on your ranking rules.

3. Suppose a setting is passed at various levels on the API and Dashboard. How are these handled?

Conflict resolution order works as follows:

• API parameters passed directly on front-end API call
• Query Override on the Dashboard
• Site-level rules on Dashboard
• Site group-level rules on the Dashboard
• Account-level rules on the Dashboard

Best practices

1. How can I use an attribute as a custom signal?

• To use an attribute as a custom signal, we expect it to be in a certain format and configuration. Ensure the attribute is sent with a positive (non-negative) value and configured in Catalog Management as ​​Numeric and Facetable (Dynamic or Manual).
• We also recommend that this numeric value represents the relevancy of the document in some form. This shows a clear order of importance or relevance for search ranking. For example, economic status with three categories (low, medium, and high), product rating with a value range of 0-5 where 0 means lowest and 5 means highest rating, discount, etc.

What is not supported
Values, string or numeric, that don’t maintain relative relation or a clear order, such as aisle_number, model_number, and categorical values like color/brand.

Notes

• If an algorithm currently uses an attribute as a signal, we recommend not changing the above attribute configuration or deleting the attribute, as this may impact the algorithm.
• If the attribute is not actively sent via feed after adding it for the first time, allow 7 days for our algorithm to train on this data before you use this attribute as an additional signal.

2. What’s Bloomreach’s recommendation for setting signal weights?

In general, Bloomreach recommends adding new signals to Ranking Studio and letting Bloomreach ML determine the appropriate balance of signal weights.

3. When is a good time to change signal weights?

Changing signal weights for an algorithm in production can be a consequential decision. We suggest that you create a new algorithm variant to assess the impact of signal weight changes and AB test the new algorithm variant to understand the impact on business metrics.