Using the Ranking Diagnostics tool
What's new in Q1 2019?
Look for a new section that provides information on how a query was processed by the Bloomreach engine.
A new table also shows where the user query (or synonyms for the user query) matches content in the feed. The answer to "Why is this product showing up for this query?" has never been clearer!
The Ranking Diagnostics tool is a utility that provides insight into how products in search results are ranked. The tool runs a specific search and returns the search results to you with data that indicates how each piece of product information contributes to the ranking of products in search results.
Diagnostic information is roughly divided into three parts. The top part of the page gives you query processing information and includes data on whether the query is being autocorrected or relaxed and the synonyms related to the entire collection of search results. The bottom part of Ranking Diagnostics displays the search results with a ranking diagnostics summary for each product. You can click a product to view more details. The third part of Ranking Diagnostics is the individual product information if you click into a product. This page gives specific metrics and scores on the product in question so you can better understand why the product shows up for the given search query.
How do I know why a product is showing up in search results?
You can run a search in the Ranking Diagnostics tool. It's located in the Merchandising → Site Search → Diagnostics section of your BloomReach Dashboard. You can run a search query with keywords, a specific product ID, category name or category ID. The diagnostic tool displays the results of your search with information to help you understand why particular products are showing up in search results and why they are in the sequence displayed.
What factors affect search results?
After you run a search, the top part of the diagnostics report shows you the Query Processing section, which highlights factors that affect the results of your search. Here's an example, showing you the query, tv, and the features that may have affected that query:
- Is this query autocorrected? - This field will say "yes" if a query is being autocorrected and "no" if it isn't. If a query is being autocorrected, this field will also display the term that the original query is being autocorrected to. A query is only autocorrected when the original query returns null results.
- Is this query relaxed? - This field will say "yes," if a query is relaxed and "no," if it isn't. If a query is relaxed, this field will also display the value for what the original query was relaxed to (eg: “black” - optional, “hiking” - optional, “boots” - required). A query is relaxed when the original query returns null results and no autocorrect match is found. You can hover over the information icon in the dashboard for more details.
- Synonyms Applied - This field lists all synonyms that are being applied to the query and now displays both the left hand side and right hand side of the synonym where previously it just showed the right hand side. This should make it easier to find the source synonym that could be introducing products into the recall set.
How does BloomReach derive synonyms?
BloomReach continually monitors which terms your customers enter into search boxes and which items they click following the search. When traffic between a search term and a clicked item reaches a threshold, then BloomReach automatically generates the appropriate synonym rule, adding it to the Auto-Generated Synonyms list on the Thesaurus page.
What can I do if I spot a synonym that doesn't belong?
You can reject derived synonyms that don't make sense for products. If there's a direct synonym that doesn't make sense, then you can modify the rule that created it.
Besides direct and derived synonyms, what else affects search results?
After you run a search, the bottom part of the page gives you a brief summary of information about each product in the search results. Here's an example for the query, tv:
You can click a product to open a detailed summary of how BloomReach used information about the product when the search results were ranked. Use this information to analyze why that particular product is part of the search results.
Performance metrics or Scores in Ranking Diagnostics have also been simplified into the following three components:
- Total Score: is the metric based on which products are ranked for a specific query/category. It is a combination of the Performance (User Behavior) and Relevance Scores. The Performance score has a much heavier weight on the Total Score than the Relevance Score.
- Performance: Normalized metric based on user behavior signals such as revenue, conversions, add-to-cart, views etc. generated by a specific product for a specific query/category.
- Relevance: Based on BloomReach Semantic Engine and Term Match to product feed data (e.g. title, category, brand etc.)
Searchable Feed Data: this table will show you where the user query (or synonyms for the user query) are matching to content in the feed. This section can be used to answer questions like "Why is this product showing up for this query?"
What do the diagnostic metrics mean?
BloomReach derives each returned product's position in search results from a set of signals. Each signal has a normalized score from 0 to 100 that is calculated specifically for that query and result set. A score of 100.0 indicates that the product had the highest score among the result set for that particular metric. When you run a specific search query in the Ranking Diagnostics tool, BloomReach returns these signals and values.
How can I see the personalization impact on search results for a specific user?
To see the impact that personalization has on a specific user, enter the user's BloomReach cookie ID in the User ID for personalization field under Optional Parameters. The user ID is the BloomReach-assigned value in the user's browser cookie (_br_uid_2). The Diagnostics view highlights personalization ranking changes by showing a personalized icon under the product in the results.
What can I do with this information?
Use the information to better understand the drivers behind a particular result set in BloomReach, and determine whether any adjustments must be made. The insights gathered from the Ranking Diagnostics panel can help you determine whether a synonym needs to be removed, a particular product attribute in the feed is incorrect, or some products need to be boosted or buried. You can then take action using the other BloomReach Dashboard tools.