Understanding Search Relevance and Improving It for Your Ecommerce Site

Looking for information on any kind of network, whether it’s your company intranet or the entire World Wide Web, is often a lot like looking for a needle in a haystack. However, these haystacks are special: they allow clever algorithms to be applied to them, so that the needle you’re looking for can be found with astonishing ease (along with similar needles!).

Understandably, this wasn’t always the case. Early search engines and protocols (at the start of the 1990’s) were pretty much entirely based on using strings to comb through the file systems of online institutions. If the string didn’t match anything, you were out of luck. With the advent of the World Wide Web in 1993, the number of websites ballooned, leading to many new search engines coming up as well, which began to provide much more relevant search results.

Google burst onto the scene in 1998, and has been at the forefront of search-related advancements ever since. Today, Artificial Intelligence (AI), Machine Learning (ML) and Natural Language Processing (NLP) are key to the underlying nuts and bolts of search engines, and enable them to provide results that are as relevant as possible.

Given that this idea of search relevance is also important for website owners to understand, let’s take a closer look at it, and also understand how website owners can improve the search relevance of their site’s search engine.

What is search relevance?

Although it might seem intuitively simple to grasp, the meaning of ‘search relevance’ is actually quite slippery and subjective. In a nutshell, it refers to the relevance of a search engine’s results to a given searcher, for a given query. Thus, there are at least two factors that go into determining the relevance of a search result: the query that produced that result, and various searcher-related factors (such as their location, past searches, etc.).

However, in practice, there is also a third factor that search engines might need to consider when ranking for relevance, particularly when it comes to site search (i.e. search that finds information only on a given website, rather than the entire World Wide Web): the business goals of website operators.

It should be clear that such search relevance is bound to have a subjective aspect to it: users might disagree on the degree of relevance of certain results. Nevertheless, it is necessary for search engines and site owners to have some way of measuring and quantifying this important variable. Below are the four metrics that are often used in this context.

1. Precision: This refers to the relevance of the entire search results set. It ignores the position of individual results.
2. Recall: This measures “the ability of a search engine to retrieve all the relevant results”
3. Average precision: This is a measure of the relevance as seen from the perspective of a user who scans through results sequentially.
4. ‍Cumulative gain: This measures the amount of information gained from a search results set.

Let’s now take a more in-depth look at the nuts and bolts of search relevance.

How does search relevance work under the hood?

The first step that a good search engine has to get right is parsing the search query and extracting as much information from it as possible. While older, keyword-based engines simply relied on the words within a query, modern search engines apply a variety of techniques to search queries, including: 

• Natural Language Processing (NLP), which is a branch of Artificial Intelligence (AI) that aims to allow computers to understand text and speech as well as humans can

• Semantic techniques that can deduce the ‘search intent’ of a given query

• User-specific information, such as their location, past searches, etc.

Unless this first crucial step is handled well, a search engine cannot hope to produce relevant results.

Moreover, before it can start calculating the search relevance of results, a site search engine must be able to analyze the entirety of the database from which the search results will be drawn. This involves creating a search index using a web crawler or API that ingests all the data on your site.

Once a search index is in place, an AI-based search engine can start deploying its AI and ML models to calculate relevance scores and rank results accordingly. Moreover, as mentioned above, there are several different metrics to measure the quality of such relevance scores, such as the normalized discounted cumulative gain (NDCG) and the mean reciprocal rank (MRR).

This then leads into the final step: the continuous refinement of search relevance over time. Modern search engines that use ML keep getting better at producing relevant results. Moreover, such search engines can also take various on-site signals (such as clicks and conversions) and use them to improve search relevance.

Lastly, site search engines today also allow website owners to tweak their ranking algorithms so as to address their specific business needs.

Why relevant search is indispensable for a seamless on-site experience

When visitors land on a new website, it is very common for their first point of interaction to be the search bar. Thus, your site’s search results are the very first impression visitors will have of your site or brand, which makes them crucial as far as your sales and lead funnels are concerned.

Netizens today are spoiled for choice and used to being catered to quickly. The days where people would have the patience to pore through search results looking for what they wanted are long gone. These days, if a site visitor is faced with unhelpful and irrelevant search results, they might not even bother to engage with the rest of your site. Instead, they might simply head back to Google and restart their hunt for whatever they were looking for. 

Such a situation is bad for the visitor as well as the website owner: the visitor has to deal with the annoyance of having wasted their time, while the website owner has lost out on a potential customer.

Thus, if you want to hold the attention of your site visitors and keep them engaged for as long as possible, then it is imperative that you provide them with powerful site search capabilities.

How website owners can give their site search a relevance boost

If you’re using a site search engine that’s relatively flexible and configurable, then there are several steps that you can take to make your site search results more relevant. In addition, if you’re using a site search engine based on more powerful technologies such as Artificial Intelligence (AI) and Machine Learning (ML), then some more advanced possibilities also open up to you.

Here are some of the steps website owners can take to improve the relevance of their site search results:

1. Use tags and categories

Tags and categories are kinds of metadata that can be attached to products or pieces of content. Categories help broadly define the group to which a given entity belongs (e.g. ‘menswear’, ‘athletic wear’, etc.) while tags enable more granular classification (e.g. ‘shorts’, ‘jersey’, etc.).

By adding accurate metadata to products or pieces of content, you can make it easier for your site search engine to match them with specific queries.

2. Leverage boosts and filters

Boosts are a way of assigning greater weight to certain aspects of a search result. For instance, if you boost the product, that product will be ranked higher in the search results if the search query matches a part of its title, as opposed to a part of its description, say.

By providing search filters to your site visitors, you effectively allow them to provide additional information about what they’re looking for, allowing your site search engine to further fine-tune its results.

3. Ensure typo tolerance 

Typos are effectively inevitable when using search on mobile devices, and are quite common even when using larger devices. To prevent them from hampering the search experience, many search engines come with configurable typo tolerance settings that automatically compensate for them. Website owners should use this feature because if they don’t, then the gap between the search intent of a visitor and the search results produced would become far too large, leading to low search relevance.

4. Personalize search results and rankings

This is one of those steps that will typically require a more advanced AI-based search engine to implement properly. In effect, such a search engine can draw on various kinds of user signals, such as location, age and past searches, to tailor search results and their rankings to individual site visitors.

By doing so, search engines can greatly increase the relevance of their results for all users, and improve conversion rates and profits for website owners.

5. Use search data to “train” your site search engine

As hinted at above, search engines that use ML under the hood are capable of improving their performance by analyzing relevant data. Thus, online business owners using an ML-based site search engine can improve its search relevance by feeding it all the search query data gathered over a period of time. 

Search relevance: a matter of great relevance today

It should be clear that if you operate an online business, you absolutely cannot afford to let potential customers walk away with a poor impression of your brand. A powerful search engine that provides highly relevant results is an essential component of engaging visitors and keeping them interested in what you have to offer.

If you’d like to implement an easy-to-integrate AI-based search engine for your business website, consider using Zevi. By combining the power of ML and NLP, Zevi ensures that your site visitors have a smooth and intuitive search experience; what’s more, it keeps getting better over time! Contact us today for a free trial.