How cutting-edge AI solutions are transforming site search

Artificial intelligence (AI) technology has made giant strides over the past two decades. In part, this is due to major advances in artificial neural networks, machine learning (ML) techniques, and natural language processing (NLP). In addition, cheaper and faster computing power, improvements in algorithms, massive datasets, better research tools, and the rise of cloud computing infrastructure have also all played a critical role in enabling the widespread application of AI that we see today. One of the key problems that AI is currently being fruitfully applied to is search and ranking.

Within a few years of the World Wide Web opening to the public in 1991, the explosion in the number of online web pages led to many search engines coming into existence, which returned results on the basis of various different algorithms and techniques. Today, however, thanks to the rise of e-commerce and online product offerings, a specific kind of search has taken on a lot of importance: site search, which is where a search engine matches a user query with a ranked list of site-internal pages, documents, or products. And it’s this kind of commercially vital search experience that AI is currently helping improve.

Let’s take a closer look at how AI augments search capabilities, and how it’s transforming site search to provide users with a frictionless and intuitive experience.

Traditional AI-based search

Even before AI began to be used by search engines, algorithms such as Google’s PageRank could deliver impressive results. PageRank, in particular, assigns each web page a score that is largely based on the number of external links that point to it; in other words, each such link is taken to be a sign of the quality of a web page. Such a scoring system, combined with standard web crawling and inverted indexes, are what gave Google most of its power in its early days.

Eventually, certain kinds of machine learning began to be incorporated into web search engines as well as site search engines. For instance, ML could be used for autocorrecting queries, providing query suggestions, named entity recognition, and so on. At this point, one could argue that AI had begun to become a critical part of the search experience.

However, there’s one major problem that such a system can’t easily address.

The key issue with traditional AI-based search

This problem stems from the fact that at the end of the day, such a system is based mainly on string matching. If the words in a query string don’t match the words in a certain document, that document will not be part of the search results, even if the query string’s words are semantically or contextually related to the words in that document.

This means that unless users happen to type in relatively specific queries, they might not get the results they’re looking for. This imposes a certain amount of additional cognitive load on users.

This situation can be partly remedied by manually tagging various key words in titles and descriptions. For example, on a blog site, the word ‘amazing’ could be tagged with synonyms or variant forms, such as ‘incredible’, ‘extraordinary’, ‘fantastic’, and ‘amazed’. As a result, someone who searches the blog using the query ‘incredible story’ will also see a post entitled ‘My Amazing Adventure’, for instance. However, manual tagging is time-consuming, can be expensive, and can fail to truly capture the richness of the associations between different words.

The commonly used string-matching-based approach to search spells bad news for e-commerce stores and content-heavy sites in particular. This is because it forces their visitors and potential customers to type in queries that partly match the title or description of certain products or pieces of content. Otherwise, those products or pieces of content won’t be shown to visitors, even if they’re exactly what visitors are looking for! And ultimately, what this means is a potential loss of revenue or engagement.

Here’s a quick example to make such a situation clearer. Imagine that you run an online clothing store. A visitor to your store wants to buy a dress with frills, so they type in ‘dress with frills’ in your store’s search bar. Now, let’s assume that although you do have a bunch of frilly dresses in your inventory, none of those dresses have the word ‘frills’ in their title or in their product description as stored in the catalog.

In such a scenario, despite your store selling frilly dresses that this visitor might be happy to buy, the visitor comes away with the impression that you don’t have any frilly dresses at all, since the search results come up empty. This would affect not just your store’s bottom line, but also the visitor’s impression of your store, thus affecting future business as well.

Similarly, for a content-heavy site, visitors might not be able to find the sort of content they were looking for, even if the site does have content in line with their search intent. This would lead to a poorer visitor experience, and a loss of engagement for the site.

Thus, a solution to this problem would translate into higher conversion and engagement rates for website owners, and a smoother and more fruitful experience for site visitors. For this reason, solving this problem is exactly what the latest, cutting-edge AI-based search engines are focussed on.

The cutting edge of AI-based search

Instead of using string matching as the primary basis for looking up queries, the latest AI-based search engines, such as Zevi, use NLP at their core. NLP is a branch of AI that aims to provide computing systems with the ability to process natural human language in such a way that they seem to understand what is being said. Behind the scenes, NLP works by chewing on massive corpora of texts and detecting various kinds of statistical patterns within them, which it then uses to respond to inputs.

Thus, the use of NLP for search means that something as abstract as ‘user intent’ can seemingly be “understood” by a site search engine. From the point of view of customers, this can seem almost magical, and such capabilities are likely to leave them delighted.

To see a concrete example of how NLP is being used for site search, let’s focus on the underlying nuts and bolts of Zevi in particular. Between the query input by a customer and the ranked list of results that they see, there are two main AI-based modules at work in the background: the AI Search Engine, and the Ranking Engine.

The AI Search Engine

This engine is built using artificial neural networks and machine learning models that have been trained on massive quantities of real-world data. Moreover, these models can be improved further by training them on newer data as and when it becomes available. This includes the data generated every time a visitor interacts with a website.

The AI Search Engine is split up into three main layers:

1. The spell-check/autocomplete layer: This layer not only helps users correct their spelling, but also uses the power of AI to suggest the most likely continuations of a query.

2. The vernacular/mixed language layer: The data that the AI Search Engine has been trained on includes multiple languages and multiple scripts. What this means is that visitors can input search queries in the language and script of their choosing; they can even mix and match scripts and languages! For instance, the query ‘kala shirt’ would, in fact, return a list of black shirts (‘kala’ is the Hindi word for ‘black’). 

Zevi supports English, Spanish, and Hindi (both in the Devanagari script and in the Roman script).

3. Contextual understanding layer: This is where the magic of NLP truly shines. Due to the large amounts of data the AI Search Engine has been trained on, it is “aware” of the typical semantic and contextual associations between words, and uses its “knowledge” of such associations to intelligently return results that are likely to be relevant to the user.

For instance, a visitor to an online store might type in ‘pimples’ with the intention of finding a cream that gets rid of pimples. However, it’s possible that none of the products in the store that can get rid of pimples have the word ‘pimples’ or ‘pimple’ in their description: they might be marketed and described as being ‘anti-acne’ or ‘for sensitive skin’ instead.

For Zevi, this would not be a problem: it would automatically be able to recognise that those words are all associated with each other, and would thus list those products in the search results, thus addressing the visitor’s search intent.

Let’s now move on to the second key AI-based module that powers Zevi: the Ranking Engine.

The Ranking Engine

The Ranking Engine takes all the hits provided by the AI Search Engine in response to a query, and presents them in a particular order to the user. In terms of its capabilities, this engine is similar to the recommendation engine used by websites such as Netflix.

This engine consists of two main layers:

1. Personalized ranking: The Ranking Engine can use the behavior and history of each individual user (i.e. user signals) to rank search results specifically for them, in such a way that the likelihood of their engagement goes up. The user signals that this layer looks at include past queries, products viewed, products purchases, previously viewed pages, and actions performed on the site (such as hovers, button clicks, etc

While such personalization could also be provided by traditional search engines without AI, they relied almost entirely on cookies to do so, which greatly limited the amount of personalization that could actually be accomplished. In contrast, Zevi’s Ranking Engine, being AI-based, is capable of providing much greater personalization.

2. Listing-performanc ranking: This layer takes into consideration various properties of products or pieces of content, such as the revenue or engagement they have generated in the past, and factors them into the final ranking.

The benefits of the latest AI-powered site search capabilities

Thus, the use of such an NLP-based core to carry out site search results in several benefits to site owners and users, with the main ones being:

• The search engine gets a contextual understanding of a user query; this results in minimal cognitive load for users, and improves engagement and conversion rates.

• There’s no need for manual tagging, as the NLP core is pre-trained on vast amounts of data, and can thus automatically extrapolate all possible tags for a particular listing. Similarly, there’s no need for a manual synonym layer either.

• Multilingual and multi script queries can be handled without any issues. Moreover, spell checks can also be handled by default in a language-specific manner: for foreign languages, the spell check focuses on the actual spelling, while for Indian languages, it focuses on the sound-to-spelling mapping.

• Rankings can be personalized for users, and can be further customized by the owner.

Business-oriented features of AI-driven site search solutions

While the AI-based capabilities discussed above are impressive enough on their own, an ideal solution of this sort must also come with several additional features that address specific business goals. For instance, Zevi also provides the following business-oriented features, among many others:

• Rank boosting: Website owners can boost search results  on the basis of various criteria (such as engagement ratings, revenue generated, etc.), towards their business goals. Moreover, these criteria can be used for different geographies.

• Hard pinning: Specific products or search results can be made to always appear in response to certain search queries. This can be especially useful for promoting new products or pieces of content.

• Detailed analytics: Zevi provides a real-time analytics suite that enables website owners to take intelligent, data-driven decisions that can improve visitor satisfaction and conversion rates.

• Search redirection: Any search query can be redirected to a specific page, a specific collection, or any campaign-related pages.

• Robust and scalable architecture: Zevi is built upon a solid, reliable and highly secure foundation that scales with website owners’ needs and provides a 99.99% uptime guarantee.

The next generation of AI-based site search is here!

To sum up, search engines that rely heavily on NLP and ML are transforming visitor journeys and improving business revenues. And with the underlying algorithms, techniques, and tools steadily improving, and with datasets growing ever larger, the future of this technology promises to shine even brighter. If you too would like to harness the power of AI to delight your visitors and get more engagement or sales, then Zevi is just the tool that you need.