In Part I of this topic, we showed you how to visualize analytics data in order to measure the relevance of search results. We now turn to accuracy, or the likelihood that a search engine will return relevant results in decreasing order of value. While relevance is the critical baseline metric in search, search accuracy really determines a search engine’s performance ceiling.
It is devilishly hard to disentangle search accuracy because many behaviors reacting to inaccurate results look a lot like users reacting to irrelevant results. The simplest way to tease them apart is to examine the interaction of two metrics:
- average rank of a selected result, and
- number of queries per session.
We assume that overall rank of a query selection should slowly increase as the number of queries increases.
The chart we like to use for this measure of accuracy is the Rank/Query ratio, which shows how far down the search results users had to go to find the article they wanted. If users find the results promising, they will tend to select results based on the search result data, like page title and description snippet. Users who find the results promising but don’t get an exact match often scroll down the results to see if there is another result in the search result list that better matches what they are looking for.
In a low-accuracy environment, the effect can look like this:
What is noteworthy about this example is that:
- the average rank of a selected article is very high, more than sixth on a ten-page search result, and
- there is a big jump in average article rank after the average queries per session increases to more than three.
- users are selecting results, *on average* near the middle of the search results page.
- persistent users are drilling deep into the search results to find what they need.
Users believe that the results are out there, but they generally are not in the first four or five search results.
Persistent users are likely to be searching large numbers of search results, so they will be willing to go farther down the page to find a search result. Nonetheless, even power users want to get accurate results if they can.
What we want to see is two-fold:
- a decrease in the average search rank, and
- a (smaller) decrease in the average search rank for multiple query sessions.
Improving search accuracy can be a lot harder than improving relevance, because it requires a deeper understanding of performance tuning for specific search engines. But if your site depends heavily on search, it will be a worthwhile undertaking. And remember the 80/20 rule: the first changes create the biggest results.
As always, feel free to contact us if you want to talk about this chart, the ways we extracted the data, or other ways we use the information!