Internal search data is hard to interpret, and it can be harder to explain. Even Lou Rosenfeld’s excellent book on search analytics assumes that a careful, detailed examination of the data is necessary for understanding. That won’t fly for a lot of reasons. Any metric that gets traction in an organization has to have two key factors:

  • It must be immediately understandable.
  • It must reflect some metric of success.

We need, in short, simpler ways to explain complex search data. In this two-part blog post, we’ll show you how to make it easier by using two killer visualizations of search behavior:

  • The Query/Selection matrix for describing relevance (see below).
  • The Rank/Query ratio graph for describing accuracy (see Part II coming soon).

Describing Relevance with the Query/Selection Matrix

Relevance is the measure of how likely a search result will return meaningful hits, regardless of the position of the hit in the search result list. A search system with good relevance should have, overall, a pretty close to 1:1 correlation between of hits and search query. Search results that have a lot of irrelevant hits, in contrast, drive users to respond with one of three approaches:

  • Thrashing: The user repeats queries with variations to the search term.
  • Pogosticking: The user looks at multiple search results to see if he can understand the search result list as a set.
  • The user punts and exits the system without even looking at the results.

Peter Morville’s and Jeffrey Callendar’s excellent book, Search Patterns, describes these characteristics in great detail.

A Query/Selection matrix can sniff out these conditions across a user set. The number of hits per session are rows; the number of queries per session are columns. The number of users are reflected by the percentages. We also used a heatmap in this matrix to help visualize the distribution of unique queries per session with actual hits.

Relevance Matrix

In this matrix, punting is a big problem; thrashing is a close second. About 20% of all users executed one query in a session, but then didn’t actually click on any of the search results. Even more telling is that about 48% of ALL users never actually clicked on a search result, suggesting that overall relevance is very low. That is some serious thrashing.

Search_Failures

What we really want to see is a pretty even distribution of queries and search results running diagonally downwards along the matrix, with fewer “clusters” along the first column and top row.

Relevant_sweet_spot

So how would we measure improvement in this situation? Basically the goal would be an increase in second queries and an increase in second article accesses.

Relevance_Improvement

Wondering about how to determine the accuracy of search results? Check out the second half of this post: Visualizing Search Analytics Data, Part II: Search Accuracy. As always, feel free to contact us if you’d like more insight into the chart or the Excel tools we used to create it.

Recommended Posts