In Analytics, Modeling
Summary: Internal search data is hard to interpret, and it can be harder to explain, but if you use visualizations it can be a snap. We'll show you how!

Internal search data is hard to interpret, and it can be harder to explain. Even Lou Rosenfeld’s excellent book on search analytics assumes that a careful, detailed examination of the data is necessary for understanding. That won’t fly for a lot of reasons. Any metric that gets traction in an organization has to have two key factors:

  • It must be immediately understandable.
  • It must reflect some metric of success.

We need, in short, simpler ways to explain complex search data. In this two-part blog post, we’ll show you how to make it easier by using two killer visualizations of search behavior:

  • The Query/Selection matrix for describing relevance (see below).
  • The Rank/Query ratio graph for describing accuracy (see Part II coming soon).

Describing Relevance with the Query/Selection Matrix

Relevance is the measure of how likely a search result will return meaningful hits, regardless of the position of the hit in the search result list. A search system with good relevance should have, overall, a pretty close to 1:1 correlation between of hits and search query. Search results that have a lot of irrelevant hits, in contrast, drive users to respond with one of three approaches:

  • Thrashing: The user repeats queries with variations to the search term.
  • Pogosticking: The user looks at multiple search results to see if he can understand the search result list as a set.
  • The user punts and exits the system without even looking at the results.

Peter Morville’s and Jeffrey Callendar’s excellent book, Search Patterns, describes these characteristics in great detail.

A Query/Selection matrix can sniff out these conditions across a user set. The number of hits per session are rows; the number of queries per session are columns. The number of users are reflected by the percentages. We also used a heatmap in this matrix to help visualize the distribution of unique queries per session with actual hits.

Relevance Matrix

In this matrix, punting is a big problem; thrashing is a close second. About 20% of all users executed one query in a session, but then didn’t actually click on any of the search results. Even more telling is that about 48% of ALL users never actually clicked on a search result, suggesting that overall relevance is very low. That is some serious thrashing.


What we really want to see is a pretty even distribution of queries and search results running diagonally downwards along the matrix, with fewer “clusters” along the first column and top row.


So how would we measure improvement in this situation? Basically the goal would be an increase in second queries and an increase in second article accesses.


Wondering about how to determine the accuracy of search results? Check out the second half of this post: Visualizing Search Analytics Data, Part II: Search Accuracy. As always, feel free to contact us if you’d like more insight into the chart or the Excel tools we used to create it.

Recommended Posts