Websort.net

Labels are a vital but often glossed-over element of successful information architecture. Designers (or worse yet, Department Managers) come up with their own labels and stick with them. A University of Minnesota grad student shows us why we’re missing out.

There’s a big gap between staring at the results of a card sort and launching a new and improved information architecture. Fortunately there are several tools and techniques to help you translate your results into usable navigation. But even experienced designers sometimes skip two other important gaps between a card sort and the user experience: what content to include, and what labels to display. Both of these questions need to be addressed before you run a card sort.

The earliest working prototype of WebSort (circa 2001) required participants to complete two distinct steps. First, they were shown long descriptions of items (which represented a feature of an application, or the content found on a page) and asked to provide a short label for each item. Then they continued on to a card sort of those items, which now showed the labels they provided. We thought this was fantastic.

Turns out, not many other people did.

As development of WebSort progressed, the whole feature was scrapped. This summer when I chatted with our long-time clients and card sorting experts at UPA, I was reminded of the importance of content labeling. Two months later, Jakob wrote about a similar issue of avoiding bias of terminology matching. Finally, Josh Carroll contacted us about a labeling study he was planning, and we were happy to provide a bit of technical support and consultation.

Below are links to a short case study of Josh’s experience, as well as a much longer paper. Let me just highlight one of the unexpected gems:

Some … responses were long, rambling narratives that would never be used on a website. However, the lengthier labels provided more insight into participants’ vocabulary and context.

Just as the organization of items in a card sort does not equal a perfect final web site architecture, the labels provided by users don’t need to be perfect usable link text; the real value is in getting a better understanding of how your users think about your content.

You can download the 5-page and 25-page versions of Josh’s report below. Josh is a Senior Usability Consultant at the University of Minnesota. You can contact him at carr0234 at umn dot edu.

Case Study – User Input on Labels (short version, pdf)

Case Study – User Input on Labels (full version, pdf)

Editor’s Note: This article was originally presented as a poster at UPA 2004.  It was also previously published on our blog on December 19th, 2007.  We recommend that you also review Jakob Nielsen’s response.

Tom Tullis, Fidelity Investments, Inc.

Larry Wood, ParallaxLC

Introduction

As card sorting has become more popular, several methodological questions/issues have arisen, such as the effect of instructions on participants, the differences in results from in-person vs. remote studies, and the number of participants needed for reliable results.  This report addresses this last issue (i.e., how many participants are needed for a study to produce reliable results).  The study was conducted online by the Usability Department at Fidelity Investments, Inc.  A total of 46 cards (items) were used in the study, many of which represented services offered internally by the the Usability Department, such as prototyping, usability testing, and card-sorting. Categories were not predefined; each user created and named their own categories (considered an “open sort”).

A total of 168 employees participated in the card-sorting study. From their data a similarity matrix was created, showing the frequency with which each pair of items was placed in the same category across all 168 participants. Therefore, the maximum similarity was 168 (all of the participants placed those two items in the same category), and the minimum was 0 (none of the participants placed the two items in the same category.

Data Analysis

The similarity matrix referred to above is the basis on which a statistical cluster analysis is performed, the result of which effectively “averages” the categorization accumulated across a set of participants. The resulting cluster analysis is then displayed as a hierarchical tree structure (known formally as a dendrogram), which shows clusters of similar items on which organization of content in a web site can be based.

The major goal of our research was to assess the degree of similarity of an organizational tree structure derived from random samples of participants to a structure based on the full set of 168 participants. This could then be used to estimate the minimum number of participants needed to produce an effective organization. As a means to that end, correlation coefficients were calculated between the similarity matrices for each sample size and the matrix for all 168 participants. The assumption is that the more similar the trees, the higher should be the correlation between the similarity matrices on which those trees are based. Thus, correlation coefficients between the sample similarity matrices and the full similarity matrix were calculated for 10 random samples each of sizes 2, 5, 8, 12, 15, 20, 30, 40, 50, 60, and 70 participants . A graph of the resulting mean correlation coefficients is shown in Figure 1.

Figure 1. Correlation coefficients for various sample sizes, with error bars.

As shown in the graph, the relationship between the sample size and the average correlation is a negatively increasing function. Thus, the increase is more dramatic at the smaller sample sizes so that as the size increases beyond 20-30, there is little increase in the size of the correlation coefficient. Also note that the variance of the values, as indicated by the error bars, is much greater for the smaller samples.

An important question is how the function shown in Figure 1 relates to the similarity of the actual tree structures as a function of sample size. One practical implication is that the structures derived from sample sizes above 30 are very similar to that derived from the full set of participants, while those based on smaller sample sizes are increasingly different with smaller sample sizes. To the extent that this is true, it would have implications for determining the minimum number of users needed to obtain valid information.

Conclusions

A general conclusion that can be drawn on the basis of this research is that it may not be cost effective to spend resources to gather information from more than 20-30 participants in a card-sorting study. However, it is important to note that even the trees based on the smallest sample sizes are probably closer to the one for all 168 participants than might be obtained from speculation by a designer who is not a potential user of the content or application for which the organization is being developed. As always, we must exercise appropriate caution in generalizing results from one study. Results will obviously differ as a function of the homogeneity of the participants in a sample and such things as the instructions given to the participants for the card-sorting task.