User-Centered Data Visualization. Part 4 – The Experiment

Tobias Komischke / Wednesday, June 19, 2013

Visualizing data instead of presenting them as ASCII in lists or tables makes sense because we’re much better in processing graphical than numerical data (the so-called pictorial superiority effect). Also, graphical visualizations are considered to be more attractive. While most people agree on this, there is a war out there between folks saying that data visualizations have to become more attractive and creative because that’s what the market wants and folks that insist that any visualization that is not according to ergonomic standards is a bad visualization. So the question seems to be: are principles of good data visualization timeless or do they go with the zeitgeist?

Take pie charts, for example. They have been around for over 200 years (invented by William Playfair). They are everywhere, including industry (de-facto standard on business dashboards), popular press, and school books. Yet some of the most prominent data visualization experts like Edward Tufte or Stephen Few condemn them with a passion that’s borderline to hate. For them, pretty much any data should be depicted in a bar or column chart. Purists like William Cleveland from Purdue even say that bar or column charts are not optimal, either, because they show length/height AND width which makes them two-dimensional and therefore unnecessarily hard to interpret. For him, the best solution is showing values through their position along a common scale, e.g. as a scatter chart.

In my humble opinion, data visualization obeys to the same rules than any other UX design challenge: there are best practices and well-founded generic rules, but what’s best in any specific case depends on the concrete circumstance. Rules are great, but there are always enough degrees of freedom (aka design space) left to design based on specific knowledge you have about the context of use. That is to say, different audiences have different needs. For a general audience like readers of USA Today, why not use info graphics and artsy charts that look great but may not be very concise? On the other hand, scientific journals like Nature or Science feature data visualizations according to the text book standard – for the reason that the authors know what’s “right” and the readers are used to those visualizations.

We did a little experiment. We got hold of a data set featuring the frequency of the occurrence of a certain event in various countries around the world for one specific year. Sorry for being so vague about it, but it’s confidential data. The data was an output from a report and the target audience for it were mid-level and senior-level managers of our company. The data set was very interesting, because the data was distributed very unevenly over the world, with 57% of all cases mapping to just one country (the United States) and the remaining 43% mapping to 51 other countries. So visualizing that kind of data was challenging in itself. The data came with one pre-canned visualization in form of a pie chart with breakouts (see Alternative 2 below). We worked out almost 20 data visualization alternatives, using various tools such as Excel, Tableau, Fireworks, etc. We produced visualizations spanning the spectrum between very classic charts in various chart types and info graphics. In an internal meeting we discussed the alternatives and voted on 5 that we wanted to have rated together with the default pie chart by the actual target audience of the data: directors and executives of Infragistics. Here are the 6 contestants:

Alternative 1

Bubbles

 

Alternative 2

Pie

 

Alternative 3

Columns

 

Alternative 4

 map pie

 

Alternative 5

GLOBEDV

 

Alternative 6

 Swirly

 

We put the visualizations into an online survey and presented them in a randomized order. While the survey explained what data the visualizations were showing, no instruction as to how to read those visualizations was provided. For each visualization we asked the same set of questions using Likert scales (0 – very bad to 5 – very good):

  1. How meaningful do you find this data representation?
  2. How easy was it for you to understand this data representation?
  3. How do you like the visual appeal of this data representation?
  4. How do you rate this data representation overall?

We also provided the option to enter comments so that participants could share more details about their ratings.

In total, we received completed surveys from 12 participants. Here are the results (average rating per visualization alternative):

 

image

This question addresses the fit between the visualizations and the data. Or in other words: does the visualization provide a true and precise picture of the underlying data? Alternative #3 (column charts with breakouts) got the highest ratings from the participants of the survey. The pie chart (Alternative #2) which was the default visualization from the report, got the lowest score. By the way, we didn’t test for statistical significance since we only had 12 participants.

 

image

This question asks about how hard it was for the managers to derive meaning from the visualizations: is an alternative immediately clear or does it take a minute to comprehend what you’re looking at? Again, the column chart (Alternative #3) got higher ratings than the others and the pie chart (Alternative #2) got the lowest score.

 

image

With this question we tried to assess the attractiveness of the visualization. We didn’t provide a definition of what visual appeal means, we left that to the individual participant. Here, Alternative #4 (map view with donut chart) got the highest ratings with the column chart (Alternative #3) only ranking number 3. The pie chart (Alternative #2) got the lowest score again.

 

image

The intent of this question was to understand the participants’ holistic assessment of the alternatives, whatever their rating criteria were. The results  are in line with the ratings of the meaningfulness, ease of understanding and visual appeal: the column chart (Alternative #3) got the highest ratings while the pie chart (Alternative #2) got the lowest.

 

Conclusion:

The initial question was: Is it still contemporary to use “standard” (equates to “boring”) charts to visualize data or is there a need to focus more on a modern appeal that stands out? For several reasons the experiment cannot answer this question conclusively:

  • We only surveyed managers whose job it is to deal with numbers and derive action items. Had we surveyed another audience, say Graphic Designers in the Marketing department, the results would’ve been probably different. We should do this survey next! :-)
  • The visualization alternatives could obviously not exhaustively represent all variations of chart types, color treatment, etc. We cannot tell what the ratings had been if for example the column chart had featured coordinated colors while the world map visualization had been shown in a dirty green hue throughout.

Yet, what the results support is that the target user group matters and that there is no reason to fear that persons in the business world would reject a classic visualization just for its looks. The clear winner in our survey is the column chart visualization which is one of the oldest ways to depict data and at the same time is also one of the most established visualization types – for a good reason. We’re much better at estimating lengths and heights than diameters (bubble charts) or angles (pie charts).

Studies have shown that visual clutter yields low attractiveness ratings AND measurably limits the users’ ability to process the information. The pie chart alternative is a prime example for this. This is not to say that each and every pie chart would be bad, but it’s definitely not the right chart type for a data set as massive as this one we used.

The experiment shows that for this audience the meaningfulness and ease of understanding is more important than the appeal of a data visualization. Not that column or bar charts could never look as attractive as other visualization types, but there are just very limited ways you can make them look more like eye-candy, because whatever effects you add also adds to the visual complexity and clutter of the visualization. I’ve seen column charts that were flat and achromatic – and they looked awesome!