Multisensory perception: beyond the visual in visualization

Mapping information onto more than one sensory modality might let us increase human bandwidth for understanding complex, multivariate data. Researchers have done significant work to explore the effective use of a single non-visual sense for data display. Unfortunately, few researchers have examined the question of what data is best expressed in what way. Lacking a theory of multisensory perception and processing of information, the critical issue is determining what data "best" maps onto what sensory input channel. Consider the problem of hydrocarbon reservoirs. Most reservoir engineers would agree that the number of variables required to characterize a reservoir is large (perhaps 30 to 50). If you depend only on visual displays limited to seven variables at a time, then it might require as many as seven such displays to cover the full range of necessary variables. You then must either mentally integrate across those seven displays or go through a process of variable selection and redisplay to achieve a specific goal.

The Challenge

Visualization should communicate data to users as information, but even the cleverest use of the visual sense-exploiting all three dimensions and using color-has limits. In 1956, George A. Miller published a fascinating paper, “The Magical Number Seven, Plus or Minus Two: Some Limits on Our Capacity for Processing Information.“^[4] One of his conclusions was that”… the span of absolute judgment and the span of immediate memory impose severe limitations on the amount of information that we are able to receive, process, and remember.” In his review of the literature of the day and in his thinking, Miller was concerned about a human employing one sensory channel at a time. If we accept his conclusion that a human can usefully perceive about seven different variables presented in a single sensory modality, then using only visual displays of information automatically limits human information perception “bandwidth.”