Hidden Bawls, Whispers, and Yelps: Can Text Convey the Sound of Speech, Beyond Words?

Whether a word was bawled, whispered, or yelped, captions will typically represent it in the same way. If they are your only way to access what is being said, subjective nuances expressed in the voice will be lost. Since so much of communication is carried by these nuances, we posit that if captions are to be used as an accurate representation of speech, embedding visual representations of paralinguistic qualities into captions could help readers use them to better understand speech beyond its mere textual content. This paper presents a model for processing vocal prosody (its loudness, pitch, and duration) and mapping it into visual dimensions of typography (respectively, font-weight, baseline shift, and letter-spacing), creating a visual representation of these lost vocal subtleties that can be embedded directly into the typographical form of text. An evaluation was carried out where participants were exposed to this speech-modulated typography and asked to match it to its originating audio, presented between similar alternatives. Participants (n=117) were able to correctly identify the original audios with an average accuracy of 65%, with no significant difference when showing them modulations as animated or static text. Additionally, participants’ comments showed their mental models of speech-modulated typography varied widely.

1 Introduction

Despite its expressive richness, when speech is represented through captions it is typically reduced to its words, and its words only. Whatever nuance was originally conveyed by the ways in which the speaker modulated their voice — their mood, emotions, dispositions, etc — is lost in this flattened textual representation. This is particularly relevant when captions are used not as a complement to a readily available audio channel, but as its replacement. This can be true for deaf and hard of hearing (DHH) persons, but will potentially affect anyone, including hearing individuals facing a situational hearing impairment, e.g., someone affected by a situational hearing impairment such as watching a film on their mobile phone in a noisy environment [1]. If so much of communication is expressed in nuances not captured by written text, what is to be said of the experience of those who have no direct access to acoustic speech but only to its written forms?