Top

Published in:

01-11-2011 | Research Article

The perception of visible speech: estimation of speech rate and detection of time reversals

Authors: Paolo Viviani, Francesca Figliozzi, Francesco Lacquaniti

Published in: Experimental Brain Research | Issue 2/2011

Abstract

Four experiments investigated the perception of visible speech. Experiment 1 addressed the perception of speech rate. Observers were shown video-clips of the lower face of actors speaking at their spontaneous rate. Then, they were shown muted versions of the video-clips, which were either accelerated or decelerated. The task (scaling) was to compare visually the speech rate of the stimulus to the spontaneous rate of the actor being shown. Rate estimates were accurate when the video-clips were shown in the normal direction (forward mode). In contrast, speech rate was underestimated when the video-clips were shown in reverse (backward mode). Experiments 2–4 (2AFC) investigated how accurately one discriminates forward and backward speech movements. Unlike in Experiment 1, observers were never exposed to the sound track of the video-clips. Performance was well above chance when playback mode was crossed with rate modulation, and the number of repetitions of the stimuli allowed some amount of speechreading to take place in forward mode (Experiment 2). In Experiment 3, speechreading was made much more difficult by using a different and larger set of muted video-clips. Yet, accuracy decreased only slightly with respect to Experiment 2. Thus, kinematic rather then speechreading cues are most important for discriminating movement direction. Performance worsened, but remained above chance level when the same stimuli of Experiment 3 were rotated upside down (Experiment 4). We argue that the results are in keeping with the hypothesis that visual perception taps into implicit motor competence. Thus, lawful instances of biological movements (forward stimuli) are processed differently from backward stimuli representing movements that the observer cannot perform.

Available only for authorised users

By convention, speed ratio (SR) denotes a property of the stimulus and speech rate what observers were asked to judge.

Because only one stimulus was presented on each trial, we did not divide d′ values by 2^1/2, as suggested for discrimination tasks in which two stimuli are presented (cf MacMillan and Creelman 2005).

Auer ET, Bernstein LE (1997) Speechreading and the structure of the lexicon: computationally modeling the effects of reduced phonetic distinctiveness on lexical uniqueness. J Acoust Soc Am 102:3704–3710PubMedCrossRef

Bassili JN (1978) Facial motion in the perception of faces and emotional expressions. J Exp Psychol Hum Percept Perform 4:373–379PubMedCrossRef

Bassili JN (1979) Emotion recognition: the role of facial movement and the relative importance of upper and lower areas of the face. J Pers Soc Psychol 37:2049–2058PubMedCrossRef

Bernstein N (1967) The coordination and regulation of movements. Pergamon Press, London

Bernstein LE, Demorest ME, Tucker PE (2000) Speech perception without hearing. Percept Psychophys 62:233–252PubMedCrossRef

Blake R, Shiffrar M (2007) Perception of human motion. Annu Rev Psychol 58:47–73PubMedCrossRef

Blakemore S-J, Frith C (2005) The role of contagion in the prediction of action. Neuropsychologia 43:260–267PubMedCrossRef

Breeuwer M, Plomp R (1986) Speechreading supplemented with auditorily presented speech parameters. J Acoust Soc Am 79:481–499PubMedCrossRef

Bruce V, Valentine T (1988) When a nod’s as good as a wink: the role of dynamic information in facial recognition. In: Gruneberg MM, Morris PE, Skyes RN (eds) Practical aspects of memory: current research and issues, vol 1. Wiley, Chichester, pp 169–174

Buffardi L (1971) Factors affecting the filled-duration illusion in the auditory, tactual, and visual modalities. Percept Psychophys 10:292–294CrossRef

Burnside W (1971) Judgment of short time intervals while performing mathematical tasks. Percept Psychophys 9:404–406CrossRef

Calvert GA, Campbell R (2003) Reading speech from still and moving faces: the neural substrates of visible speech. J Cogn Neurosci 15:57–70PubMedCrossRef

Calvo-Merino B, Ehrengerg S, Leung D, Haggard P (2010) Experts see it all: configural effects in action observation. Psychol Res 74:400–406PubMedCrossRef

Campbell R (2008) The processing of audio-visual speech: empirical and neural bases. Philos Trans R Soc B 363:1001–1010CrossRef

Campbell R, De Gelder B, De Haan E (1996) The lateralization of lip-reading: a second look. Neuropsychologia 34:1235–1240PubMedCrossRef

Campbell R, MacSweeney M, Surguladze S, Calvert G, McGuire P, Suckling J, Brammer MJ, David AS (2001) Cortical substrates for the perception of face actions: an fMRI study of the specificity of activation for seen speech and for meaningless lower-face acts (gurning). Cogn Brain Res 12:233–243CrossRef

Cantor NE, Thomas EC (1976) Visual masking effects on duration, size, and form discrimination. Percept Psychophys 19:321–327CrossRef

Caruso AJ, Abbs JH, Gracco VL (1988) Kinematic analysis of multiple movement coordination during speech in stutterers. Brain 111:439–455PubMedCrossRef

Chang DHF, Troje NF (2008) Perception of animacy and direction from local biological motion signals. J Vis 8:1–10PubMedCrossRef

Chang DHF, Troje NF (2009) Acceleration carries the local inversion effect in biological motion perception. J Vis 9:1–17

de’Sperati C, Viviani P (1997) The relationship between curvature and velocity in two-dimensional smooth pursuit eye movements. J Neurosci 15:3932–3945

Dekle DJ, Fowler CA, Funnell MG (1992) Audiovisual integration in perception of real words. Percept Psychophys 51:355–362PubMedCrossRef

Demorest ME, Bernstein LE (1992) Sources of variability in speechreading sentences: a generalizability analysis. J Speech Hear Res 35:876–891PubMed

Di Pellegrino G, Fadiga L, Fogassi L, Gallese V, Rizzolatti G (1992) Understanding motor events: a neurophysiological study. Exp Brain Res 91:176–180PubMed

Diamond R, Carey S (1986) Why faces are and are not special: an effect of expertise. J Exp Psychol Gen 115:107–117PubMedCrossRef

Eefting W, Rietveld A (1989) Just noticeable differences of articulation rate at the sentence level. Speech Commun 8:355–361CrossRef

Eklund R (2008) Pulmonic ingressive phonation: diachronic and synchronic characteristics, distribution and function in animal and human sound production and in human speech. J Int Phonetic Ass 38:235–324CrossRef

Elphick R (1996) Issues in comparing the speechreading abilities of hearing-impaired and hearing 15 to 16 year-old pupils. Br J Educ Psycol 66:357–365CrossRef

Fadiga L, Craighero L, Buccino G, Rizzolatti G (2002) Speech listening specifically modulates the excitability of tongue muscles: a TMS study. Eur J Neurosci 15:399–402PubMedCrossRef

Farah MJ, Tanaka JN, Drain M (1995) What causes the face inversion effect. J Exp Psychol Hum 21:628–634CrossRef

Feld JE, Sommers MS (2009) Lipreading, processing speed, and working memory in younger and older adults. J Speech Lang Hear R 52:1555–1565CrossRef

Fisher CG (1968) Confusion among visually perceived consonants. J Speech Hear Res 11:796–804PubMed

Freyd JJ (1987) Dynamic mental representations. Psychol Rev 94:427–438PubMedCrossRef

Freyd JJ, Finke RA (1984) Representational momentum. J Exp Psychol Learn 10:126–132CrossRef

Fridriksson J, Moss J, Davis B, Baylis GC, Bonilha L, Rorden C (2008) Motor speech perception modulates the cortical language areas. Neuroimage 41:605–613PubMedCrossRef

Galantucci B, Fowler CA, Turvey MT (2006) The motor theory of speech perception reviewed. Psychon B Rev 13:361–377CrossRef

Galantucci B, Fowler CA, Goldstein L (2009) Perceptuomotor compatibility effects in speech. Atten Percept Psycho 71:1138–1149CrossRef

Gibbon J, Malapani C, Dale CL, Gallistel CR (1997) Toward a neurobiology of temporal cognition: advances and challenges. Curr Opin Biol 2:170–184

Gracco VL (1988) Timing factors in the coordination of speech movements. J Neurosci 8:4628–4639PubMed

Gracco VL, Abbs JH (1986) Variant and invariant characteristics of speech movements. Exp Brain Res 65:156–166PubMedCrossRef

Gracco VL, Löfqvist A (1994) Speech motor coordination and control: evidence from lip, jaw, and laryngeal movements. J Neurosci 14:6585–6597PubMed

Grasso R, Bianchi L, Lacquaniti F (1998) Motor patterns for human gait: backward versus forward locomotion. J Neurophysiol 80:1868–1885PubMed

Jacobs A, Pinto J, Shiffrar M (2004) Experience, context, and the visual perception of human movement. J Exp Psychol Hum 30:822–835CrossRef

Johansson G (1973) Visual perception of biological motion and a model for its analysis. Percep Psychophys 14:201–211CrossRef

Kelso JAS, Saltzman EL, Tuller B (1986) The dynamical perspective on speech production. J Phonetics 14:29–59

Kerzel D, Bekkering H (2000) Motor activation from visible speech: evidence from stimulus response compatibility. J Exp Psychol Hum 26:634–647CrossRef

Keysers C, Kohler E, Umiltà MA, Nanetti L, Fogassi L, Gallese V (2003) Audiovisual mirror neurons and action recognition. Exp Brain Res 153:628–636PubMedCrossRef

Kohler W (1940) Dynamics in psychology. Liveright, New York

Kohler E, Keysers C, Umiltà MA, Fogassi L, Gallese V, Rizzolatti G (2002) Hearing sounds, understanding actions: action representation in mirror neurons. Science 297:846–848PubMedCrossRef

Kollia HB, Gracco VL, Harris KS (1995) Articulatory organization of mandibular, labial, and velar movements during speech. J Acoustic Soc Am 98:1313–1324CrossRef

Krakow RA (1999) Physiological organization of syllables: a review. J Phonetics 27:23–54CrossRef

Leder H, Bruce V (2000) When inverted faces are recognized: the role of configural information in face recognition. Q J Exp Psychol 53:513–536CrossRef

Lehiste I (1970) Suprasegmentals. MIT Press, Cambridge

Liberman AM (1996) Speech: a special code. MIT Press, Cambridge

Liberman AM, Mattingly IG (1985) The motor theory of speech perception revised. Cognition 21:1–36PubMedCrossRef

Liberman AM, Whalen DH (2000) On the relation of speech to language. Trends Cogn Sci 4:187–196PubMedCrossRef

Liberman AM, Cooper FS, Shankweiler DP, Studdert-Kennedy M (1967) Perception of speech code. Psychol Rev 74:431–461PubMedCrossRef

Lyxell B, Rönnberg J (1992) The relation between verbal ability and sentence-based speechreading. Scand Audiol 21:67–72PubMedCrossRef

Mach E (1885) Beiträge zur Analyse der Empfindungen [english translation: Contributions to the analysis of sensations. Open Court (1897), La Salle, IL]

MacMillan NA, Creelman CD (2005) Detection theory: a user’s guide. Lawrence Erlbaum Associates, 2nd edn. New York, London

Massaro DW (1998) Perceiving talking faces: from speech perception to a behavioral principle. MIT Press, Cambridge

McGurk H, MacDonald J (1976) Hearing lips and seeing voices. Nature 264:746–748PubMedCrossRef

Michon JA (1965) Studies in subjective duration. II. Subjective time measurements during task with different information content. Acta Psychol 24:205–219CrossRef

Middleweerd MJ, Plomp R (1987) The effect of speechreading on the speech-reception threshold of sentences in noise. J Acoust Soc Am 82:2145–2147CrossRef

Mitkin AA, Pavlova MA (1990) Changing a natural orientation: recognition of biological motion patterns by children and adults. Psychol Beitr 32:28–35

Möttönen R, Watkins KE (2009) Motor representations of articulators contribute to categorical perception of speech sounds. J Neurosci 29:9819–9825PubMedCrossRef

Mowbray GH, Gebhard JW (1955) Differential sensitivity of the eye to intermittent white light. Science 121:173–175PubMedCrossRef

Munhall KG, Ostry DJ, Parush A (1985) Characteristics of velocity profiles of speech movements. J Exp Psychol Hum 11:457–474CrossRef

Nishitani N, Hari R (2002) Viewing lip forms: cortical dynamics. Neuron 36:1211–1220PubMedCrossRef

Nishitani N, Schürmann N, Amunts K, Hari R (2005) Broca’s region: from action to language. Physiology 20:60–69PubMedCrossRef

Noteboom S, Eefting W (1994) Evidence of adaptive nature of speech on the phrase level and below. Phonetica 51:92–98CrossRef

Ojanen V, Möttönen R, Pekkola J, Jääskeläinen IP, Joensuu R, Autti T, Sams M (2005) Processing of audiovisual speech in Broca’s area. Neuroimage 25:333–338PubMedCrossRef

Ornstein R (1969) On the experience of time. Penguin Books, Baltimore

Ostry DJ, Munhall KG (1985) Control of rate and duration of speech movements. J Acoust Soc Am 77:640–648PubMedCrossRef

Paulesu E, Perani D, Blasi V, Silani G, Borghese NA, De Giovanni U, Sensolo S, Fazio F (2003) A functional-anatomical model for lip-reading. J Neurophysiol 90:2005–2013PubMedCrossRef

Pavlova M, Sokolov A (2000) Orientation specificity in biological motion perception. Percept Psychophys 62:889–899PubMedCrossRef

Pekkola J, Ojanen V, Autti T, Jääskeläinen IP, Möttönen R, Tarkiainen A, Sams M (2005) Primary auditory cortex activation by visual speech: an fMRI study at 3T. Neuroreport 16:125–128PubMedCrossRef

Pfitzinger HR, Tamashima M (2006) Comparing perceptual local speech rate of German and Japanese speech. In: Proceedings of the 3rd international conference on speech prosody, pp 105–108

Pulvermüller F, Huss M, Kherif F, Moscoso del Prado Martin F, Hauk O, Shtyrov Y (2006) Motor cortex maps articulatory features of speech sounds. Proc Natl Acad Sci USA 103:7865–7870PubMedCrossRef

Quené H (2007) On the just noticeable difference for tempo in speech. J Phonetics 35:353–362CrossRef

Rizzolatti G, Arbib MA (1998) Language within our grasp. Trends Neurosci 21:188–194PubMedCrossRef

Rizzolatti G, Craighero L (2004) The mirror-neuron system. Annu Rev Neurosci 27:169–192PubMedCrossRef

Rönnberg J (1995) What makes a skilled speechreader? In: Plant G, Spens K (eds) Profound deafness and speech communication. Whurr Publications, London, pp 393–416

Rönnberg J, Samuelsson S, Lyxell B (1998) Conceptual constraints in sentence-based lipreading in hearing-impaired. In: Campbell R, Dodds D, Burnham DK (eds) Hearing by eye II: advances in the psychology of speechreading and auditory-visual speech. Psychology Press, Hove, pp 143–153

Rosenblum LD, Saldaña HM (1996) An audiovisual test of kinematic primitives for visual speech perception. J Exp Psychol Hum 22:318–331CrossRef

Ruytjens L, Albers F, van Dijk P, Wit H, Willemsen A (2006) Neural responses to silent lip-reading in normal hearing male and female subjects. Eur J Neurosci 24:1835–1844PubMedCrossRef

Sams M, Aulanko R, Hämäläinen M, Hari R, Lounasmaa OV, Lu S-T, Simola J (1991) Seeing speech: visual information from lip movement modifies activity in the human auditory cortex. Neurosci Lett 127:141–145PubMedCrossRef

Santi A, Servos P, Vatikiotis-Bateson E, Kuratate T, Munhall K (2003) Perceiving biological motion: dissociating visible speech from walking. J Cogn Neurosci 15:800–809PubMedCrossRef

Shaiman S, Adams SG, Kimelman MDZ (1995) Timing relationships of the upper lip and jaw across changes in speaking rate. J Phonetics 23:119–128CrossRef

Shepard RN (1984) Ecological constraints on internal representation: resonant kinematics of perceiving, imagining, thinking, and dreaming. Psychol Rev 91:417–447PubMedCrossRef

Shepard RN (1994) Perceptual-cognitive universals as reflections of the world. Psychon B Rev 1:2–28CrossRef

Simion F, Regolin L, Bulf H (2008) A predisposition for biological motion in the newborn baby. Proc Natl Acad Sci USA 105:809–813PubMedCrossRef

Smith A, Goffman L (1998) Stability and patterning of speech movement sequences in children and adults. J Speech Lang Hear R 41:18–30

Stevens JC, Shickman GM (1959) The perception of repetition rate. J Exp Psychol 58:433–440PubMedCrossRef

Sumby WH, Pollack I (1954) Visual contribution to speech intelligibility in noise. J Acoust Soc Am 26:212–215CrossRef

Sumi S (1984) Upside-down presentation of the Johansson moving light-spot pattern. Perception 13:283–286PubMedCrossRef

Summerfield Q (1987) Some preliminaries to a comprehensive account of audio-visual speech perception. In: Dodd D, Campbell R (eds) Hearing by eye: the psychology of lip-reading. Lawrence Erlbaum Associates, London, pp 3–51

Summerfield Q (1991) Visual perception of phonetic gestures. In: Mattingly IG, Studdert-Kennedy M (eds) Modularity and the motor theory of speech perception. Erlbaum, Hillsdale, pp 117–137

Summerfield Q (1992) Lipreading and audio-visual speech perception. Philos Trans R Soc Lon B 335:71–78CrossRef

Sussman H (1989) Neural coding of relational invariance in speech: human language analogue to the barn owl. Psychol Rev 96:631–642PubMedCrossRef

Tasko SM, McClean MD (2004) Variations in articulatory movements with changes in speech task. J Speech Lang Hear R 47:85–100CrossRef

Thomas EC, Brown I Jr (1974) Time perception and the filled-duration illusion. Percept Psychophys 16:449–458CrossRef

Thompson P (1980) Margaret Thatcher: a new illusion. Perception 9:483–484PubMedCrossRef

Thorstensson A (1986) How is the normal locomotor program modified to produce backward walking? Exp Brain Res 61:644–648CrossRef

Troje NF, Weshoff C (2006) The inversion effect in biological motion perception: evidence for a “life detector”? Curr Biol 16:821–824PubMedCrossRef

Tuller B, Kelso JAS, Harris KS (1982) Interarticular phasing as an index of temporal regularity in speech. J Exp Psychol Hum 8:460–472CrossRef

Tuller B, Kelso JAS, Harris KS (1983) Converging evidence for the role of relative timing in speech. J Exp Psychol Hum 9:829–833CrossRef

Turner TH, Fridriksson J, Baker J, Eoute D Jr, Bonilha L, Rorden C (2009) Obligatory Broca’s area modulation associated with passive speech perception. Neuroreport 20:492–496PubMedCrossRef

Turvey MT (1977) Preliminaries to a theory of action with reference to vision. In: Shaw R, Bransford J (eds) Perceiving, acting, and knowing: towards an ecological psychology. Erlbaum, Hillsdale, pp 211–263

Valentine T (1988) Upside-down faces: a review of the effect of inversion upon face recognition. Br J Psychol 79:471–491PubMedCrossRef

Valentine T, Bruce V (1985) What’s up? The Margaret Thatcher illusion revisited. Perception 14:515–516PubMedCrossRef

Viviani P (2002) Motor competence in the perception of dynamic events: a tutorial. In: Prinz W, Hommel B (eds) Attention & performance XIX: common mechanisms in perception and action. Oxford University Press, Oxford, pp 406–442

Viviani P, Stucchi N (1989) The effect of movement velocity on form perception: geometric illusions in dynamic displays. Percept Psychophys 46:266–274PubMedCrossRef

Viviani P, Stucchi N (1992) Biological movements look constant: evidence of motor-perceptual interactions. J Exp Psychol Hum 18:603–623CrossRef

Viviani P, Baud-Bovy G, Redolfi M (1997) Perceiving and tracking kinesthetic stimuli: further evidence of motor-perceptual interactions. J Exp Psychol Hum 23:1232–1252CrossRef

Viviani P, Figliozzi F, Campione GC, Lacquaniti F (2011) Detecting temporal reversals in human locomotion. Exp Brain Res 214:93–103

Watkins KE, Strafella AP, Paus T (2003) Seeing and hearing speech excites the motor system involved in speech production. Neuropsychologia 41:989–994PubMedCrossRef

Welch RB, DuttonHurt LD, Warren DH (1986) Contribution of audition and vision to temporal rate perception. Percept Psychophys 39:294–300PubMedCrossRef

Wilson SM, Saygin AP, Sereno MI, Iacoboni M (2004) Listening to speech activates motor areas involved in speech production. Nat Neurosci 7:701–702PubMedCrossRef

Yin RK (1969) Looking at upside down faces. J Exp Psychol 81:141–145CrossRef

Title: The perception of visible speech: estimation of speech rate and detection of time reversals
Authors: Paolo Viviani
Francesca Figliozzi
Francesco Lacquaniti
Publication date: 01-11-2011
Publisher: Springer-Verlag
Published in: Experimental Brain Research / Issue 2/2011
Print ISSN: 0014-4819
Electronic ISSN: 1432-1106
DOI: https://doi.org/10.1007/s00221-011-2883-9

Keynote webinar | Spotlight on medication adherence

Springer Medicine

The perception of visible speech: estimation of speech rate and detection of time reversals

Abstract

Keynote webinar | Spotlight on medication adherence

Springer Medicine

Abstract

Please log in to get access to this content

Other articles of this Issue 2/2011

Effects of experimentally induced low back pain on the sit-to-stand movement and electroencephalographic contingent negative variation

The stability of rhythmic movement coordination depends on relative speed: the Bingham model supported

Task-related variations in the surface EMG of the human first dorsal interosseous muscle

Excitability changes induced in the human auditory cortex by transcranial direct current stimulation: direct electrophysiological evidence

Self-touch affects motor imagery: a study on posture interference effect

Early-life environmental intervention may increase the number of neurons, astrocytes, and cellular proliferation in the hippocampus of rats