Skip to main content
Key Specialties
Cardiology Diabetology Endocrinology Gastroenterology Geriatrics and Gerontology Gynecology and Obstetrics Hematology Infectious Disease Internal Medicine Nephrology Neurology Oncology Pediatrics Respiratory Medicine Rheumatology Surgery
Congress Coverage
2024 ACC Congress 2023 ESMO Congress 2023 EASD Congress 2023 ERS Congress 2023 ESC Congress
Clinical Collections
Advances in Alzheimer’s

At a glance: The STEP trials

A round-up of the STEP phase 3 clinical trials evaluating semaglutide for weight loss in people with overweight or obesity.
ADVANCED SEARCH
Log in
Top
Brain Topography
Published in: Brain Topography 3/2015

01-05-2015 | Original Paper

Learning to Associate Auditory and Visual Stimuli: Behavioral and Neural Mechanisms

Authors: Nicholas Altieri, Ryan A. Stevenson, Mark T. Wallace, Michael J. Wenger

Published in: Brain Topography | Issue 3/2015

Login to get access

Abstract

The ability to effectively combine sensory inputs across modalities is vital for acquiring a unified percept of events. For example, watching a hammer hit a nail while simultaneously identifying the sound as originating from the event requires the ability to identify spatio-temporal congruencies and statistical regularities. In this study, we applied a reaction time and hazard function measure known as capacity (e.g., Townsend and AshbyCognitive Theory 200–239, 1978) to quantify the extent to which observers learn paired associations between simple auditory and visual patterns in a model theoretic manner. As expected, results showed that learning was associated with an increase in accuracy, but more significantly, an increase in capacity. The aim of this study was to associate capacity measures of multisensory learning, with neural based measures, namely mean global field power (GFP). We observed a co-variation between an increase in capacity, and a decrease in GFP amplitude as learning occurred. This suggests that capacity constitutes a reliable behavioral index of efficient energy expenditure in the neural domain.
Footnotes
1
It is worth noting that these principles are not necessarily associated with stimulus features per se, but rather neural features including receptive field properties, effectiveness in eliciting action potentials, etc.
 
2
Contingencies, or probabilistic stimulus configurations that could facilitate/inhibit audiovisual target (i.e., matched) trial responses (Mordkoff and Yantis 1991), were reduced by appropriately balancing redundant, single, and target-absent trials. Although this yielded a difference in the number of matched and mismatched audiovisual trials, the proportion remained constant across training days. Hence, holding the proportion of audiovisual matched versus mismatched trials constant across days provides a valid way to examine the effects of training on changes in capacity and neural signals.
 
Literature
Allison PD (1996) Fixed-effects partial likelihood for repeated events. Sociol Methods Res 25:2207–2222CrossRef
Altieri N, Townsend JT (2011) An assessment of behavioral dynamic information processing measures in audiovisual speech perception. Front Psychol 2(238):1–15
Atkinson RC, Shiffrin RM (1968) Human memory: a proposed system and its control processes. In: Spence KW, Spence JT (eds) The psychology of learning and motivation, vol 2. Academic Press, New York, pp 89–195
Beauchamp MS, Lee KE, Argall BD, Martin A (2004) Integration of auditory and visual information about objects in superior temporal sulcus. Neuron 41(5):809–823CrossRefPubMed
Belardinelli MO, Sestieri C, Di Matteo R, Delogu F, Del Gratta C, Ferretti A et al (2004) Audio-visual crossmodal interactions in environmental perception: an fMRI investigation. Cogn Process 5(3):167–174CrossRef
Calvert GA, Campbell R, Brammer MJ (2000) Evidence from functional magnetic resonance imaging of crossmodal binding in the human heteromodal cortex. Curr Biol 10(11):649–657CrossRefPubMed
Cappe C, Thut G, Romei V, Murray MM (2010) Auditory-visual multisensory interactions in humans: timing, topography, directionality, and sources. J Neurosci 30(38):12572–12580CrossRefPubMed
Colonius H, Diederich A (2004) Multisensory interaction in saccadic reaction time: a time-window-of-integration model. J Cogn Neurosci 16(6):1000–1009CrossRefPubMed
Colonius H, Diederich A (2010) The optimal time window of visual-auditory integration: a reaction time analysis. Front Integr Neurosci 4:11PubMedCentralPubMed
Conrey B, Pisoni DB (2006) Auditory-visual speech perception and synchrony detection for speech and nonspeech signals. J Acoust Soc Am 119(6):4065–4073CrossRefPubMedCentralPubMed
Cowan N, Elliott EM, Saults JS, Morey CC, Mattox S, Hismjatullina A, Conway ARA (2005) On the capacity of attention: its estimation and its role in working memory and cognitive aptitudes. Cogn Psychol 51:42–100CrossRefPubMedCentralPubMed
Cox DR (1972) Regresion models and life tables (with discussion). J Roy Stat Soc B 34:187–220
Diederich A, Colonius H (2004) Bimodal and trimodal multisensory enhancement: effects of stimulus onset and intensity on reaction time. Percept Psychophys 66(8):1388–1404CrossRefPubMed
Diederich A, Colonius H (2009) Crossmodal interaction in speeded responses: time window of integration model. Prog Brain Res 174:119–135CrossRefPubMed
Doehrmann O, Naumer MJ (2008) Semantics and the multisensory brain: how meaning modulates processes of audio-visual integration. Brain Res 1242:136–150CrossRefPubMed
Erber NP (2003) Use of hearing aids by older people: influence of non-auditory factors (vision, manual dexterity). Int J Audiol 42:2S21–2S25CrossRefPubMed
Fiebelkorn IC, Foxe JJ, Schwartz TH, Molholm S (2010) Staying within the lines: the formation of visuospatial boundaries influences multisensory feature integration. Eur J Neurosci 31(10):1737–1743CrossRefPubMed
Frens MA, Van Opstal AJ, Van der Willigen RF (1995) Spatial and temporal factors determine auditory-visual interactions in human saccadic eye movements. Percept Psychophys 57(6):802–816CrossRefPubMed
Hecht D, Reiner M, Karni A (2008) Multisensory enhancement: gains in choice and in simple response times. Exp Brain Res 189(2):133–143CrossRefPubMed
Hein G, Doehrmann O, Muller NG, Kaiser J, Muckli L, Naumer MJ (2007) Object familiarity and semantic congruency modulate responses in cortical audiovisual integration areas. J Neurosci 27(30):7881–7887CrossRefPubMed
Hershenson M (1962) Reaction time as a measure of intersensory facilitation. J Exp Psychol 63:289–293CrossRefPubMed
Hillock AR, Powers AR, Wallace MT (2011) Binding of sights and sounds: age-related changes in multisensory temporal processing. Neuropsychologia 49(3):461–467CrossRefPubMedCentralPubMed
James W (1890) The principles of psychology. Harvard University Press, BostonCrossRef
James TW, Stevenson RA (2012) The use of fMRI to assess multisensory integration. In: Wallace MH, Murray MM (eds) Frontiers in the neural basis of multisensory processes. Taylor & Francis, London
James TW, VanDerKlok RM, Stevenson RA, James KH (2011) Multisensory perception of action in posterior temporal and parietal cortices. Neuropsychologia 49(1):108–114CrossRefPubMedCentralPubMed
James TW, Stevenson RA, Kim S (2012) Inverse effectiveness in multisensory processing. In: Stein BE (ed) The new handbook of multisensory processes. MIT Press, Cambridge
Johnson SA, Blaha LM, Houpt JW, Townsend JT (2010) Systems Factorial Technology provides new insights on global-local information processing in autism spectrum disorders. J Math Psychol 54:53–72CrossRefPubMedCentralPubMed
Keetels M, Vroomen J (2005) The role of spatial disparity and hemifields in audio-visual temporal order judgments. Exp Brain Res 167(4):635–640CrossRefPubMed
Laurienti PJ, Wallace MT, Maldjian JA, Susi CM, Stein BE, Burdette JH (2003) Cross-modal sensory processing in the anterior cingulate and medial prefrontal cortices. Hum Brain Mapp 19(4):213–223CrossRefPubMed
Lovelace CT, Stein BE, Wallace MT (2003) An irrelevant light enhances auditory detection in humans: a psychophysical analysis of multisensory integration in stimulus detection. Brain Res Cogn Brain Res 17(2):447–453CrossRefPubMed
Luce RD (1986) Response times: their role in inferring elementary mental organization. Oxford University Press, New York
Macaluso E, George N, Dolan R, Spence C, Driver J (2004) Spatial and temporal factors during processing of audiovisual speech: a PET study. Neuroimage 21(2):725–732CrossRefPubMed
McKee SP, Westheimer G (1978) Improvement in vernier acuity with practice. Percept Psychophys 24:258–262CrossRefPubMed
Meredith MA, Stein BE (1986a) Spatial factors determine the activity of multisensory neurons in cat superior colliculus. Brain Res 365(2):350–354CrossRefPubMed
Meredith MA, Stein BE (1986b) Visual, auditory, and somatosensory convergence on cells in superior colliculus results in multisensory integration. J Neurophysiol 56(3):640–662PubMed
Meredith MA, Stein BE (1996) Spatial determinants of multisensory integration in cat superior colliculus neurons. J Neurophysiol 75(5):1843–1857PubMed
Meredith MA, Nemitz JW, Stein BE (1987) Determinants of multisensory integration in superior colliculus neurons I. Temporal factors. J Neurosci 7(10):3215–3229PubMed
Miller LM, D’Esposito M (2005) Perceptual fusion and stimulus coincidence in the cross-modal integration of speech. J Neurosci 25(25):5884–5893CrossRefPubMed
Molholm S, Ritter W, Murray M, Javitt DC, Schroeder CE, Foxe JJ (2002) Multisensory auditory-visual interactions during early sensory processing in humans: a high-density electrical mapping study. Cogn Brain Res 14:115–128CrossRef
Mordkoff JT, Yantis S (1991) An interactive race model of divided attention. J Exp Psychol Hum Percept Perform 17:520–538CrossRefPubMed
Munhall K, Vatikiotis-Bateson E (2004) Spatial and Temporal Constraints on Audiovisual Speech Perception. In: Calvert G, Spence C, Stein BE (eds) The handbook of multisensory processes. MIT Press, Cambridge, pp 177–188
Murray MM, Michel CM, Grave de Peralta R, Ortigue S, Brunet D, Andino SG, Schnider A (2004) Rapid discrimination of visual and multisensory memories revealed by electrical neuroimaging. Neuroimage 21:125–135CrossRefPubMed
Naue N, Rach S, Strüber D, Huster RJ, Zaehle T, Körner U, Herrmann CS (2011) Auditory event-related responses in visual cortex modulates subsequent visual responses in humans. J Neurosci 31(21):7729–7736CrossRefPubMed
Nelson WT, Hettinger LJ, Cunningham JA, Brickman BJ, Haas MW, McKinley, RL (1998) Effects of localized auditory information on visual target detection performance using helmet-mounted display. Hum Factors 40(3):452–460CrossRefPubMed
Neufeld RWJ, Townsend JT, Jetté J (2007) Quantitative response time technology for measuring cognitive-processing capacity in clinical studies. In: Neufeld RWJ (ed) Advances in clinical cognitive science: formal modeling and assessment of processes and symptoms. American Psychological Association, Washington DCCrossRef
Noppeney U, Josephs O, Hocking J, Price CJ, Friston KJ (2008) The effect of prior visual information on recognition of speech and sounds. Cereb Cortex 18(3):598–609CrossRefPubMed
Pilling M (2009) Auditory event-related potentials (ERPs) in audiovisual speech perception. J Speech Lang Hearing Res 52:1073–1081CrossRef
Saffran JR, Johnson EK, Aslin RN, Newport EL (1999) Statistical learning of tone sequences by human infants and adults. Cognition 70:27–52CrossRefPubMed
Seitz A, Kim, Van Wassenhove V, Shams M (2007) Simultaneous and Independent Acquisition of Multisensory and Unisensory Associations. Perception 36:1445–1453CrossRefPubMed
Skrandies W (1990) Global field power and topographic similarity. Brain Topogr 3:147–541CrossRef
Stein BE, Stanford TR, Ramachandran R, Perrault TJ Jr, Rowland BA (2009) Challenges in quantifying multisensory integration: alternative criteria, models, and inverse effectiveness. Exp Brain Res 198(2–3):113–126
Stevenson RA, James TW (2009) Audiovisual integration in human superior temporal sulcus: inverse effectiveness and the neural processing of speech and object recognition. Neuroimage 44(3):1210–1223CrossRefPubMed
Stevenson RA, Geoghegan ML, James TW (2007) Superadditive BOLD activation in superior temporal sulcus with threshold non-speech objects. Exp Brain Res 179(1):85–95CrossRefPubMed
Stevenson RA, Kim S, James TW (2009) An additive-factors design to disambiguate neuronal and areal convergence: measuring multisensory interactions between audio, visual, and haptic sensory streams using fMRI. Exp Brain Res 198(2–3):183–194CrossRefPubMed
Stevenson RA, Altieri NA, Kim S, Pisoni DB, James TW (2010) Neural processing of asynchronous audiovisual speech perception. Neuroimage 49(4):3308–3318CrossRefPubMedCentralPubMed
Stevenson RA, VanDerKlok RM, Pisoni DB, James TW (2011) Discrete neural substrates underlie complementary audiovisual speech integration processes. Neuroimage 55(3):1339–1345CrossRefPubMedCentralPubMed
Stevenson RA, Krueger Fister J, Barnett ZP, Nidiffer AR, Wallace MT (2012) Interactions between the spatial and temporal stimulus factors that influence multisensory integration in human performance. Exp Brain Res 219:121–137CrossRefPubMedCentralPubMed
Talsma D, Senkowski D, Woldorff MG (2009) Intermodal attention affects the processing of the temporal alignment of audiovisual stimuli. Exp Brain Res 198(2–3):313–328CrossRefPubMedCentralPubMed
Tanabe S, Doi T, Umeda K, Fujita I (2005) Disparity-tuning characteristics of neuronal responses to dynamic random-dot stereograms in macaque visual area V4. J Neurophysiol 94(4):2683–2699CrossRefPubMed
Taylor KI, Moss HE, Stamatakis EA, Tyler LK (2006) Binding crossmodal object features in perirhinal cortex. Proc Natl Acad Sci U S A 103(21):8239–8244CrossRefPubMedCentralPubMed
Thelen A, Cappe C, Murray MM (2012) Electrical neuroimaging of memory discrimination based on single-trial multisensory learning. Neuroimage 62(3):1478–1488CrossRefPubMed
Townsend JT (1990) The truth and consequences of ordinal differences in statistical distributions: toward a theory of hierarchical inference. Psychol Bull 108:551–567CrossRefPubMed
Townsend JT, Altieri NA (2012) An accuracy-response time capacity assessment function that measures performance against standard parallel predictions. Psychol Rev. doi:10.​1037/​a0028448 PubMed
Townsend JT, Ashby FG (1978) Methods of modeling capacity in simple processing systems. In: Castellan J, Restle F (eds) Cognitive theory, vol III. Erlbaum Associates, Hillsdale, pp 200–239
Townsend JT, Nozawa G (1995) Spatio-temporal properties of elementary perception: an investigation of parallel, serial and coactive theories. J Math Psychol 39:321–360CrossRef
Townsend JT, Wenger MJ (2004) A theory of interactive parallel processing: new capacity measures and predictions for a response time inequality series. Psychol Rev 111(4):1003–1035CrossRefPubMed
van Atteveldt NM, Formisano E, Blomert L, Goebel R (2007) The effect of temporal asynchrony on the multisensory integration of letters and speech sounds. Cereb Cortex 17(4):962–974CrossRefPubMed
van Atteveldt N, Formisano E, Goebel R, Blomert, L (2004) Integration of letters and speech sounds in the human brain. Neuron 43:271–282CrossRefPubMed
van Wassenhove V, Grant KW, Poeppel D (2007) Temporal window of integration in auditory-visual speech perception. Neuropsychologia 45(3):598–607CrossRefPubMed
Vroomen J, Keetels M, de Gelder B, Bertelson P (2004) Recalibration of temporal order perception by exposure to audio-visual asynchrony. Cogn Brain Res 22:32–35CrossRef
Wallace MT, Wilkinson LK, Stein BE (1996) Representation and integration of multiple sensory inputs in primate superior colliculus. J Neurophysiol 76(2):1246–1266PubMed
Wallace MT, Roberson GE, Hairston WD, Stein BE, Vaughan JW, Schirillo JA (2004) Unifying multisensory signals across time and space. Exp Brain Res 158(2):252–258CrossRefPubMed
Wallace MT, Carriere BN, Perrault TJ Jr, Vaughan JW, Stein BE (2006) The development of cortical multisensory integration. J Neurosci 26(46):11844–11849CrossRefPubMed
Wenger MJ, Gibson BS (2004) Using hazard functions to assess changes in processing capacity in an attentional cuing paradigm. J Exp Psychol Hum Percept Perform 30:708–719CrossRefPubMed
Wenger MJ, Negash S, Petersen RC, Petersen L (2010) Modeling and estimating recall processing capacity: sensitivity and diagnostic utility in application to mild cognitive impairment. J Math Psychol 54:73–89CrossRefPubMedCentralPubMed
Zampini M, Guest S, Shore DI, Spence C (2005) Audio-visual simultaneity judgments. Percept Psychophys 67(3):531–544CrossRefPubMed
Metadata
Title
Learning to Associate Auditory and Visual Stimuli: Behavioral and Neural Mechanisms
Authors
Nicholas Altieri
Ryan A. Stevenson
Mark T. Wallace
Michael J. Wenger
Publication date
01-05-2015
Publisher
Springer US
Published in
Brain Topography / Issue 3/2015
Print ISSN: 0896-0267
Electronic ISSN: 1573-6792
DOI
https://doi.org/10.1007/s10548-013-0333-7

Other articles of this Issue 3/2015

Brain Topography 3/2015 Go to the issue

Original Paper

Learning-Stage-Dependent Plasticity of Temporal Coherence in the Auditory Cortex of Rats

Original Paper

Brain Dynamics of Distractibility: Interaction Between Top-Down and Bottom-Up Mechanisms of Auditory Attention

Original Paper

Temporal Evolution of Neural Activity Underlying Auditory Discrimination of Frequency Increase and Decrease

Original Paper

Brain Prediction of Auditory Emphasis by Facial Expressions During Audiovisual Continuous Speech

Original Paper

Analyzing the Auditory Scene: Neurophysiologic Evidence of a Dissociation Between Detection of Regularity and Detection of Change

Original Paper

Hierarchical Neural Encoding of Temporal Regularity in the Human Auditory Cortex