Prosodic influence in face emotion perception: evidence from functional near-infrared spectroscopy


  • 1.

    Schirmer, A. & Adolphs, R. Emotion perception from face, voice, and touch: comparisons and convergence. Trends Cognit. Sci. 21, 216–228. https://doi.org/10.1016/j.tics.2017.01.001 (2017).

    Article 

    Google Scholar
     

  • 2.

    Ekman, P. & Friesen, W. V. Pictures of Facial Affect (Consulting Psychological Press, Palo Alto, CA, 1976).


    Google Scholar
     

  • 3.

    Bartlett, M. S. et al. Classifying facial action. IEEE Trans. Pattern Anal. Mach. Intell. 21, 974–989 (1996).


    Google Scholar
     

  • 4.

    Ekman, P. An argument for basic emotions. Cognit. Emot. 6, 169–200. https://doi.org/10.1080/02699939208411068 (1992).

    Article 

    Google Scholar
     

  • 5.

    Schröder, M. Experimental study of affect bursts. Speech Commun. 40, 99–116. https://doi.org/10.1016/S0167-6393(02)00078-X (2003).

    Article 
    MATH 

    Google Scholar
     

  • 6.

    Grandjean, D. et al. The voices of wrath: brain responses to angry prosody in meaningless speech. Nat. Neurosci. 8, 145–146. https://doi.org/10.1038/nn1392 (2005).

    CAS 
    Article 
    PubMed 

    Google Scholar
     

  • 7.

    Belin, P., Fillion-Bilodeau, S. & Gosselin, F. The montreal affective voices: a validated set of nonverbal affect bursts for research on auditory affective processing. Behav. Res. Methods 40, 531–539. https://doi.org/10.3758/BRM.40.2.53 (2008).

    Article 
    PubMed 

    Google Scholar
     

  • 8.

    Patel, S., Scherer, K. R., Björkner, E. & Sundberg, J. Mapping emotions into acoustic space: the role of voice production. Biol. Psychol. 87, 93–98. https://doi.org/10.1016/j.biopsycho.2011.02.010 (2011).

    Article 
    PubMed 

    Google Scholar
     

  • 9.

    Adolphs, R. Recognizing emotion from facial expressions: psychological and neurological mechanisms. Behav. Cognit. Neurosci. Rev. 1, 21–62. https://doi.org/10.1177/1534582302001001003 (2002).

    Article 

    Google Scholar
     

  • 10.

    Leppänen, J. M. & Nelson, C. A. Tuning the developing brain to social signals of emotions. Nat. Rev. Neurosci. 10, 37–47. https://doi.org/10.1038/nrn2554 (2009).

    CAS 
    Article 
    PubMed 

    Google Scholar
     

  • 11.

    Haxby, J. V., Hoffman, E. A. & Gobbini, M. I. The distributed human neural system for face perception. Trends Cognit. Sci. 4, 223–233. https://doi.org/10.1016/S1364-6613(00)01482-0 (2000).

    CAS 
    Article 

    Google Scholar
     

  • 12.

    Winston, J. S., Vuilleumier, P. & Dolan, R. J. Effects of low-spatial frequency components of fearful faces on fusiform cortex activity. Curr. Biol. 13, 1824–1829. https://doi.org/10.1016/j.cub.2003.09.038 (2003).

    CAS 
    Article 
    PubMed 

    Google Scholar
     

  • 13.

    Rotshtein, P., Henson, R. N. A., Treves, A., Driver, J. & Dolan, R. J. Morphing Marilyn into Maggie dissociates physical and identity face representations in the brain. Nat. Neurosci. 8, 107–113. https://doi.org/10.1016/j.tics.2017.01.0011 (2005).

    CAS 
    Article 
    PubMed 

    Google Scholar
     

  • 14.

    Fox, C. J., Moon, S. Y., Iaria, G. & Barton, J. J. S. The correlates of subjective perception of identity and expression in the face network: an fMRI adaptation study. NeuroImage 44, 569–580. https://doi.org/10.1016/j.tics.2017.01.0012 (2009).

    Article 
    PubMed 

    Google Scholar
     

  • 15.

    De Winter, F.-L. et al. Lateralization for dynamic facial expressions in human superior temporal sulcus. NeuroImage 106, 340–352. https://doi.org/10.1016/j.tics.2017.01.0013 (2015).

    Article 
    PubMed 

    Google Scholar
     

  • 16.

    Hoffman, E. A. & Haxby, J. V. Distinct representations of eye gaze and identity in the distributed human neural system for face perception. Nat. Neurosci. 3, 80–84. https://doi.org/10.1016/j.tics.2017.01.0014 (2000).

    CAS 
    Article 
    PubMed 

    Google Scholar
     

  • 17.

    Yang, D.Y.-J., Rosenblau, G., Keifer, C. & Pelphrey, K. A. An integrative neural model of social perception, action observation, and theory of mind. Neurosci. Biobehav. Rev. 51, 263–275. https://doi.org/10.1016/j.tics.2017.01.0015 (2015).

    Article 
    PubMed 
    PubMed Central 

    Google Scholar
     

  • 18.

    Calvert, G. A. et al. Activation of auditory cortex during silent lipreading. Science (New York, N.Y.) 276, 593–596 (1997).

    CAS 
    Article 

    Google Scholar
     

  • 19.

    Ross, E. D. & Monnot, M. Neurology of affective prosody and its functional-anatomic organization in right hemisphere. Brain Lang. 104, 51–74. https://doi.org/10.1016/j.tics.2017.01.0016 (2008).

    Article 
    PubMed 

    Google Scholar
     

  • 20.

    Gorelick, P. B. & Ross, E. D. The aprosodias: further functional-anatomical evidence for the organisation of affective language in the right hemisphere. J. Neurol. Neurosurg. Psychiatry 50, 553–560 (1987).

    CAS 
    Article 

    Google Scholar
     

  • 21.

    Belin, P., Zatorre, R. J. & Ahad, P. Human temporal-lobe response to vocal sounds. Brain Res. Cognit. Brain Res. 13, 17–26. https://doi.org/10.1016/S0926-6410(01)00084-2 (2002).

    Article 

    Google Scholar
     

  • 22.

    Kriegstein, K. V. & Giraud, A.-L. Distinct functional substrates along the right superior temporal sulcus for the processing of voices. NeuroImage 22, 948–955. https://doi.org/10.1016/j.tics.2017.01.0018 (2004).

    Article 
    PubMed 

    Google Scholar
     

  • 23.

    Ethofer, T., Van De Ville, D., Scherer, K. & Vuilleumier, P. Decoding of emotional information in voice-sensitive cortices. Curr. Biol. CB 19, 1028–1033. https://doi.org/10.1016/j.cub.2009.04.054 (2009).

    CAS 
    Article 
    PubMed 

    Google Scholar
     

  • 24.

    Johnstone, T., van Reekum, C. M., Oakes, T. R. & Davidson, R. J. The voice of emotion: an FMRI study of neural responses to angry and happy vocal expressions. Soc. Cognit. Affect. Neurosci. 1, 242–249. https://doi.org/10.1093/scan/nsl027 (2006).

    Article 

    Google Scholar
     

  • 25.

    Schirmer, A. & Kotz, S. A. Beyond the right hemisphere: brain mechanisms mediating vocal emotional processing. Trends Cognit. Sci. 10, 24–30. https://doi.org/10.1016/j.tics.2005.11.009 (2006).

    Article 

    Google Scholar
     

  • 26.

    Belin, P., Fecteau, S. & Bédard, C. Thinking the voice: neural correlates of voice perception. Trends Cognit. Sci. 8, 129–135. https://doi.org/10.1016/j.tics.2004.01.008 (2004).

    Article 

    Google Scholar
     

  • 27.

    Wright, T. M., Pelphrey, K. A., Allison, T., McKeown, M. J. & McCarthy, G. Polysensory interactions along lateral temporal regions evoked by audiovisual speech. Cereb. Cortex (New York, N.Y.: 1991) 13, 1034–1043. https://doi.org/10.1093/cercor/13.10.1034 (2003).

    Article 

    Google Scholar
     

  • 28.

    Beauchamp, M. S., Argall, B. D., Bodurka, J., Duyn, J. H. & Martin, A. Unraveling multisensory integration: patchy organization within human STS multisensory cortex. Nat. Neurosci. 7, 1190–1192. https://doi.org/10.1080/026999392084110684 (2004).

    CAS 
    Article 
    PubMed 

    Google Scholar
     

  • 29.

    Vuilleumier, P. & Pourtois, G. Distributed and interactive brain mechanisms during emotion face perception: evidence from functional neuroimaging. Neuropsychologia 45, 174–194. https://doi.org/10.1080/026999392084110685 (2007).

    Article 
    PubMed 

    Google Scholar
     

  • 30.

    Massaro, D. W. & Egan, P. B. Perceiving affect from the voice and the face. Psychon. Bull. Rev. 3, 215–221. https://doi.org/10.3758/BF03212421 (1996).

    CAS 
    Article 
    PubMed 

    Google Scholar
     

  • 31.

    de Gelder, B. & Vroomen, J. The perception of emotions by ear and by eye. Cognit. Emot. 14, 289–311. https://doi.org/10.1080/026999300378824 (2000).

    Article 

    Google Scholar
     

  • 32.

    Kreifelts, B., Ethofer, T., Grodd, W., Erb, M. & Wildgruber, D. Audiovisual integration of emotional signals in voice and face: an event-related fMRI study. NeuroImage 37, 1445–1456. https://doi.org/10.1080/026999392084110688 (2007).

    Article 
    PubMed 

    Google Scholar
     

  • 33.

    Peelen, M. V., Atkinson, A. P. & Vuilleumier, P. Supramodal representations of perceived emotions in the human brain. J. Neurosci. 30, 10127–10134. https://doi.org/10.1523/JNEUROSCI.2161-10.2010 (2010).

    CAS 
    Article 
    PubMed 
    PubMed Central 

    Google Scholar
     

  • 34.

    Watson, R. et al. Dissociating task difficulty from incongruence in face-voice emotion integration. Front. Hum. Neurosci. 7, 744. https://doi.org/10.3389/fnhum.2013.00744 (2013).

    Article 
    PubMed 
    PubMed Central 

    Google Scholar
     

  • 35.

    Davidson, R. J. Cerebral Asymmetry, Emotion, and Affective Style. In Brain asymmetry, 361–387 (The MIT Press, Cambridge, MA, US, 1995).

  • 36.

    Carter, R. M. & Huettel, S. A. A nexus model of the temporal-parietal junction. Trends Cognit. Sci. 17, 328–336. https://doi.org/10.1016/j.tics.2013.05.007 (2013).

    Article 

    Google Scholar
     

  • 37.

    Amodio, D. M. & Frith, C. D. Meeting of minds: the medial frontal cortex and social cognition. Nat. Rev. Neurosci. 7, 268–277. https://doi.org/10.1016/S0167-6393(02)00078-X2 (2006).

    CAS 
    Article 
    PubMed 

    Google Scholar
     

  • 38.

    Keysers, C. & Gazzola, V. Integrating simulation and theory of mind: from self to social cognition. Trends Cognit. Sci. 11, 194–196. https://doi.org/10.1016/j.tics.2007.02.002 (2007).

    Article 

    Google Scholar
     

  • 39.

    Hess, U. & Fischer, A. Emotional mimicry: why and when we mimic emotions—emotional mimicry. Soc. Pers. Psychol. Compass 8, 45–57. https://doi.org/10.1111/spc3.12083 (2014).

    Article 

    Google Scholar
     

  • 40.

    Ross, E. D. Right hemispheres role in language, affective behavior and emotion. Trends Neurosci. 7, 342–346. https://doi.org/10.1016/S0167-6393(02)00078-X5 (1984).

    Article 

    Google Scholar
     

  • 41.

    Blair, R. J. R. Impaired social response reversal: a case of acquired sociopathy. Brain 123, 1122–1141. https://doi.org/10.1093/brain/123.6.1122 (2000).

    Article 
    PubMed 

    Google Scholar
     

  • 42.

    Bailey, P. E., Henry, J. D. & Varcin, K. J. Right frontal cortical lesions disrupt anger mimicry. Neuropsychologia 50, 1632–1638. https://doi.org/10.1016/S0167-6393(02)00078-X7 (2012).

    Article 
    PubMed 

    Google Scholar
     

  • 43.

    Schmidt, K. L. & Cohn, J. F. Human facial expressions as adaptations: evolutionary questions in facial expression research. Am. J. Phys. Anthropol. Suppl 33, 3–24 (2001).

    Article 

    Google Scholar
     

  • 44.

    Cannon, P. R., Hayes, A. E. & Tipper, S. P. An electromyographic investigation of the impact of task relevance on facial mimicry. Cognit. Emot. 23, 918–929. https://doi.org/10.1080/02699930802234864 (2009).

    Article 

    Google Scholar
     

  • 45.

    Hatfield, E., Cacioppo, J. T. & Rapson, R. L. Emotional contagion. Studies in emotion and social interaction (Cambridge University Press ; Editions de la Maison des sciences de l’homme, Cambridge [England] ; New York : Paris, 1994).

  • 46.

    Seidel, E.-M., Habel, U., Kirschner, M., Gur, R. C. & Derntl, B. The impact of facial emotional expressions on behavioral tendencies in women and men. J. Exp. Psychol. Hum. Percept. Perform. 36, 500–507. https://doi.org/10.1016/S0167-6393(02)00078-X9 (2010).

    Article 
    PubMed 
    PubMed Central 

    Google Scholar
     

  • 47.

    Pourtois, G., de Gelder, B., Vroomen, J., Rossion, B. & Crommelinck, M. The time-course of intermodal binding between seeing and hearing affective information. NeuroReport 11, 1329–1333. https://doi.org/10.1038/nn13920 (2000).

    CAS 
    Article 
    PubMed 

    Google Scholar
     

  • 48.

    Klasen, M., Chen, Y.-H. & Mathiak, K. Multisensory emotions: perception, combination and underlying neural processes. Rev. Neurosci.https://doi.org/10.1515/revneuro-2012-0040 (2012).

    Article 
    PubMed 

    Google Scholar
     

  • 49.

    Gerdes, A. B. M., Wieser, M. J. & Alpers, G. W. Emotional pictures and sounds: a review of multimodal interactions of emotion cues in multiple domains. Front. Psychol. 5, 1351. https://doi.org/10.3389/fpsyg.2014.01351 (2014).

    Article 
    PubMed 
    PubMed Central 

    Google Scholar
     

  • 50.

    Korb, S., Frühholz, S. & Grandjean, D. Reappraising the voices of wrath. Soc. Cognit. Affect. Neurosci. 10, 1644–1660. https://doi.org/10.1093/scan/nsv051 (2015).

    Article 

    Google Scholar
     

  • 51.

    King, A. J. & Nelken, I. Unraveling the principles of auditory cortical processing: can we learn from the visual system?. Nat. Neurosci. 12, 698–701. https://doi.org/10.1038/nn13923 (2009).

    CAS 
    Article 
    PubMed 
    PubMed Central 

    Google Scholar
     

  • 52.

    Borod, J. C. et al. Relationships among facial, prosodic, and lexical channels of emotional perceptual processing. Cognit. Emot. 14, 193–211. https://doi.org/10.1080/026999300378932 (2000).

    Article 

    Google Scholar
     

  • 53.

    Frühholz, S. & Grandjean, D. Multiple subregions in superior temporal cortex are differentially sensitive to vocal expressions: a quantitative meta-analysis. Neurosci. Biobehav. Rev. 37, 24–35. https://doi.org/10.1038/nn13925 (2013).

    Article 
    PubMed 

    Google Scholar
     

  • 54.

    Balconi, M. & Vanutelli, M. E. Hemodynamic (fNIRS) and EEG (N200) correlates of emotional inter-species interactions modulated by visual and auditory stimulation. Sci. Rep. 6, 23083. https://doi.org/10.1038/srep23083 (2016).

    ADS 
    CAS 
    Article 
    PubMed 
    PubMed Central 

    Google Scholar
     

  • 55.

    Köchel, A. et al. Affective perception and imagery: a NIRS study. Int. J. Psychophysiol. 80, 192–197. https://doi.org/10.1016/j.ijpsycho.2011.03.006 (2011).

    Article 
    PubMed 

    Google Scholar
     

  • 56.

    Plichta, M. et al. Event-related functional near-infrared spectroscopy (fNIRS): are the measurements reliable?. NeuroImage 31, 116–124. https://doi.org/10.1016/j.neuroimage.2005.12.008 (2006).

    CAS 
    Article 
    PubMed 

    Google Scholar
     

  • 57.

    Zhang, D., Zhou, Y. & Yuan, J. Speech prosodies of different emotional categories activate different brain regions in adult cortex: an fNIRS study. Sci. Rep.https://doi.org/10.1038/s41598-017-18683-2 (2018).

    Article 
    PubMed 
    PubMed Central 

    Google Scholar
     

  • 58.

    Yuvaraj, R., Murugappan, M., Norlinah, M. I., Sundaraj, K. & Khairiyah, M. Review of emotion recognition in stroke patients. Dement. Geriatr. Cogn. Disord. 36, 179–196. https://doi.org/10.1159/000353440 (2013).

    Article 
    PubMed 

    Google Scholar
     

  • 59.

    Brancucci, A., Lucci, G., Mazzatenta, A. & Tommasi, L. Asymmetries of the human social brain in the visual, auditory and chemical modalities. Philos. Trans. R. Soc. B Biol. Sci. 364, 895–914. https://doi.org/10.1098/rstb.2008.0279 (2009).

    CAS 
    Article 

    Google Scholar
     

  • 60.

    Harciarek, M., Heilman, K. M. & Jodzio, K. Defective comprehension of emotional faces and prosody as a result of right hemisphere stroke: modality versus emotion-type specificity. J. Int. Neuropsychol. Soc. 12, 774–781. https://doi.org/10.3758/BRM.40.2.531 (2006).

    Article 
    PubMed 

    Google Scholar
     

  • 61.

    Adolphs, R., Jansari, A. & Tranel, D. Hemispheric perception of emotional valence from facial expressions. Neuropsychology 15, 516–524. https://doi.org/10.3758/BRM.40.2.532 (2001).

    CAS 
    Article 
    PubMed 

    Google Scholar
     

  • 62.

    Adolphs, R., Tranel, D. & Damasio, A. R. Dissociable neural systems for recognizing emotions. Brain Cogn. 52, 61–69. https://doi.org/10.3758/BRM.40.2.533 (2003).

    Article 
    PubMed 

    Google Scholar
     

  • 63.

    Capilla, A., Belin, P. & Gross, J. The early spatio-temporal correlates and task independence of cerebral voice processing studied with MEG. Cereb. Cortex 23, 1388–1395. https://doi.org/10.1093/cercor/bhs119 (2013).

    Article 
    PubMed 

    Google Scholar
     

  • 64.

    Stevenson, R. A. et al. Multisensory temporal integration in autism spectrum disorders. J. Neurosci. 34, 691–697. https://doi.org/10.1523/JNEUROSCI.3615-13.2014 (2014).

    CAS 
    Article 
    PubMed 
    PubMed Central 

    Google Scholar
     

  • 65.

    Dionne-Dostie, E., Paquette, N., Lassonde, M. & Gallagher, A. Multisensory integration and child neurodevelopment. Brain Sci. 5, 32–57. https://doi.org/10.3390/brainsci5010032 (2015).

    CAS 
    Article 
    PubMed 
    PubMed Central 

    Google Scholar
     

  • 66.

    Ferrari, M. & Quaresima, V. A brief review on the history of human functional near-infrared spectroscopy (fNIRS) development and fields of application. NeuroImage 63, 921–935. https://doi.org/10.3758/BRM.40.2.537 (2012).

    Article 
    PubMed 

    Google Scholar
     

  • 67.

    Shekhar, S. et al. Hemodynamic responses to emotional speech in two-month-old infants imaged using diffuse optical tomography. Sci. Rep.https://doi.org/10.1038/s41598-019-39993-7 (2019).

    Article 
    PubMed 
    PubMed Central 

    Google Scholar
     

  • 68.

    Anderson, A. W., Marois, R., Colson, Peterson, B. . S. & Dun, C. . C. Neonatal auditory activation detected by functional magnetic resonance imaging. Magn. Reson. Imaging 19, 1–5. https://doi.org/10.3758/BRM.40.2.538 (2001).

    Article 

    Google Scholar
     

  • 69.

    Doi, H., Nishitani, S. & Shinohara, K. NIRS as a tool for assaying emotional function in the prefrontal cortex. Front. Hum. Neurosci.https://doi.org/10.3389/fnhum.2013.00770 (2013).

    Article 
    PubMed 
    PubMed Central 

    Google Scholar
     

  • 70.

    Bendall, R. C. A., Eachus, P. & Thompson, C. A brief review of research using near-infrared spectroscopy to measure activation of the prefrontal cortex during emotional processing: the importance of experimental design. Front. Hum. Neurosci.https://doi.org/10.3389/fnhum.2016.00529 (2016).

    Article 
    PubMed 
    PubMed Central 

    Google Scholar
     

  • 71.

    Tottenham, N. et al. The NimStim set of facial expressions: Judgments from untrained research participants. Psychiatry Res. 168, 242–249. https://doi.org/10.3758/BRM.40.2.539 (2009).

    Article 
    PubMed 
    PubMed Central 

    Google Scholar
     

  • 72.

    Tiddeman, B., Burt, M. & Perrett, D. Prototyping and transforming facial textures for perception research. IEEE Comput. Graphics Appl. 21, 42–50. https://doi.org/10.1016/j.biopsycho.2011.02.0100 (2001).

    Article 

    Google Scholar
     

  • 73.

    Tiddeman, B. & Perrett, D. Transformation of dynamic facial image sequences using static 2d prototypes. Vis. Comput. 18, 218–225. https://doi.org/10.1007/s003710100142 (2002).

    Article 

    Google Scholar
     

  • 74.

    Scholkmann, F. et al. A review on continuous wave functional near-infrared spectroscopy and imaging instrumentation and methodology. NeuroImage 85, 6–27. https://doi.org/10.1016/j.biopsycho.2011.02.0102 (2014).

    Article 
    PubMed 

    Google Scholar
     

  • 75.

    Tak, S., Uga, M., Flandin, G., Dan, I. & Penny, W. Sensor space group analysis for fNIRS data. J. Neurosci. Methods 264, 103–112. https://doi.org/10.1016/j.biopsycho.2011.02.0103 (2016).

    CAS 
    Article 
    PubMed 
    PubMed Central 

    Google Scholar
     

  • 76.

    Scholkmann, F., Spichtig, S., Muehlemann, T. & Wolf, M. How to detect and reduce movement artifacts in near-infrared imaging using moving standard deviation and spline interpolation. Physiol. Meas. 31, 649–662. https://doi.org/10.1016/j.biopsycho.2011.02.0104 (2010).

    CAS 
    Article 
    PubMed 

    Google Scholar
     



  • Source link

    Leave a Reply

    Your email address will not be published. Required fields are marked *