Mondays 1pm in Anderson Conference Room (5^th floor McGill Hall)

Instructor: Viola Störmer (vstoermer@ucsd.edu)

Course Overview:

In this course we will do readings every week on topics of everybody’s choosing, centered on the theme of multisensory integration. Every week there will be a student presentation about the papers and then a discussion of the papers and how they fit in the broader literature. We will cover a wide range of topics within the context of multisensory interactions, including animal studies focusing on the neural mechanisms and circuits involved in multisensory integration, cross-modal attention effects in humans, the emergence of multisensory systems during development and changes thereafter (plasticity), low-level cross-modal interactions as well as higher-level interactions, for example during speech processing, and “special cases” of multisensory integration (synesthesia).

Course Meetings:

January 9^th. Overview of course and discussion of topics. These are the broad topics we discussed:

Neural basis of multisensory integration – superior colliculus & cortical circuits

Development / plasticity of multisensory systems

Basic principles of multisensory integration

Cross-modal spatial attention; spatial interactions

Are primary cortices really unisensory?

Speech processing

Amodal (Crossmodal) object representations

Synesthesia

January 23. Chaipat & Sasen - Superior colliculus and basic principles of multisensory integration

STEIN&STANFORD, 2008_REVIEW.PDF  - What is a multisensory neuron? (REVIEW)

Stanford, Quessy, Stein, 2005.pdf - Response properties of multisensory neurons in the superior colliculus (SC)

Cappe, Thelen, Romei, Thut, & Murray, 2012.pdf - Principles of multisensory integration in humans

January 30. Eric & Michael - Cortical circuits

Ghazanfar & Schroeder, 2006.pdf - Is the neocortex multisensory? (REVIEW)

Alvarado ,Stanford, Vaughan,& Stein, 2007.pdf - Cortex mediates multisensory integration in the SC

Ghazanfar, Chandrasekaran, Logothetis, 2008.pdf - Cortical integration zones during face/voice integration

February 6. Sasen - Computational approaches 

Ahshiro,Angelaki,DeAngelis, 2011.pdf - Normalization model of multisensory integration

Alais&Burr, 2004.pdf - Optimal integration explains the ventriloquist effect

Körding, Beierholm, Ma, Quartz, Tenenbaum, Shams, 2007.pdf - Causal bayesian inference for audio-visual localization

     *optional -  Knill & Pouget, 2004.pdf  - "The Bayesian Brain"

February 13. Eric & Parla - Crossmodal attention

March 6. Michael - Crossmodal (amodal) object representations 

Held & Sinha, 2011.pdf - Does tactile object knowledge transfer to visual object representations?  

** see also TED talk: https://www.ted.com/talks/pawan_sinha_on_how_brains_learn_to_see

Chen et al.,2016.pdf - Tactile and visual object representations - a case study

Peelen et al., 2014.pdf - Shared object representations in occipital cortex between blind and sighted

 ***MAYBE ***: Hasan, Valdes-Sosa, Gross, Belin, 2016.pdf - Shared identity representations of faces and voices

March 13. Eric & Parla - Crossmodal correspondences and synesthesia

Spence, 2011_REVIEW.pdf - Crossmodal correspondences (REVIEW)

Newell&Mitchell, 2016_REVIEW.pdf -  Is synesthesia an amplified version of "normal" crossmodal integration? (REVIEW)

Lacey, Martinez, McCormick, Sathian, 2016.pdf - Synesthesia strengthens sound-symbolic correspondences

Rouw & Scholte, 2007.pdf - Stronger connectivity in synesthetes

Course Requirements:

1. Attendance and in-class participation.

2. You are responsible for presenting overviews of the materials and organizing the discussion for the week of the class you are assigned for.

Course Grading:

Your grade will be calculated based on 1) class participation (50%) and 2) presentations and leading of discussion (50%).

Preparing for class discussion:

1. Make sure you can summarize the key points of all the articles that will be discussed the day of the class. If you have questions that prevent you from understanding any of the papers, you should first contact the student in charge of the presentation for a given week, and then ask me. You should be ready to answer the following questions:

      * What is the theoretical or empirical issue being addressed?

      * What is the method (for an empirical paper)? Make sure you understand the method and task described in the paper. If         no figure is given, it might be worth sketching the task design out yourself based on the Methods section to be sure you         understand it.

      * What were the main results of the study? Try to keep in mind the figure with the key results of that study that illustrates         the main point of the study.

      * What are the implications of the results with regards to that paper, and also with regards to other papers we’ve read or         you know about? Most papers relate in some ways to papers read during the previous weeks – make sure to spend               some time thinking about how the results fit into the big picture.

      * Try to critique methods and conclusions. What could the authors have done differently? What additional experiments             can be done to follow up on the results?

2. Make sure you are prepared to participate in the conversation each week. It might help you to write down a few sentences detailing your thoughts on the papers, and how they connect to previous classes. This will ensure that you are comfortable discussing something in class. Try to think in particular how the different topics link to each other because many good ideas in science come from making connections between seemingly disparate papers and lines of research, so this is a good way to practice drawing connections just to see where it takes you.

Organizing in-class presentation and discussion:

Students are responsible for presenting the papers (with some slides) and leading the discussion in class. They should make sure they have some discussion points prepared. Furthermore, they are responsible for acting as a resource for other students to turn to prior to class with questions, if they have any. You may also schedule a meeting with me in advance of the course for which you are responsible for to discuss the readings.

--------------------------------------------------------------------------------------------------------------

Overview of all course topics we discussed on January 9th in more detail:

Neural basis of multisensory integration – superior colliculus & cortical circuits

-       The superior colliculus:

o   What is a multisensory neuron? (REVIEW by Stein & Stanford, 2008)

o   SC represents a common map of space (e.g., Jay & Sparks, 1984)

o   response properties of multisensory neurons in the SC (e.g., Stanford, Quessy, Stein, 2005; Perrault et al., 2004)

-       Cortical circuits:

o   Is the neocortex multisensory? (REVIEW- Ghazanfar & Schroeder, 2006)

o   multisensory integration is mediated by cortical inputs (e.g., Alvarado, Stanford, Vaughan, Stein, 2007)

o   auditory and visual communication signals are integrated in the ventrolateral PFC (Sugihara et al., 2006)

-       Multisensory integration effects in sensory areas

o   Audiovisual illusions show multisensory effects in early perception (e.g., Shams et al. 2000, 2002; Mishra et al., 2997; Watkins et al., 2006)

Development / plasticity of multisensory systems

-       Neurons in the superior colliculus:

o   Is visual experience necessary for the development of multisensory integration in the SC? (e.g., Wallace et al., 2004)

o   Plasticity in the cat SC: short term changes in the adult superior colliculus (Yu, Stein, Rowland, 2009; Yu et al., 2014)

o   How does an individual multisensory neuron develop? (– REVIEW by Wallace & Stein, 2014) 

-       Development/plasticity in humans:

o   Cortical reorganization due to early deprivation (e.g., Finney, Fine, & Dobkins, 200; Collignon et al. 2015; Röder et al., 2013; Bavelier & Neville, 2002 – REVIEW)

 Basic principles of multisensory integration

-       Spatial proximity, temporal proximity, inverse effectiveness - How context-dependent are these principles across tasks and stimuli? (e.g., Cappe et al., 2012; Stevenson & Wallace, 2013)

-       Canonincal operations of multisensory processing: divisive normalization & oscillatory phase resetting (Fetsch et al., 2013; Ohshiro et al., 2011; Reynolds & Heeger, 2009; Kayser et al., 2008; Lakatos eta l., 2005, 2007)

-       Bayesian approaches to multisensory integration (e.g., Beierholm et al., 2009; Cheng et al., 2007; Knill & Pouget, 2004)

-       How is multisensory integration different from unisensory integration? (Is multisensory integration “special”? e.g., Alvarado, Stanford, Vaughan, Stein, 2007; Alvarado et al., 2008; Rowland et al., 2007)

 Cross-modal spatial attention; spatial interactions

-       Is attention supramodal? (Spence & Driver, 2004)

-       Cross-modal effects of endogenous attention (Macaluso et al., 2002, 2002; Macaluso & Driver, 2004)

-       Exogenous cross-modal attention (McDonald et al., 2000; 2003; Störmer et al., 2009; Frassetti et al., 2002)

Are primary cortices really unisensory?

-       Top-down (voluntary) auditory attention can activate visual cortex (e.g., Cate et al., 2009; Wu et al., 2007)

-       Involuntarily orienting to sound enhances visual cortex (e.g., McDonald et al., 2013; Romei, Gross, Thut, 2012; Romei, Murray, Merabet, Thut, 2007)

-       Functional role of responses to non-preferred stimuli in sensory areas (e.g., Lemus et al., 2010)

Speech processing

-       Is speech a special case of multisensory integration?  (Ross et al., 2007; Vatakis et al., 2008)

-       Potential neural mechanisms: Vision increases the speed auditory neural processing (e.g., Wassenhoeve, et al., 2005)

-       Language acquisition (e.g., Pons et al., 2009)

 Amodal (Crossmodal) object representations

-       Shared object representations between modalities (Peelen & Caramazza, 2014; Massone t al., 2015)

-       Does haptic knowledge transfer to vision, even without visual experience? (Held & Sinha, 2011; Chen et al., 2016)

-       Shared representation for a person’s voice and face? (Hasan et al., 2016) 

-       Cross-modal / amodal representations of space

Synesthesia

-       Learned and inherent aspects of synesthesia (Newell & Mitchell, 2016)

-       Brain structure underlying synesthesia, e.g., hyperconnectivity (e.g., Rouw & Scholte, 2007)

-       Sensory substitution as a “special case” of synesthesia (perceptual learning; e.g., Ward & Wright, 2014)

-       Cross-modal correspondences