Editing Multisensory integration

{{Short description|Study of senses and nervous system}}
'''Multisensory integration''', also known as '''multimodal integration''', is the study of how information from the different [[sensory modality|sensory modalities]] (such as sight, sound, touch, smell, self-motion, and taste) may be integrated by the [[nervous system]].<ref name="Stein Stanford 2009">{{Cite journal  | last1 = Stein | first1 = BE. | last2 = Stanford | first2 = TR. | last3 = Rowland | first3 = BA. | title = The neural basis of multisensory integration in the midbrain: its organization and maturation. | journal = Hear Res | volume = 258 | issue = 1–2 | pages = 4–15 |date=Dec 2009 | doi = 10.1016/j.heares.2009.03.012 | pmid = 19345256 | pmc = 2787841}}</ref> A coherent representation of objects combining modalities enables animals to have meaningful perceptual experiences. Indeed, multisensory integration is central to adaptive behavior because it allows animals to perceive a world of coherent perceptual entities.<ref name="Lewkowicz 2009">{{cite journal |vauthors=Lewkowicz DJ, Ghazanfar AA |title=The emergence of multisensory systems through perceptual narrowing |journal=Trends Cogn. Sci. (Regul. Ed.) |volume=13 |issue=11 |pages=470–8 |date=November 2009 |pmid=19748305 |doi=10.1016/j.tics.2009.08.004 |url=http://asifg.mycpanel.princeton.edu/publications/pdfs/Lewkowicz&Ghazanfar2009.pdf|citeseerx=10.1.1.554.4323 |s2cid=14289579 }}</ref> Multisensory integration also deals with how different sensory modalities interact with one another and alter each other's processing.

==General introduction==
Multimodal perception is how animals form coherent, valid, and robust perception by [[sensory processing|processing sensory]] stimuli from various modalities. Surrounded by multiple objects and receiving multiple sensory stimulations, the brain is faced with the decision of how to categorize the stimuli resulting from different objects or events in the physical world.  The nervous system is thus responsible for whether to integrate or segregate certain groups of signals.  Multimodal perception has been widely studied in cognitive science, behavioral science, and neuroscience.

===Stimuli and sensory modalities===
There are four attributes of stimulus: modality, intensity, location, and duration. The [[neocortex]] in the mammalian brain has parcellations that primarily process sensory input from one modality. For example, primary visual area, V1, or primary [[somatosensory system|somatosensory]] area, S1. These areas mostly deal with low-level stimulus features such as brightness, orientation, intensity, etc. These areas have extensive connections to each other as well as to higher association areas that further process the stimuli and are believed to integrate sensory input from various modalities. However, multisensory effects have been shown to occur in primary sensory areas as well.<ref name="Lemus 2010">{{cite journal |vauthors=Lemus L, Hernández A, Luna R, Zainos A, Romo R |title=Do sensory cortices process more than one sensory modality during perceptual judgements? |journal=Neuron |volume=67 |issue=2 |pages=335–48 |date=July 2010 |pmid=20670839 |doi=10.1016/j.neuron.2010.06.015 |s2cid=16043442 |doi-access=free }}</ref>

===Binding problem===
{{main|Binding problem}}
The relationship between the binding problem and multisensory perception can be thought of as a question – the binding problem, and potential solution – multisensory perception. The binding problem stemmed from unanswered questions about how mammals (particularly higher primates) generate a unified, coherent perception of their surroundings from the cacophony of [[Electromagnetic radiation|electromagnetic waves]], chemical interactions, and pressure fluctuations that forms the physical basis of the world around us. It was investigated initially in the [[visual system|visual domain]] (colour, motion, depth, and form), then in the auditory domain, and recently in the multisensory areas. It can be said therefore, that the binding problem is central to multisensory [[perception]].<ref name="Zmigrod 2010">{{Cite journal  | last1 = Zmigrod | first1 = S. | last2 = Hommel | first2 = B. | title = Temporal dynamics of unimodal and multimodal feature binding. | journal = Atten Percept Psychophys | volume = 72 | issue = 1 | pages = 142–52 |date=Jan 2010 | doi = 10.3758/APP.72.1.142 | pmid = 20045885 | s2cid = 7055915 |url = http://www.bernhard-hommel.eu/The%20temporal%20dynamics.pdf | doi-access = free }}</ref>

However, considerations of how unified conscious representations are formed are not the full focus of multisensory Integration research. It is obviously important for the senses to interact in order to maximize how efficiently people interact with the environment. For perceptual experience and behavior to benefit from the simultaneous stimulation of multiple sensory modalities, integration of the information from these modalities is necessary. Some of the mechanisms mediating this phenomenon and its subsequent effects on cognitive and behavioural processes will be examined hereafter. Perception is often defined as one's conscious experience, and thereby combines inputs from all relevant senses and prior knowledge. Perception is also defined and studied in terms of feature extraction, which is several hundred milliseconds away from conscious experience. Notwithstanding the existence of [[Gestalt psychology]] schools that advocate a holistic approach to the operation of the brain,<ref name="Wagemans 2012a">{{Cite journal  | last1 = Wagemans | first1 = J. | last2 = Elder | first2 = JH. | last3 = Kubovy | first3 = M. | last4 = Palmer | first4 = SE. | last5 = Peterson | first5 = MA. | last6 = Singh | first6 = M. | last7 = von der Heydt | first7 = R. | title = A century of Gestalt psychology in visual perception: I. Perceptual grouping and figure-ground organization. | journal = Psychol Bull | volume = 138 | issue = 6 | pages = 1172–217 |date=Nov 2012 | doi = 10.1037/a0029333 | pmid = 22845751 | pmc=3482144| citeseerx = 10.1.1.452.8394 }}</ref><ref name="Wagemans 2012b">{{Cite journal  | last1 = Wagemans | first1 = J. | last2 = Feldman | first2 = J. | last3 = Gepshtein | first3 = S. | last4 = Kimchi | first4 = R. | last5 = Pomerantz | first5 = JR. | last6 = van der Helm | first6 = PA. | last7 = van Leeuwen | first7 = C. | title = A century of Gestalt psychology in visual perception: II. Conceptual and theoretical foundations. | journal = Psychol Bull | volume = 138 | issue = 6 | pages = 1218–52 |date=Nov 2012 | doi = 10.1037/a0029334 | pmid = 22845750 | pmc=3728284}}</ref> the physiological processes underlying the formation of percepts and conscious experience have been vastly understudied. Nevertheless, burgeoning neuroscience research continues to enrich our understanding of the many details of the brain, including neural structures implicated in multisensory integration such as the [[superior colliculus|superior colliculus (SC)]]<ref name="Stein 2011">{{Cite book  | last1 = Stein | first1 = BE. | last2 = Rowland | first2 = BA. | title = Enhancing Performance for Action and Perception - Multisensory Integration, Neuroplasticity and Neuroprosthetics, Part I | chapter = Organization and plasticity in multisensory integration | journal = Prog Brain Res | volume = 191 | pages = 145–63 | year = 2011 | doi = 10.1016/B978-0-444-53752-2.00007-2 | pmid = 21741550 | pmc = 3245961 | series = Progress in Brain Research | isbn = 9780444537522 }}</ref> and various cortical structures such as the [[superior temporal gyrus|superior temporal gyrus (GT)]] and visual and auditory association areas. Although the structure and function of the SC are well known, the cortex and the relationship between its constituent parts are presently the subject of much investigation. Concurrently, the recent impetus on integration has enabled investigation into perceptual phenomena such as the [[ventriloquism]] effect,<ref name="Recanzone 2009">{{Cite journal  | last1 = Recanzone | first1 = GH. | title = Interactions of auditory and visual stimuli in space and time. | journal = Hear Res | volume = 258 | issue = 1–2 | pages = 89–99 |date=Dec 2009 | doi = 10.1016/j.heares.2009.04.009 | pmid = 19393306 | pmc = 2787663}}</ref> rapid localization of stimuli and the [[McGurk effect]];<ref name="Smith 2013">{{Cite journal| last1=Smith| first1=E.| last2=Duede| first2=S.| last3=Hanrahan| first3=S.| last4=Davis| first4=T.| last5=House| first5=P.| last6=Greger| first6=B.| title= Seeing is believing: neural representations of visual stimuli in human auditory cortex correlate with illusory auditory perceptions.| journal=PLOS ONE| volume=8| issue=9 | pages=e73148| year=2013 | doi=10.1371/journal.pone.0073148 | pmid=24023823 | pmc=3762867| bibcode=2013PLoSO...873148S| doi-access=free}}</ref> culminating in a more thorough understanding of the human brain and its functions.

===History===
Studies of sensory processing in humans and other animals has traditionally been performed one sense at a time,<ref name="Fodor 1983">{{cite book |author=Fodor, Jerry A. |title=Modularity of mind: an essay on faculty psychology |publisher=MIT Press |location=Cambridge, Mass |year=1983 |isbn=978-0-262-06084-4 |oclc=551957787 }}</ref> and to the present day, numerous academic societies and journals are largely restricted to considering sensory modalities separately ([[Vision science|'Vision Research']], [[Audiology|'Hearing Research']] etc.). 
However, there is also a long and parallel history of multisensory research. An example is the  [[George M. Stratton|Stratton's]] (1896) experiments on the somatosensory effects of wearing vision-distorting prism glasses.<ref name="Stratton 1896">{{cite journal| last1=Stratton| first1=George M.| title=Some preliminary experiments on vision without inversion of the retinal image. |journal=Psychological Review| volume=3|issue=6| year=1896| pages=611–617 |doi=10.1037/h0072918}}</ref><ref name="Stratton 1897">{{cite journal| last1=Stratton| first1=George M.| title=Vision without inversion of the retinal image. |journal=Psychological Review| volume=4| issue=4| year=1897| pages=341–360, 463–481 | doi = 10.1037/h0075482 | url=https://zenodo.org/record/1429122}}</ref>
Multisensory interactions or [[crossmodal]] effects
in which the perception of a stimulus is influenced by
the presence of another type of stimulus are  referred since very
early in the past. They were reviewed by Hartmann<ref name="Hartman1935">{{cite book| last1=Hartmann| first1=G.M.| title=Gestalt Psychology | publisher = The Ronald Press | year = 1935 | location = New York}}</ref> in a fundamental book where, among several
references to different types of multisensory interactions,
reference is made to the work of Urbantschitsch in 1888<ref name="Urbantschitsch1888">{{cite journal| last1=Urbantschitsch| first1=V.| title=Über den Einfluss einer Sinneserregung auf die übrigen Sinnesempfindungen. |journal=Pflügers Archiv| volume=42| year=1888| pages=154–182| doi=10.1007/bf01669354| s2cid=42136599| url=https://zenodo.org/record/1428330}}</ref> who reported on the improvement of visual acuity by auditive stimuli in subjects
with damaged brains. This effect was also found later in individuals with undamaged brains by Krakov<ref name="Kravkov1930">{{cite journal| last1=Kravkov| first1=S.V.| title=Über die Abhängigkeit der Sehschärfe vom Schallreiz. |journal=Arch. Ophthalmol.| volume=124| issue=2| year=1930| pages=334–338| doi=10.1007/bf01853661| s2cid=30040170}}</ref> and  Hartmann,<ref name="Hartmann1933">{{cite journal| last1=Hartmann| first1=G.W.| title=Changes in Visual Acuity through Simultaneous Stimulation of Other Sense Organs. |journal=J. Exp. Psychol.| volume=16| issue=3| year=1933| pages=393–407| doi=10.1037/h0074549}}</ref> as well as the fact that the visual acuity could be improved  by
other type of stimuli.<ref name="Hartmann1933" />  It is also noteworthy the amount
of work in the early 1930s on intersensory relations in the Soviet
Union, reviewed by London.<ref name="London1954">{{cite journal| last1=London| first1=I.D.| title=Research of sensory interaction in the Soviet Union. |journal=Psychol. Bull.| volume=51| issue=6| year=1954| pages=531–568| doi=10.1037/h0056730| pmid=13215683}}</ref> A remarkable multisensory research is the extensive and pioneering work of [[Justo Gonzalo|Gonzalo]]
<ref name=Dinamica>Gonzalo, J. (1945, 1950, 1952, 2010, 2023). [[hdl:10347/4341|''Dinámica Cerebral'', Open Access]]. Edición facsímil 2010 del Vol. 1 1945, Vol. 2 1950 (Madrid: Inst. S. Ramón y Cajal, CSIC), Suplemento I 1952 (Trab. Inst. Cajal Invest. Biol.) y 1ª ed. Suplemento II 2010. Red Temática en Tecnologías de Computación Artificial/Natural (RTNAC) y Universidad de Santiago de Compostela (USC). ISBN 978-84-9887-458-7. [http://libros.csic.es/product_info.php?products_id=1762&language=en ''Brain Dynamics'', English edition 2023 (Vols. 1 and 2, Supplements I and II),  I. Gonzalo-Fonrodona (ed.) Editorial CSIC, Open Access].</ref> in the mid-20th century on the characterization of a multisensory syndrome in patients with parieto-occipital cortical lesions. In this syndrome,  all the sensory functions are affected, and with symmetric bilaterality, in spite of being a unilateral lesion where the primary areas were not involved. A feature of this syndrome is the great permeability to crossmodal effects between visual, tactile, auditive stimuli as well as muscular effort to improve the perception, also decreasing the reaction times. The improvement by crossmodal effect was found to be greater as
the primary stimulus to be perceived was weaker, and as the cortical lesion was greater (Vol 1 and 2  of reference<ref name=Dinamica />). This author interpreted these phenomena under a dynamic physiological concept, and from a model based on functional gradients through the cortex and scaling laws of dynamical systems, thus highlighting the functional unity of the cortex. According to the functional cortical gradients,  the specificity of the cortex would be distributed in gradation, and the overlap of different specific gradients would be related to multisensory interactions.<ref name=Gradients>[[Justo Gonzalo|Gonzalo, J.]] (1952). "Las funciones cerebrales humanas según nuevos datos y bases fisiológicas. Una introducción a los estudios de Dinámica Cerebral". ''Trabajos del Inst. Cajal de Investigaciones Biológicas'', '''XLIV''': pp.&nbsp;95–157. It is the Supplement I of [http://libros.csic.es/product_info.php?products_id=1762&language=en ''Brain Dynamics'', English edition 2023 (Vols. 1 and 2, Supplements I and II),  I. Gonzalo-Fonrodona (ed.) Editorial CSIC, Open Access].</ref> 
Multisensory research has recently gained enormous interest and popularity.

===Example of spatial and structural congruence===
When we hear a [[vehicle horn|car honk]], we would determine which car triggers the honk by which car we see is the spatially closest to the honk. It's a spatially congruent example by combining visual and auditory stimuli. On the other hand, the sound and the pictures of a TV program would be integrated as structurally congruent by combining visual and auditory stimuli. However, if the sound and the pictures did not meaningfully fit, we would segregate the two stimuli. Therefore, spatial or structural congruence comes from not only combining the stimuli but is also determined by our understanding.

==Theories and approaches==

===Visual dominance===
Literature on spatial [[crossmodal]] biases suggests that visual modality often influences information from other senses.<ref name="Witten 2005">{{Cite journal  | last1 = Witten | first1 = IB. | last2 = Knudsen | first2 = EI. | title = Why seeing is believing: merging auditory and visual worlds. | journal = Neuron | volume = 48 | issue = 3 | pages = 489–96 |date=Nov 2005 | doi = 10.1016/j.neuron.2005.10.020 | pmid = 16269365 | s2cid = 17244783 | doi-access = free }}</ref> Some research indicates that vision dominates what we hear, when varying the degree of spatial congruency. This is known as the ventriloquist effect.<ref name="Shams 2010">{{Cite journal  | last1 = Shams | first1 = L. | last2 = Beierholm | first2 = UR. | title = Causal inference in perception. | journal = Trends Cogn Sci | volume = 14 | issue = 9 | pages = 425–32 |date=Sep 2010 | doi = 10.1016/j.tics.2010.07.001 | pmid = 20705502 | s2cid = 7750709 }}</ref> In cases of visual and haptic integration, children younger than 8 years of age show visual dominance when required to identify object orientation. However, haptic dominance occurs when the factor to identify is object size.<ref name="Gori 2008">{{Cite journal  | last1 = Gori | first1 = M. | last2 = Del Viva | first2 = M. | last3 = Sandini | first3 = G. | last4 = Burr | first4 = DC. | title = Young children do not integrate visual and haptic form information. | journal = Curr Biol | volume = 18 | issue = 9 | pages = 694–8 |date=May 2008 | doi = 10.1016/j.cub.2008.04.036 | pmid = 18450446 | s2cid = 13899031 | url = https://api.research-repository.uwa.edu.au/files/1490966/11970_PID11970.pdf | doi-access = free }}</ref><ref name="Gori 2012">{{Cite journal  | last1 = Gori | first1 = M. | last2 = Sandini | first2 = G. | last3 = Burr | first3 = D. | title = Development of visuo-auditory integration in space and time. | journal = Front Integr Neurosci | volume = 6 | pages = 77 | year = 2012 | doi = 10.3389/fnint.2012.00077 | pmid = 23060759 | pmc = 3443931 | doi-access = free }}</ref>

===Modality appropriateness===
According to Welch and Warren (1980), the Modality Appropriateness Hypothesis states that the influence of perception in each modality in multisensory integration depends on that modality's appropriateness for the given task. Thus, vision has a greater influence on integrated localization than hearing, and hearing and touch have a greater bearing on timing estimates than vision.<ref name="Welch 1980">{{cite journal |vauthors=Welch RB, Warren DH |title=Immediate perceptual response to intersensory discrepancy |journal=Psychol Bull |volume=88 |issue=3 |pages=638–67 |date=November 1980 |pmid=7003641 |doi= 10.1037/0033-2909.88.3.638}}</ref><ref name="Lederman and Klatzky 2004">{{cite book|last1=Lederman|first1=Susan J.|author-link1=Susan Lederman|last2=Klatzky|first2=Roberta L.|editor1-last=Calvert|editor1-first=Gemma A.|editor2-last=Spence|editor2-first=Charles|editor3-last=Stein|editor3-first=Barry E.|title=The Handbook of Multisensory Processing|date=2004|publisher=MIT Press|location=Cambridge, MA|isbn=978-0-262-03321-3|pages=[https://archive.org/details/handbookofmultis0000unse/page/107 107–122]|chapter=Multisensory Texture Perception|chapter-url=https://archive.org/details/handbookofmultis0000unse/page/107}}</ref>

More recent studies refine this early qualitative account of multisensory integration. Alais and Burr (2004), found that following progressive degradation in the quality of a visual stimulus, participants' perception of spatial location was determined progressively more by a simultaneous auditory cue.<ref name="Alais 2004">{{cite journal |vauthors=Alais D, Burr D |title=The ventriloquist effect results from near-optimal bimodal integration |journal=Curr. Biol. |volume=14 |issue=3 |pages=257–62 |date=February 2004 |pmid=14761661 |doi=10.1016/j.cub.2004.01.029 |citeseerx=10.1.1.220.4159 |s2cid=3125842 }}</ref> However, they also progressively changed the temporal uncertainty of the auditory cue; eventually concluding that it is the uncertainty of individual modalities that determine to what extent information from each modality is considered when forming a percept.<ref name="Alais 2004"/> This conclusion is similar in some respects to the 'inverse effectiveness rule'. The extent to which multisensory integration occurs may vary according to the ambiguity of the relevant stimuli. In support of this notion, a recent study shows that weak senses such as olfaction can even modulate the perception of visual information as long as the reliability of visual signals is adequately compromised.<ref>{{cite journal|last1=Kuang|first1=S.|last2=Zhang|first2=T.|title=Smelling directions: Olfaction modulates ambiguous visual motion perception|journal=Scientific Reports|date=2014|volume=4|pages=5796|doi=10.1038/srep05796|pmid=25052162|pmc=4107342|bibcode=2014NatSR...4E5796K}}</ref>

===Bayesian integration===
The theory of Bayesian integration is based on the fact that the brain must deal with a number of inputs, which vary in reliability.<ref name="Deneve 2004">{{cite journal |vauthors=Deneve S, Pouget A |title=Bayesian multisensory integration and cross-modal spatial links |journal=J. Physiol. Paris |volume=98 |issue=1–3 |pages=249–58 |year=2004 |pmid=15477036 |doi=10.1016/j.jphysparis.2004.03.011 |url=http://www.gnt.ens.fr/reprints/deneve_jphysparis_2004.pdf|citeseerx=10.1.1.133.7694 |s2cid=9831111 }}</ref> In dealing with these inputs, it must construct a coherent representation of the world that corresponds to reality. The Bayesian integration view is that the brain uses a form of [[Bayesian inference]].<ref name="Pouget 2002">{{cite journal |vauthors=Pouget A, Deneve S, Duhamel JR |title=A computational perspective on the neural basis of multisensory spatial representations |journal=Nature Reviews Neuroscience |volume=3 |issue=9 |pages=741–7 |date=September 2002 |pmid=12209122 |doi=10.1038/nrn914 |s2cid=1035721 }}</ref> This view has been backed up by computational modeling of such a Bayesian inference from signals to coherent representation, which shows similar characteristics to integration in the brain.<ref name="Pouget 2002"/>

====Cue combination vs. causal inference models====
With the assumption of independence between various sources, the traditional cue combination model is successful in modality integration. However, depending on the discrepancies between modalities, there might be different forms of stimuli fusion: integration, partial integration, and segregation. To fully understand the other two types, we have to use causal inference model without the assumption as cue combination model. This freedom gives us general combination of any numbers of signals and modalities by using Bayes' rule to make causal inference of sensory signals.<ref name="Vilares 2011">{{Cite journal  | last1 = Vilares | first1 = I. | last2 = Kording | first2 = K. | title = Bayesian models: the structure of the world, uncertainty, behavior, and the brain. | journal = Annals of the New York Academy of Sciences | volume = 1224 | issue =  1| pages = 22–39 |date=Apr 2011 | doi = 10.1111/j.1749-6632.2011.05965.x | pmid = 21486294 | pmc=3079291| bibcode = 2011NYASA1224...22V }}</ref>

====The hierarchical vs. non-hierarchical models====
The difference between two models is that hierarchical model can explicitly make causal inference to predict certain stimulus while non-hierarchical model can only predict joint probability of stimuli. However, hierarchical model is actually a special case of non-hierarchical model by setting joint prior as a weighted average of the prior to common and independent causes, each weighted by their prior probability. Based on the correspondence of these two models, we can also say that hierarchical is a mixture modal of non-hierarchical model.

====Independence of likelihoods and priors====
For [[Bayesian network|Bayesian model]], the prior and likelihood generally represent the statistics of the environment and the sensory representations. The independence of priors and likelihoods is not assured since the prior may vary with likelihood only by the representations. However, the independence has been proved by Shams with series of parameter control in multi sensory perception experiment.<ref name="Beierholm 2009">{{Cite journal  | last1 = Beierholm | first1 = UR. | last2 = Quartz | first2 = SR. | last3 = Shams | first3 = L. | title = Bayesian priors are encoded independently from likelihoods in human multisensory perception. | journal = J Vis | volume = 9 | issue = 5 | pages = 23.1–9 | year = 2009 | doi = 10.1167/9.5.23 | pmid = 19757901 | doi-access = free }}</ref>

===Shared intentionality===
The [[shared intentionality]] approach proposes the holistic explanation of the neurophysiological processes underlying the formation of percepts and conscious experience. The psychological construct of shared intentionality was introduced at the end of 20th century.<ref>Gilbert M. (1989). ''On social facts.'' London: [[Routledge]].</ref><ref>Searle JR. (1992). ''The rediscovery of the mind.'' London: [[MIT Press]].</ref><ref>Tuomela R. (1995). ''The importance of us.'' SUP. Stanford, CA: [[Stanford University Press]].</ref> Michael Tomasello developed it to explain cognition beginning in the earlier developmental stage through unaware collaboration in mother-child dyads.<ref>Tomasello, M. (1999). ''The Cultural Origins of Human Cognition.'' Cambridge, Massachusetts: [[Harvard University Press]].</ref><ref>Tomasello, M. (2019). ''Becoming Human: A Theory of Ontogeny.'' Cambridge, Massachusetts: [[Harvard University Press]].</ref> Over the past twenty years, knowledge of this notion has evolved through observing shared intentionality from different perspectives, e.g., [[psychophysiology]],<ref>McClung, J. S., Placì, S., Bangerter, A., Clément, F., & Bshary, R. (2017). "The language of cooperation: shared intentionality drives variation in helping as a function of group membership." ''Proceedings of the Royal Society B: Biological Sciences,'' 284(1863), 20171682. http://dx.doi.org/10.1098/rspb.2017.1682.</ref><ref>Shteynberg, G., & Galinsky, A. D. (2011). "Implicit coordination: Sharing goals with similar others intensifies goal pursuit." ''Journal of Experimental Social Psychology,'' 47(6), 1291-1294., https://doi.org/10.1016/j.jesp.
2011.04.012.</ref> and [[neurobiology]].<ref>Fishburn, F. A., Murty, V. P., Hlutkowsky, C. O., MacGillivray, C. E., Bemis, L. M., Murphy, M. E., ... & Perlman, S. B. (2018). "Putting our heads together: interpersonal neural synchronization as a biological mechanism for shared intentionality." ''Social cognitive and affective neuroscience,'' 13(8), 841-849.</ref> According to Igor Val Danilov, shared intentionality enables the mother-child pair to share the essential sensory stimulus of the actual cognitive problem.<ref>Val Danilov I. (2023). "Theoretical Grounds of Shared Intentionality for Neuroscience in Developing Bioengineering Systems." ''OBM Neurobiology'' 2023; 7(1): 156; doi:10.21926/obm.neurobiol.2301156</ref> The hypothesis of neurophysiological processes occurring during shared intentionality explains its integrative complexity from neuronal to interpersonal dynamics levels.<ref>Val Danilov, I. & Mihailova S. (2021). "Neuronal Coherence Agent for Shared Intentionality: A Hypothesis of Neurobiological Processes Occurring during Social Interaction." ''OBM Neurobiology'' 2021;5(4):26; doi:10.21926/obm.neurobiol.2104113</ref> This collaborative interaction provides environmental learning of the immature organism, starting at the reflexes stage of development, for processing the organization, identification, and interpretation of sensory information in developing perception.<ref>Val Danilov I. (2023)."Low-Frequency Oscillations for Nonlocal Neuronal Coupling in Shared Intentionality Before and After Birth: Toward the Origin of Perception." ''OBM Neurobiology'' 2023; 7(4): 192; doi:10.21926/obm.neurobiol.2304192. https://www.lidsen.com/journals/neurobiology/neurobiology-07-04-192</ref> From this perspective, Shared intentionality contributes to the formation of percepts and conscious experiences, solving the [[binding problem]], or at least complements one of the mechanisms noted above.<ref>Val Danilov I. (2023). "Shared Intentionality Modulation at the Cell Level: Low-Frequency Oscillations for Temporal Coordination in Bioengineering Systems." ''OBM Neurobiology'' 2023; 7(4): 185; doi:10.21926/obm.neurobiol.2304185. https://www.lidsen.com/journals/neurobiology/neurobiology-07-04-185</ref>

==Principles==
The contributions of Barry Stein, Alex Meredith, and their colleagues (e.g."The merging of the senses" 1993,<ref name="Stein 1993">{{cite book |author1=Stein, Barry |author2=Meredith, M. Alex |title=The merging of the senses |publisher=MIT Press |location=Cambridge, Mass |year=1993 |isbn=978-0-262-19331-3 |oclc=25869284 }}</ref>) are widely considered to be the groundbreaking work in the modern field of multisensory integration. Through detailed long-term study of the neurophysiology of the superior colliculus, they distilled three general principles by which multisensory integration may best be described.

*'''The spatial rule'''<ref name="Meredith 1986">{{Cite journal  | last1 = Meredith | first1 = MA. | last2 = Stein | first2 = BE. | title = Spatial factors determine the activity of multisensory neurons in cat superior colliculus. | journal = Brain Res | volume = 365 | issue = 2 | pages = 350–4 |date=Feb 1986 | doi =  10.1016/0006-8993(86)91648-3| pmid = 3947999 | s2cid = 12807282 }}</ref><ref name="King 1985">{{cite journal |vauthors=King AJ, Palmer AR |title=Integration of visual and auditory information in bimodal neurones in the guinea-pig superior colliculus |journal=Exp Brain Res |volume=60 |issue=3 |pages=492–500 |year=1985 |pmid=4076371 |doi= 10.1007/bf00236934|s2cid=25796198 }}</ref> states that multisensory integration is more likely or stronger when the constituent unisensory stimuli arise from approximately the same location.
*'''The temporal rule'''<ref name="King 1985"/><ref name="Meredith Nemitz 1987">{{Cite journal  | last1 = Meredith | first1 = MA. | last2 = Nemitz | first2 = JW. | last3 = Stein | first3 = BE. | title = Determinants of multisensory integration in superior colliculus neurons. I. Temporal factors. | journal = J Neurosci | volume = 7 | issue = 10 | pages = 3215–29 |date=Oct 1987 | pmid = 3668625 | pmc = 6569162 | doi = 10.1523/JNEUROSCI.07-10-03215.1987 }}</ref> states that multisensory integration is more likely or stronger when the constituent unisensory stimuli arise at approximately the same time.
*'''The principle of inverse effectiveness'''<ref name="Meredith 1983">{{cite journal |vauthors=Meredith MA, Stein BE |title=Interactions among converging sensory inputs in the superior colliculus |journal=Science |volume=221 |issue=4608 |pages=389–91 |date=July 1983 |pmid=6867718 |doi=10.1126/science.6867718|bibcode=1983Sci...221..389M }}</ref><ref name="Meredith Stein 1986">{{Cite journal  | last1 = Meredith | first1 = MA. | last2 = Stein | first2 = BE. | title = Visual, auditory, and somatosensory convergence on cells in superior colliculus results in multisensory integration. | journal = J Neurophysiol | volume = 56 | issue = 3 | pages = 640–62 |date=Sep 1986 | pmid = 3537225 | doi = 10.1152/jn.1986.56.3.640 }}</ref> states that multisensory integration is more likely or stronger when the constituent unisensory stimuli evoke relatively weak responses when presented in isolation.

===Perceptual and behavioral consequences ===
A unimodal approach dominated scientific literature until the beginning of this century. Although this enabled rapid progression of neural mapping, and an improved understanding of neural structures, the investigation of perception remained relatively stagnant, with a few exceptions.  The recent revitalized enthusiasm into perceptual research is indicative of a substantial shift away from reductionism and toward gestalt methodologies. Gestalt theory, dominant in the late 19th and early 20th centuries espoused two general principles: the 'principle of totality' in which conscious experience must be considered globally, and the 'principle of psychophysical isomorphism' which states that perceptual phenomena are correlated with cerebral activity. Just these ideas were already applied by [[Justo Gonzalo]] in his work of brain dynamics, where a sensory-cerebral correspondence is considered in the formulation of the  "development of the sensory field due to a psychophysical isomorphism" (pag. 23 of the English translation of ref.<ref name=Gradients />).  Both ideas 'principle of totality' and 'psychophysical isomorphism' are particularly relevant in the current climate and have driven researchers to investigate the behavioural benefits of multisensory integration.

===Decreasing sensory uncertainty===
It has been widely acknowledged that uncertainty in sensory domains results in an increased dependence of multisensory integration.<ref name="Alais 2004"/> Hence, it follows that cues from multiple modalities that are both temporally and spatially synchronous are viewed neurally and perceptually as emanating from the same source. The degree of synchrony that is required for this 'binding' to occur is currently being investigated in a variety of approaches.<ref>{{cite journal |last1=Cervantes Constantino |first1=F. |last2=Sánchez-Costa |first2=T. |last3=Cipriani |first3=G.A. |last4=Carboni |first4=A. |title=Visuospatial attention revamps cortical processing of sound amid audiovisual uncertainty |journal=Psychophysiology |date=2023 |volume=60 |issue=10 |pages=e14329 |doi=10.1111/psyp.14329 |pmid=37166096|s2cid=258617930 }}</ref> The integrative function only occurs to a point beyond which the subject can differentiate them as two opposing stimuli. Concurrently, a significant intermediate conclusion can be drawn from the research thus far. Multisensory stimuli that are bound into a single percept, are also bound on the same receptive fields of multisensory neurons in the SC and cortex.<ref name="Alais 2004"/>

===Decreasing reaction time===
Responses to multiple simultaneous sensory stimuli can be faster than responses to the same stimuli presented in isolation. Hershenson (1962) presented a light and tone simultaneously and separately, and asked human participants to respond as rapidly as possible to them. As the asynchrony between the onsets of both stimuli was varied, it was observed that for certain degrees of asynchrony, reaction times were decreased.<ref name="Hershenson 1962">{{cite journal |author=Hershenson M |title=Reaction time as a measure of intersensory facilitation |journal=J Exp Psychol |volume=63 |issue= 3|pages=289–93 |date=March 1962 |pmid=13906889 |doi= 10.1037/h0039516}}</ref> These levels of asynchrony were quite small, perhaps reflecting the temporal window that exists in multisensory neurons of the SC. Further studies have analysed the reaction times of saccadic eye movements;<ref name="Hughes 1994">{{Cite journal  | last1 = Hughes | first1 = HC. | last2 = Reuter-Lorenz | first2 = PA. | last3 = Nozawa | first3 = G. | last4 = Fendrich | first4 = R. | title = Visual-auditory interactions in sensorimotor processing: saccades versus manual responses. | journal = J Exp Psychol Hum Percept Perform | volume = 20 | issue = 1 | pages = 131–53 |date=Feb 1994 | doi = 10.1037/0096-1523.20.1.131 | pmid = 8133219 }}</ref> and more recently correlated these findings to neural phenomena.<ref name="Wallace 2004">{{cite journal|last1=Wallace|first1=Mark T.|title=The development of multisensory processes|journal=Cognitive Processing|volume=5|issue=2|year=2004|pages=69–83|issn=1612-4782|doi=10.1007/s10339-004-0017-z|s2cid=16710851}}</ref> In patients studied by  [[Justo Gonzalo|Gonzalo]],<ref name=Dinamica />  with lesions in the parieto-occipital cortex, the decrease in the reaction time to a given stimulus by means of intersensory facilitation was shown to be very remarkable.

====Redundant target effects====
The redundant target effect is the observation that people typically respond faster to double targets (two targets presented simultaneously) than to either of the targets presented alone. This difference in latency is termed the redundancy gain (RG).<ref name="Ridgway 2008">{{cite journal |vauthors=Ridgway N, Milders M, Sahraie A |title=Redundant target effect and the processing of colour and luminance |journal=Exp Brain Res |volume=187 |issue=1 |pages=153–60 |date=May 2008 |pmid=18264703 |doi=10.1007/s00221-008-1293-0 |s2cid=23092762 }}</ref>

In a study done by Forster, Cavina-Pratesi, Aglioti, and Berlucchi (2001), normal observers responded faster to simultaneous visual and
tactile stimuli than to single visual or tactile stimuli. RT to simultaneous visual and tactile stimuli was also faster than RT to simultaneous dual visual or tactile stimuli. The advantage for RT to combined visual-tactile stimuli over RT to the other types of stimulation could be accounted for by intersensory neural facilitation rather than by probability summation. These effects can be ascribed to the convergence of tactile and visual inputs onto neural centers which contain flexible multisensory representations of body parts.<ref name="Forster 2002">{{cite journal |vauthors=Forster B, Cavina-Pratesi C, Aglioti SM, Berlucchi G |title=Redundant target effect and intersensory facilitation from visual-tactile interactions in simple reaction time |journal=Exp Brain Res |volume=143 |issue=4 |pages=480–7 |date=April 2002 |pmid=11914794 |doi=10.1007/s00221-002-1017-9 |s2cid=115844 }}</ref>

===Multisensory illusions===

====McGurk effect====
It has been found that two converging bimodal stimuli can produce a perception that is not only different in magnitude than the sum of its parts, but also quite different in quality. In a classic study labeled the [[McGurk effect]],<ref name="McGurk 1976">{{cite journal |vauthors=McGurk H, MacDonald J |title=Hearing lips and seeing voices |journal=Nature |volume=264 |issue=5588 |pages=746–8 |year=1976 |pmid=1012311 |doi=10.1038/264746a0 |bibcode=1976Natur.264..746M |s2cid=4171157 }}</ref> a person's phoneme production was dubbed with a video of that person speaking a different phoneme.<ref>{{Cite journal | last1=Nath| first1=AR.| last2=Beauchamp| first2=MS.| title=A neural basis for interindividual differences in the McGurk effect, a multisensory speech illusion.| journal=NeuroImage| volume=59| issue =1| pages=781–7|date=Jan 2012| doi=10.1016/j.neuroimage.2011.07.024| pmid=21787869| pmc=3196040}}</ref> The result was the perception of a third, different phoneme. McGurk and MacDonald (1976) explained that phonemes such as ba, da, ka, ta, ga and pa can be divided into four groups, those that can be visually confused, i.e. (da, ga, ka, ta) and (ba and pa), and those that can be audibly confused. Hence, when ba – voice and ga lips are processed together, the visual modality sees ga or da, and the auditory modality hears ba or da, combining to form the percept da.<ref name="McGurk 1976"/>

====Ventriloquism====
[[Ventriloquism]] has been used as the evidence for the modality appropriateness hypothesis. The [[ventriloquism effect]] is the situation in which auditory location perception is shifted toward a visual cue. The original study describing this phenomenon was conducted by Howard and Templeton, (1966) after which several studies have replicated and built upon the conclusions they reached.<ref name="Hairston 2003">{{cite journal |vauthors=Hairston WD, Wallace MT, Vaughan JW, Stein BE, Norris JL, Schirillo JA |title=Visual localization ability influences cross-modal bias |journal=J Cogn Neurosci |volume=15 |issue=1 |pages=20–9 |date=January 2003 |pmid=12590840 |doi=10.1162/089892903321107792 |s2cid=13636325 }}</ref> In conditions in which the visual cue is unambiguous, [[visual capture]] reliably occurs. Thus to test the influence of sound on perceived location, the visual stimulus must be progressively degraded.<ref name="Alais 2004"/> Furthermore, given that auditory stimuli are more attuned to temporal changes, recent studies have tested the ability of temporal characteristics to influence the spatial location of visual stimuli. Some types of EVP – [[electronic voice phenomenon]], mainly the ones using sound bubbles are considered a kind of modern ventriloquism technique and is played by the use of sophisticated software, computers and sound equipment.

====Double-flash illusion====
The double flash illusion was reported as the first illusion to show that visual stimuli can be qualitatively altered by audio stimuli.<ref name="Shams 2000">{{cite journal |vauthors=Shams L, Kamitani Y, Shimojo S |title=Illusions. What you see is what you hear |journal=Nature |volume=408 |issue=6814 |pages=788 |date=December 2000 |pmid=11130706 |doi=10.1038/35048669 |bibcode=2000Natur.408..788S |s2cid=205012107 |doi-access=free }}</ref> In the standard paradigm participants are presented combinations of one to four flashes accompanied by zero to 4 beeps. They were then asked to say how many flashes they perceived. Participants perceived illusory flashes when there were more beeps than flashes. fMRI studies have shown that there is crossmodal activation in early, low level visual areas, which was qualitatively similar to the perception of a real flash. This suggests that the illusion reflects subjective perception of the extra flash.<ref name="Watkins 2007">{{cite journal |vauthors=Watkins S, Shams L, Josephs O, Rees G |title=Activity in human V1 follows multisensory perception |journal=NeuroImage |volume=37 |issue=2 |pages=572–8 |date=August 2007 |pmid=17604652 |doi=10.1016/j.neuroimage.2007.05.027 |s2cid=17477883 }}</ref> Further, studies suggest that timing of multisensory activation in unisensory cortexes is too fast to be mediated by a higher order integration suggesting feed forward or lateral connections.<ref name="Shams 2005">{{cite journal |vauthors=Shams L, Iwaki S, Chawla A, Bhattacharya J |title=Early modulation of visual cortex by sound: an MEG study |journal=Neurosci. Lett. |volume=378 |issue=2 |pages=76–81 |date=April 2005 |pmid=15774261 |doi=10.1016/j.neulet.2004.12.035 |s2cid=4675944 }}</ref> One study has revealed the same effect but from vision to audition, as well as fission rather than fusion effects, although the level of the auditory stimulus was reduced to make it less salient for those illusions affecting audition.<ref name="Andersen 2004">{{cite journal |vauthors=Andersen TS, Tiippana K, Sams M |title=Factors influencing audiovisual fission and fusion illusions |journal=Brain Res Cogn Brain Res |volume=21 |issue=3 |pages=301–308 |date=November 2004 |pmid=15511646 |doi=10.1016/j.cogbrainres.2004.06.004 }}</ref>

====Rubber hand illusion====
{{see also|Body transfer illusion#Rubber hand illusion}}
[[File:Rubber hand illusion.png|thumb|Schematic diagram of the experimental set-up in the rubber hand illusion task.]]
In the rubber hand illusion (RHI),<ref name="Botvinick 1998"/> human participants view a dummy hand being stroked with a paintbrush, while they feel a series of identical brushstrokes applied to their own hand, which is hidden from view. If this visual and tactile information is applied synchronously, and if the visual appearance and position of the dummy hand is similar to one's own hand, then people may feel that the touches on their own hand are coming from the dummy hand, and even that the dummy hand is, in some way, their own hand.<ref name="Botvinick 1998">{{cite journal |vauthors=Botvinick M, Cohen J |title=Rubber hands 'feel' touch that eyes see |journal=Nature |volume=391 |issue=6669 |pages=756 |date=February 1998 |pmid=9486643 |doi=10.1038/35784 |url=http://www.pni.princeton.edu/ncc/publications/1998/BotvinickCohen1998Nature.pdf|bibcode=1998Natur.391..756B |s2cid=205024516 |doi-access=free }}</ref> This is an early form of [[body transfer illusion]].

The RHI is an illusion of vision, touch, and posture ([[proprioception]]), but a similar illusion can also be induced with touch and proprioception.<ref name="Ehrsson 2005">{{cite journal |vauthors=Ehrsson HH, Holmes NP, Passingham RE |title=Touching a rubber hand: feeling of body ownership is associated with activity in multisensory brain areas |journal=J. Neurosci. |volume=25 |issue=45 |pages=10564–73 |date=November 2005 |pmid=16280594 |pmc=1395356 |doi=10.1523/JNEUROSCI.0800-05.2005 }}</ref> It has also been found that the illusion may not require tactile stimulation at all, but can be completely induced using mere vision of the rubber hand being in a congruent posture with the hidden real hand.<ref name="Samad 2015">{{cite journal |vauthors=Samad M, Chung A, Shams L |title=Perception of Body Ownership is Driven by Bayesian Sensory Inference |journal=PLOS ONE |date=February 2015 |volume=10 |issue=2 |doi=10.1371/journal.pone.0117178 |pages=e0117178 |pmid=25658822 |pmc=4320053|bibcode=2015PLoSO..1017178S |doi-access=free }}</ref>  

The first report of this kind of illusion may have been as early as 1937 (Tastevin, 1937).<ref name="Holmes 2004">{{cite journal |vauthors=Holmes NP, Crozier G, Spence C |title=When mirrors lie: "visual capture" of arm position impairs reaching performance |journal=Cogn Affect Behav Neurosci |volume=4 |issue=2 |pages=193–200 |date=June 2004 |pmid=15460925 |pmc=1314973 |doi=10.3758/CABN.4.2.193 |url=http://neurobiography.info/web_docs/holmes-et-al-2004-cabn.pdf}}</ref><ref>{{cite journal | url=https://gallica.bnf.fr/ark:/12148/bpt6k96512101/f67.item | author=J. Tastevin | title=En partant de l'expérience d'Aristote: Les déplacements artificiels des parties du corps ne sont pas suivis par le sentiment de ces parties ni par les sensations qu'on peut y produire | trans-title=Starting from Aristotle's experiment: The artificial displacements of parts of the body are not followed by feeling in these parts or by the sensations which can be produced there | journal={{ill|L'Encephale|fr}} | volume=32 | number=2 | pages=57&ndash;84 | date=Feb 1937 | language=fr }} &nbsp; ([https://psycnet.apa.org/record/1938-00125-001 English abstract])</ref><ref>{{cite journal | url=https://gallica.bnf.fr/ark:/12148/bpt6k96512101/f163.item | author=J. Tastevin | title=En partant de l'expérience d'Aristote | journal=L'Encephale | volume=32 | number=3 | pages=140&ndash;158 | date=Mar 1937 | language=fr }}</ref>

====Body transfer illusion====
[[Body transfer illusion]] typically involves the use of virtual reality devices to induce the illusion in the subject that the body of another person or being is the subject's own body.

==Neural mechanisms==

===Subcortical areas===

====Superior colliculus====
[[File:Slide5ff.JPG|thumb|Superior colliculus]]
The [[superior colliculus]] (SC) or optic tectum (OT) is part of the tectum, located in the midbrain, superior to the brainstem and inferior to the thalamus. It contains seven layers of alternating white and grey matter, of which the superficial contain topographic maps of the visual field; and deeper layers contain overlapping spatial maps of the visual, auditory and somatosensory modalities.<ref name="Affifi">{{cite book |author1=Bergman, Ronald A. |author2=Afifi, Adel K. |title=Functional neuroanatomy: text and atlas |publisher=McGraw-Hill |location=New York |year=2005 |isbn=978-0-07-140812-7 |oclc=475017241}}</ref> The structure receives afferents directly from the retina, as well as from various regions of the cortex (primarily the occipital lobe), the [[spinal cord]] and the inferior colliculus. It sends efferents to the spinal cord, cerebellum, thalamus and occipital lobe via the [[lateral geniculate nucleus]] (LGN). The structure contains a high proportion of multisensory neurons and plays a role in the motor control of orientation behaviours of the eyes, ears and head.<ref name="Wallace 2004"/>

Receptive fields from somatosensory, visual and auditory modalities converge in the deeper layers to form a two-dimensional multisensory map of the external world. Here, objects straight ahead are represented caudally and objects on the periphery are represented rosterally. Similarly, locations in superior sensory space are represented medially, and inferior locations are represented laterally.<ref name="Stein 1993"/>

However, in contrast to simple convergence, the SC integrates information to create an output that differs from the sum of its inputs. Following a phenomenon labelled the 'spatial rule', neurons are excited if stimuli from multiple modalities fall on the same or adjacent receptive fields, but are inhibited if the stimuli fall on disparate fields.<ref name="Giard 1999">{{cite journal |vauthors=Giard MH, Peronnet F |title=Auditory-visual integration during multimodal object recognition in humans: a behavioral and electrophysiological study |journal=J Cogn Neurosci |volume=11 |issue=5 |pages=473–90 |date=September 1999 |pmid=10511637 |doi= 10.1162/089892999563544|s2cid=5735865 }}</ref> Excited neurons may then proceed to innervate various muscles and neural structures to orient an individual's behaviour and attention toward the stimulus. Neurons in the SC also adhere to the 'temporal rule', in which stimulation must occur within close temporal proximity to excite neurons. However, due to the varying processing time between modalities and the relatively slower speed of sound to light, it has been found the neurons may be optimally excited when stimulated some time apart.<ref name="Miller 2005">{{cite journal |vauthors=Miller LM, D'Esposito M |title=Perceptual fusion and stimulus coincidence in the cross-modal integration of speech |journal=J. Neurosci. |volume=25 |issue=25 |pages=5884–93 |date=June 2005 |pmid=15976077 |pmc=6724802 |doi=10.1523/JNEUROSCI.0896-05.2005 }}</ref>

====Putamen====
Single neurons in the macaque putamen have been shown to have visual and somatosensory responses closely related to those in the polysensory zone of the premotor cortex and area 7b in the parietal lobe.<ref name="Graziano 1993">{{cite journal |vauthors=Graziano MS, Gross CG |title=A bimodal map of space: somatosensory receptive fields in the macaque putamen with corresponding visual receptive fields |journal=Exp Brain Res |volume=97 |issue=1 |pages=96–109 |year=1993 |pmid=8131835 |doi=10.1007/BF00228820 |s2cid=9387557 |url=http://neuroinformatics.usc.edu/mediawiki/images/3/39/Graziano_Gross_1993.pdf}}</ref><ref name="Gentile 2011">{{Cite journal  | last1 = Gentile | first1 = G. | last2 = Petkova | first2 = VI. | last3 = Ehrsson | first3 = HH. | title = Integration of visual and tactile signals from the hand in the human brain: an FMRI study. | journal = J Neurophysiol | volume = 105 | issue = 2 | pages = 910–22 |date=Feb 2011 | doi = 10.1152/jn.00840.2010 | pmid = 21148091 | pmc = 3059180 }}</ref>

===Cortical areas===
Multisensory neurons exist in a large number of locations, often integrated with unimodal neurons. They have recently been discovered in areas previously thought to be modality specific, such as the somatosensory cortex; as well as in clusters at the borders between the major cerebral lobes, such as the occipito-parietal space and the occipito-temporal space.<ref>Thesen, T., Vibell, J., Calvert, G.A., & Osterbauer, R. (2004). Neuroimaging of multisensory processing in vision, audition, touch and olfaction. Cognitive Processing, 5, 84-93.</ref><ref name="Wallace 2004"/><ref name="Wallace Ramachandran 2004">{{cite journal |vauthors=Wallace MT, Ramachandran R, Stein BE |title=A revised view of sensory cortical parcellation |journal=Proc. Natl. Acad. Sci. U.S.A. |volume=101 |issue=7 |pages=2167–72 |date=February 2004 |pmid=14766982 |pmc=357070 |doi=10.1073/pnas.0305697101 |bibcode=2004PNAS..101.2167W |doi-access=free }}</ref>

However, in order to undergo such physiological changes, there must exist continuous connectivity between these multisensory structures. It is generally agreed that information flow within the cortex follows a hierarchical configuration.<ref name="Clavagnier 2004">{{cite journal |vauthors=Clavagnier S, Falchier A, Kennedy H |title=Long-distance feedback projections to area V1: implications for multisensory integration, spatial awareness, and visual consciousness |journal=Cogn Affect Behav Neurosci |volume=4 |issue=2 |pages=117–26 |date=June 2004 |pmid=15460918 |doi=10.3758/CABN.4.2.117 |s2cid=6907281 |url=http://www.seco-project.eu/dev/labo/files/publications/clavagnier%2004%20cabneuro.pdf|doi-access=free }}</ref>  Hubel and Wiesel showed that receptive fields and thus the function of cortical structures, as one proceeds out from V1 along the visual pathways, become increasingly complex and specialized.<ref name="Clavagnier 2004"/> From this it was postulated that information flowed outwards in a feed-forward fashion; the complex end products eventually binding to form a percept. However, via fMRI and intracranial recording technologies, it has been observed that the activation time of successive levels of the hierarchy does not correlate with a feed-forward structure. That is, late activation has been observed in the striate cortex, markedly after activation of the prefrontal cortex in response to the same stimulus.<ref name="Foxe 2002">{{cite journal |vauthors=Foxe JJ, Simpson GV |title=Flow of activation from V1 to frontal cortex in humans. A framework for defining "early" visual processing |journal=Exp Brain Res |volume=142 |issue=1 |pages=139–50 |date=January 2002 |pmid=11797091 |doi=10.1007/s00221-001-0906-7 |s2cid=25506401 }}</ref>

Complementing this, afferent nerve fibres have been found that project to early visual areas such as the lingual gyrus from late in the dorsal (action) and ventral (perception) visual streams, as well as from the auditory association cortex.<ref name="Macaluso 2000">{{cite journal |vauthors=Macaluso E, Frith CD, Driver J |title=Modulation of human visual cortex by crossmodal spatial attention |journal=Science |volume=289 |issue=5482 |pages=1206–8 |date=August 2000 |pmid=10947990 |doi=10.1126/science.289.5482.1206 |citeseerx=10.1.1.420.5403 |bibcode=2000Sci...289.1206M }}</ref> Feedback projections have also been observed in the opossum directly from the auditory association cortex to V1.<ref name="Clavagnier 2004"/> This last observation currently highlights a point of controversy within the neuroscientific community. Sadato ''et al.'' (2004) concluded, in line with Bernstein ''et al.'' (2002), that the primary auditory cortex (A1) was functionally distinct from the auditory association cortex, in that it was void of any interaction with the visual modality. They hence concluded that A1 would not at all be effected by [[cross modal plasticity]].<ref name="Sadato 2004">{{cite journal  |vauthors=Sadato N, Yamada H, Okada T, etal |title=Age-dependent plasticity in the superior temporal sulcus in deaf humans: a functional MRI study |journal=BMC Neurosci |volume=5 |pages=56 |date=December 2004 |pmid=15588277 |pmc=539237 |doi=10.1186/1471-2202-5-56 |doi-access=free }}</ref><ref name="Bernstein 2002">{{cite journal |vauthors=Bernstein LE, Auer ET, Moore JK, Ponton CW, Don M, Singh M |title=Visual speech perception without primary auditory cortex activation |journal=NeuroReport |volume=13 |issue=3 |pages=311–5 |date=March 2002 |pmid=11930129  |doi=10.1097/00001756-200203040-00013|s2cid=44484836 }}</ref> This concurs with Jones and Powell's (1970) contention that primary sensory areas are connected only to other areas of the same modality.<ref name="Jones 1970">{{cite journal |vauthors=Jones EG, Powell TP |title=An anatomical study of converging sensory pathways within the cerebral cortex of the monkey |journal=Brain |volume=93 |issue=4 |pages=793–820 |year=1970 |pmid=4992433 |doi=10.1093/brain/93.4.793}}</ref>

In contrast, the dorsal auditory pathway, projecting from the temporal lobe is largely concerned with processing spatial information, and contains receptive fields that are topographically organized. Fibers from this region project directly to neurons governing corresponding receptive fields in V1.<ref name="Clavagnier 2004"/> The perceptual consequences of this have not yet been empirically acknowledged. However, it can be hypothesized that these projections may be the precursors of increased acuity and emphasis of visual stimuli in relevant areas of perceptual space. Consequently, this finding rejects Jones and Powell's (1970) hypothesis<ref name="Jones 1970"/> and thus is in conflict with Sadato ''et al.''{{'s}} (2004) findings.<ref name="Sadato 2004"/> A resolution to this discrepancy includes the possibility that primary sensory areas can not be classified as a single group, and thus may be far more different from what was previously thought.

The multisensory syndrome with symmetric bilaterality, characterized by [[Justo Gonzalo|Gonzalo]] and called by this author 'central syndrome of the cortex',<ref name=Dinamica /><ref name=Gradients /> was originated from a unilateral parieto-occipital cortical lesion equidistant from the visual, tactile, and auditory projection areas (the middle of area 19, the anterior part of area 18 and the most posterior of area 39, in Brodmann terminology) that was called 'central zone'. The gradation observed between syndromes led this author to propose a functional gradient scheme in which the specificity of the cortex is distributed with a continuous variation,<ref name=Gradients /> the overlap of the specific gradients would be high or maximum  in that 'central zone'.

====Frontal lobe====
{{context|section|date=July 2013}}
Area F4 in macaques

Area F5 in macaques<ref name="Grefkes 2005">{{Cite journal  | last1 = Grefkes | first1 = C. | last2 = Fink | first2 = GR. | title = The functional organization of the intraparietal sulcus in humans and monkeys. | journal = J Anat | volume = 207 | issue = 1 | pages = 3–17 |date=Jul 2005 | doi = 10.1111/j.1469-7580.2005.00426.x | pmid = 16011542 | pmc=1571496}}</ref><ref name="Murata 1997">{{Cite journal  | last1 = Murata | first1 = A. | last2 = Fadiga | first2 = L. | last3 = Fogassi | first3 = L. | last4 = Gallese | first4 = V. | last5 = Raos | first5 = V. | last6 = Rizzolatti | first6 = G. | title = Object representation in the ventral premotor cortex (area F5) of the monkey. | journal = J Neurophysiol | volume = 78 | issue = 4 | pages = 2226–30 |date=Oct 1997 | pmid = 9325390 | doi = 10.1152/jn.1997.78.4.2226 }}</ref>

Polysensory zone of premotor cortex (PZ) in macaques<ref name="Smiley 2009">{{Cite journal  | last1 = Smiley | first1 = JF. | last2 = Falchier | first2 = A. | title = Multisensory connections of monkey auditory cerebral cortex. | journal = Hear Res | volume = 258 | issue = 1–2 | pages = 37–46 |date=Dec 2009 | doi = 10.1016/j.heares.2009.06.019 | pmid = 19619628 |pmc = 2788085}}</ref>

====Occipital lobe====
{{context|section|date=July 2013}}
Primary visual cortex (V1)<ref name="Sharma 2003">{{Cite journal| last1=Sharma| first1=J.| last2=Dragoi| first2=V.| last3=Tenenbaum| first3=JB.| last4=Miller| first4=EK.| last5=Sur| first5=M. | title=V1 neurons signal acquisition of an internal representation of stimulus location.| journal=Science| volume=300| issue=5626| pages=1758–63|date=Jun 2003| doi=10.1126/science.1081721 | pmid=12805552 | bibcode=2003Sci...300.1758S| s2cid=14716502| url=https://zenodo.org/record/895347}}</ref>

Lingual gyrus in humans

Lateral occipital complex (LOC), including lateral occipital tactile visual area (LOtv)<ref name="Lacey 2009">{{Cite journal  | last1 = Lacey | first1 = S. | last2 = Tal | first2 = N. | last3 = Amedi | first3 = A. | last4 = Sathian | first4 = K. | title = A putative model of multisensory object representation. | journal = Brain Topogr | volume = 21 | issue = 3–4 | pages = 269–74 |date=May 2009 | doi = 10.1007/s10548-009-0087-4 | pmid = 19330441 | pmc = 3156680}}</ref>

====Parietal lobe====
{{context|section|date=July 2013}}
Ventral intraparietal sulcus (VIP) in macaques<ref name="Grefkes 2005"/>

Lateral intraparietal sulcus (LIP) in macaques<ref name="Grefkes 2005"/>

Area 7b in macaques<ref name="Neal 1990">{{Cite journal  | last1 = Neal | first1 = JW. | last2 = Pearson | first2 = RC. | last3 = Powell | first3 = TP. | title = The ipsilateral cortico-cortical connections of area 7b, PF, in the parietal and temporal lobes of the monkey. | journal = Brain Res | volume = 524 | issue = 1 | pages = 119–32 |date=Jul 1990 | doi = 10.1016/0006-8993(90)90500-B | pmid = 1698108 | s2cid = 24535669 }}</ref>

Second somatosensory cortex (SII)<ref name="Eickhoff 2006">{{Cite journal  | last1 = Eickhoff | first1 = SB. | last2 = Schleicher | first2 = A. | last3 = Zilles | first3 = K. | last4 = Amunts | first4 = K. | title = The human parietal operculum. I. Cytoarchitectonic mapping of subdivisions. | journal = Cereb Cortex | volume = 16 | issue = 2 | pages = 254–67 |date=Feb 2006 | doi = 10.1093/cercor/bhi105 | pmid = 15888607 | doi-access = free }}</ref>

====Temporal lobe====
{{context|section|date=July 2013}}
Primary auditory cortex (A1)

Superior temporal cortex (STG/STS/PT) Audio visual cross modal interactions are known to occur in the auditory association cortex which lies directly inferior to the Sylvian fissure in the temporal lobe.<ref name="Sadato 2004"/> Plasticity was observed in the superior temporal gyrus (STG) by Petitto ''et al.'' (2000).<ref name="Petitto 2000">{{cite journal |vauthors=Petitto LA, Zatorre RJ, Gauna K, Nikelski EJ, Dostie D, Evans AC |title=Speech-like cerebral activity in profoundly deaf people processing signed languages: implications for the neural basis of human language |journal=Proc. Natl. Acad. Sci. U.S.A. |volume=97 |issue=25 |pages=13961–6 |date=December 2000 |pmid=11106400 |doi=10.1073/pnas.97.25.13961 |pmc=17683|doi-access=free }}</ref> Here, it was found that the STG was more active during stimulation in native deaf signers compared to hearing non signers. Concurrently, further research has revealed differences in the activation of the Planum temporale (PT) in response to non linguistic lip movements between the hearing and deaf; as well as progressively increasing activation of the auditory association cortex as previously deaf participants gain hearing experience via a cochlear implant.<ref name="Sadato 2004"/>

Anterior ectosylvian sulus (AES) in cats<ref name="Meredith Clemo 1989">{{Cite journal | last1=Meredith| first1=MA.| last2=Clemo| first2=HR.| title=Auditory cortical projection from the anterior ectosylvian sulcus (Field AES) to the superior colliculus in the cat: an anatomical and electrophysiological study.| journal=J Comp Neurol | volume=289| issue=4| pages=687–707|date=Nov 1989 | doi=10.1002/cne.902890412| pmid=2592605| s2cid=221577963}}</ref><ref name="Jiang 2001">{{Cite journal | last1=Jiang| first1=W.| last2=Wallace| first2=MT.| last3=Jiang| first3=H.| last4=Vaughan| first4=JW.| last5=Stein| first5=BE.| title=Two cortical areas mediate multisensory integration in superior colliculus neurons.| journal=J Neurophysiol | volume=85| issue=2 | pages=506–22|date=Feb 2001 | pmid=11160489 | doi=10.1152/jn.2001.85.2.506| s2cid=2499047}}</ref><ref name="Wallace 2006">{{Cite journal | last1=Wallace| first1=MT.| last2=Carriere| first2=BN.| last3=Perrault| first3=TJ. | last4=Vaughan| first4=JW. | last5=Stein| first5=BE. | title=The development of cortical multisensory integration.| journal=Journal of Neuroscience | volume=26| issue=46| pages=11844–9|date=Nov 2006| doi=10.1523/JNEUROSCI.3295-06.2006| pmid=17108157 | pmc=6674880}}</ref>

Rostral lateral suprasylvian sulcus (rLS) in cats<ref name="Jiang 2001"/>

===Cortical-subcortical interactions===
The most significant interaction between these two systems (corticotectal interactions) is the connection between the anterior ectosylvian sulcus (AES), which lies at the junction of the parietal, temporal and frontal lobes, and the SC. The AES is divided into three unimodal regions with multisensory neurons at the junctions between these sections.<ref name="Jiang 2003">{{cite journal |vauthors=Jiang W, Stein BE |title=Cortex controls multisensory depression in superior colliculus |journal=J. Neurophysiol. |volume=90 |issue=4 |pages=2123–35 |date=October 2003 |pmid=14534263 |doi=10.1152/jn.00369.2003 }}</ref> (Jiang & Stein, 2003). Neurons from the unimodal regions project to the deep layers of the SC and influence the multiplicative integration effect. That is, although they can receive inputs from all modalities as normal, the SC can not enhance or depress the effect of multisensory stimulation without input from the AES.<ref name="Jiang 2003"/>

Concurrently, the multisensory neurons of the AES, although also integrally connected to unimodal AES neurons, are not directly connected to the SC. This pattern of division is reflected in other areas of the cortex, resulting in the observation that cortical and tectal multisensory systems are somewhat dissociated.<ref name="Wallace 1993">{{cite journal |vauthors=Wallace MT, Meredith MA, Stein BE |title=Converging influences from visual, auditory, and somatosensory cortices onto output neurons of the superior colliculus |journal=J. Neurophysiol. |volume=69 |issue=6 |pages=1797–809 |date=June 1993 |pmid=8350124 |doi= 10.1152/jn.1993.69.6.1797}}</ref> Stein, London, Wilkinson and Price (1996) analysed the perceived luminance of an LED in the context of spatially disparate auditory distracters of various types. A significant finding was that a sound increased the perceived brightness of the light, regardless of their relative spatial locations, provided the light's image was projected onto the fovea.<ref name="Stein London 1996">{{cite journal|last1=Stein|first1=Barry E.|last2=London|first2=Nancy|last3=Wilkinson|first3=Lee K.|last4=Price|first4=Donald D.|title=Enhancement of Perceived Visual Intensity by Auditory Stimuli: A Psychophysical Analysis|journal=Journal of Cognitive Neuroscience|volume=8| issue=6|year=1996| pages=497–506|doi=10.1162/jocn.1996.8.6.497|pmid=23961981|s2cid=43705477}}</ref> Here, the apparent lack of the spatial rule, further differentiates cortical and tectal multisensory neurons. Little empirical evidence exists to justify this dichotomy. Nevertheless, cortical neurons governing perception, and a separate sub cortical system governing action (orientation behavior) is synonymous with the perception action hypothesis of the visual stream.<ref name="Goodale 1992">{{cite journal |vauthors=Goodale MA, Milner AD |title=Separate visual pathways for perception and action |journal=Trends Neurosci. |volume=15 |issue=1 |pages=20–5 |date=January 1992 |pmid=1374953 |doi=10.1016/0166-2236(92)90344-8 |url=http://www.sybergroup.com/docs/Pathways2.pdf|citeseerx=10.1.1.207.6873 |s2cid=793980 }}</ref> Further investigation into this field is necessary before any substantial claims can be made.

===Dual "what" and "where" multisensory routes===
Research suggests the existence of two multisensory routes for "what" and "where".  The "what" route identifying the identity of things involving area [[Brodmann area 9]] in the right [[inferior frontal gyrus]] and right [[middle frontal gyrus]], [[Brodmann area 13]] and [[Brodmann area 45]] in the right [[Insular cortex|insula]]-inferior frontal gyrus area, and [[Brodmann area 13]] bilaterally in the insula. The  "where" route detecting their spatial attributes involving the [[Brodmann area 40]] in the right and left [[inferior parietal lobule]] and the [[Brodmann area 7]] in the right [[precuneus]]-[[superior parietal lobule]] and [[Brodmann area 7]] in
the left superior parietal lobule.<ref name="Renier 2009">{{cite journal |vauthors=Renier LA, Anurova I, De Volder AG, Carlson S, VanMeter J, Rauschecker JP |title=Multisensory integration of sounds and vibrotactile stimuli in processing streams for "what" and "where" |journal=J. Neurosci. |volume=29 |issue=35 |pages=10950–60 |date=September 2009 |pmid=19726653 |pmc=3343457 |doi=10.1523/JNEUROSCI.0910-09.2009 }}</ref>

==Development of multisensory operations==

=== Theories of development ===

All species equipped with multiple sensory systems, utilize them in an integrative manner to achieve action and perception.<ref name="Stein 1993"/> However, in most species, especially higher mammals and humans, the ability to integrate develops in parallel with physical and cognitive maturity. Children until certain ages do not show mature integration patterns.<ref name="Nardini et al. 2010" /><ref name="Gori et al. 2008" /> Classically, two opposing views that are principally modern manifestations of the nativist/empiricist dichotomy have been put forth. The integration (empiricist) view states that at birth, sensory modalities are not at all connected. Hence, it is only through active exploration that plastic changes can occur in the nervous system to initiate holistic perceptions and actions. Conversely, the differentiation (nativist) perspective asserts that the young nervous system is highly interconnected; and that during development, modalities are gradually differentiated as relevant connections are rehearsed and the irrelevant are discarded.<ref name="Lewkowicz 2004">{{cite book | last1 = Lewkowicz | first1 = D | last2 = Kraebel | first2 = K | title = The value of multisensory redundancy in the development of intersensory perception | editor = Gemma Calvert | editor2 = Charles Spence | editor3 = Barry E Stein | work = The Handbook of multisensory processes | publisher = MIT Press, cop. | year = 2004 | location = Cambridge, Mass | pages = [https://archive.org/details/handbookofmultis0000unse/page/655 655–78] | isbn = 9780262033213 | oclc = 803222288 | url = https://archive.org/details/handbookofmultis0000unse/page/655 }}</ref>

Using the SC as a model, the nature of this dichotomy can be analysed. In the newborn cat, deep layers of the SC contain only neurons responding to the somatosensory modality. Within a week, auditory neurons begin to occur, but it is not until two weeks after birth that the first multisensory neurons appear. Further changes continue, with the arrival of visual neurons after three weeks, until the SC has achieved its fully mature structure after three to four months. Concurrently in species of monkey, newborns are endowed with a significant complement of multisensory cells; however, along with cats there is no integration effect apparent until much later.<ref name="Wallace 2004"/> This delay is thought to be the result of the relatively slower development of cortical structures including the AES; which as stated above, is essential for the existence of the integration effect.<ref name="Jiang 2003"/>

Furthermore, it was found by Wallace (2004) that cats raised in a light deprived environment had severely underdeveloped visual receptive fields in deep layers of the SC.<ref name="Wallace 2004"/> Although, receptive field size has been shown to decrease with maturity, the above finding suggests that integration in the SC is a function of experience. Nevertheless, the existence of visual multisensory neurons, despite a complete lack of visual experience, highlights the apparent relevance of nativist viewpoints. Multisensory development in the cortex has been studied to a lesser extent, however a similar study to that presented above was performed on cats whose optic nerves had been severed. These cats displayed a marked improvement in their ability to localize stimuli through audition; and consequently also showed increased neural connectivity between V1 and the auditory cortex.<ref name="Clavagnier 2004"/>  Such plasticity in early childhood allows for greater adaptability, and thus more normal development in other areas for those with a sensory deficit.

In contrast, following the initial formative period, the SC does not appear to display any neural plasticity. Despite this, habituation and sensititisation over the long term is known to exist in orientation behaviors. This apparent plasticity in function has been attributed to the adaptability of the AES. That is, although neurons in the SC have a fixed magnitude of output per unit input, and essentially operate an all or nothing response, the level of neural firing can be more finely tuned by variations in input by the AES.

Although there is evidence for either perspective of the integration/differentiation dichotomy, a significant body of evidence also exists for a combination of factors from either view. Thus, analogous to the broader nativist/empiricist argument, it is apparent that rather than a dichotomy, there exists a continuum, such that the integration and differentiation hypotheses are extremes at either end.

=== Psychophysical development of integration ===
Not much is known about the development of the ability to integrate multiple estimates such as vision and touch.<ref name="Nardini et al. 2010"/> Some multisensory abilities are present from early infancy, but it is not until children are eight years or older before they use multiple modalities to reduce sensory uncertainty.<ref name="Nardini et al. 2010"/>
 
One study demonstrated that cross-modal visual and auditory integration is present from within 1 year of life.<ref name="Neil et al. 2006">{{cite journal|last=Neil|first=PA|author2=Chee-Ruiter, C |author3=Scheier, C |author4=Lewkowicz, DJ |author5= Shimojo, S |title=Development of multisensory spatial integration and perception in humans.|journal=Developmental Science|date=Sep 2006|volume=9|issue=5|pages=454–64|pmid=16911447|doi=10.1111/j.1467-7687.2006.00512.x|s2cid=25690976 |url=https://authors.library.caltech.edu/101479/}}</ref> This study measured response time for orientating towards a source. Infants who were 8–10 months old showed significantly decreased response times when the source was presented through both [[visual perception|visual]] and [[Hearing|auditory]] information compared to a single [[Stimulus modality|modality]]. Younger infants, however, showed no such change in response times to these different conditions. Indeed, the results of the study indicates that children potentially have the capacity to integrate sensory sources at any age. However, in certain cases, for example [[visual cues#visual cues|visual cues]], intermodal integration is avoided.<ref name="Nardini et al. 2010" /><br />
Another study found that cross-modal integration of [[Haptic perception|touch]] and [[visual perception|vision]] for distinguishing size and [[Orientation (mental)|orientation]] is available from at least 8 years of age.<ref name="Gori et al. 2008">{{cite journal|last=Gori|first=M |author2=Del Viva, M |author3=Sandini, G |author4=Burr, DC |title=Young children do not integrate visual and haptic form information.|journal=Current Biology|date=May 6, 2008|volume=18|issue=9|pages=694–8|pmid=18450446|doi=10.1016/j.cub.2008.04.036|s2cid=13899031 |url=https://api.research-repository.uwa.edu.au/files/1490966/11970_PID11970.pdf |doi-access=free}}</ref>  For pre-integration age groups, one sense dominates depending on the characteristic discerned (see [[#Visual dominance|visual dominance]]).<ref name="Gori et al. 2008"/>

A study investigating sensory integration ''within'' a single [[Stimulus modality|modality]] ([[Visual perception|vision]]) found that it cannot be established until age 12 and above.<ref name="Nardini et al. 2010">{{cite journal|last=Nardini|first=M|author2=Bedford, R |author3=Mareschal, D |title=Fusion of visual cues is not mandatory in children.|journal=Proceedings of the National Academy of Sciences of the United States of America|date=Sep 28, 2010|volume=107|issue=39|pages=17041–6|pmid=20837526|doi=10.1073/pnas.1001699107|pmc=2947870|bibcode=2010PNAS..10717041N|doi-access=free}}</ref> This particular study assessed the integration of [[Binocular disparity|disparity]] and [[Texture gradient|texture]] cues to resolve surface slant. Though younger age groups showed a somewhat better performance when combining disparity and texture cues compared to using only disparity or texture cues, this difference was not statistically significant.<ref name="Nardini et al. 2010"/> In adults, the sensory integration can be mandatory, meaning that they no longer have access to the individual sensory sources.<ref name="Hills et al. 2002">{{cite journal|last=Hillis|first=JM|author2=Ernst, MO |author3=Banks, MS |author4= Landy, MS |title=Combining sensory information: mandatory fusion within, but not between, senses.|journal=Science|date=Nov 22, 2002|volume=298|issue=5598|pages=1627–30|pmid=12446912|doi=10.1126/science.1075396|citeseerx=10.1.1.278.6134|bibcode=2002Sci...298.1627H|s2cid=15607270}}</ref>

Acknowledging these variations, many hypotheses have been established to reflect why these observations are task-dependent. Given that different senses develop at different rates, it has been proposed that cross-modal integration does not appear until both [[Stimulus modality|modalities]] have reached maturity.<ref name="Gori et al. 2008"/><ref name="Ernst 2008">{{cite journal|last=Ernst|first=MO|title=Multisensory integration: a late bloomer.|journal=Current Biology|date=Jun 24, 2008|volume=18|issue=12|pages=R519–21|pmid=18579094|doi=10.1016/j.cub.2008.05.002|s2cid=130911|doi-access=free}}</ref> The human body undergoes significant physical transformation throughout childhood. Not only is there growth in size and [[human height|stature]] (affecting viewing height), but there is also change in inter-ocular distance and [[eyeball]] length. Therefore, sensory signals need to be constantly re-evaluated to appreciate these various physiological changes.<ref name="Gori et al. 2008"/> Some support comes from animal studies that explore the neurobiology behind integration. Adult monkeys have deep inter-neuronal connections within the [[superior colliculus]] providing strong, accelerated visuo-auditory integration.<ref>{{Cite book|last=Stein|first=BE|author2=Meredith, MA |author3=Wallace, MT |title=Chapter 8 the visually responsive neuron and beyond: Multisensory integration in cat and monkey |series=Progress in Brain Research|year=1993|volume=95|pages=79–90|pmid=8493355|doi=10.1016/s0079-6123(08)60359-3|isbn=9780444894922}}</ref> Young animals conversely, do not have this enhancement until unimodal properties are fully developed.<ref>{{cite journal|last=Stein|first=BE|author2=Labos, E |author3=Kruger, L |title=Sequence of changes in properties of neurons of superior colliculus of the kitten during maturation.|journal=Journal of Neurophysiology|date=Jul 1973|volume=36|issue=4|pages=667–79|pmid=4713313|doi=10.1152/jn.1973.36.4.667}}</ref><ref>{{cite journal|last=Wallace|first=MT|author2=Stein, BE|title=Sensory and multisensory responses in the newborn monkey superior colliculus.|journal=The Journal of Neuroscience|date=Nov 15, 2001|volume=21|issue=22|pages=8886–94|pmid=11698600|pmc=6762279|doi=10.1523/JNEUROSCI.21-22-08886.2001}}</ref>

Additionally, to rationalize sensory dominance, Gori et al. (2008) advocates that the brain utilises the most direct source of information during sensory immaturity.<ref name="Gori et al. 2008"/> In this case, [[Orientation (mental)|orientation]] is primarily a visual characteristic. It can be derived directly from the object image that forms on the [[retina]], irrespective of other visual factors. In fact, data shows that a functional property of [[neurons]] within [[primate]] [[visual cortex|visual cortices']] are their discernment to orientation.<ref>{{cite journal|last=Tootell|first=RB|author2=Hadjikhani, NK |author3=Vanduffel, W |author4=Liu, AK |author5=Mendola, JD |author6=Sereno, MI |author7= Dale, AM |title=Functional analysis of primary visual cortex (V1) in humans.|journal=Proceedings of the National Academy of Sciences of the United States of America|date=Feb 3, 1998|volume=95|issue=3|pages=811–7|pmid=9448245|doi=10.1073/pnas.95.3.811 |pmc=33802|bibcode=1998PNAS...95..811T|doi-access=free}}</ref> In contrast, [[haptic perception|haptic]] orientation judgements are recovered through collaborated patterned stimulations, evidently an indirect source susceptible to interference. Likewise, when size is concerned [[haptic perception|haptic]] information coming from positions of the fingers is more immediate. Visual-size perceptions, alternatively, have to be computed using parameters such as [[angle|slant]] and [[distance]]. Considering this, sensory dominance is a useful instinct to assist with calibration. During sensory immaturity, the more simple and robust information source could be used to tweak the accuracy of the alternate source.<ref name="Gori et al. 2008"/> Follow-up work by Gori et al. (2012) showed that, at all ages, vision-size perceptions are near perfect when viewing objects within the [[haptic perception|haptic]] workspace (i.e. at arm's reach).<ref name="Gori et al. 2012">{{cite journal|last=Gori|first=M|author2=Giuliana, L |author3=Sandini, G |author4= Burr, D |title=Visual size perception and haptic calibration during development.|journal=Developmental Science|date=Nov 2012|volume=15|issue=6|pages=854–62|pmid=23106739|doi=10.1111/j.1467-7687.2012.2012.01183.x}}</ref> However, systematic errors in perception appeared when the object was positioned beyond this zone.<ref>{{cite journal|last=Granrud|first=CE|author2=Schmechel, TT|title=Development of size constancy in children: a test of the proximal mode sensitivity hypothesis.|journal=Perception & Psychophysics|date=Nov 2006|volume=68|issue=8|pages=1372–81|pmid=17378423|doi=10.3758/bf03193736|doi-access=free}}</ref> Children younger than 14 years tend to underestimate object size, whereas adults overestimated. However, if the object was returned to the haptic workspace, those visual biases disappeared.<ref name="Gori et al. 2012"/> These results support the hypothesis that haptic information may educate visual perceptions. If sources are used for cross-calibration they cannot, therefore, be combined (integrated). Maintaining access to individual estimates is a trade-off for extra [[neuroplasticity|plasticity]] over accuracy, which could be beneficial in retrospect to the developing body.<ref name="Gori et al. 2008"/><ref name="Ernst 2008"/>
 
Alternatively, Ernst (2008) advocates that efficient integration initially relies upon establishing correspondence – which sensory signals belong together.<ref name="Ernst 2008"/> Indeed, studies have shown that visuo-haptic integration fails in adults when there is a perceived [[space|spatial]] separation, suggesting sensory information is coming from different targets.<ref>{{cite journal|last=Gepshtein|first=S|author2=Burge, J |author3=Ernst, MO |author4= Banks, MS |title=The combination of vision and touch depends on spatial proximity.|journal=Journal of Vision|date=Dec 28, 2005|volume=5|issue=11|pages=1013–23|pmid=16441199 |doi=10.1167/5.11.7 |pmc=2632311}}</ref> Furthermore, if the separation can be explained, for example viewing an object through a [[mirror]], integration is re-established and can even be optimal.<ref>{{cite journal|last=Helbig|first=HB|author2=Ernst, MO|title=Optimal integration of shape information from vision and touch.|journal=Experimental Brain Research|date=Jun 2007|volume=179|issue=4|pages=595–606|pmid=17225091|doi=10.1007/s00221-006-0814-y|s2cid=12049308}}</ref><ref>{{cite journal|last=Helbig|first=HB|author2=Ernst, MO|title=Knowledge about a common source can promote visual- haptic integration.|journal=Perception|year=2007|volume=36|issue=10|pages=1523–33|pmid=18265835|doi=10.1068/p5851|s2cid=14884284}}</ref> Ernst (2008) suggests that adults can obtain this knowledge from previous experiences to quickly determine which sensory sources depict the same target, but young children could be deficient in this area.<ref name="Ernst 2008"/> Once there is a sufficient bank of experiences, confidence to correctly integrate sensory signals can then be introduced in their behaviour.

Lastly, Nardini et al. (2010) recently hypothesised that young children have optimized their sensory appreciation for speed over accuracy.<ref name="Nardini et al. 2010" /> When information is presented in two forms, children may derive an estimate from the fastest available source, subsequently ignoring the alternate, even if it contains redundant information. Nardini et al. (2010) provides evidence that children's (aged 6 years) response latencies are significantly lower when stimuli are presented in multi-cue over single-cue conditions.<ref name="Nardini et al. 2010" /> Conversely, adults showed no change between these conditions. Indeed, adults display mandatory fusion of signals, therefore they can only ever aim for maximum accuracy.<ref name="Nardini et al. 2010" /><ref name="Hills et al. 2002" /> However, the overall mean latencies for children were not faster than adults, which suggests that speed optimization merely enable them to keep up with the mature pace. Considering the haste of real-world events, this strategy may prove necessary to counteract the general slower processing of children and maintain effective vision-action coupling.<ref>{{cite journal|last=Kail|first=RV|author2=Ferrer, E|title=Processing speed in childhood and adolescence: longitudinal models for examining developmental change.|journal=Child Development|date=Nov–Dec 2007|volume=78|issue=6|pages=1760–70|pmid=17988319|doi=10.1111/j.1467-8624.2007.01088.x}}</ref><ref>{{cite journal|last=Kail|first=R|title=Developmental change in speed of processing during childhood and adolescence.|journal=Psychological Bulletin|date=May 1991|volume=109|issue=3|pages=490–501|pmid=2062981|doi=10.1037/0033-2909.109.3.490}}</ref><ref>{{cite journal|last=Ballard|first=DH|author2=Hayhoe, MM |author3=Pook, PK |author4= Rao, RP |title=Deictic codes for the embodiment of cognition.|journal=The Behavioral and Brain Sciences|date=Dec 1997|volume=20|issue=4|pages=723–42; discussion 743–67|pmid=10097009|doi=10.1017/s0140525x97001611|citeseerx=10.1.1.49.3813|s2cid=1961389 }}</ref> Ultimately the developing sensory system may preferentially adapt for different goals – ''speed and detecting sensory conflicts'' – those typical of objective learning.

The late development of efficient integration has also been investigated from computational point of view.<ref>{{Cite journal|title = Reward Maximization Justifies the Transition from Sensory Selection at Childhood to Sensory Integration at Adulthood|journal = PLOS ONE|pmc = 4110011|pmid = 25058591|volume = 9|issue = 7|doi = 10.1371/journal.pone.0103143|first1 = Pedram|last1 = Daee|first2 = Maryam S.|last2 = Mirian|first3 = Majid Nili|last3 = Ahmadabadi|pages=e103143|year = 2014|bibcode = 2014PLoSO...9j3143D|doi-access = free}}</ref> Daee et al. (2014) showed that having one dominant sensory source at early age, rather than integrating all sources, facilitates the overall development of cross-modal integrations.

=== Applications ===

==== Prosthesis ====

Prosthetics designers should carefully consider the nature of dimensionality alteration of sensorimotor signaling from and to the [[Central nervous system|CNS]] when designing prosthetic devices. As reported in literatures, neural signaling from the CNS to the motors is organized in a way that the dimensionalities of the signals are gradually increased as you approach the muscles, also called muscle synergies. In the same principal, but in opposite ordering, on the other hand, signals dimensionalities from the sensory receptors are gradually integrated, also called sensory synergies, as they approaches the CNS. This bow tie like signaling formation enables the CNS to process abstract yet valuable information only. Such as process will decrease complexity of the data, handle the noises and guarantee to the CNS the optimum energy consumption. 
Although the current commercially available prosthetic devices mainly focusing in implementing the motor side by simply uses EMG sensors to switch between different activation states of the prosthesis. Very limited works have proposed a system to involve by integrating the sensory side. The integration of tactile sense and proprioception is regarded as essential for implementing the ability to perceive environmental input.<ref>{{cite journal |vauthors=Rincon-Gonzalez L, WarrenJ P | year = 2011 | title = Haptic interaction of touch and proprioception: implications for neuroprosthetics | journal = IEEE Trans. Neural Syst. Rehabil. Eng. | volume = 19 | issue = 5| pages = 490–500 | doi=10.1109/tnsre.2011.2166808| pmid = 21984518 | s2cid = 20575638 }}</ref>

==== Visual rehabilitation ====

Multisensory integration has also been shown to ameliorate visual hemianopia.  Through the repeated presentation of multisensory stimuli in the blind hemifield, the ability to respond to purely visual stimuli gradually returns to that hemifield in a central to peripheral manner.  These benefits persist even after the explicit multisensory training ceases.<ref>{{Cite journal|last1=Jiang|first1=Huai|last2=Stein|first2=Barry E.|last3=McHaffie|first3=John G.|date=2015-05-29|title=Multisensory training reverses midbrain lesion-induced changes and ameliorates haemianopia|journal=Nature Communications|volume=6|pages=7263|doi=10.1038/ncomms8263|issn=2041-1723|pmc=6193257|pmid=26021613|bibcode=2015NatCo...6.7263J}}</ref>

==See also==
*[[Body transfer illusion]]
*[[Sensory processing disorder]]
*[[Synesthesia]]

==References==
{{reflist|30em}}

==Further reading==
*{{Cite journal  | last1 = Kujala | first1 = T. | last2 = Alho | first2 = K. | last3 = Huotilainen | first3 = M. | last4 = Ilmoniemi | first4 = RJ. | last5 = Lehtokoski | first5 = A. | last6 = Leinonen | first6 = A. | last7 = Rinne | first7 = T. | last8 = et | first8 = al. | title = Electrophysiological evidence for cross-modal plasticity in humans with early- and late-onset blindness. | journal = Psychophysiology | volume = 34 | issue = 2 | pages = 213–6 |date=Mar 1997 | doi = 10.1111/j.1469-8986.1997.tb02134.x  | pmid = 9090272 }}
*{{cite book | last1 = Pascual-Leone  | first1 = A. | last2 = Theoret  | first2 = H. | last3 = et | first3 = al. | title = The Role of Visual Cortex in Tactile Processing: A Metamodal Brain. |editor = Morton A Heller |editor2=Soledad Ballesteros | work = Touch and blindness : psychology and neuroscience | publisher = Lawrence Erlbaum Associates | year = 2006 | location = Mahwah, N.J.  | isbn = 9780805847260 |oclc = 6124743}}
*{{cite book |author1=Wallace, Mark |author2=Murray, Micah Middelmann |title=The Neural Bases of Multisensory Processes (Frontiers in Neuroscience) |publisher=CRC Press |location=Boca Raton |year=2012 |isbn=978-1-4398-1217-4 |oclc=707710852|url=https://www.ncbi.nlm.nih.gov/books/NBK92848/}}

==External links==
*[http://www.sciencedirect.com/science/journal/03785955/258/1 Hearing Research special edition 2009 '''"Multisensory integration in auditory and auditory-related areas of cortex"''']

<!--Categories listed below-->

[[Category:Perception]]
[[Category:Sensory systems]]
[[Category:Neurology]]