Jump to ContentJump to Main Navigation
Sensory Substitution and Augmentation$

Fiona Macpherson

Print publication date: 2018

Print ISBN-13: 9780197266441

Published to British Academy Scholarship Online: September 2019

DOI: 10.5871/bacad/9780197266441.001.0001

Show Summary Details
Page of

PRINTED FROM BRITISH ACADEMY SCHOLARSHIP ONLINE (www.britishacademy.universitypressscholarship.com). (c) Copyright British Academy, 2020. All Rights Reserved. An individual user may print out a PDF of a single chapter of a monograph in BASO for personal use.date: 01 June 2020

Rewired Animals and Sensory Substitution: The Cause Is Not Cortical Plasticity

Rewired Animals and Sensory Substitution: The Cause Is Not Cortical Plasticity

(p.167) 10 Rewired Animals and Sensory Substitution: The Cause Is Not Cortical Plasticity
Sensory Substitution and Augmentation

J. Kevin O’regan

, Fiona Macpherson
British Academy

Abstract and Keywords

Cortical plasticity is often invoked to explain changes in the quality or location of experience observed in rewired animals, in sensory substitution, in extension of the body through tool use, and in the rubber hand illusion. However this appeal to cortical plasticity may be misleading, because it suggests that the cortical areas that are plastic are themselves the loci of generation of experience. This would be an error, I claim, since cortical areas do not generate experience. Cortical areas participate in enabling the interaction of an agent with its environment, and the quality of this interaction constitutes the quality of experience. Thus it is not plasticity in itself, but the change in modes of interaction which plasticity allows, which gives rise to the change of experience observed in these studies.

Keywords:   plasticity, sensory substitution, rewired animals

WILIAM JAMES SAID THAT if we were able to re-direct nerves from the eye so that they connected to the brain centres concerned with hearing, and vice versa, then we would ‘hear the lightning and see the thunder’ (James 1890).

Underlying this statement is the idea that what determines the quality of sensations is the particular brain area that processes the neural influx. Since James’ time, an enormous literature has lent support to this idea, including several iconic results. Among these is Penfield’s demonstration that tactile sensations in different body locations can be elicited by stimulation of corresponding areas of a somatosensory map or ‘Penfield homunculus’, and that stimulation of other areas can actually generate visual or auditory experiences (Penfield 1958).

Since those days the idea of ‘localization of function’ in the brain has become a doctrine (cf. Phillips et al. 1984), with a large portion of neuroscience research devoted to determining the particular functions of the different brain areas.

Applied to perception, the localization of function doctrine is often taken to suggest that there is something special about each kind of cortex that underlies (p.168) the particular quality of sensations that that kind of cortex allows. Just as there is something special about Penfield’s somatosensory cortex which allows it to generate tactile sensations, there is something special about visual, auditory, etc., cortices which allows them to generate visual, auditory, etc., sensations.

Within this way of thinking, the studies of ‘rewired’ animals and visual cortex in blind individuals have added an interesting twist, which reveals a subtle issue concerning the causal status of the notion of cortical plasticity.

10.1 Rewired Animals and Visual Cortex in Blind Persons

The studies of rewired animals investigate what happens when animals are surgically rewired at birth in such a way that information from the eyes, usually transmitted to visual cortex, is re-routed so that it projects into auditory cortex (Frost et al. 2000; Pallas et al. 1990; Pallas and Sur 1993; Sharma et al. 2000). The results show that when mature, the rewired animals are able to perform visual behaviours almost as efficiently as normal animals using their visual cortex. Furthermore, the rewired auditory cortex reorganizes and displays a two-dimensional orientation-sensitive structure similar to that observed in normal visual cortex.

Of particular interest is one very ingenious study which was able to approach an answer to the question of whether the quality of the sensations experienced by such rewired animals via their auditory cortex was actually visual rather than auditory (Von Melchner et al. 2000). The authors used ferrets that were only rewired in one hemisphere. They first trained these animals to distinguish between sounds and lights that were presented in such a way as to be processed only by the normal hemisphere. When completely trained, the animals were tested with lights that were processed only by the rewired hemisphere, that is, through auditory cortex. The animals invariably turned in a way suggesting that they perceived the lights visually, even though the perception occurred via their auditory cortex. When the auditory cortex was removed, the animals behaved as though they were blind.

Note that the authors did not observe any lateral connections between the ‘converted’ auditory cortex and existing visual cortex. Thus it truly appears that the visual sensations were related to the presence of the auditory cortex, and not to stimulation spreading to visual cortex. We thus seem to have a case in which, contrary to William James’ prediction, it seems that the animals end up ‘seeing the lightning rather than hearing it’.

Another well-publicized result showing adaptation of sensation generated in cortical areas has emerged recently in studies on blind people. For such persons it has been shown that visual cortex can rewire itself so as to participate in other functions, including language processing, memory, motion detection and navigation. The results are particularly strong in the congenitally blind, where visual cortex has never been exposed to visual stimulation. Of interest as regards the question of the (p.169) experienced quality of sensations are the results of Kupers et al. (2006) and Ptito et al. (2008). They trained subjects to use a sensory substitution device called the ‘tongue display unit’, which converts visual stimuli captured by a video camera into tactile stimulation on the tongue. After learning to do orientation, form or motion discrimination tasks with the device, early blind subjects given transcranial magnetic stimulation (TMS) directed onto their visual cortex felt tingling sensations on their tongue. Visual cortex thus seems now to underlie the generation of tactile sensations on the tongue.

10.2 Cortical Plasticity

All these results show that whatever it is about a particular cortical area that gives rise to a particular quality of sensation (e.g. visual, auditory, tactile), it does not lie in the specific cortical area or type of neurons, but in something that can be modified as a function of adaptation to extrinsic stimulation. As noted by Hurley and Noë (2003), the cortex ‘defers’ to the particular sensory input channel.

Such a modification is generally referred to as being an instance of ‘cortical plasticity’. Researchers will tend to say that cortical plasticity has allowed auditory cortex to change its function, going from generating auditory sensations to generating visual sensations. The implicit underlying assumption is that cortex generates sensations, and so to have different sensations, cortex must change its mode of functioning.

But to say this is to fall into a ‘category mistake’ (Ryle 1949). For suppose it made sense to say that different cortical areas ‘generated’ different types of sensations. Then we would be faced with the problem of explaining exactly how this came about. Exactly what is it about the different areas that allows them to generate the particular sensations they provide? This is the vexed problem of ‘qualia’, for which a variety of esoteric hypotheses have been advanced, going from quantum gravity effects in microtubules, through synchronous oscillations in widespread cortical areas, to cortico-thalamic reverberations—see the list provided by Chalmers (1997). For any such hypothesis, as noted by Dennett (1991), an infinite regress of questions (‘And then what happens?’) can always be asked about what exactly it is about these mechanisms that produces feel rather than no feel. And why are the feels the way they are, rather than some other way? Any explanation for similarities and differences in feels that is made in terms of physico-chemical mechanisms requires making a link between the language of neural processes and the language of sensations: and there is no non-arbitrary way to make that link.

For that reason the sensorimotor approach (O’Regan and Noë 2001; O’Regan 2011) proposes to reject the notion that sensations are the kinds of things that can be generated. The sensorimotor approach proposes instead that it is scientifically more fruitful to consider the qualities of sensations as laws of interaction (p.170) with the world. For example the sensation of softness consists in the law that links the sensory inputs you receive from your fingers with the pressure you exert on the sponge. The sponge is soft if it squishes under your fingers. Taking this stance is not just a philosophical trick, it actually leads to scientific predictions and paradigms, like the work on change blindness, inattentional blindness, sensory substitution and colour psychophysics.

In conclusion, if the sensorimotor approach is correct, then it is misleading to say that in rewired animals cortical plasticity has allowed ‘auditory cortex to change its function, going from generating auditory sensations to generating visual sensations’.

10.3 The Real Cause

What then is the correct thing to say about what happens with rewired animals?

We must say that the animal has become able to use its auditory cortex instead of its visual cortex in overseeing the sensorimotor interactions it has with the world using its eyes. Initially it tries to react appropriately to input from the eyes, but because at first auditory cortex is not wired up to do the task efficiently, the animal does not process the input or react quite as efficiently as if it were using visual cortex. But gradually there is learning, and auditory cortex reorganizes itself to do the task more efficiently. When things stabilize, however, it would be a mistake to say that auditory cortex is ‘generating’ visual qualia. No: auditory cortex is now participating in visual-type interactions in a way that is almost as efficient as visual cortex usually would be. Such a situation is called ‘seeing’.

Of course we could call the change from inefficient to efficient use of the auditory cortex ‘cortical plasticity’. But it is important to be aware that this ‘plasticity’ is not in itself what is causing the transition to seeing with the auditory cortex. The plasticity is allowing the animal to interact in a visual fashion, but it is not effecting a magical transition from generating auditory to generating visual qualia.

Indeed there is another argument supporting the view that cortical plasticity is not the cause of the radical changes in sensory quality observed in rewired animals. It comes from other phenomena where drastic changes in the locus, modality or nature of sensations can occur in a very short time, making cortical plasticity, which is presumably a fairly slow process, a highly unlikely cause.

One such phenomenon is the feeling of ‘projecting’ your tactile sensation onto the tip of a tool: for example, when you write, you feel the roughness of the paper at the tip of the pen; and this happens even though you may hold the pen in any number of ways, and even though the vibrations from the paper are transmitted to widely different areas of your hand. The effect is immediate on setting pen to paper, and is thus presumably not the product of cortical plasticity. A similar example is the feeling you get immediately after learning to drive, of the tyres touching the (p.171) road or the curb when you park your car. The rapidity of these adaptations argues against cortical plasticity as an explanation.

On the other hand, cortical plasticity does of course occur. As has been shown by an extensive literature, cortical changes occur when you use a tool: receptive fields normally situated at the hand can extend to the end of the tool (cf. Maravita and Iriki 2004). This cortical plasticity, which may take weeks to set up in a macaque monkey, participates in improving the monkey’s ability to move objects with a rake. But it is not this cortical plasticity which is the original cause of a change in felt location of touch. On the contrary, the felt location of touch is the brain’s way of best accounting for the currently occurring sensorimotor contingencies. Cortical plasticity may improve this estimation, but it is not itself the cause of the change in sensation.

The rubber hand illusion is another example: in this illusion, which takes only minutes to establish, touch actually occurs on the observer’s arm, but is perceived as occurring on the rubber arm (Botvinick and Cohen 1998). This is not because of cortical plasticity, but because the sensorimotor contingencies established artificially by the experimental manipulation can best be interpreted by the brain in terms of touch on the rubber arm rather than on the observer’s arm.

Finally sensory substitution, though often ‘explained’ in terms of cortical plasticity (Bach-y-Rita et al. 2005; Bach-y-Rita 1972), is in fact also not essentially caused by cortical plasticity. People using the TVSS or the vOICe report after only a few hours, if not within minutes, that they are getting ‘something like’ a visual sensation. Of course the sensation can only be visual to the extent that the apparatus, and the observer’s motor, sensory and cognitive faculties allow the interaction to approximate visual-type interactions with the world. But the effect occurs almost immediately, and is thus presumably not caused by cortical plasticity.

Again, it is of course the case that over time, when a person becomes experienced with sensory substitution devices, cortical changes will occur. Brain structures will adapt so as to more efficiently do the processing involved in entertaining interaction with the world corresponding to the perceived modality. The experienced sensation will then more accurately approximate the normal sensation in that modality. But again, the cause for the experienced sensations lies not in the plasticity itself, but in the fact that the plasticity has enabled a mode of interaction with the environment which better approximates the new sensory modality.

10.4 Conclusion

It seems that the notion of cortical plasticity that researchers often refer to in the context of rewiring and sensory substitution may be somewhat misleading. Whereas the dramatic changes in experienced sensations seem to warrant appeal to some equally significant cortical modifications, I claim that taking this view would (p.172) be to fall into Ryle’s category mistake of thinking that sensations are the kinds of things that can be generated in the brain.

In fact, if we admit that the quality of sensory experiences is constituted by the modes of interaction with the environment that they involve, then we see that the real cause of modification of sensations in cases like rewired animals and sensory substitution will necessarily lie in changes in the modes of interaction which are involved. These changes in modes of interaction can be enabled, or favoured, by cortical plasticity, but cortical plasticity is not in itself what causes the changes. What causes the modification in sensory experience is just the fact that, through the use of special instrumentation or interfaces, a person or an animal interacts with the world in a way which approximates a sensory modality that involves cortical areas or sensory apparatus usually used for other modalities.


Bibliography references:

Bach-y-Rita, P. (1972) Brain Mechanisms in Sensory Substitution. New York: Academic Press.

Bach-y-Rita, P., Danilov, Y., Tyler, M. E. and Grimm, R. J. (2005) Late human brain plasticity: vestibular substitution with a tongue BrainPort human-machine interface. Intellectica 40: 115–122.

Botvinick, M. and Cohen, J. (1998) Rubber hands ‘feel’ touch that eyes see. Nature 391(6669): 756. doi:10.1038/35784.

Chalmers, D. J. (1997) The Conscious Mind: In Search of a Fundamental Theory. New York: Oxford University Press.

Dennett, D. C. (1991) Consciousness Explained. Boston: Little, Brown & Co.

Frost, D. O., Boire, D., Gingras, G. and Ptito, M. (2000) Surgically created neural pathways mediate visual pattern discrimination. Proceedings of the National Academy of Sciences USA 97(20): 11068–11073. doi:VL-97.

Hurley, S. and Noë, A. (2003) Neural plasticity and consciousness. Biology and Philosophy 18(1): 131–168.

James, W. (1890) The Principles of Psychology. New York: Dover Publications.

Kupers, R., Fumal, A., de Noordhout, A. M., Gjedde, A., Schoenen, J. and Ptito, M. (2006) Transcranial magnetic stimulation of the visual cortex induces somatotopically organized qualia in blind subjects. Proceedings of the National Academy of Sciences USA 103(35): 13256–13260.

Maravita, A. and Iriki, A. (2004) Tools for the body (schema). Trends in Cognitive Sciences 8(2): 79–86. doi:10.1016/j.tics.2003.12.008.

O’Regan, J. K. (2011) Why Red Doesn’t Sound Like a Bell: Understanding the Feel of Consciousness. New York: Oxford University Press.

O’Regan, J. K. and Noë, A. (2001) A sensorimotor account of vision and visual consciousness. Behavioral and Brain Sciences 24(5): 883–917.

Pallas, S. L. and Sur, M. (1993) Visual projections induced into the auditory pathway of ferrets: II. Corticocortical connections of primary auditory cortex. Journal of Comparative Neurology 337(2): 317–333. doi:10.1002/cne.903370212.

(p.173) Pallas, S. L., Roe, A. W. and Sur, M. (1990) Visual projections induced into the auditory pathway of ferrets. I. Novel inputs to primary auditory cortex (AI) from the LP/ pulvinar complex and the topography of the MGN-AI projection. Journal of Comparative Neurology 298(1): 50–68. doi:10.1002/cne.902980105.

Penfield, W. (1958) Some mechanisms of consciousness discovered during electrical stimulation of the brain. Proceedings of the National Academy of Sciences USA 44(2): 51–66.

Phillips, C. G., Zeki, S. and Barlow, H. B. (1984) Localization of function in the cerebral cortex: past, present and future. Brain 107(1): 328–361. doi:10.1093/brain/107.1.328.

Ptito, M., Schneider, F. C. G., Paulson, O. B. and Kupers, R. (2008) Alterations of the visual pathways in congenital blindness. Experimental Brain Research 187(1): 41–49. doi:10.1007/s00221-008-1273-4.

Ryle, G. (1949) The Concept of Mind. Chicago: University of Chicago Press.

Sharma, J., Angelucci, A. and Sur, M. (2000) Induction of visual orientation modules in auditory cortex. Nature 404(6780): 841–847.

Von Melchner, L., Pallas, S. L. and Sur, M. (2000) Visual behaviour mediated by retinal projections directed to the auditory pathway. Nature 404(6780): 871–876. doi:10.1038/ 35009102.