Sensorimotor theory of consciousness
| Jan Degenaar and J. Kevin O'Regan (2015), Scholarpedia, 10(5):4952. | doi:10.4249/scholarpedia.4952 | revision #190809 [link to/cite this article] |
The sensorimotor theory of perceptual consciousness (e.g. O’Regan & Noë 2001; O’Regan 2011) aims to account for the phenomenal character of perceptual experience (sometimes referred to as ‘qualia’ [1]). The theory rejects traditional accounts appealing to inner representational models, stressing instead patterns of sensorimotor dependencies (or ‘sensorimotor contingencies’), defined as the regularities in how sensory stimulation depends on the activity of the perceiver. The theory addresses both the particular quality of experiences (e.g. the differences between different experiences) and why there is conscious experience at all (e.g. why some things are consciously experienced while others are not). Sensorimotor theory offers a new way to think about experience, and has given rise to a productive empirical research program.
Contents |
The quality of perceptual experience
The challenge in explaining the quality of experience is to avoid an ‘explanatory gap’ (Levine 1983) between descriptions of the biological or physical processes involved in experience and descriptions of the phenomenal quality of experience. For example, consider that visual experiences involve neural activity in visual brain areas and auditory experiences involve neural activity in the auditory areas. As long as there is no way of making an intelligible link between the language used to describe the neural activity and the language used to describe the experience, the challenge remains to explain why visual brain activity is associated with visual experience and auditory brain activity is associated with auditory experience.
The sensorimotor theory addresses these issues by considering the sensorimotor patterns characterizing our interaction with the environment. Instead of focusing on local properties of neural activity, the theory explains the quality of perceptual experiences by appeal to the perceiver’s exercise of particular sensorimotor capacities. The idea is that each experience implies a bodily engagement with the environment, characterized by a particular set of sensorimotor dependencies, and that the quality of experience is constituted by the laws of sensorimotor engagement.
While a traditional approach focusing on neural processes opens up an explanatory gap, there need not be such a gap when we conceive of experience as a matter of sensorimotor engagement. On the contrary, we can appeal to sensorimotor patterns to articulate what the experiences are like. An example that is often used to illustrate the point is that of the experience of the softness of a sponge. While it may be hard to account for the experience of softness in terms of neural activity, the sensorimotor theory claims that softness consists in the particular way in which the sponge yields when it is squeezed. To explicate what the experience is like and to explain the quality of experience we can appeal to the language used to describe the bodily activity and the sensorimotor regularities it involves.
From this general perspective the theory addresses a broad range of explanatory issues. In particular, regarding the particular quality of perceptual consciousness, a distinction can be made between two groups of explanatory gaps, the intermodal gaps and the intramodal gaps (Hurley & Noë 2003). The intermodal gaps concern the differences between the sensory modalities (e.g. visual experience versus auditory experiences); the challenge here is to align physical/biological descriptions of the sense modalities to phenomenal descriptions of the modalities. Intra-modal gaps concern the differences within a sensory modality (e.g. the difference between visual experience of a straight line versus the visual experience of a curved line, or the difference between the experiences of different colors); again the challenge is to avoid a gap between the physical/biological and the phenomenal. Both groups of explanatory challenges are taken up by the sensorimotor theory, and both are addressed in fundamentally the same way.
Examples explicating the intermodal gap: What makes visual experience phenomenally different from other modalities are the particular physical/biological ways in which sensory stimulation depends on the perceiver’s activity. In vision, head movements can bring things in or out of the visual field, while there is no clearly analogous ‘auditory field’ capturing the distinction between in and out of hearing. Visual experience is further characterized by eye blinks, optic flow, the laws of occlusion, etc., which have no precise analogs in hearing.
Examples explicating the intramodal gap: Within the visual modality, the experience of a straight line is characterized by facts such as that sensory stimulation does not change when you move your eyes along the line; curved lines imply a different set of sensorimotor dependencies. Preliminary attempts to describe sensorimotor dependencies governing our engagement with different colors have been proposed by Philipona and O’Regan (2006).
In addition to these issues concerning intermodal and intramodal differences in the qualities of perceptual consciousness, the sensorimotor theory is also applied to explicate the differences between cases where sensory experiences can be said to be properly sensory, and cases where experiences are not so clearly sensory, as in the experience of thinking, or for visceral goings-on. Characteristic for sensory experience, compared for example to such not properly sensory experiences, is the high degree of ‘bodiliness’ (or ‘corporality’), ‘insubordinateness’, and ‘grabbiness’ (or ‘alerting capacity’) associated with sensory inputs (O’Regan, Myin & Noë 2005; O’Regan 2011). Bodiliness refers to the objectively quantifiable way in which bodily changes modify sensory input; for example, turning your head alters visual input, but has no effect on thoughts. Insubordinateness is the fact that bodily changes, though they have a systematic effect, do not completely determine sensory changes (sensory input can change without bodily changes occurring). Grabbiness concerns the fact that, due to basic properties of sensory systems, sudden transitory changes in sensory input strongly grab our attention and cause perceptual processing to be focused on the sudden event. Sensory experiences differ from non-sensory experiences like thinking or unfelt visceral goings-on in these respects: the experienced “sensory presence” of the world is a consequence of bodiliness, insubordinateness, and grabbiness. These are objective properties of properly sensory engagement with the world, which are lacking or less pronounced in non-sensory experiences like thinking and unfelt visceral processes.
Bodily engagement as exercising implicit grasp of sensorimotor dependencies
In sensorimotor theory, bodily engagement is often spelled out in terms of the exercise of one’s implicit grasp of sensorimotor dependencies (e.g. O’Regan & Noë 2001). Suppose a newborn has not yet developed the capacity to experience the redness of objects. The objective sensorimotor dependencies characteristic of the experience of red may already be in place (retinal stimulation systematically depends on action the same way as in adults), but the newborn would still lack the capacity to engage with the sensorimotor regularities: it has no implicit grasp of the dependencies. Once the child has developed this grasp, the experience of red occurs when the capacity is put to use: the experience occurs in the exercise of this grasp.
The notion of implicit grasp needs careful unpacking. Some interpretations of the notion of implicit grasp or ‘implicit knowledge’ of sensorimotor dependencies suggest or even imply commitment to representationalism, as criticized by Hutto (2005). It should be clear that such an interpretation is not intended in the sensorimotor approach (see also Noë 2001; Myin & O’Regan 2002; Myin & Degenaar 2014). When one literally grasps on object with one’s hand, the hand gets a grip on the object without representing it (in the sense of forming a model of it). Similarly, it is claimed, when one ‘grasps’ the obtaining sensorimotor dependencies, one gets a grip on the environment without having to represent it. Thus, having grasp of sensorimotor dependencies is no more (and no less) than being attuned to these sensorimotor regularities. When one is attuned to the environment, this means that one can get under the influence of the obtaining sensorimotor dependencies, which in turn implies that one potentially can act accordingly and differentially, e.g. by picking out red objects or soft objects.
The existence of conscious experience
In addition to explaining the differences between possible experiences, a theory of consciousness should also capture the fact that there is experience at all. Again, the sensorimotor theory addresses the issue by situating perceptual consciousness at the level of the exercise of perceptual capacities.
This question of the existence of consciousness is sometimes referred to as the ‘absolute gap,’ because we are no longer comparing experiences and asking why a particular process comes with a particular experience rather than another. Nevertheless, the absolute question can also be operationalized into comparative questions. For example, what explains that some environmental properties are consciously experienced while others are not? What explains that we sometimes are conscious while in other cases we are not (e.g. knocked out)? What explains that some systems (e.g. humans) have conscious experience while others (e.g. thermostats) do not?
Sensorimotor theory notes that in everyday usage, being conscious of a stimulus implies that the stimulus can have an impact on further action (including speech behavior) and on thought and rational reflection (O’Regan & Noë 2001). Not all sensorimotor capacities imply consciousness in this sense. For example, a skilled driver may negotiate the curves of a road while absorbed in conversation, without being able to report any details on the perceptual basis for the driving – indeed, the person may even deny having experienced certain curves. What is lacking here, on a sensorimotor account, is the use of perception in one’s thought and reflective action. When something is not integrated into the appropriate cognitive activities, it cannot be said to be the object of consciousness. At best the object can then form part of the background of one’s conscious experience.
The issue of the presence of consciousness links up with the issue of the phenomenal character of consciousness in the following way. If and only if we can speak of conscious experience, we can speak of its qualitative character. According to the sensorimotor theory, the absolute gap is a matter of our access to the environment, it concerns functional capacities for making use of this access (the question of access has sometimes considered to be an ‘easy problem’, Chalmers 1996). But rather than allowing for a dissociation between (‘easy’) access to the environment and (‘hard’) phenomenal consciousness, as if there could be 'zombies’ displaying full-blown access to the environment but lacking experiences with a phenomenal character (Block 1995; Chalmers 1996), the sensorimotor theory offers an account of the phenomenal character of the process of accessing the environment. The idea here is that conscious access is a matter of functional capacities, and that it always involves a process of interaction which itself necessarily possesses phenomenal character of some sort, and this phenomenal character is constituted by the sensorimotor dependencies that are involved. For example, why is my present experience characterized by the sensory presence of the environment, what gives the experience its visual feel, and what explains the particular quality of the experience of red? Sensorimotor theory addresses precisely such questions concerning the quality of conscious experiences by appealing to the underlying sensorimotor dependencies that are constitutive of those experiences. Note that these questions are only raised in cases of conscious access: for the question of the quality of conscious experience to make sense, the perceiver must be exercising the grasp of sensorimotor dependencies making use of the relevant cognitive capacities.
It remains as a matter of agreement on the definition of “consciousness” to decide to what extent we want the cognitive capacities defining the presence of consciousness to be capacities associated with the use of language. Clearly, the conscious experience of typical human adults often involves explicit conceptual capacities. A prototypical case is one in which someone verbally reports to have an experience: we tend to take this as a good indication of consciousness, as an exemplar of what we mean by saying that something is consciously experienced. Similarly, when something plays an explicit role in someone’s deliberation, this implies that the person is conscious of the thing. But we could also allow that certain sufficiently sophisticated non-linguistic capacities be considered forms of cognitive access that we agree to label as “conscious”.
Note that the appeal to cognitive capacities in accounting for the contrast between conscious and not-conscious does not imply that we should turn to something like ‘cognizing’ for explaining the particular quality of perceptual experience. It is true that to some extent explicit thought has its own phenomenology, so that thinking changes the overall experience of the perceiver. For example, we may think about what we see ‘in English’, and this differs phenomenally from thinking ‘in French’ (the difference may be described by reference to the sounds of words). However, in normal (e.g. non-pathological) cases, sensorimotor theory claims that thought does not fundamentally distort perceptual phenomenology as such. Thus, for example, the theory explains the qualitative character of the experience of red by appeal to the particular regularities implied in the sensorimotor engagement with red surfaces. It may be on the basis of the occurrence of a red-related thought that we speak of conscious experience at all, but whether one thinks about that redness ‘in English’ or ‘in French’, it is claimed that there is something important about the quality of experience that is addressed without addressing the particular phenomenal character of the thought.
The sensorimotor research program
The sensorimotor theory is more than a philosophical account of conscious experience. It offers a framework for an empirical and theoretical research program elucidating why people say what they say about different kinds of sensory experience. For this, the key question is always: Which are the relevant patterns of sensorimotor engagement constituting the experience? This approach has successfully been applied to a range of phenomena. We shall first discuss a few prominent examples as discussed in the literature, after which we address some challenges and open questions.
Change blindness, the richness of experience, and the sense of presence of the world
Given the apparent ‘defects’ of the visual apparatus, such as the presence of a blind spot, the low spatial and color resolution in visual periphery, and the continual interruptions due to blinks and eye movements, how can it be that we experience the world as continuous and rich in detail? A traditional way to approach the question is to assume that the visual system continuously compensates for the flaws of the visual apparatus by ‘filling in’ the lacking information. The sensorimotor account rejects such compensation mechanisms as being unnecessary. The account starts from the premise that: “seeing constitutes an active process of probing the external environment as though it were a continuously available external memory” (O’Regan 1992). If this is true, then the experience of richness or continuity should not be sought in richness or continuity of some internal representation of the world, but in the richness and continuity of the process of accessing the world. We have the impression of seeing continually because whenever we seek information about something, this information is immediately available at the slightest flick of attention or of the eye. There is no reason why we should be aware of blind spots, blinks, saccades or retinal inhomogeneity, since if we want information, we can obtain it immediately. It is the continual availability of details of the environment which constitutes the experience of continuity and richness of vision (Noë 2001). We feel that the world is present before us in all its detail because we know how to get information about it at will (O’Regan 1992; Noë 2001).
An interesting consequence of this analysis is that it can potentially simplify our view of the activity of the visual system. Indeed, the idea that the visual system must be ‘filling in’ experience at the blind spot, can be discarded as an artifact of the supposition that vision depends on the construction of an internal model of the world (Pessoa, Thompson & Noë 1998). The point is not to deny that there are perceptual completion phenomena involving spreading neural activity – such neural activity undoubtedly exists (Pessoa, Thompson & Noë 1998). But in the sensorimotor theory such activity is not interpreted as part of a process of constructing a model of the world inside the head: instead it is seen as providing the mechanisms which allow an engagement with the world which is similar to one’s engagement with the physically completed stimulus. Both in sensorimotor accounts, as well as in internal model based accounts, neural activity is required for engagement with the environment; the sensorimotor account is more straightforward or simpler because it does not require that the neural activity have the additional function of building up a model of the environment inside the head.
If it is true that our impression of continued presence of elements in the visual scene derives from our having confidence that information about them is immediately available at the slightest flick of attention or the eye, then this has a surprising consequence. It implies that our impression of presence of scene elements derives not from them actually being there; the richness of experience derives from our in some sense implicitly assuming that they are there. Usually of course our assumptions are correct, and usually we can rely on the grabbiness of sensory changes (that is, their capacity to alert us) in order to detect changes in the environment. But if it were possible to surreptitiously make a change in a scene without triggering the normal alerting reactions that direct attention to sudden transients that usually accompany changes in the visual field, then we should remain ignorant of these changes unless we happened for endogenous reasons to be attending to the changing scene element.
This is what happens in the phenomenon of ‘change blindness’. For example if large areas of a scene change so slowly that automatic transient detectors in the visual system are not triggered, then such changes very often go completely undetected, even if they are currently in the center of the visual field (Simons, Franconeri & Reimer 2000). Similarly, if a large flicker or ‘mudsplash’ occurs in a scene at the same time as a picture change, then transients occur all over the scene, hiding the transient caused by the searched-for picture change. An observer is consequently highly likely to miss an otherwise obvious picture change (Rensink, O’Regan & Clark 2000; O’Regan, Rensink & Clark 1999).
The plasticity of experience: sensory substitution and visual inversions
Sensorimotor capacities develop over time. Given the notion that experiencing consists in exercising sensorimotor capacities, sensorimotor theory predicts that perceptual experiences should change when new capacities of sensorimotor exploration are acquired.
Confirmation of this idea can be found in Bach-y-Rita’s work on the ‘tactile vision substitution system’ or TVSS, a sensory substitution device transforming the optical information recorded by a camera into a tactile display of vibrating pins which can be applied to a subject’s back or belly (Bach-y-Rita et al. 1969; Bach-y-Rita 1984). After training, subjects reported experiences that could best be described with visual terminology, namely the occlusion of parts of the scene by objects, and looming of objects in the camera’s visual field. Since that early work, other studies using not only visual to tactile but also visual to auditory substitution have confirmed that sensory substitution devices can support experiences that are in some respects like vision (Auvray et al. 2007).
Changes of experience have also been investigated in studies disrupting the patterns within a sensory modality, for example in classic work on the effect of wearing inverting goggles. Here the sensorimotor theory emphasizes that adaptation to such disruptions should not be conceived as involving a ‘righting’ of a visual image. Instead, sensorimotor theory emphasizes the various sensorimotor capacities of vision that are each independently adaptable, involving a mesh of different laws more or less independently governing the sensory consequences of eye, head and body movements. This results in a more variegated analysis of changes of visual experience while wearing inverting goggles (Degenaar 2014). Sensory plasticity has further been investigated using radically new modes of interaction, for example with a belt providing a tactile indication of the magnetic north (e.g. Kaspar et al. 2014). Subjects wearing such a sensory augmentation device report changes of spatial experience associated with the use of the device, demonstrating an attunement to the new sensorimotor contingencies.
The experience of colors
A remarkable success of the sensorimotor theory is in the domain of color experience, where the theory has inspired a new approach to color experience. As is well-known in color science, there is a mismatch between the activation of opponent channels in the brain and findings about color naming and psychophysical measures of perceptually ‘unique’ hues (DeValois & Webster 2011 [2]). In response, a classic philosophical picture of experience would turn attention to ‘further stages’ in the visual system, assuming that that is where the explanation for color experience is to be found. The sensorimotor approach instead takes the stance that the quality of color experience must be constituted by the laws that govern the way colored surfaces change the light reflected into the eye as those surfaces are moved around under different illuminants – or as differently illuminated parts of a surface are sampled with the eyes. A mathematical analysis then shows that these laws can be accurately described by 3 x 3 matrices. The surprising finding is that the properties of these matrices accurately predict which surfaces tend to be given names across different cultures across the world, and which hues are judged by observers to be “pure” (Philipona & O’Regan 2006). Focusing on the dynamics of our sensorimotor engagement with colored surfaces thus helps to explain key aspects of color experience. Further support for a sensorimotor approach to color experience comes from experiments suggesting that the experience of color is also partially constituted by the way color sampling changes as a colored patch is seen in central vs. peripheral vision (e.g. Bompas & O’Regan 2006).
The quest for artificial consciousness
The sensorimotor approach offers guidance for research aiming towards systems with artificial consciousness. First, the answer to the question ‘what should we build into a system to make it conscious’ is that we should focus on sensorimotor capacities. No additional ingredient is required. This implies that, for example, the creation of artificial life (the synthesis of organizational principles of life) would be useful for artificial consciousness only to the extent that it enables a range of sensorimotor capacities. Second, sensorimotor theory provides an outline of an answer to the question concerning which sensorimotor capacities are required for perceptual consciousness. As explained above, engagement with sensorimotor dependencies is not sufficient for perceptual consciousness because engagement with sensorimotor dependencies can remain unconscious. The sensorimotor account therefore proposes further criteria to guarantee perceptual consciousness: we only ascribe consciousness when a system is able to make use of its grasp of sensorimotor dependencies in more advanced abilities such as verbal report or planning. Thus research aimed at artificial consciousness should aim for robotic systems with such more advanced capacities. In turn, making robotic implementations will help us delimit more precisely the properties of systems to which we are willing to ascribe consciousness.
Theoretical developments
That the sensorimotor theory offers a successful research program is evident from the empirical work mentioned above. In addition, there are theoretical developments worth pointing out.
One development concerns the notion of space. Biological agents possess spatial skills allowing them to navigate and perceive the location of objects in space, and much research has concerned the fine-tuning of such spatial skills. But more fundamentally, spatial capacities must have emerged somehow during the course of evolution, and/or they must develop in agents during their maturation. How is this possible, given that a priori the only information available to brains is undifferentiated neural spike trains? The sensorimotor theory, building on ideas of Poincaré and Jean Nicod, suggests that the only way the notion of space could emerge would be by discovering invariants in the sensorimotor laws that govern the relations between an agent’s motor commands and its sensory inputs. The experience of space, from this point of view, efficiently captures these sensorimotor regularities (Philipona, O’Regan & Nadal 2003; Terekhov & O’Regan forthcoming).
At a conceptual level, work has recently been done on differentiating various notions of sensorimotor contingency/dependency. In particular, four distinct notions of sensorimotor dependency have been introduced: “the notions of sensorimotor environment (open-loop motor-induced sensory variations), sensorimotor habitat (closed-loop sensorimotor trajectories), sensorimotor coordination (reliable sensorimotor patterns playing a functional role) and sensorimotor strategy (normative organization of sensorimotor coordinations)” (Buhrmann, Di Paolo & Barandiaran 2013). These distinctions can facilitate detailed investigations of the contribution to perceptual development of particular aspects of our sensorimotor engagement with the environment. For example, our sensorimotor environment puts constraints on perceptual development, and the existence of systematic patterns of sensorimotor coordination can have important implications for perceptual development, because it introduces biases in the opportunity to get attuned to particular sensorimotor patterns.
The notion of sensory presence has also been developed in some detail. Consider that when we look at an object we have a sense of its back side. A tomato for example appears to us as a three-dimensional object whose parts seem present to us even if they are currently out of view. To capture this sense of presence of unseen parts of the world, Noë speaks of ‘virtual presence’ (Noë 2004; 2012). The idea is that this sense of presence reflects our capacity to bring unseen features into view: we have a sense of presence of the backside of objects because they are available to us for exploration. The same even applies to elements of the scene which are in view, since whatever we attend to, there is always more detail available, and more ways of attending to it. The notion can be applied to other perceptual modalities as well (e.g. the tactile presence of the tomato). Importantly therefore, even those aspects that are presently under scrutiny can be understood only as part of a larger sensorimotor web of possibilities: perceptual content is ‘virtual all the way in’ (Noë 2004). Our sense of the color of an object for example is not ‘given’ to our sense organs to be passively received, it rather is an ongoing perceptual activity, involving our attunement to the way in which the object might respond to changes of the light. The idea that perceptual content is virtual ‘all the way in’ also implies that, if one were to look for the content of experience inside the head, there likely will be no states fully matching this content. The brain contains mechanisms enabling access to the world; there need be no content-carrying states themselves ‘containing’ the world that is present to us. (More about neural processes in the section below.)
While much research has focused on the laws describing perceptual interactions, the sensorimotor approach has also been extended to address atypical phenomena such as synesthesia and phantom limb experiences. Having a synesthetic color experience consists in exercising one’s attunement to color, but triggered by other stimuli such as shapes or spoken words: the sensorimotor theory claims that we can only make sense of the experience in terms of the sensorimotor laws describing our interaction with colored surfaces, for that’s why it makes sense to speak of color experience in the first place. Similarly, to describe phantom limb experiences, the sensorimotor theory must appeal to sensorimotor relations defining the spatial presence of a limb (which need not involve motor action of the limb itself, since one might have the experience as of a paralyzed limb). A key issue then is why such experiences do not always adapt away, while in other cases (e.g. when wearing inverting goggles) experiences adapt under influence of interaction with the environment. The sensorimotor account suggests that the reason that such experiences do not adapt away is that they depend on ‘dangling’ cortical activity, i.e. activity that is not part of the ongoing sensorimotor interactions with the environment (Hurley & Noë 2003).
Another issue that has been addressed by sensorimotor theory concerns similarities between vision and visual imagery. This may also be understood in terms of similarities in the subject’s active mode of engagement (e.g. Thomas 2014). To the extent that the perceptual systems of a subject are active as if the subject is interacting with the environment, the experience of imagery will be similar to perception. Just as is the case for our perceptual capacities, capacities of imagery also derive from, and can be made sense of in terms of our attunement to the sensorimotor patterns characterizing active engagement with the environment (e.g. Myin & Degenaar 2014). The vividness of experience during imagery or dreaming can be understood from the same perspective. To the extent that, during imagery or dreaming, our brains can be active as if we are engaging with the world, our experience may be as vivid as when we are actually engaging with the world..
Importantly, from the perspective of the sensorimotor account, we should not interpret for example imagining an object as having an ‘imagined object’ in our heads; rather, imagery is a case of apparent engagement with the object. Moreover the sensorimotor theory has an edge on traditional theories, because it can additionally explain why imagery typically does not completely feel like real seeing: the theory explains this in terms of the lack of bodiliness, grabbiness and insubordinateness involved in the interaction we have with the world when we are imagining.
In short, the sensorimotor account generates testable predictions and it provides a fruitful perspective for the investigation of experience. While the ramifications of this perspective for neuroscience are still largely to be explored, progress is being made on key aspects of the theory, which is maturing to capture an increasing range of applications. Along the way, we may expect offshoots of the account to develop into semi-autonomous domains of research, raising the continuing challenge to maintain conceptual coherence in accounting for a plethora of phenomena. Hopes are that the fundamental commitments of sensorimotor theory continue to provide unity in an expanding research program.
Open questions, challenges and scope
There are two major challenges for developing sensorimotor theory, and these are closely related. One is to make links to the development of perceptual capacities. The other is to specify the neural processes enabling our attunement to sensorimotor dependencies. Sensorimotor theory offers a constitutive account of perceptual experience, an account of what experiencing is, aiming to bring together physical/biological and phenomenological descriptions of the character of perceptual consciousness. As such the theory is not committed to any particular account of how our perceptual capacities come into being. Below we address development first, then we turn to the brain. We end the section pointing at some experiences that are not addressed by the present sensorimotor theory.
Development
The developmental question concerns how sensorimotor dependencies are actually learnt: what kind of learning and reinforcement mechanisms are at play? What is the role of constraints imposed by the perceiver’s perceptual and motor apparatus? How do the statistics of environmental stimulation influence learning? How does the perceiver’s prior experience interact with to-be-acquired sensorimotor laws? And of course a full account of perceptual development would address ontogenesis of perceptual capacities as well as the factors at play over evolutionary timescales (the latter explaining constraints on ontogenesis). The guidance sensorimotor theory can give here is to provide an account of what it is that develops, namely a skillful engagement with the environment, which avoids representationalist presuppositions. For some steps towards a theory of the development of perceptual sensorimotor capacities, see Di Paolo et al. (2014).
Neural processes
Another challenge for sensorimotor theory is to specify what neural processes enable conscious perceptual engagement with the environment. Present sensorimotor theory primarily aims to clarify the problem to be solved, leaving proposals for specific neural mechanisms still largely to be worked out or to be brought into connection with the sensorimotor framework.
First consider the conscious/non-conscious contrast. While the sensorimotor theory offers the outlines of a descriptive account of the contrast between conscious and unconscious, the neural processes explaining or correlating to these descriptive features remain to be given. A connection may be made here with neural workspace theories, which offer an account of when one’s sensitivity to the environment is such that it can be qualified as an instance of conscious experience (Degenaar & Keijzer 2009). According to neural workspace theories, some neural activity has a larger impact on further processes in the brain – thus potentially resulting in an impact on deliberate action, further thought and verbal report – and this neural activity is proposed to be particularly relevant to consciousness (e.g. Baars 1988; Dehaene & Naccache 2001). Sensorimotor theory can appeal to such theories as a way of implementing neurally the differences between the modes of engagement with the environment that the theory considers to correspond to conscious and non-conscious engagement (see also the relation to the Multiple Drafts Model below).
But note that workspace theories tend to explicate perceptual consciousness in terms of the availability of information to brain subsystems. For compatibility with the sensorimotor theory, this availability would have to be cashed out in terms of the degree to which these subsystems come under the influence of potential interactions with the environment. The sensorimotor theory then additionally provides an account of why this availability is accompanied by an experiential quality: the phenomenal character of experience corresponds to the particular skills involved in the ongoing engagement with the environment. The workspace theories, because they only talk about brain availability, have difficulties making the link with phenomenal quality.
Second, consider the particular quality of experience. If, as the sensorimotor theory claims, phenomenal quality derives from attunement to particular sensorimotor dependencies, the question then becomes how neural activity could enable this attunement.
Importantly, being attuned to the sensorimotor dependencies pertaining to a situation need not be understood as involving active simulation of the sensory consequences of all possible movements. To be attuned to an environmental property (e.g. the shape of a coin) is having sensorimotor expectancies, but the expectancies can be implicit: when a movement is made the resulting change in sensory stimulation is as expected and there’s no change of one’s experience of the environmental property. An idea then is to regard development of a sensitivity to a particular set of sensorimotor dependencies as an increasing insensitivity to differences within the set of dependencies. For example, when one learns to see from various angles that the coin is round, one’s experience of the shape of the coin becomes relatively insensitive to the angle of observation. A possible neural correlate to this desensitization to changes in viewing conditions might consist in convergence of sensory responses on an internal process. Such a convergence is proposed for example by predictive processing accounts of perception, which appeal to hierarchical structures of interactions within the brain (e.g. Friston 2008; Clark 2013; more on this approach below).
Another important point for the sensorimotor theory concerns the relation between phenomenal experience and ongoing neural activity. For example, consider how the sense of presence or reality of the environment depends on the ‘bodiliness’ of sensory experience (O’Regan, Myin & Noë 2005). The possibility of bodily exploration of the environment (by touch, visual exploration, etc.) and actions like closing and opening the eyes, according to the theory, are what provide the sense of presence of the environment. While the phenomenal character of sensory experience depends on (is partly described by) such structural features of our temporally extended engagement with the environment, we should not project these features onto any moment of the exploratory activity. It would therefore be a mistake to propose particular neural mechanisms relating to a continuous buzzing sense of presence, for such mechanisms may simply not exist.
The scope of the theory
Further issues may be raised about the scope of the theory, in particular concerning the range of experiential phenomena addressed. Sensorimotor theory has been developed as an account of perceptual consciousness and it has been applied to derivative experiences like imagery and synesthetic experiences. It remains an open question as to how, or to what extent, the approach can be applied to other experiences such as emotional experiences (joy, anger, sadness etc.), sensations such as pains and itches, and experiences of bodily condition (e.g. feeling energetic, sleepy, or weary).
One might think that a sensorimotor perspective could encompass (some of) these experiences by viewing these as ‘perceptions’ of bodily state, hormone levels, visceral goings-on etc. One may then attempt to describe the experiences in terms of our attunement to the way in which interoceptive and proprioceptive signals and/or pain receptor activity depend on what we do. However, such dependencies arguably fail to explain the crucial affective aspects of the experiences. For example, we would still need to account for the fact that pain is something to avoid, and that there’s something agreeable about joy. Moreover, given that we do not interact with bodily states, it is hard to make sense of the notion of ‘perception’ of such states from a sensorimotor perspective. A more natural way to develop a sensorimotor perspective to apply to (some of) the experiences mentioned above seems therefore to describe the experiences as modulations of our sensorimotor engagement with the environment. Different emotions, sensations and bodily conditions all imply different behavioral tendencies, they may come with different bodily postures and changes in our sensitivities to the environment. This could potentially provide possibilities for developing sensorimotor accounts of emotions, sensations, and experiences of one’s bodily condition.
Relation to other accounts
Let us highlight some contrasts and affinities between sensorimotor theory and other accounts.
Multiple Drafts Model
Sensorimotor theory has clear affinities with Dennett’s (1991) Multiple Drafts Model, despite the fact that the latter is framed in representational terms. Given that the sensorimotor account claims that consciousness constitutively involves bodily engagement, so that experiencing is ‘something we do’ (O’Regan & Noë 2001), it must claim that there is nothing intrinsically special about the neural processes underlying consciousness: This is also a key element of Dennett’s Multiple Drafts Model (Dennett 1991; [3]). Rather than postulating a Cartesian Theater in the head ‘where it all comes together’, the multiple drafts model puts emphasis on the capacities implied in conscious experience. Relevant here is Dennett’s metaphor of ‘fame in the brain’: Just as one cannot be famous ‘intrinsically’, independently from the reactions of other people, whether or not some neural activity is involved in consciousness depends ultimately on the (potential) behavioral influence the activity has, and it is not determined by the intrinsic properties of the neural activity (Dennett 2001).
While the Multiple Drafts Model has been accused of “explaining away” the qualitative character of experience, sensorimotor theory expressly takes up the challenge of characterizing the phenomenal quality of experience. Importantly, it does so without postulating the existence of qualia as something that can vary independently of our interactions with the environment (Noë 2004; Degenaar 2013). Arguably, sensorimotor theory thus provides what is lacking in the Multiple Drafts Model, and it does so without committing to a Cartesian Theater or a questionable notion of qualia as something that becomes disconnected from what we say and do. Without commitment to such a notion of qualia, sensorimotor theory aims to explain what people say about the character of their experiences, by giving a positive characterization of sensory ‘feel’ (O’Regan 2010).
Ecological and active perception approaches
Ecological approaches (e.g. Gibson 1979) and active perception approaches (e.g. Ballard 1991) stress the active exploration of the environment. But although these approaches emphasize that perception depends on action, they tend to focus more on patterns of sensory stimulation irrespective of whether they derive from one’s own actions, rather than on the sensorimotor dependencies. In sensorimotor theory, by contrast, action is not merely an instrument for acquiring sensory stimulation: motor activity and sensory stimulation are on an equal footing, with the perceiver’s activity ‘directly’ or ‘non-instrumentally’ contributing to perception (Taylor 1962; Hurley 1998; Mossio & Taraborelli 2008). There is also an important difference in emphasis between ecological approaches and sensorimotor theory: ecological approaches rightly stress the importance of dynamic invariants in allowing agents to act in an adaptive way in their environments. But they do not extend this idea further to claim that the sensorimotor contingencies actually can be taken to characterize the ‘feel’ of sensory experience. Sensorimotor theory explicitly addresses the phenomenal quality of conscious experience.
Enactivism
Sensorimotor theory is an enactive account, in the broad sense that it is focused on a perceiver’s active bodily engagement with the environment rather than on alleged internal representations. The enactive focus on bodily engagement with the environment has strong affinity with the phenomenological work of for example Husserl (1907) and Merleau-Ponty (1945). But what kind of body is required for conscious engagement? Some forms of enactivism, which we may call ‘autopoietic enactivism’, claim that biological life, or autopoiesis (self-production) is necessary for consciousness (e.g. Thompson 2007). Sensorimotor theory need not be committed to this necessity claim: it implies that if life is necessary, it is so only because without life we would not have the appropriate sensorimotor capacities necessary for perceptual consciousness. If it so happens that the appropriate capacities are actually possible without life, then consciousness is possible without life (Degenaar & O’Regan forthcoming).
The extended mind and vehicle externalism
Sensorimotor theory emphasizes capacities rather than committing to representationalism (content-carrying vehicles) or to computational functionalism (trafficking in contents). Still the theory has some affinities with the (functionalist) extended mind thesis (Clark & Chalmers 1998) and (representationalist) vehicle externalism (Hurley 1998, 2010; Rowlands 2003), both of which emphasize extracranial processes as co-constituting the processes underlying mental phenomena. In fact, sensorimotor theory has particularly important antecedents in the work of Hurley, stressing that perception and action are interdependent and that both are dependent on sensorimotor dynamics (Hurley 1998; 2001). Like externalists, sensorimotor theory is skeptical about an exclusive focus on internal processes, for it views perception as essentially an engagement with the environment (Noë 2004). However, given the sensorimotor theory’s constitutive claim about consciousness as lying in the exercise of capacities, the question of the (subpersonal) location of consciousness becomes moot: ‘does consciousness extend beyond the head’ would be no more a serious question than ‘does cycling extend beyond the head.’ While sensorimotor theory appeals to the environment we interact with, under its interpretation of experiencing as the exercise of capacities the question of the material ‘vehicles’ of consciousness lacks any clear interpretation.
Higher order theories
Like sensorimotor theory, higher order theories appeal to cognitive capacities to characterize the contrast between conscious and not conscious. However, higher order theories conceive of these capacities in terms of accessing mental states; according to Rosenthal & Weisberg (2008): “Higher-order theories all embrace the idea that a mental state is conscious when the subject is appropriately conscious of that state” ([4]). Sensorimotor theory differs from this approach. It stipulates that for us to be conscious of something in the environment (or of our way of interacting with the environment), the thing must play a role in subsequent thought or behavior (for example, when we reflect on something in the environment we are conscious of the thing in the environment, as may be evident from our subsequent behavior). Furthermore, sensorimotor theory claims that the phenomenal character of the experience is then explained by our implicit grasp of sensorimotor regularities. No appeal is made here to ‘higher order’ thoughts or to mental states accessing other mental states. Sensorimotor theory appeals to different activities defining the difference between conscious and not-conscious, contrasting for example making use of something in planning on the one hand, with fleeting responses leaving no recognizable trace in further thought or action on the other (see the Multiple Drafts Model). Higher order theories would claim that making use of a perceptual state in deliberate planning necessitates a higher order state accessing the perceptual state, if the resulting behavior is to count as evidence for conscious perceptual experience. The sensorimotor theory is not committed to such a model of the difference between conscious and not conscious: given that sensorimotor theory claims that what is fundamental for accounting for consciousness are the different capacities implied by conscious experience compared to unconscious engagement, it does not need to make the additional assertion that conscious experience involves (or must involve) higher order access to (otherwise unconscious) mental states.
Helmholtz and Bayesian models of perception
Sensorimotor theory differs from standard approaches stressing the relevance of action-related signals, such as familiar from Helmholtz and classically-interpreted Bayesian approaches to perception, including predictive processes accounts of perception (e.g. Clark 2013). Helmholtz claimed that: “when we perceive before us the objects distributed in space, this perception is the acknowledgement of a lawlike connection between our movements and the therewith occurring sensations” (Helmholtz 1878/1977: 138). A key difference with Helmholtz is that in sensorimotor theory it is the bodily engagement itself that constitutes experience, not any alleged internal representation, or internal ‘conclusion’ reached according to a Helmholtzian ‘unconscious inferences’. Note that while perceptual development may be described by Bayesian models, this does not imply the existence of such Bayesian models within the brain. Moreover, sensorimotor theory rejects the identification of perceptual experience with a representational ‘content’ ascribed to the brain, replacing such notions with a fundamental emphasis on skillful engagement. Internal models may of course play a role in a theory of perceptual experience, as they do for example in Seth’s (2014) predictive processing account of sensory presence, which aims to provide an account of attunement to sensorimotor dependencies. However, from the perspective of sensorimotor theory, the focus of theorizing about experience should never be the content ascribed to such models, but always the patterns of engagement they enable.
Acknowledgments
This work was supported by ERC Advanced Grant 323674 to J. Kevin O'Regan
References
Auvray, M., Hanneton, S., & O’Regan, J. K. (2007). Learning to perceive with a visuo-auditory substitution system: localisation and object recognition with “the vOICe.” Perception, 36(3), 416–430.
Baars, B. J. (1988). A Cognitive Theory of Consciousness. Cambridge: Cambridge University Press.
Bach-y-Rita, P. (1984). The relationship between motor processes and cognition in tactile vision substitution. In Prinz, P. D. W., & Sanders, P. D. A. F. (eds.), Cognition and Motor Processes, 149–160. Berlin: Springer.
Bach-y-Rita, P., Collins, C. C., Saunders, F. A., White, B., & Scadden, L. (1969). Vision substitution by tactile image projection. Nature, 221(5184), 963–964.
Block, N. (1995). On a confusion about a function of consciousness. Behavioral and Brain Sciences, 18(2), 227–247.
Bompas, A., & O’Regan, J. K. (2006). Evidence for a role of action in colour perception. Perception, 35(1), 65–78.
Buhrmann, T., Di Paolo, E., Barandiaran, X. (2013). A dynamical systems account of sensorimotor contingencies.‖ Frontiers in Psychology, 4, 285.
Chalmers, D. J. (1996). The Conscious Mind: In Search of a Fundamental Theory. Oxford University Press.
Clark, A. (2013). Whatever next? Predictive brains, situated agents, and the future of cognitive science. Behavioral and Brain Sciences, 36(03), 181–204.
Clark, A., & Chalmers, D. (1998). The extended mind. Analysis, 58(1), 7–19.
Degenaar, J. (2013). Perception from the phenomenal stance. Logique et Analyse, 223, 273–286.
Degenaar, J. (2014). Through the inverting glass: first-person observations on spatial vision and imagery. Phenomenology and the Cognitive Sciences, 13(2), 373–393.
Degenaar, J., & Keijzer, F. (2009). Workspace and sensorimotor theories: complementary approaches to experience. Journal of Consciousness Studies, 16(9), 77–102.
Degenaar, J., & O’Regan, J. K. (forthcoming). Sensorimotor theory and enactivism.
Dehaene, S., & Naccache, L. (2001). Towards a cognitive neuroscience of consciousness: basic evidence and a workspace framework. Cognition, 79(1-2), 1–37.
Dennett, D. C. (1991). Consciousness Explained. Little, Brown and Company.
Dennett, D. C. (2001). Are we explaining consciousness yet? Cognition, 79(1-2), 221–237.
DeValois, K. K., Webster, M. A. (2011). Color vision. Scholarpedia, 6(4):3073.
Di Paolo, E. A., Barandiaran, X. E., Beaton, M., & Buhrmann, T. (2014). Learning to perceive in the sensorimotor approach: Piaget’s theory of equilibration interpreted dynamically. Frontiers in Human Neuroscience, 8, 551.
Friston, K. (2008). Hierarchical models in the brain. PLoS Computational Biology 4 (11) e1000211.
Gibson, J. J. (1979). The Ecological Approach to Visual Perception. Houghton Mifflin.
Helmholtz, H. von ([1878]/1977). The facts in perception (with notes and comments by Moritz Schlick), in Cohen, R. S.; Elkana, Y. (eds.) Hermann von Helmholtz: Epistemological Writings, 115-185. Dordrecht: Reidel Publishing Company.
Hurley, S. L. (1998). Consciousness in Action. Harvard University Press.
Hurley, S. L. (2001). Perception and action: alternative views. Synthese, 129(1), 3–40.
Hurley, S. L. (2010). Varieties of externalism. In Menary, R. (ed.) The Extended Mind, 101-153. Cambridge, Mass: MIT Pres.
Hurley, S. L., & Noë, A. (2003). Neural plasticity and consciousness. Biology and Philosophy, 18(1), 131–168.
Husserl, E. (1997). Thing and Space: Lectures of 1907. Rojcewicz, R. (ed., transl.) Kluwer.
Hutto, D. D. (2005). Knowing what? Radical versus conservative enactivism. Phenomenology and the Cognitive Sciences, 4, 389-405.
Kaspar, K., König, S., Schwandt, J., König, P. (2014). The experience of new sensorimotor contingencies by sensory augmentation. Consciousness and Cognition, 28, 47–63.
Levine, J. (1983). Materialism and qualia: the explanatory gap. Pacific Philosophical Quarterly, 64, 354–61.
Merleau-Ponty, M. (1945). Phenomenology of Perception. Smith, C. (transl. 2002). Routledge.
Mossio, M., & Taraborelli, D. (2008). Action-dependent perceptual invariants: from ecological to sensorimotor approaches. Consciousness and Cognition, 17(4), 1324–1340.
Myin, E., & Degenaar, J. (2014). Enactive vision. In Shapiro, L. (ed.) Routledge Handbook of Embodied Cognition, 90-98. Oxon: Routledge.
Myin, E., & O'Regan, J. K. (2002). Perceptual consciousness, access to modality and skill theories: A way to naturalize phenomenology? Journal of Consciousness Studies, 9 (1), 27-45.
Noë, A. (2001). Experience and the active mind. Synthese, 61(1), 41-60.
Noë, A. (2004). Action in Perception. MIT Press.
Noë, A. (2012). Varieties of Presence. Harvard University Press.
O'Regan, J. K. (1992). Solving the "real" mysteries of visual perception: the world as an outside memory. Canadian Journal of Psychology, 46, 461-88.
O’Regan, J. K. (2010). Explaining what people say about sensory qualia. In Gangopadhyay, N., Madary, M., & Spicer F. (eds.) Perception, Action and Consciousness: Sensorimotor Dynamics and Two Visual Systems, 31-50. Oxford: Oxford University Press.
O’Regan, J. K. (2011). Why Red Doesn’t Sound Like a Bell: Understanding the Feel of Consciousness. New York: Oxford University Press.
O’Regan, J. K., Myin, E., & Noë, A. (2005). Skill, corporality and alerting capacity in an account of sensory consciousness. Progress in Brain Research, 150, 55–68.
O’Regan, J. K., & Noë, A. (2001). A sensorimotor account of vision and visual consciousness. Behavioral and Brain Sciences, 24(5), 883–917.
O’Regan, J. K., Rensink, R. A., & Clark, J. J. (1999). Change-blindness as a result of “mudsplashes.” Nature, 398(6722), 34–34.
Pessoa, L., Thompson, E., & Noë, A. (1998). Finding out about filling-in: a guide to perceptual completion for visual science and the philosophy of perception. Behavioral and Brain Sciences, 21(21), 723–802.
Philipona, D. L., & O’Regan, J. (2006). Color naming, unique hues, and hue cancellation predicted from singularities in reflection properties. Visual Neuroscience, 23(3-4), 331–339.
Philipona, D., O’Regan, J. K., & Nadal, J. P. (2003). Is there something out there? Inferring space from sensorimotor dependencies. Neural Computation, 15(9), 2029–2049.
Rensink, R. A., O’Regan, K. J., & Clark, J. J. (2000). On failures to detect changes in scenes across brief interruptions. Visual Cognition, 7(1-3), 127–145.
Seth, A. K. (2014). A predictive processing theory of sensorimotor contingencies: Explaining the puzzle of perceptual presence and its absence in synesthesia, Cognitive Neuroscience, 5(2), 97-118.
Simons, D. J., Franconeri, S. L., & Reimer, R. L. (2000). Change blindness in the absence of a visual disruption. Perception, 29(10), 1143–1154.
Taylor, J. G. (1962). The Behavioral Basis of Perception. Yale University Press.
Terekhov, A. V., & O’Regan, J. K. (forthcoming). Space as an invention of biological organisms.
Thomas, N. J. T. (2014). The multidimensional spectrum of imagination: images, dreams, hallucinations, and active, imaginative perception. Humanities, 3(2), 132–184.
Thompson, E. (2007). Mind in Life: Biology, Phenomenology, and the Sciences of Mind. Harvard University Press.
