Some of the external links on this page lead to PDF files, and you will need Adobe Acrobat Reader® to view them. If you do not have it, get it here, free and legal.

Discussion

1. Direct Evidence for Content-Appropriate Exploratory Perceptual Activity During Imagery.

The heart of the Perceptual Activity Theory of Imagery (PA theory) is the contention that the experience of imagery arises from our more or less covert reenactment of the specific exploratory perceptual behavior that would be appropriate for exploring the imagined object if it were actually present. Direct experimental support for this now comes from recent work on eye movements during imagery. In the account of PA theory given in ATOITOI I stressed covert reenactment processes occurring within the perceptual instruments or even entirely within the brain, assuming that reenactment of more overt exploratory actions, such as eye movements, would be largely inhibited. Although I still think such covert processes play the major role in imagery, I am now gratified to discover that the inhibition of certain more overt reenactment processes is not so strong or so pervasive as I had feared. A real correlation between eye movements during perceptual exploration and movements during imagery has now been demonstrated in several experiments.

The most direct demonstration of this builds upon earlier research by Stark and others that shows that looking at a visual presentation results in cognitively directed saccadic eye movements that actively explore the display in a structured, way (Ellis & Stark, 1979; Stark & Ellis, 1981; Noton, 1970; Bozkov, Bohdanecky, & Radil-Weiss, 1982; Gaarder, 1975; Kowler, 1996; Yarbus, 1967). In fact, it was demonstrated that under the right conditions, and with suitable stimuli, people soon establish a regular scanpath, a sequence of saccades and intervening fixations through which they cycle repeatedly, apparently exploring the stimulus over and over in a stereotyped sequence (Noton & Stark, 1971a, 1971b, 1971c). (It should be noted that scanpaths vary both from person to person and from stimulus to stimulus: i.e. each experimental subject established their own distinctive set of scanpath patterns, a different one for each stimulus drawing.)

Brandt & Stark (1997) have now shown that when people form mental images of previously learned visual stimuli, they also spontaneously move their eyes in a regular scanpath demonstrably similar to the one established whilst viewing the specific stimulus now being imagined. The subjects were first presented with a simple visual pattern to be memorized, and shortly afterwards they were asked to imagine that pattern. Their eye movements were recorded throughout the procedure, and it was possible to show that the unique scanpaths established during presentation of each stimulus pattern spontaneously reappeared when the subjects tried to imagine that same pattern. Laeng & Teodorescu (2002) have since replicated and confirmed these findings, demonstrating that "Eye scanpaths during visual imagery reenact those of perception of the same visual scene." Furthermore, they found that when subjects were free to move their eyes whilst visually inspecting a pattern, but were required to fix their gaze when recalling it (via imagery), they did not remember the pattern as well as the did when allowed to move their eyes freely during both inspection and recall phases of the experiment. This suggests that the reenacted eye movements play a real functional role in the production of the imagery, and are not mere "epiphenomena" of internal processes. This clearly constitutes direct experimental support for a PA theory of imagery (Laeng & Teodorescu, 2002).

Further confirmation, but this time with imagery of moving (rotating) objects, comes from the finding that the pattern of saccades made by subjects in visually following the motion of a rotating disk is spontaneously reenacted when they are asked to asked to visualize such a rotation (de'Sperati, 2003). There is also evidence that, under the right conditions, the tracking (as opposed to saccadic) eye movements that people make when following a moving stimulus are also reenacted when they are merely imagine such a motion (Antrobus, Antrobus, & Singer, 1964; Deckert, 1964; Zikmund, 1972; but see Brown, 1968).

Both Brandt & Stark (1997) and Laeng & Teodorescu (2002) used rather simple visual stimuli in their experiments, which thus, presumably, gave rise to rather simple mental images. However, very similar results have now been found using more complex and "naturalistic" stimuli. Johansson, Holsanova, & Holmqvist (2005) used either a picture of a visual scene or a verbal description of a relatively complex visual scene as stimuli, and recorded their subjects' eye movements both as they encoded the stimulus, and as they recalled the scene from memory (by verbally describing it - they were not explicitly asked to recall it as a mental image). As in the earlier experiments, the scanpath patterns in the recall phase were measurably similar to those recorded during encoding (presumably the subjects who heard a verbal description were forming an image as they heard it).

Two other recent studies, by Demarais & Cohen (1998) and by Spivey & Geng (2001), also provide evidence that eye movement patterns during imagery reflect the spatial structures even of complex, naturalistic mental images.

The imagery in these experiments was generated in response to verbal descriptions of naturalistic scenes, and must surely have been relatively complex and detailed. Demarais & Cohen (1998) studied eye movements made whilst subjects solved transitive inference ("n-term series") reasoning problems involving such tasks as the following (concerning an arrangement of items on imaginary kitchen shelves): "A jar of pickles is below a box of tea bags; the jar of pickles is above a can of coffee; where's the can of coffee?" (Demarais & Cohen, 1998 p. 231). The subjects were not explicitly told to generate imagery, but it is well known that most subjects report spontaneously generating and using visual images in thinking about problems of this type (Huttenlocher, 1968; Shaver, Pierson, & Lang 1974; Sternberg, 1980; Nyíri, 2001). Demarais & Cohen found that, with problems such as the one quoted, subjects (who were not aware that their eye movements were being recorded) made significantly more up and down eye movements than they did during otherwise similar control tasks that did not involve linearly arrayed items. When solving similar problems in which the items were described as arrayed in a horizontal rather than a vertical line, the subjects made more side to side eye movements than when doing control tasks. Clearly (modulo assumptions about what aspects of perceptual testing get suppressed during imagery) this result is just what PA theory would predict would happen during the experiencing of such images.

Demarais & Cohen (1998) also suggest an interesting and quite plausible explanation of why the results of some (not all) earlier investigations imply that subjects' eyes tend to move less when they are doing mental tasks that supposedly depend upon imagery than when they are doing other sorts of task (e.g. Weiner & Ehrlichman, 1976; Hiscock & Bergstrom, 1981; Ehrlichman & Barrett, 1983; Bergstrom & Hiscock, 1988). Many experimental imagery tasks, it is suggested, do not involve the inspection of the extremities of a linearly extended image, but, rather, internal structural details of an image that may be relatively compact. Most eye movements they evoke are thus likely to be low amplitude "fixation" movements (microsaccades) rather than the large, more easily measurable saccades evoked by spatially linear transitive inference tasks.

Spivey & Geng (2001 experiment 1) found very similar results in a related experiment. This differed from the experiment by Demarais & Cohen mainly in that the subjects were given explicit instructions to form imagery (on the basis of verbal descriptions) rather than the imagery being induced indirectly, via the demands of a transitive inference task. Subjects were instructed to form images in accordance with passages such as the following:

Imagine you are standing across the street from a 40 story apartment building. At the bottom there is a doorman in blue. On the 10th floor, a woman is hanging her laundry out the window. On the 29th floor, two kids are sitting on the fire escape smoking cigarettes. On the very top floor, two people are screaming. (Spivey & Geng, 2001 p. 237)

The subjects' eye movements were recorded whilst they were hearing the italicized portion of this passage, and it was found that they made a significantly greater proportion of upwardly directed saccades than were seen during a control imagery task (forming an image in response to a similar descriptive passage, but without the strong suggestion of the more salient aspects of the imagined scene being high up, or, indeed, in any particular direction). Similar passages that suggested the interesting aspects of the imagined scene would be in a downwards direction (e.g., watching from the top of a canyon as people rappel down its opposite wall), or to the left or right, produced significantly more saccades directed downwards, leftwards, and rightwards respectively, as compared to the control (and the earlier results of Jacobson (1932) suggest that even more such eye movements would probably be observed if they were not often being actively inhibited during imagery). Once again, this is very much what one would expect on the basis of PA theory, and nothing inherent in picture theory or description theory suggests the likelihood of such results (or those of Demarais & Cohen).

Further corroboration comes from Richardson & Spivey (2000) (and experiment 2 of Spivey & Geng (2001); and see also Richarson & Spivey, 2004 part 2), where it is shown that saccadic eye movements during the recall of visually presented information tend to reflect the spatial structure of the original stimulus (which is, presumably, being visualized). Also, Hong et al. (1997) found that the pattern of REMs (rapid eye movements) of dreaming subjects are correlated with the reported contents of the subjects' dream imagery. Admittedly, it is now well recognized that dreams, including dreams involving vivid visual imagery, may occur during periods of sleep when there are no such eye movements, but this may be simply because in non-REM sleep the eye movements associated with the dream imagery have been inhibited (as is most motor activity during sleep), and have thus become covert (Nielsen, 2000). After all, according to PA theory, overt exploratory activity during imagery is the exception rather than the rule. [It should be noted that the question of whether the eye movements of REM sleep are directly related to dream imagery was also much studied and debated during the 1960s and early 1970s (Dement & Kleitman, 1957; Roffwarg et al., 1962; Moskowitz & Berger, 1969; Jacobs, Feldman & Bender, 1972; Firth & Oswald, 1975). No very clear answers emerged from this research, however. Koulack (1972) concluded, from his review of this literature, that although some of the eye movements of REM sleep may well be related to dream imagery, others are not.]

The experiments discussed above indicate that (as PA theory predicts) eye movements, or the eye movement control system, are closely involved in the generation of the experience of visual mental imagery, and the specific pattern of such movements (or the specific sequence of states of the control system) is specific to the particular experiential and intentional content of the imagery. This suggests that if the eyes of someone who is experiencing imagery are made to move in ways that are inappropriate or irrelevant to what is being imagined (i.e., that would not be those made in visually exploring the thing being imagined) then the imagery will be disrupted. This prediction is experimentally confirmed both by Ruggieri (1999), who found that if subjects who are imagining a moving object are instructed to stop moving their eyes or their head, the motion in the image is (more often than not) stopped or noticeably slowed, and by Andrade, Kavanaugh, & Baddeley (1997), who found that the vividness (and, to some extent, the emotional impact) of people's mental images was significantly reduced when they simultaneously had to carry out a visual task requiring eye movements. Perhaps unsurprisingly, the vividness was also somewhat attenuated when the subjects were required to do an otherwise similar visual task that did not require eye movement, and when they had to do a task that called for spatial control of hand rather than eye movements. However, the effect of the eye movement task was markedly stronger. These findings also make sense of the fact that when people want to get rid of an unpleasant or disturbing mental image they may deliberately move their eyes about in a violent and random manner (Antrobus, Antrobus, & Singer (1964). (It may also help to explain why Marks (1973) found that when they recall pictures from memory, vivid imagers tend to display less eye movement than do those whose imagery is less vivid. Perhaps a greater ability to keep irrelevant and disruptive eye movements under control is a part of the "skill" of being a vivid imager.) [ Andrade, Kavanaugh, & Baddeley (1997), discuss their results in the light of Baddeley's theory of the "visuo-spatial sketchpad" function of working memory (Baddeley, 1976, 1994; Baddeley & Hitch, 1974). This, however, is essentially a theory about the function of imagery in the broader cognitive system, and, as such, there is no reason to think that it is incompatible with PA theory, which is a theory of the nature of imagery itself (see Thomas, 2005 §4.2.1).] .]

Of course, as noted in ATOITOI (§2) none of the extant imagery theories (or rather, "research programs") is such that it can be decisively refuted (or verified) by any particular set of experimental results. Certainly Kosslyn's quasi-pictorialism is sufficiently loosely formulated that just about any empirical evidence can be accommodated by making suitable auxiliary assumptions or ad hoc maneuvers (and as Pylyshyn (1981) has noted, defenses of pictorialism have frequently taken advantage of this fact). Thus it is no surprise to find that Mast & Kosslyn (2002) have wasted no time in spinning the Laeng & Teodorescu (2002) results in such a way as to make them seem consistent with pictorialism. Given the pervasive influence of pictorialism, and the general lack of awareness that there might be any alternative explanations of imagery (apart from descriptionism) (Thomas, 2003), it should come as no surprise that Demarais & Cohen (1998) also sketch a pictorialist account of their results along similar lines, and even Laeng & Teodorescu (2002) themselves seem to be very open to this sort of interpretation of their work (although they also explicitly recognize how well their findings jibe with PA Theory and a thoroughgoing rejection of inner pictures). Mast & Kosslyn (2002) suggests that the eye movements observed during imagery reflect the movement of pictorial images (as based on individual "visual snapshots," corresponding to eye fixations) across, or in and out of, the visual buffer where they are supposedly displayed. What seems to be being suggested is a process akin to the stitching together of a number of relatively small inner pictures into a more panoramic imaginal experience of a scene. (It is implied that this may be necessary because of the fact that, in a single eye fixation, only about 2° of visual angle is focused on the fovea, and thus seen with high acuity. Some theorists, such as O'Regan (1992) and, implicitly, Ballard (1991) take this as evidence that vision itself does not involve any picture-like inner display of the sort envisaged by Kosslyn. However, from a determinedly pictorialist perspective it means that an individual image, as displayed at any one moment on the visual buffer, would only be detailed in its central 2°.) But although, when suitable auxiliary assumptions are introduced, the new eye-movement results can undoubtedly be rendered compatible with the core commitments of pictorialism, it is far from clear that these assumptions are consistent with ways that Kosslyn has interpreted his pictorialism in the past in dealing with other empirical findings (such as his favored interpretation of his own "mental scanning" research (Kosslyn, Ball, & Reiser, 1978; Kosslyn, 1980, 1994), and his vaunted claims to have measured the "visual angle of the mind's eye" (Kosslyn, 1978), which he at that time held to be very much greater than 2°). In any case, it is important to stress that, unlike picture theory, PA theory does not merely accommodate but, rather, strongly suggests the likelihood of phenomena of the sort found by the recent eye movement studies. This prediction arises not from any auxiliary assumptions, but rather from the core commitments of the PA research program.

Furthermore, there is recent evidence for the involvement of overt perceptual exploratory actions in olfactory as well as in visual imagery. Bensafi et al (2003) measured the sniffing behavior of subjects instructed to imagine various smells, and found that it parallels what occurs when real smells are being perceived. If subjects were prevented from sniffing (by a nasal clip) the vividness of their olfactory imagery was significantly decreased. Furthermore, when they imagined pleasant odors they spontaneously took large, deep sniffs, and when they imagined unpleasant odors they took short, shallow ones (just as happens when people really smell pleasant or unpleasant smells). This experiment is significant not only, because it extends the sorts of findings discussed above to a non-visual mode of imagery, but also because the sort of quasi-pictorialist interpretation of the eye-movement findings that is suggested by Mast & Kosslyn (2002) does not seem applicable in this case. What could possibly be the olfactory equivalent of the little pictorial sub-images that Mast & Kosslyn imagine as being stitched together by the imagery eye movements?

There is also evidence for yet another sort of eye movement being involved in visual imagery (in addition to the saccadic and tracking movements already discussed). Ruggieri & Alfieri (1992) measured changes in the length of the optical axis (i.e., the thickness) of the crystalline lens of people's eyes when they were asked to imagine either nearby objects ("reading a word on the page of a book"), or faraway ones ("a ship on the horizon"). Sure enough, when asked to imagine the nearby objects, the length of the optical axis increased, just as it does when we are bringing a real nearby stimulus into focus. Once again, these results make good sense if we conceive of the adjustments in the lens during imagery as part of the acting out of the actual perceptual tests we would make in actually seeing something, but it is not at all clear how they could be accounted for in terms of a pictorialist (or, come to that, a descriptionist) story about imagery, even one amended in the way that Mast & Kosslyn (2002) suggest.

According to Bridgeman (1992) "most human behaviors are eye movements. . . . Saccades occur several times per second throughout waking life: even during what we think of as steady fixation, microsaccades continue at the same rate." Although we are not conscious of making them, and have little if any conscious control over them, even these microsaccades appear to be under cognitive control, adjusted appropriately to the stimulus (Gaarder, 1975). Thus it is not so implausible as it might at first seem that such movements are closely involved in such a ubiquitous aspect of our mental lives as imagery. (Much the same could probably be said for sniffs.) Although we normally have little if any conscious awareness of these exploratory perceptual behaviors, we are conscious of both the perceptions and the mental imagery that they make possible.

Nevertheless, it is worth re-emphasizing that, as I conceive PA theory, there is a lot more to visual exploration than just eye movements (and, I am sure, a lot more to olfaction than sniffing), and I doubt whether actual eye movement reenactments are either sufficient, or even, in most cases, necessary for experiencing visual imagery. During visual perception, much of the visual exploration process, as I conceive it, goes on internally, within the brain, and involves the exploration (or querying) of not only the (external) optic array, but also its internal informationally correlated causal products, such as the momentary bottom-up excitation pattern of the retinotopically mapped brain areas, such as V1 (what Pylyshyn (2002b) calls the Tootell Display). During imagery,although there is neither a relevant optic array nor a relevant Tootell Display, we reenact the relevant exploratory procedures as if there were.

Furthermore, what you see and what you imagine will depend not only on where you direct your attention, but, equally, on what you look for there. Differences in what you are looking for (what perceptible features or characteristics) ought to be reflected in fine grained differences at the neural level, in the brain, but could hardly be expected to produce reliably and consistently measurable differences in eye movements. Although eye movements are certainly a significant aspect of visual exploration, the focus on eye movements by many researchers inclined toward active theories of perception seems to be largely a result of the fact that they are much easier to observe and measure (at the requisite fine level of detail) than the relevant brain events. It is gratifying that, despite these limitations, eye movement studies have now produced clear evidence in favor of the PA theory of imagery.

[Note: The development of my version of of PA theory was, in fact, significantly influenced by the scanpath research of Noton & Stark (1971a, 1971b; Noton, 1970), and I discussed this work in my doctoral thesis (Thomas, 1987). Unfortunately I did not have the space, in ATOITOI, to cover it adequately. I would like to thank Lawrence Stark, for alerting me to his work with Brandt that now explicitly extends the scanpath research program to encompass imagery. Thanks also to Bruno Laeng for alerting me to Spivey's work.]

Back to table

2. Support from Neurology (Clinical Studies of People with Brain Lesions).

Neuroscientist Paolo Bartolomeo (2002) has recently reviewed, in some detail, the clinical literature on deficits in visual mental imagery (and related deficits in visual perception) resulting from brain injury. He concludes that this evidence is not consistent with the quasi-pictorial theory of Kosslyn, and argues instead that the perceptual activity (or PA) theory, as outlined in ATOITOI, provides a better account of the full range of relevant neuroscientific and phenomenological evidence.

Bartolomeo & Chokron (2002) come to the same conclusion in a brief discussion that focuses specifically on the strange neurological syndrome of unilateral neglect wherein (in typical cases) people with damage to the right parietal cortex of their brain show a strong tendency to ignore or fail to notice things to their left (although they are not blind to things to the left, and can see them if their attention is explicitly drawn to them). People with this syndrome may fail to eat food on the left hand side of their plate (even if they are hungry), or fail to notice that a picture of a house shows it as being on fire, if the flames and smoke are coming only from what is, from their perspective, its left hand side (Young & de Haan, 1990; Bartolomeo & Chokron, 2001). Bisiach & Luzzatti (1978; Bisiach, Luzzatti & Perani, 1979) showed that patients with this syndrome may show a similar effect when they are asked to recall visual features of a familiar scene from memory by forming a mental image of it as seen from a particular vantage point: The patients fail to report many features that should be to the left of their image, and do much better with those to the right.

The reality of this latter phenomenon, sometimes called representational neglect, is now quite well established (see, e.g., Ogden, 1985; Bisiach & Berti, 1990; Bartolomeo & Chokron, 2001), and I did discuss it briefly in the penultimate draft of ATOITOI. It seemed to me that it was very hard to see how this phenomenon might be explained in a principled way from the perspective of description theory. (Pylyshyn (2002 b §R53) does have a try, but in the process seems to move a long way toward embracing PA theory!) The phenomenon might conceivably be explained in terms of the patients failing to pay attention to (or perhaps failing to properly construct) the left hand side of an inner quasi-picture, but why should we expect attention to an inner picture to be affected in the same way as attention to the actual visual environment? Peripheral actions such as eye and head movements play a large and crucial role in the directing of attention during visual perception, but they clearly cannot be playing anything like the same role in directing the putative "attentional spotlight" around inside the brain.

On the other hand, it seemed to me that, because unilateral neglect seems to be essentially a defect of active attention (a failure to look for things to the left, to explore to the left), and because PA theory regards imagery as a product of just such active, attentional, exploratory processes, PA theory would accommodate the representational neglect phenomenon in a very straightforward and natural way. I was particularly encouraged in this view by the interpretations of the phenomenon offered by Sunderland (1990), and especially Bisiach, Luzzatti & Perani (1979), who explicitly discuss the phenomenon in terms of schemata (although, admittedly, they use the term in a sense that is not quite identical to that given it by Neisser (1976) and myself). I would probably have been even more encouraged if I had been aware of the interpretation of the phenomenon offered by Bartolomeo, D'Erme, & Gainotti (1994). However, I decided to cut the discussion of this out of ATOITOI, mainly because I became aware that occasional reports had appeared in the clinical literature of visual neglect apparently occuring without representational neglect, and of representational neglect without apparent visual neglect (Coslett, 1997), and I came to feel that my armchair grasp of the complex clinical neurological issues was insufficient for me to be able to give an authoritative account of them. It is very gratifying for me now to find that experts on the unilateral neglect syndrome, with first hand clinical experience of its manifestations and complexities, have come to much the same conclusions as I initially did. Bartolomeo & Chokron (2002; Bartolomeo, 2002) argue that PA theory (in the form outlined in ATOITOI) provides much the most plausible and parsimonious framework for explaining the representational neglect phenomenon. (It is notable also that, despite his ingrained commitment to the description theory of imagery, Pylyshyn (2002b) essentially accepts their arguments.)

This conclusion has now received further support from a study by Chokron, Colliot, & Bartolomeo (2004), who have found that (in most of the cases they studied) representational neglect appears noticeably less severe when the patient's eyes are closed. They had their subjects, all patients suffering from neglect, reproduce drawings form memory (i.e., presumably from an image) in both eyes-open and eyes-closed conditions. Despite the obvious difficulties of drawing with one's eyes closed, patients in this condition generally succeeded in drawing in the left as well as the right side of the object to be depicted, whereas with their eyes open they neglected to draw the left side. For example, neglect patients asked to draw the numbers of a clock face on a circle they were given, generally failed, when their eyes were open, to put in the numbers 7 to 11, although 12 to 6 were accurately placed. When their eyes were closed, the same subjects succeed in making a complete ring of all 12 numbers (even if, as one might expect even from an unimpaired subject drawing with eyes closed, the arrangement was often neither very circular nor very well fitted to the supplied circle).

Apparently, then, it is seeing itself that causes these patients to neglect the left-hand side of their world. This is because their condition makes it difficult for them to disengage visual attention from things they see on their right (Posner et al., 1984). It thus appears that the representational neglect phenomenon does not arise because one side of an inner representation is missing, or even because of a failure to pay attention to the left-representing parts of such a representation (whether it pictorial, propositional, or whatever). In fact, the eyes-closed results seem to show that there is little or nothing wrong with the patient's knowledge of what things look like (at least things whose appearance they learned before they began to suffer from neglect). Rather, their inability to imagine things to the left (when their eyes are open) appears to be a result of a deficit of control of visual exploratory behavior. It is only to be expected that when a patient's actual visual attention cannot be disengaged from the right-hand side of space, this will interfere not only with their ability to pay attention to actual things to the left, but also with their ability to simulate or enact paying attention to hypothetical things to the left (which, according to PA theory, is what constitutes imagining things to the left).

ADD THE STUFF ON PRISM ADAPTATION AND REPRESENTAIONAL NEGLECT (RODE ET AL 1998, 2001) HERE. THIS LINKS THE NEGLECT EVIDENCE TO THE EYE MOVEMENT EVIDENCE.

In another recent study of representational neglect, Rode et al. (2004) note the consistency of their results with PA theory, and express doubt as to whether they can be explained in terms of an inner "mental screen". The study involved a patient suffering from visual neglect, but with an intimate knowledge of the geography of France, who was asked to form a mental image of a map of the country and, in two minutes, to name as many as possible of the towns that he could "see" in his image. Nearly all the towns he named were toward the right of the map (i.e. in eastern France). When asked to name French towns without being instructed to imagine the map, he was able to name a similar number of towns in the same amount of time, but in this condition they were more or less evenly distributed around the country.

Admittedly the direct support this particular study lends to PA theory is relatively weak (alternative accounts of the results are probably possible), but I take note of it because of something the authors themselves somehow fail to mention: the results are clearly difficult to reconcile with common coding theories of memory (which are are usually associated with description theories of imagery such as those of Pylyshyn (1973, 1981, 2002a, 2002b, 2003b) or Hinton (1979)). On the other hand, these results are very much what would be expected from the perspective of the rival Dual Coding theory of memory (Paivio, 1971, 1986, 1991; Sadoski & Paivio, 2001). (As argued in ATOITOI, Dual Coding theory is equally consistent with both PA and pictorial theories of imagery.)

Back to table

9. An Argument from Psychological Phenomenology.

Kubovy (2003) points out certain aspects of the phenomenology of imagery, in particular: the intentionality of imagery (and perception); the fact that in imagery (unlike perception) we experience the intentional object as absent; and the fact that when we visually imagine a scene (such as imagining the front of one's house, in order to count how many windows it has) we necessarily imagine it as seen from a particular vantage point (pace Gibson, 1974). For Kubovy, this last fact implies that the "act of imagining contains the experiencer." He then argues as follows:

A theory of mental imagery must take account of these observations. But current theories do not. Regarding the fact that you knew where you stood when you counted windows, perhaps we need a theory of embodied imagination (perhaps in the direction proposed by Ballard et al. (1997)), according to which imagining a visible object is a partial reenactment of many of the bodily activities involved in perceiving something, not just an activation of the visual system. (Kubovy, 2003 p. 581)

It is quite clear from an earlier passage in Kubovy's article that the "current theories" of imagery that are being judged as inadequate here are the quasi-pictorial theory of Kosslyn and the description theory of Pylyshyn. Like so many, Kubovy appears to be unaware that there might be any alternative theories of imagery, such as PA theory, in the literature. (He seems unaware not only of my version of PA theory, but also of the several other earlier versions: see ATOITOI §2.3 for references. The article by Ballard et al. (1997) that he cites [and that I cite above] presents an active, embodied theory of perception, but does not address the issue of imagery at all.) Nevertheless, with its call for a "theory of embodied imagination," and its description of imagining as "a partial reenactment of many of the bodily activities involved in perceiving something," this is clearly an argument for some form of PA theory. (Note that ATOITOI endorsed, and drew significantly on, Ballard's earlier (1991) work on active perception.)

As it stands, Kubovy's very terse argument does not seem particularly compelling to me, but perhaps it might become more persuasive if it were fleshed out a little more. In any case, it is an argument for PA theory quite different from any that I have made (and, so far as I know, from any anyone else has made) and I am happy to find a fresh line of support.

Back to table

10. Misrepresentations of Perceptual Activity Theory.

(a) Kosslyn & Thompson (2003).

The main concern of Kosslyn & Thompson (2003) is to suggest an explanation for why it is that many neuroimaging studies of visual imagery have found evidence for elevated activity in primary visual cortex during imagery, many other such studies indicate that there is no such elevated activity. The explanation that they suggest seems quite plausible to me, and is broadly consistent with the commitments of Perceptual Activity (PA) theory as outlined in ATOITOI (Thomas, 1999). However, although it does not really vitiate their main argument, in the course of their exposition they seriously misrepresent the nature of PA theory.

Their argument involves the classification of imagery theories into two broad classes, which they call "perceptual anticipation theories" and "propositional theories" respectively. The latter class, of course, corresponds to the well known "propositional" or "description" theory associated mainly with Pylyshyn (see Thomas, 2003). However, the class of "perceptual anticipation theories" is supposed to encompass both quasi-pictorial theories of the sort long championed by Kosslyn (e.g., 1980, 1994; also Miyashita, 1995), and PA theories (they cite both ATOITOI (Thomas, 1999) and Neisser's (1976) version of PA theory, which was what originally inspired my interest in this way of thinking about imagery). I think they make a good case that "perceptual anticipation theories" (of either subtype) are able to give a much more plausible account of the range of neuroimaging data than are "propositional" theories. My objection is to their account of what "perceptual anticipation theories" hold:

According to perceptual anticipation theory, mental images arise when one anticipates perceiving an object or scene so strongly that a depictive representation of the stimulus is created in early visual cortex (cf. Kosslyn, 1994; McCrone, 2001; Miyashita, 1995; Neisser, 1976; Thomas, 1999). According to this class of theories, visual long-term memories of shapes are stored in an abstract code in the inferior temporal lobes. . . . According to perceptual anticipation theory, the local geometry of shapes is only implicit in the longterm memory (LTM) representation, and is made explicit by generating patterns of activation in topographically organized early visual cortex. (Kosslyn & Thompson, 2003 pp.724-5)

The first claim here, about anticipation, is an appropriate, if rather elliptical, characterization of PA theory (the concept of perceptual anticipation plays a particularly central role in Neisser's (1976) version). [I am not sure how well it really applies to quasi-pictorial theory, I do not recall anything like anticipation playing any very large role in Kosslyn's earlier theoretical work on imagery, but that is another matter.]

However, the second claim, about depictive representation, is completely at odds with a central commitment of PA theory (certainly as I understand it, and I think as Neisser understood it too (see Neisser, 1979)). PA theory explicitly and emphatically rejects the notion that mental images are depictive representations or spatial patterns of neural activation (whether in early, topographically organized visual cortex or anywhere else). Although it is quite possible that such activation patterns, isomorphic to the spatial structure of whatever is being imagined, may sometimes arise in topographically organized cortex during imagery, according to PA theory they are not to be identified with the mental image, nor are they the cause of the experience of imagery. Rather, they are collateral effects of the covert enactment of exploratory perceptual behavior. It is this enactment, and not any pattern of brain activation that may arise from it, that constitutes imagery, and gives rise to quasi-perceptual experience. (I have no strong views about where in the brain the information that guides this enactment, what Neisser and I call the schema, is stored long-term. Perhaps, as Kosslyn & Thompson imply, it is all in the inferior temporal lobes, although I am inclined to suspect that it is distributed much more widely.)

Although the Kosslyn & Thompson paper is well worth reading for its plausible explanation of the discrepancies between the different neuroimaging studies, and for the good case it makes for thinking that both quasi-pictorial and PA theories can account for these findings better than "propositional" theory can, please do not be misled by its quite false account of PA theory itself.

(b) Dartnall (2005)

Dartnall provides a brief account of PA theory that, although not necessarily incorrect, if carefully interpreted, may be misleading. He says that, according to ATOITOI, "rather than storing inner analogs of the external world, we generate them by running our perceptual abilities offline" (Dartnall, 2005 p. 139, emphasis in original). What concerns me, here, is the metaphor of generation (which he repeats twice more). Perhaps it is fair to say that according to PA theory, the process of running our perceptual abilities offline generates imagery experience (i.e., quasi-perceptual experience). However, it would be very natural to read Dartnall's words as implying the generation of some sort of representational product (an inner analog, an activation pattern in a topographical map, or some other sort of representational token), and that this product is the mental image (which then, perhaps, causes the quasi-perceptual experience). For PA theory, however, there is no such product. Imagery is the running of our perceptual abilities offline, not something generated by it.

Dartnall's mistake (if it is one) probably does not affect the cogency of the overall argument of his paper. I just want to forestall the possibility of his readers being misled as to the real commitments of PA theory.

Back to table

Bibliography
(other items)

Baddeley, A.D. (1976). The Psychology of Memory. London: Harper.

Baddeley, A.D. (1994). Working Memory. Oxford: Oxford University Press.

Baddeley, A.D. & Hitch, G. (1974). Working Memory. In G.H. Bower (Ed.) The Psychology of Learning and Motivation, Vol. 8. New York & London: Academic Press.

Ballard, D. H. (1991). Animate vision. Artificial Intelligence (48) 57-86.

Bartolomeo, P. & Chokron, S. (2001). Levels of Impairment in Unilateral Neglect. In M. Behermann (ed.), Disorders of Visual Behavior (Handbook of Neuropsychology, 2nd edn. Vol. 3) (pp. 67- 98). Amsterdam: Elsevier Science.

Bartolomeo, P., D'Erme, P., & Gainotti G. (1994). The Relation between Visuospatial Neglect and Representational Neglect. Neurology (44) 1710-1714.

Bergstrom, K.J. & Hiscock, M. (1988). Factors Influencing Ocular Motility During the Performance of Cognitive Tasks. Canadian Journal of Psychology (42) 1–23.

Bisiach, E. & Berti, A. (1990). Waking Images and Neural Activity. In R.G. Kunzendorf & A.A. Sheikh (eds.). The Psychophysiology of Mental Imagery: Theory, Research and Application (pp. 67-88). Amityville, NY: Baywood.

Bozkov, V., Bohdanecky, Z., & Radil-Weiss, T. (1982). Role of the Physical and Semantic Contents of Form During Eye Scanning. In R. Groner & P. Fraise (Eds.). Cognition and Eye Movements. Amsterdam: North Holland.

Bridgeman, B. (1992). Conscious vs Unconscious Processes: The Case of Vision. Theory & Psychology (2) 73-88.

Brown, B.B. (1968). Visual Recall Ability and Eye Movements. Psychophysiology (4) 300-306.

Coslett, H.B. (1997) Neglect in Vision and Visual Imagery: A Double Dissociation. Brain (120) 1163-1171.

Dement, W.C. & Kleitman, N. (1957). The Relation of Eye Movements During Sleep to Dream Activity: An Objective Method for the Study of Dreaming. Journal of Experimental Psychology (53) 339-346.

Ehrlichman, H. & Barrett, J. (1983). ‘Random' Saccadic Eye Movements During Verbal-linguistic and Visual-imaginal Tasks. Acta Psychologia (53) 9–26.

Ellis, S.R. & Stark, L. (1979). Reply to Piggins. Perception (8) 721-722.

Firth, H. & Oswald, I. (1975). Eye Movements and Visually Active Dreams. Psychophysiology (12) 602-606.

Gaarder, K.R. (1975). Eye Movements, Vision and Behavior. Washington D.C.: Hemisphere Publishing.

Gibson, J.J. (1974). Visualizing Conceived as Visual Apprehending Without Any Particular Point of Observation. Leonardo (7) 41-42.

Hinton, G. (1979). Some Demonstrations of the Effects of Structural Descriptions in Mental Imagery. Cognitive Science (3) 231-250.

Hiscock, M. & Bergstrom, K.J. (1981). Ocular Motility as an Indicator of Verbal and Visuospatial Processing. Memory and Cognition (9) 332–338.

Huttenlocher, J. (1968). Constructing Spatial Images: A Strategy in Reasoning. Psychological Review (75) 550-560.

Jacobs, L., Feldman, M., & Bender, M.B. (1972). Are the Eye Movements of Dreaming Sleep Related to the Visual Images of the Dreams? Psychophysiology (9) 393-401.

Jacobson, E. (1932). Electrophysiology of Mental Activities. American Journal of Psychology (44) 677-694.

Kosslyn, S.M. (1978). Measuring the Visual Angle of the Mind's Eye. Cognitive Psychology (10) 356-389.

Kosslyn, S.M. (1980). Image and Mind. Cambridge, MA: Harvard University Press.

Kosslyn, S.M. (1994). Image and Brain: The Resolution of the Imagery Debate. Cambridge, MA: MIT Press.

Kosslyn, S.M., Ball, T.M, &. Reiser, B.J. (1978). Visual Images Preserve Metric Spatial Information: Evidence from Studies of Image Scanning. Journal of Experimental Psychology: Human Perception and Performance (4) 47-60.

Koulack, D. (1972). Rapid Eye Movements and Visual Imagery During Sleep. Psychological Bulletin (78) 155-158.

Kowler, E. (1996). Cogito Ergo Moveo: Cognitive Control of Eye Movement. In M.S. Landy, L.T. Maloney & M. Pavel (eds.). Exploratory Vision: the Active Eye (pp. 51-77). New York: Springer-Verlag.

Marks, D. F. (1973). Visual Imagery Differences and Eye Movements in the Recall of Pictures. Perception and Psychophysics (14) 407-412.

Mast, F.W. & Kosslyn, S.M. (2002). Eye Movements During Visual Mental Imagery. Trends in Cognitive Sciences (6) 271-272.

McCrone, J. (2001). Going Inside: A Tour Round a Single Moment of Consciousness. New York: International.

Miyashita, Y. (1995). How the Brain Creates Imagery: Projection to Primary Visual Cortex. Science (268) 1719–1720.

Moskowitz, E. & Berger, R. (1969). Rapid Eye Movements and Dream Imagery: Are They Related? Nature (224) 613-614.

Nielsen, T.A. (2000). A Review of Mentation in REM and NREM Sleep: “Covert” REM Sleep as a Possible Reconciliation of Two Opposing Models. Behavioral and Brain Sciences (23) 851-866.

Neisser, U. (1976). Cognition and Reality. San Francisco: Freeman. [Already cited in ATOITOI - indeed, Neisser here provides the most developed earlier version of Perceptual Activity theory, the principal inspiration for the version presented in ATOITOI]

Neisser, U. (1979). Images, Models, and Human Nature. Behavioral and Brain Sciences (2) 561.

Noton, D. (1970). A Theory of Visual Pattern Perception. IEEE Transactions on Systems Science and Cybernetics (6) 349-357.

Noton, D. & Stark, L. (1971a). Eye Movements and Visual Perception. Scientific American (224 - vi) 34-43.

Noton, D. & Stark, L. (1971b). Scanpath in Eye Movements During Pattern Perception. Science (171) 308-311.

Noton, D. & Stark, L. (1971c). Scanpath in Saccadic Eye Movements While Viewing and Recognizing Patterns. Vision Research (11) 929-942.

Nyíri, J.C. (2001). The Picture theory of Reason. In B. Brogaard & B. Smith (Eds.), Rationality and Irrationality (Schriftenreihe-Wittgenstein Gesellschaft, Vol. 29). Vienna: Öbv&hpt.

Ogden, J.A. (1985). Contralesional Neglect of Constructed Visual Images in Right and Left Brain-Damaged Patients. Neuropsychologia (23) 273-277.

Paivio, A. (1971). Imagery and Verbal Processes. New York: Holt, Rinehart and Winston. [Republished in 1979 -- Hillsdale, NJ: Erlbaum]

Paivio, A. (1986). Mental Representations: A Dual Coding Approach. New York: Oxford University Press.

Paivio, A. (1991). Dual Coding Theory: Retrospect and Current Status. Canadian Journal of Psychology (45) 255-287.

Posner, M.I., Walker, J.A., Friedrich, F.J. & Rafal, R.D. (1984) Effects of Parietal Injury on Covert Orienting of Attention. Journal of Neuroscience (4) 1863-1864. [Abstract available online]

Pylyshyn, Z.W. (1973). What the Mind's Eye Tells the Mind's Brain: A Critique of Mental Imagery. Psychological Bulletin (80) 1-25.

Pylyshyn, Z. W. (1981). The Imagery Debate: Analogue Media Versus Tacit Knowledge. Psychological Review (88) 16-45.

Richardson, D.C. & Spivey, M.J. (2000). Representation, Space and Hollywood Squares: Looking at Things That Aren't There Anymore. Cognition (76) 269-295.

Richardson, D.C. & Spivey, M.J. (2004). Eye Tracking. In G.E. Wnek & G.L. Bowlin (Eds.), Encyclopedia of Biomaterials and Biomedical Engineering (pp. 568-582). New York : Marcel Dekker. [Preprint available online]

Roffwarg, H., Dement, W., Muzio, J., & Fisher, C. (1962). Dream Imagery: Relationship to Rapid Eye Movements. Archives of General Psychiatry (7) 235-238.

Sadoski, M. & Paivio, A. (2001). Imagery and Text: A Dual Coding Theory of Reading and Writing. Mahwah, NJ: Erlbaum.

Shaver, P., Pierson, L., Lang, S., 1974. Converging Evidence for the Functional Significance of Imagery in Problem Solving. Cognition (3) 359–375.

Stark, L. & Ellis, S.R. (1981). Scanpaths Revisited: Cognitive Models Direct Active Looking. In D.F. Fisher, R.A. Monty & J.W. Senders (eds.). Eye Movements: Cognition and Visual Perception (pp. 193-226). Hillsdale, NJ: Erlbaum. [Already cited in ATOITOI]

Sternberg, R.J., 1980. Representation and Process in Linear Syllogistic Reasoning. Journal of Experimental Psychology: General (109) 119–159.

Sunderland, A. (1990). The Bisected Image? Visual Memory in Patients With Visual Neglect. In P.J. Hampson, D.F. Marks & J.T.E. Richardson (eds.) Imagery: Current Developments (pp. 333-350). London: Routledge.

Thomas, N.J.T. (1987). The Psychology of Perception, Imagination and Mental Representation, and Twentieth Century Philosophies of Science. Unpublished Ph.D. thesis, Leeds University, Leeds, U.K. (A.S.L.I.B. Index to Theses 37-iii No. 37-4561).

Thomas, N.J.T. (1999). Are Theories of Imagery Theories of Imagination? An Active Perception Approach to Conscious Mental Content. Cognitive Science (23) 207-245. [ATOITOI]

Thomas, N.J.T. (2003). Mental Imagery, Philosophical Issues About. In L. Nadel (ed.) Encyclopedia of Cognitive Science (Volume 2, pp. 1147-1153). London: Nature Publishing/Macmillan.

Thomas, N.J.T. (2005). Mental Imagery. In E. Zalta (Ed.). The Stanford Encyclopedia of Philosophy. URL: http://plato.stanford.edu/entries/mental-imagery/

Weiner, S.L. & Ehrlichman, H. (1976). Ocular Motility and Cognitive Process. Cognition (4) 31–43.

Yarbus, A.L. (1967). Eye Movements and Vision. Plenum Press. New York. (Translated from the Russian of 1965 by Basil Haigh.)

Young, A.W. & de Haan, E.F. (1990). Impairments of Visual Awareness. Mind and Language (5) 29-48.