Facial expression analysis

• Dr. David Matsumoto, San Francisco State University

• Dr. Paul Ekman, Paul Ekman Group, LLC., Oakland, CA

Facial expressions refer to movements of the mimetic musculature of the face. The vast majority of these muscles are innervated by the VIIth cranial nerve, emanating from the brainstem between the pons and medulla (Figure 1). ¹ The nerve includes a motor root that supplies somatic muscle fibers to the muscles of the face, scalp, and outer ear, enabling the muscle movements that comprise facial expressions. The sensory part of the nerve enables and augments some aspects of taste and sound (Standring, 2005). (The upper eyelid is innervated by a different nerve – the oculomotor nerve; this muscle is used in expressions of surprise, fear and anger.)

The facial nerve receives impulses from multiple brain areas. Lower face muscles are represented more fully in the motor cortex than the upper face, allowing for more voluntary and learned control of the lower face; this provides the fine controls of that facial region required for speech articulation. The amount of bilateral v. contralateral fibers to the facial muscles differs depending on region, with the lower face being primarily contralateral and bilateral fibers increasing in the upper face (Matsumoto & Lee, 1993). (Note that there are large individual differences in this regard, and that involuntary expressions for the most part provide bilateral activation.) Voluntary and involuntary expressions are under the control of different neural tracts (Rinn, 1991), with voluntary expressions controlled by impulses from the motor strip through the pyramidal tract, and involuntary expressions controlled by impulses from subcortical areas through the extrapyramidal tract. The activation of facial movements that have become habitual, although acquired voluntarily, might resemble involuntary activation, but no research on this has been reported.

Once innervated, the face is intricate and differentiated, making it one of the most complex signal systems available to humans. It includes over 40 structurally and functionally anatomically independent muscles, each of which can innervate independently of each other. The facial musculature is fairly unique. They include the only somatic muscles in the body attached on one side to bone and the other to skin; thus facial movements are specialized for expression. The face is also one of the few places in the body where some muscles are not attached to any bone at all (e.g., orbicularis oculi, the muscle surrounding the eyes; orbicularis oris, the muscle in the lips). Finally, there isn’t a one to one correspondence between structure and function in some facial muscles. The corrugator muscle group, for instance, which brings the brows down and together, is comprised of three muscles that usually act together when innervated. (Note it is possible to activate just the muscle that lowers the brows without drawing them together, although it is not common.) Although the frontalis muscle is a single muscle that spans the forehead, the inner and outer parts of this muscle can move independently of each other, allowing for just the inner or outer corners of the eyebrows to rise (or the entire brow if both portions are innervated). Because this requires separate neural supplies to these two strands of frontalis, from a functional viewpoint this should be regarded as two not one muscle. Thus when decomposing facial behaviors, it is important to understand them from the perspective of functional, not structural anatomy (i.e., how the muscles function in a living, not dead, individual). Moreover, each of the functional muscle units of the face can be innervated with different timing, intensity, and laterality characteristics. These characteristics produce the ability to create thousands of different expressions.

Facial behaviors are used for various functions, including

• Speech illustration. For instance, people often raise their brows when being inquisitive, and lower their brows when they lower their voices.
• Conversation regulation. People can cue others that they are either done talking and it’s their turn, or not, through their faces (and voice).
• Emblematic Gestures. These are movements that symbolically give verbal meaning that can be conveyed by words, such as the doubtful look produced by raising the upper lip and pushing the lower lip up.
• Cognition. People often furrow their brows when concentrating or are perplexed. They also purse their lips when conducting mental searches.
• Talking and eating. People use the muscles around the mouth area for talking and eating, and especially speech articulation.
• Emotion signaling. People use the facial muscles to signal their emotional states.
• Expressive regulation. People also use the facial muscles to regulate their emotion signals.

Emotion Signaling

By far the greatest amount of research on facial behaviors has been on the emotion signaling function of the face. The notion that emotions are linked discretely with facial expressions has roots in the work in the work of Darwin (1872/1998), and those who have refined and elaborated his evolutionist claims (Ekman, 1992). Darwin claimed, in his principle of serviceable habits, that facial expressions are the residual actions of more complete behavioral responses, and occur in combination with other bodily responses – vocal, postural, gestural, skeletal muscle movements, and physiological responses. Thus, we express anger by furrowing the brow and tightening the lips with teeth displayed because these actions are part of an attack response; we express disgust with an open mouth, nose wrinkle, and tongue protrusion as part of a vomiting response. Facial expressions, then, are elements of a coordinated response involving multiple response systems. Table 1 lists the descriptions of the facial muscles involved in each of the emotions Darwin considered universal.

Table 1
Descriptions of Facial Muscles (and Other Nonverbal Behaviors) Involved in the Emotions Darwin Considered Universal and that Research has Shown to be Universally Expressed and Recognized

Emotion	Darwin’s Description (non facial elements in parentheses)	FACS AUs Associated with Darwin’s Description	FACS AUs Found to be Associated with this Emotion in Research with Humans (optional AUs in parentheses)	Homologous or Analogous AUs Found in Chimpanzees
Anger	nostrils raised, mouth compressed, furrowed brow, eyes wide open, head erect, (chest expanded, arms rigid by sides, stamp ground, body sways backwards/forwards, tremble]	4, 5, 24, 38	4, 5 and/or 7, 22, 23, 24	4,22,23,24
Contempt	lip protrusion, nose wrinkle, partial closure of eyelids, turn away eyes, upper lip raised, (snort, body expiration, expiration)	9, 10, 22, 41, 61 or 62	Unilateral 12, unilateral 14	9,10
Disgust	lower lip turned down, upper lip raised, expiration, mouth open, spitting, blowing out protruding lips, clear throat sound, lower lip, tongue protruded	10, 16, 22, 25 or 26	9 and/or 10, (25 or 26)	9, 10
Fear	eyes open, mouth open, lips retracted, eye brows raised, (crouch, pale, perspiration, hair stands on end, muscles shiver, yawn, tremble)	1, 2, 5, 20	1, 2, 4, 5, 7, 20, (25 or 26)	1,2,4
Happiness	eyes sparkle, skin under eyes wrinkled, mouth drawn back at corners	6, 12	6, 12	6, 12
Joy	zygomatic, orbicularis, upper lip raised, naso labial fold formed (muscle tremble, purposeless movements, laughter, clapping hands, jumping, dancing about, stamping, chuckle/giggle)	6, 7, 12	6, 12	6, 12
Sadness	corner mouth depressed, inner corner eyebrows raised (low spirits)	1, 15	1, (4), 15, (17)	1, 4, 15, 17
Surprise	eyebrows raised, mouth open, eyes open, lips protruded, (expiration, blowing/hissing, open hands high above head, palms toward person with straightened fingers, arms backwards)	1, 2, 5, 25 or 26	1, 2, 5, 25 o 26	1,2

Darwin claimed that all people, regardless of race or culture, possess the ability to express some emotions in exactly the same ways through their faces. Darwin wrote The Expression of the Emotions in Man and Animals to refute the claims of Sir Charles Bell, the leading facial anatomist of his time and a teacher of Darwin’s, about how God designed humans with unique facial muscles to express uniquely human emotions. (To wit, Darwin penciled in the margin of Bell’s book, “he never looked at a monkey.”) Relying on advances in photography and anatomy, Darwin study the facial muscle actions involved in emotion in detail, and concluded that the muscle actions are universal, and that their precursors can be seen in nonhuman primates and other mammals.

Early research testing Darwin’s ideas was inconclusive (Ekman, Friesen, & Ellsworth, 1972), and for many years the dominant perspective in psychology was the opposite – that facial expressions were culture-specific, much like language. Darwin’s claims were resurrected by Tomkins (1962, 1963), who suggested that emotion was the basis of human motivation, and that the seat of emotion was in the face. Tomkins, in turn, joined forces with Paul Ekman, who conducted what is known in the field today as the original universality studies.

There are four sources of evidence that comprise these studies. In the first, Ekman and his colleagues (Ekman, Sorenson, & Friesen, 1969) showed pictures of many different facial expressions to observers in different cultures, who were asked to judge which emotion was portrayed in the face. If emotional expressions were universal, there would be high agreement within and across cultures in judgments. If emotional expressions were culture-specific, there might be agreement within a culture, but disagreement across cultures. The results revealed agreement both within and across cultures for six emotional expressions – anger, disgust, fear, happiness, sadness, and surprise. These data were the first systematic evidence for the universality of emotions and their expressions.

Ekman’s next two studies assessed whether or not the results might be due to shared visual input. All of the previous participants were from relatively industrialized countries and might have learned to recognize emotion through mass media (television, movies, magazines, etc.) and hence interpreted the emotions similarly. He (Ekman & Friesen, 1971) went to the highlands of southeast New Guinea to study a preliterate, stone-age culture that was isolated, void of mass media, and whose members had never seen westerners before. In one study the tribespersons accurately matched facial expressions with stories depicting the relevant emotions. This finding was the second source of evidence for universality and addressed the concern that the earlier results might be due to shared visual input. In the next study, the researchers filmed the faces of individual tribespersons as they portrayed what they would look like if they were the person in the stories, and then showed those film clips to Americans had never seen the tribes people before. Nonetheless, they accurately identified the emotions the New Guineans intended to portray. This was the third source of evidence for universality.

Still, opponents levied criticism against the research, arguing that all the previous studies involved judgments, not productions, of facial expressions. Ekman (1972) therefore conducted another study in which he videotaped American and Japanese participants alone in a room as each watched a strong emotion-provoking film. Analysis of the specific muscles in the face that moved in the participants showed that both Americans and Japanese produced the same expressions of emotion, which were the same as the expressions previous studies demonstrated were universally recognized. These data provided the fourth source of evidence for universality of facial expression of emotion.

Since then there have been many other judgment studies around the world that have demonstrated convincingly that a small set of facial expressions are universally recognized (Matsumoto, 2001). And, many other studies have demonstrated that when emotions are aroused, the same facial expressions of emotion are reliably produced by people all around the world and from all walks of life (Matsumoto, Keltner, O'Sullivan, & Frank, 2007). Subsequent research has also demonstrated the universal recognition of contempt (Ekman & Friesen, 1986), and that facial expressions of emotion are part of a coordinated response system that involves unique physiological signatures, specific cognitive activities, preparation for motor behaviors, and specific feelings. They are signals of a rich, complex response system all humans have. Of the literally thousands of expressions that can possibly be produced, the facial configurations associated with discrete emotional states represent a relatively small set of specific combinations of the available repertoire. Figure 2 presents examples of the prototypic, full-face expressions of each of the seven emotions that research has demonstrated to be universally recognized and expressed.

Facial Measurement

Documenting the universality of facial expressions of emotion led to a wide range of research on facial expressions, much but certainly not all of it focusing on emotion signaling. This led to the need to develop methodologies to measure facial behavior validly and reliably. While several ways to do so have been developed over the years, the Facial Action Coding System (FACS) (Ekman & Friesen, 1978) is widely acknowledged to be the most comprehensive and objective system available.

FACS is an anatomically-based system for measuring any and all kinds of visible facial behavior. It identifies 44 possible independent movements of the facial musculature (called Action Units or AUs) – the entire range of all possible movements that can occur in the human face. This identification is based on functional, not structural, anatomy, organizing muscles or muscle combinations into AUs depending on how they function. For instance, as mentioned previously, although the frontalis muscle is a single muscle, the inner (frontalis, pars medialis) and outer (frontalis, pars lateralis) portions can move independently of each other. FACS distinguishes between these two actions (AUs 1 and 2, respectively), despite the fact they originate in what heretofore anatomists regarded as a single muscle. Likewise, the action that brings the brows down and together is actually a combination of three anatomically separate muscles that usually do not act independently in humans (corrugator supercilli, depressor supercilli); it is classified as a single FACS AU (4).

FACS coding is atheoretical and can describe any facial behavior, not only those related to emotion signals. Coding involves minimally the identification of which facial muscles are innervated in any expression based on the visible appearance changes that occur in the face when expressions occur. Coding can also involve the rating of the intensity, laterality, and timing (onset, apex, and offset) of individual AUs. Because any of the 44 AUs can occur independently of each other, and depending on whether each AU is coded for intensity, laterality, and timing characteristics, FACS coding is typically comprehensive, complex, and labor intensive. Table 1 lists the AU codes for the expressions Darwin considered universal, expressions subsequent research has shown to be universal, and analogous or homologous expressions of nonhuman primates. Figure 3 shows an example of a FACS coded expression of fear.

There are a number of derivatives of FACS. For instance, Ekman and Friesen (1982) developed an abbreviated version of it called Emotion FACS (EMFACS), which identifies only those AUs that were theoretically or empirically related to emotion. EMFACS coding, therefore, focuses only on the identification of a smaller set of AUs or AU combinations, resulting in a more time efficient coding procedure.

Although all of the facial musculature in the human adult exists and is fully functional at birth (Ekman & Oster, 1979), infant faces differ from adult faces in several ways, such as in the proportions and dimensions of the bony structures of the face, fat deposits and elastic skin, and in the presence of specialized features such as the sucking lip and buccal sucking pad (Oster, 2005). For these reasons, the appearance changes produced by facial movements in infants differ slightly than those that occur in adults. Thus, there is a modified version of FACS specialized for use with infants called BabyFACS (Oster, 2004).

For years ethologists have examined the facial behaviors of nonhuman primates, noting the similarities between human and nonhuman expressions displayed in similar contexts (de Waal, 2003). Previous research in this area, however, was limited because there was no method of objectively identifying nonhuman primate expressions; expressions that appeared alike were merely described and classified by researchers according to their overall appearance. Recently, Vick and colleagues (2007) developed a modified version of FACS for chimpanzees (Pan troglodytes) called ChimpFACS. The issues associated with the translation of human adult FACS to chimpanzees are similar to those associated with its translation to the study of infants; for example, the forehead musculature of chimps is less well developed than that of humans. Nevertheless, there are minimal differences in the underlying musculature for the AUs that are common between humans and chimps (Vick et al., 2007), and many muscles that are related to human facial expressions of emotion appear to have the same location and functional effect in chimpanzees, thus representing homologues.

Factors that Moderate the Production of Facial Expressions of Emotion

Human social life requires expression regulation, because the non-regulated, unadulterated expression of emotion would lead to social chaos (Matsumoto, Yoo, Nakagawa et al., in press); that is, humans cannot just act automatically on their impulses whenever strong emotions are aroused if they are to live harmoniously with others. Fortunately, humans differ greatly from other animals in that they have been endowed with an elaborate set of neuroanatomical structures that allow the alteration of the linkage between the tendency to respond and the actual response (Levenson, 1999). With regard to facial expressions of emotion, these regulatory mechanisms are known as display rules (Ekman & Friesen, 1969).

Display rules are learned early in life and dictate the management and modification of facial expressions depending on social circumstances. There are seven ways in which expressions can be regulated (Ekman & Friesen, 1969). Emotions can be

1. Expressed as is
2. Deamplified, showing less than what is actually felt
3. Neutralized, showing nothing
4. Qualified, shown with other emotions
5. Masked, concealed by showing other emotions altogether
6. Amplified, shown more intensely than what is actually felt
7. Simulated, shown when not felt at all

The first study to document the existence of display rules was Ekman’s (1972) study of Americans and Japanese described earlier. Recall that in the first condition of the experiment, members of both cultures displayed the same facial expressions when viewing the stress films. In a second condition, however, the participants viewed the same films in the presence of a higher-status experimenter. This time, the Japanese masked their feelings with smiles, while the Americans tended to show the same expressions they did the first time. That is, the Japanese had a display rule to conceal their feelings of disgust and to be pleasant by smiling in that context.

One of the major functions of every human culture is to proscribe rules by how expression regulation should occur. One way this is accomplished is through the ascription of cultural meaning to different social situations (Matsumoto, 2007). For example, expressing emotions such as anger, contempt, or disgust to ingroup members or higher-status others may be dangerous because it disrupts ingroup harmony or dominance hierarchies. In fact, this display rule is universal to all human cultures (Matsumoto, Yoo, Fontaine et al., in press)

At the same time, different cultures may proscribe different rules about exactly how this inhibition occurs. Some cultures, for example, may proscribe a deamplification of expression, while others neutralization, qualification, or even masking, each of which refer to inhibition of the original emotion. So, while all cultures believe that emotions such as anger, contempt, and disgust need to be regulated when with a higher-status person, Americans may believe that the expression should be deamplified, while Hong Kong Chinese (or Japanese, such as in Ekman’s, 1972, study) believe it should be masked. Studies are just beginning to examine exactly what these cultural meanings are, and how cultural norms for expression differ depending on the nature of social context.

Suggested Reading

Ekman, P. (2003). Emotions revealed. New York: Times Books.

References

• Darwin, C. (1872/1998). The expression of emotion in man and animals. New York: Oxford University Press.
• de Waal, F. B. M. (2003). Darwin's legacy and the study of primate visual communication. In P. Ekman, J. Campos, R. J. Davidson & F. B. M. De Waal (Eds.), Emotions inside out: 130 years after Darwin's The Expression of Emotion in Man and Animals (pp. 7-31). New York: New York Academy of Sciences.
• Ekman, P. (1972). Universal and cultural differences in facial expression of emotion. In J. R. Cole (Ed.), Nebraska Symposium on Motivation', 1971 (Vol. 19, pp. 207-283). Lincoln, NE: Nebraska University Press.
• Ekman, P. (1992). Facial expressions of emotion: New findings, new questions. Psychological Science, 3(1), 34-38.
• Ekman, P., & Friesen, W. V. (1969). The repertoire of nonverbal behavior: Categories, origins, usage, and coding. Semiotica, 1, 49-98.
• Ekman, P., & Friesen, W. V. (1971). Constants across culture in the face and emotion. Journal of Personality and Social Psychology, 17, 124-129.
• Ekman, P., & Friesen, W. V. (1978). Facial action coding system: Investigator's guide. Palo Alto, Calif.: Consulting Psychologists Press.
• Ekman, P., & Friesen, W. V. (1982). EMFACS.Unpublished manuscript, San Francisco, CA.
• Ekman, P., & Friesen, W. V. (1986). A new pan-cultural facial expression of emotion. Motivation & Emotion, 10(2), 159-168.
• Ekman, P., Friesen, W. V., & Ellsworth, P. (1972). Emotion in the human face: guide-lines for research and an integration of findings. New York: Pergamon Press.
• Ekman, P., & Oster, H. (1979). Facial expressions of emotion. Annual Review of Psychology, 30, 527-554.
• Ekman, P., Sorenson, E. R., & Friesen, W. V. (1969). Pancultural elements in facial displays of emotion. Science, 164(3875), 86-88.
• Levenson, R. W. (1999). The intrapersonal functions of emotion. Cognition and Emotion, 13(5), 481-504.
  
• Matsumoto, D. (2001). Culture and Emotion. In D. Matsumoto (Ed.), The Handbook of Culture and Psychology (pp. 171-194). New York: Oxford University Press.
  
• Matsumoto, D. (2007). Culture, context, and behavior. Journal of Personality, 75(6), 1285-1319.
  
• Matsumoto, D., Keltner, D., O'Sullivan, M., & Frank, M. G. (2007). What's in a face? Facial expressions as signals of discrete emotions. Manuscript currently submitted for publication.
  
• Matsumoto, D., & Lee, M. (1993). Consciousness, volition, and the neuropsychology of facial expressions of emotion. Consciousness & Cognition: an International Journal, 2(3), 237-254.
  
• Matsumoto, D., Yoo, S. H., Fontaine, J. R. J., Anguas-Wong, A. M., Arriola, M., Ataca, B., et al. (in press). Mapping expressive differences around the world: The relationship between emotional display rules and Individualism v. Collectivism. Journal of Cross-Cultural Psychology.
  
• Matsumoto, D., Yoo, S. H., Nakagawa, S., Alexandre, J., Altarriba, J., Anguas-Wong, A. M., et al. (in press). Culture, emotion regulation, and adjustment. Journal of Personality and Social Psychology.
  
• Oster, H. (2004). BabyFACS: Facial Action Coding System for infants and young children.Unpublished manuscript, New York: New York University.
  
• Oster, H. (2005). The repertoire of infant facial expressions: An ontogenetic perspective. In J. Nadel & D. Muir (Eds.), Emotional development (pp. 261-292). New York: Oxford University Press.
  
• Rinn, W. E. (1991). Neuropsychology of facial expression. In R. Feldman & B. Rime (Eds.), Fundamentals of nonverbal behavior (pp. 3-70). New York: Cambridge University Press.
  
• Standring, S. (2005). Gray's anatomy (39th ed.). New York: Elsevier/Churchill Livingstone.
  
• Tomkins, S. S. (1962). Affect, imagery, and consciousness (Vol. 1: The positive affects). New York: Springer.
  
• Tomkins, S. S. (1963). Affect, imagery, and consciousness (Vol. 2: The negative affects). New York: Springer.
  
• Vick, S.-J., Waller, B. M., Parr, L. A., Pasqualini, M. S., & Bard, K. A. (2007). A cross species comparison of facial morphology and movement in humans and chimpanzees using the Facial Action Coding System (FACS). Journal of Nonverbal Behavior, 31, 1-20.

Internal references

• Valentino Braitenberg (2007) Brain. Scholarpedia, 2(11):2918.

• Mark Aronoff (2007) Language. Scholarpedia, 2(5):3175.

• Peter Jonas and Gyorgy Buzsaki (2007) Neural inhibition. Scholarpedia, 2(9):3286.

• David Myers (2007) Psychology of happiness. Scholarpedia, 2(8):3149.

See also

Sponsored by: Eugene M. Izhikevich, Editor-in-Chief of Scholarpedia, the peer-reviewed open-access encyclopedia

Sponsored by: Prof. Ke CHEN, School of Computer Science, The University of Manchester, U.K.

Reviewed by: Anonymous

Accepted on: 2008-05-14 22:33:38 GMT