Local Binary Patterns

• Matti Pietikäinen, Machine Vision Group, Department of Electrical and Information Engineering, University of Oulu, Finland

Figure 1: Description of facial expressions with local binary patterns.

Local Binary Pattern (LBP) is a simple yet very efficient texture operator which labels the pixels of an image by thresholding the neighborhood of each pixel and considers the result as a binary number. Due to its discriminative power and computational simplicity, LBP texture operator has become a popular approach in various applications. It can be seen as a unifying approach to the traditionally divergent statistical and structural models of texture analysis. Perhaps the most important property of the LBP operator in real-world applications is its robustness to monotonic gray-scale changes caused, for example, by illumination variations. Another important property is its computational simplicity, which makes it possible to analyze images in challenging real-time settings.

LBP in the spatial domain

The basic idea for developing the LBP operator was that two-dimensional surface textures can be described by two complementary measures: local spatial patterns and gray scale contrast. The original LBP operator (Ojala et al. 1996) forms labels for the image pixels by thresholding the 3 x 3 neighborhood of each pixel with the center value and considering the result as a binary number. The histogram of these 2⁸ = 256 different labels can then be used as a texture descriptor. This operator used jointly with a simple local contrast measure provided very good performance in unsupervised texture segmentation (Ojala and Pietikäinen 1999). After this, many related approaches have been developed for texture and color texture segmentation.

The LBP operator was extended to use neighborhoods of different sizes (Ojala et al. 2002). Using a circular neighborhood and bilinearly interpolating values at non-integer pixel coordinates allow any radius and number of pixels in the neighborhood. The gray scale variance of the local neighborhood can be used as the complementary contrast measure. In the following, the notation (P,R) will be used for pixel neighborhoods which means P sampling points on a circle of radius of R. See Fig. 2 for an example of LBP computation.

Figure 2: An example of LBP computation.

Another extension to the original operator is the definition of so-called uniform patterns, which can be used to reduce the length of the feature vector and implement a simple rotation-invariant descriptor. This extension was inspired by the fact that some binary patterns occur more commonly in texture images than others. A local binary pattern is called uniform if the binary pattern contains at most two bitwise transitions from 0 to 1 or vice versa when the bit pattern is traversed circularly. For example, the patterns 00000000 (0 transitions), 01110000 (2 transitions) and 11001111 (2 transitions) are uniform whereas the patterns 11001001 (4 transitions) and 01010010 (6 transitions) are not. In the computation of the LBP labels, uniform patterns are used so that there is a separate label for each uniform pattern and all the non-uniform patterns are labeled with a single label. For example, when using (8,R) neighborhood, there are a total of 256 patterns, 58 of which are uniform, which yields in 59 different labels.

Ojala et al. (2002) noticed in their experiments with texture images that uniform patterns account for a little less than 90% of all patterns when using the (8,1) neighborhood and for around 70% in the (16,2) neighborhood. Each bin (LBP code) can be regarded as a micro-texton. Local primitives which are codified by these bins include different types of curved edges, spots, flat areas etc.

The following notation is used for the LBP operator: LBP_P,R^u2. The subscript represents using the operator in a (P,R) neighborhood. Superscript u2 stands for using only uniform patterns and labeling all remaining patterns with a single label. After the LBP labeled image f_l(x,y) has been obtained, the LBP histogram can be defined as

\[\tag{1} H_i=\sum_{x,y} I\left\{f_l(x,y)=i\right\}, i=0,\ldots,n-1, \]

in which n is the number of different labels produced by the LBP operator, and I{A} is 1 if A is true and 0 if A is false.

When the image patches whose histograms are to be compared have different sizes, the histograms must be normalized to get a coherent description:

\[\tag{2} N_i=\frac{H_i}{\sum_{j=0}^{n-1} H_j} . \]

Spatiotemporal LBP

The original LBP operator was defined to only deal with the spatial information. Later, it was extended to a spatiotemporal representation for dynamic texture analysis. For this purpose, the so-called Volume Local Binary Pattern (VLBP) operator was proposed (Zhao and Pietikäinen 2007). The idea behind VLBP consists of looking at dynamic texture as a set of volumes in the (X,Y,T) space where X and Y denote the spatial coordinates and T denotes the frame index (time). The neighborhood of each pixel is thus defined in three dimensional space. Then, similarly to LBP in spatial domain, volume textons can be defined and extracted into histograms. Therefore, VLBP combines motion and appearance together to describe dynamic textures.

To make VLBP computationally simple and easy to extend, an operator based on co-occurrences of local binary patterns on three orthogonal planes (LBP-TOP) was also introduced. LBP-TOP considers three orthogonal planes: XY, XT and YT, and concatenates local binary pattern co-occurrence statistics in these three directions as shown in Figure 1. The circular neighborhoods are generalized to elliptical sampling to fit to the space-time statistics.

Fig. 3 shows example images from three planes. The XY plane represents appearance information, while the XT plane gives a visual impression of one row changing in time and YT describes the motion of one column in temporal space. The LBP codes are extracted for all pixels from the XY, XT and YT planes, denoted as XY-LBP, XT-LBP and YT-LBP, and histograms from these planes are computed and concatenated into a single histogram. In such a representation, a dynamic texture is encoded by an appearance (XY-LBP) and two spatial temporal (XT-LBP and YT-LBP) co-occurrence statistics.

Figure 3: LBP from three orthogonal planes.

Setting the radius in the time axis to be equal to the radius in the space axis is not reasonable for dynamic textures. So we have different radius parameters in space and time to set. In the XT and YT planes, different radii can be assigned to sample neighboring points in space and time. More generally, the radii in axes X, Y and T, and the number of neighboring points in the XY, XT and YT planes can also be different.

Face description using LBP

In the LBP approach for texture classification, the occurrences of the LBP codes in an image are collected into a histogram. The classification is then performed by computing simple histogram similarities. However, considering a similar approach for facial image representation results in a loss of spatial information and therefore one should codify the texture information while retaining also their locations. One way to achieve this goal is to use the LBP texture descriptors to build several local descriptions of the face and combine them into a global description. Such local descriptions have been gaining interest lately which is understandable given the limitations of the holistic representations. These local feature based methods are more robust against variations in pose or illumination than holistic methods.

The basic methodology for LBP based face description proposed by Ahonen et al. (2006) is as follows: The facial image is divided into local regions and LBP texture descriptors are extracted from each region independently. The descriptors are then concatenated to form a global description of the face, as shown in Fig. 4.

Figure 4: Face description with local binary patterns.

This histogram effectively has a description of the face on three different levels of locality: the LBP labels for the histogram contain information about the patterns on a pixel-level, the labels are summed over a small region to produce information on a regional level and the regional histograms are concatenated to build a global description of the face.

It should be noted that when using the histogram based methods the regions do not need to be rectangular. Neither do they need to be of the same size or shape, and they do not necessarily have to cover the whole image. It is also possible to have partially overlapping regions.

The two-dimensional face description method has been extended into spatiotemporal domain (Zhao and Pietikäinen 2007). Fig. 1 depicts facial expression description using LBP-TOP. Excellent facial expression recognition performance has been obtained with this approach.

Since the publication of the LBP based face description, the methodology has already attained an established position in face analysis research and applications. A notable example is illumination-invariant face recognition system proposed by Li et al. (2007), combining NIR imaging with LBP features and Adaboost learning. Zhang et al. (2005) proposed the extraction of LBP features from images obtained by filtering a facial image with 40 Gabor filters of different scales and orientations, obtaining outstanding results. Hadid and Pietikäinen (2009) used spatiotemporal LBPs for face and gender recognition from video sequences, while Zhao et al. (2009) adopted the LBP-TOP approach to visual speech recognition achieving leading-edge performance without error-prone segmentation of moving lips. The LBP has also been successfully used in recognizing spontaneous facial microexpressions, a comprehensive survey of microexpressions is presented in (Zhao et al. 2023).

Extensions and applications

The LBP methodology has led to significant progress in texture analysis. It is widely used all over the world both in research and applications. Due to its discriminative power and computational simplicity, the method has been very successful in many such computer vision problems which were not earlier even regarded as texture problems, such as face analysis and motion analysis (Pietikäinen et al. 2011).

To increase the applicability of LBP, various extensions and modifications of it have been proposed. For example, Liao et al. (2009) proposed dominant local binary patterns which make use of the most frequently occurred patterns of LBP to improve the recognition accuracy. The use of interest region descriptors (such as SIFT) to various computer vision problems has been of great interest recently. For this purpose, a novel descriptor combining the strengths of SIFT and LBP was proposed (Heikkilä et al. 2009), in which center-symmetric local binary patterns (CS-LBP) were used to replace the gradient operator used by the SIFT operator. Mäenpää and Pietikäinen (2004) proposed an opponent color LBP, and investigated joint and separate use of color and texture in classification. The combination of the LBPs and Gabor features has been investigated (Tan et al. 2007, Wang et al. 2009).

The first texture-based method for background subtraction was proposed by Heikkilä and Pietikäinen (2006). Each pixel is modeled as a group of adaptive local binary pattern histograms that are calculated over a circular region around the pixel. The was shown to be tolerant to illumination variations, the multimodality of the background, and the introduction or removal of background objects. Furthermore, the method is capable for real-time processing. A preprocessing algorithm based on the LBPs has been developed to handle variations in illumination in a face authentication system (Heusch et al. 2006). The use of LBPs in the recognition of actions was considered by (Kellokumpu et al. 2010). The use of LBPs in the facial age classification has been investigated (Wang et al. 2009). Other related LBP-based approaches to these problems have been proposed recently.

In addition to face and facial expression recognition, the LBP has also been used in many other applications of biometrics, including eye localization, iris recognition, fingerprint recognition, palmprint recognition, gait recognition and facial age classification.

Recent developments

The success of LBP has continued since 2011. A large number of new LBP variants have been proposed, including, for example, the Median Robust Extended Local Binary Pattern (MRELBP) operator (Liu et al. 2016a). An extensive experimental evaluation of different LBP and deep texture descriptors is presented in (Liu et al. 2016b). The robustness of texture operators against different classification challenges is considered, including changes in rotation, scale, illumination, viewpoint, number of classes, different types of image degradation, and computational complexity. The best overall performance is obtained for MRELBP when distinctiveness, robustness and computational complexity are all taken into consideration. After this study, a taxonomy of LBP variants is presented together with extensive experimental evaluation (Liu et al. 2017). LBP can be seen as a simple Bag-of-Words (BoW) descriptor, in which each LBP pattern corresponds to a word. A comprehensive survey of BoW and deep CNN texture representations is presented in (Liu et al. 2019).

Among the weaknesses of CNN are its need for massive amounts of training data and high-performance computing capabilities. The paper by (Su et al. 2023) combines the strengths of LBP and CNN computations by proposing lightweight pixel difference networks for efficient visual representation learning. Excellent results are obtained for edge detection, object recognition and face recognition with low computational cost. A hort history of LBP is presented in (Pietikäinen, 2024).

References

• Ahonen, T., Hadid, A. and Pietikäinen, M. (2006), Face Description with Local Binary Patterns: Application to Face Recognition. IEEE Trans. Pattern Analysis and Machine Intelligence 28(12):2037-2041.

• Hadid, A. and Pietikäinen, M. (2009), Combining Appearance and Motion for Face and Gender Recognition from Videos. Pattern Recognition 42(11):2818-2827.

• Heikkilä, M. and Pietikäinen, M. (2006), A Texture-Based Method for Modeling the Background and Detecting Moving Objects. IEEE Trans. Pattern Analysis and Machine Intelligence 28(4):657-662.

• Heikkilä, M., Pietikäinen, M. and Schmid, C. (2009), Description of Interest Regions with Local Binary Patterns. Pattern Recognition 42(3):425-436.

• Heusch G., Rodriguez Y. and Marcel S. (2006), Local Binary Patterns as an Image Processing for Face Authentication, Proceedings of the IEEE International Conference on Automatic Face and Gesture Recognition.

• Kellokumpu, V., Zhao, G. and Pietikäinen, M. (2011), Recognition of Human Actions Using Texture Descriptors. Machine Vision and Applications 22(5):767-780.

• Li, S.Z., Chu, R.F., Liao, S.C. and Zhang, L.(2007), Illumination Invariant Face Recognition Using Near-Infrared Images. IEEE Trans. Pattern Analysis and Machine Intelligence 29(4):627-639.

• Liao, S., Law M.W.K. and Chung A.C.S. (2009), Dominant Local Binary Patterns for Texture Classification. IEEE Trans. Image Processing 18(5):1107-1118.

• Liu, L., Lao, S., Fieguth, P., Guo, Y., Wang, X. and Pietikäinen, M. (2016a), Median Robust Extended Local Binary Pattern for Texture Classification. IEEE Trans. Image Processing 25(3):1368-1381.

• Liu, L., Fieguth, P., Wang, X., Pietikäinen, M. and Hu, D. (2016b), Evaluation of LBP and Deep Texture Descriptors with a New Robustness Benchmark. Proceedings of the European Conference on Computer Vision.

• Liu, L., Fieguth, P., Guo, Y., Wang, X. and Pietikäinen, M. (2017), Local Binary Features for Texture Classification: Taxonomy and Experimental Study. Pattern Recognition 62:135-160.

• Liu,L., Chen, J., Fieguth, P., Zhao,G., Chellappa, R. and Pietikäinen, M. (2019), From BoW to CNN: Two Decades of Texture Representation for Texture Classification. International Journal of Computer Vision 127(1):74-109.

• Su, Z., Zhang, J., Wang, L. Zhang, H., Liu, Z., Pietikäinen, M., and Liu, L. (2023) Lightweight Pixel Difference Networks for Efficient Visual Representation Learning. IEEE Trans. Pattern Analysis and Machine Intelligence August 2023.

• Mäenpää, T. and Pietikäinen, M. (2004), Classification with Color and Texture: Jointly or Separately? Pattern Recognition 37(8):1629-1640.

• Ojala, T., Pietikäinen, M. and Harwood, D. (1996), A Comparative Study of Texture Measures with Classification Based on Feature Distributions. Pattern Recognition 19(3):51-59.

• Ojala, T. and Pietikäinen, M. (1999), Unsupervised Texture Segmentation Using Feature Distributions. Pattern Recognition 32:477-486.

• Ojala, T., Pietikäinen, M. and Mäenpää, T. (2002), Multiresolution Gray-scale and Rotation Invariant Texture Classification with Local Binary Patterns. IEEE Trans. Pattern Analysis and Machine Intelligence 24(7): 971-987.

• Pietikäinen, M., Hadid, A., Zhao, G. and Ahonen, T. (2011), Computer Vision Using Local Binary Patterns, Springer, 207 p.

• Tan X. and Triggs B. (2007), Fusing Gabor and LBP feature Sets for Kernel-based Face Recognition, Proceedings of the IEEE International Workshop on Analysis and Modeling of Face and Gesture, pp.235-249.

• Wang J.-G., Yau W.-Y. and Wang H. L. (2009), Age Categorization via ECOC with Fused Gabor and LBP Features. Proceedings of the IEEE Workshop on Applications of Computer Vision (WACV), pp.313-318.

• Zhao, G. and Pietikäinen, M. (2007), Dynamic Texture Recognition Using Local Binary Patterns with an Application to Facial Expressions. IEEE Trans. Pattern Analysis and Machine Intelligence 29(6:915-928.

• Zhang, W., Shan, S., Gao, W., Chen, X. and Zhang H. (2005), Local Gabor Binary Pattern Histogram Sequence (LGBPHS): A Novel Non-statistical Model for Face Representation and Recognition. In Proc. Tenth IEEE International Conference on Computer Vision, 1:786-791.

• Zhao, G., Barnard, M. and Pietikäinen, M. (2009), Lipreading with Local Spatiotemporal Descriptors. IEEE Trans. Multimedia 11(7):1254-1265.

• Zhao, G., Li, X., Li, Y. and Pietikäinen M. (2023) Facial Microexpressions: An Overview. Proceedings of the IEEE, October 2023.

Internal references

• Valentino Braitenberg (2007) Brain. Scholarpedia, 2(11):2918.

• Rodolfo Llinas (2008) Neuron. Scholarpedia, 3(8):1490.

Further reading

• Pietikäinen, M., Hadid, A., Zhao, G. and Ahonen, T. (2011), Computer Vision Using Local Binary Patterns, Springer. http://www.springer.com/mathematics/book/978-0-85729-747-1. This book presents a detailed description of LBP methods and their variants, and provides an overview as how texture methods can be used for solving different kinds of computer vision problems.

• Pietikäinen, M. and Zhao, G. (2015), Two Decades of Local Binary Patterns: A Survey. In: Bingham, E., Kaski, S., Laaksonen, J. and Lampinen, J., eds., Advances in Independent Component Analysis and Learning Machines, Elsevier, pp. 175-210. This article presents a survey to different LBP variants. Link: http://www.ee.oulu.fi/research/imag/lbp/LBP_surveyReport.htm

• Mirmehdi, M., Xie, X. and Suril, J., eds. (2008), Handbook of Texture Analysis. Imperial College Press. This book offers a good overview to texture analysis and its applications, including a chapter on face analysis using local binary patterns.

• Pietikäinen, M. (2024) From the Middle of Nowhere to a World-Class Scientist in Computer Vision and Artificial Intelligence. Link: https://urn.fi/URN:NBN:fi-oulu202408085262

External links

Matti Pietikäinen's website

See also

Brain, Neuron

Sponsored by: Jian-Gang Wang, Institute for Infocomm Research, Singapore

Reviewed by: Anonymous

Accepted on: 2010-03-03 06:10:14 GMT