3D Morphable Model

In computer vision and computer graphics, the 3D Morphable Model (3DMM) is a generative technique that uses methods of statistical shape analysis to model 3D objects. The model follows an analysis-by-synthesis approach over a dataset of 3D example shapes of a single class of objects (e.g., face, hand). The main prerequisite is that all the 3D shapes are in a dense point-to-point correspondence, namely each point has the same semantical meaning over all the shapes. In this way, we can extract meaningful statistics from the dataset and use them to represent new plausible shapes of the object's class. Given a 2D image, we can represent its 3D shape via a fitting process or generate novel shapes by directly sampling from the statistical shape distribution of that class. The question that initiated the research on 3DMMs was to understand how a visual system could handle the vast variety of images produced by a single class of objects and how these can be represented. The primary assumption in developing 3DMMs was that prior knowledge about object classes was crucial in vision. 3D Face Morphable Models are the most popular 3DMMs since they were the first to be developed in the field of facial recognition. It has also been applied to the whole human body, the hand, the ear, cars, and animals.

In computer vision and computer graphics, the 3D Morphable Model (3DMM) is a generative technique that uses methods of statistical shape analysis to model 3D objects. The model follows an analysis-by-synthesis approach over a dataset of 3D example shapes of a single class of objects (e.g., face, hand). The main prerequisite is that all the 3D shapes are in a dense point-to-point correspondence, namely each point has the same semantical meaning over all the shapes. In this way, we can extract meaningful statistics from the dataset and use them to represent new plausible shapes of the object's class. Given a 2D image, we can represent its 3D shape via a fitting process or generate novel shapes by directly sampling from the statistical shape distribution of that class. The question that initiated the research on 3DMMs was to understand how a visual system could handle the vast variety of images produced by a single class of objects and how these can be represented. The primary assumption in developing 3DMMs was that prior knowledge about object classes was crucial in vision. 3D Face Morphable Models are the most popular 3DMMs since they were the first to be developed in the field of facial recognition. It has also been applied to the whole human body, the hand, the ear, cars, and animals. The 3DFMM uses statistical analysis to define a statistical shape space, a vectorial space equipped with a probability distribution, or prior. To extract the prior from the example dataset, all the 3D faces must be in a dense point-to-point correspondence. This means that each point has the same semantic meaning on each face (e.g., nose tip, edge of the eye). In this way, by fixing a point, we can, for example, derive the probability distribution of the texture's red channel values over all the faces. A face shape S of n vertices is defined as the vector containing the 3D coordinates of the n vertices in a specified order, that is S \in \mathbb{R}^{3n} . A shape space is regarded as a d -dimensional space that generates plausible 3D faces by performing a lower-dimensional ( d \ll n ) parametrization of the database. Thus, a shape S can be represented through a generator function \mathbf{c}: \mathbb{R}^d \rightarrow \mathbb{R}^{3n} by the parameters \mathbf{w} \in \mathbb{R}^d , \mathbf{c}(\mathbf{w}) = S \in \mathbb{R}^{3n} . The most common statistical technique used in 3DFMM to generate the shape space is Principal Component Analysis (PCA), that generates a basis that maximizes the variance of the data. Performing PCA, the generator function is linear and defined as \mathbf{c}(\mathbf{w}) = \mathbf{\bar c} + \mathbf{E}\mathbf{w} where \mathbf{\bar c} is the mean over the training data and \mathbf{E} \in \mathbb{R}^{3n \times d} is the matrix that contains the d most dominant eigenvectors. Using a unique generator function for the whole face leads to the imperfect representation of finer details. A solution is to use local models of the face by segmenting important parts such as the eyes, mouth, and nose. The additive model is defined as a linear model and the expression is an additive offset with respect to the identity \mathbf{c}(\mathbf{w}^s, \mathbf{w}^w) = \mathbf{\bar c} + \mathbf{E}^s\mathbf{w}^s + \mathbf{E}^e\mathbf{w}^e where \mathbf{E}^s , \mathbf{E}^{e} and \mathbf{w}^{s} , \mathbf{w}^e are the matrices basis and the coefficients vectors of the shape and expression space, respectively. With this model, given the 3D shape of a subject in a neutral expression \mathbf{c}_{ne} and in a particular expression \mathbf{c}^{exp} , we can transfer the expression to a different subject by adding the offset \Delta_{\mathbf{c}} = \mathbf{c}^{exp} - \mathbf{c}^{ne} . Two PCAs can be performed to learn two different spaces for shape and expression. In a multiplicative model, shape and expression can be combined in different ways. For example, by exploiting d_e operators \mathbf{T}_j: \mathbb{R}^{3n} \rightarrow \mathbb{R}^{3n} that transform a neutral expression into a target blendshape we can write \mathbf{c}(\mathbf{w}^s, \mathbf{w}^e) = \sum_{j=1}^{d_e}w_j^e\mathbf{T}_j(\mathbf{c}(\mathbf{w}^s) + \mathbf{\delta}^s) + \mathbf{\delta}_j^e where \mathbf{\delta}^s and \mathbf{\delta}^s_j are vectors to correct to the target expression. The nonlinear model uses nonlinear transformations to represent an expression. The color information id often associated to each vertex of a 3D shape. This one-to-one correspondence allows us to represent appearance analogously to the linear shape model \mathbf{d}(\mathbf{w}^t) = \mathbf{\bar d} + \mathbf{E}^{t}\mathbf{w}^{t} where \mathbf{w}^t is the coefficients vector defined over the basis matrix \mathbf{E}^t . PCA can be again be used to learn the appearance space. In the original definition of Blanz and Vetter, the shape of a face is represented as the vector S = (X_1, Y_1, Z_1, ..., X_n, Y_n, Z_n)^T \in \mathbb{R}^{3n} that contains the 3D coordinates of the n vertices. Similarly, the texture is represented as a vector T = (R_1, G_1, B_1, ..., R_n, G_n, B_n)^T \in \mathbb{R}^{3n} that contains the three RGB color channels associated with each corresponding vertex. Due to the full correspondence between exemplar 3D faces, new shapes \mathbf{S}_{models} and textures \mathbf{T}_{models} can be defined as a linear combination of the m example faces: \mathbf{S}_{model} =\sum_{i=1}^m a_i \mathbf{S}_i \qquad \mathbf{T}_{model} =\sum_{i=1}^m b_i \mathbf{T}_i \qquad \text{with} \; \sum_{i=1}^m a_i = \sum_{i=1}^m b_i = 1 Thus, a new face shape and texture is parametrized by the shape \mathbf{a} = (a_1, a_2,..., a_m)^T and texture coefficients \mathbf{b} = (b_1, b_2,..., b_m)^T . To extract the statistics from the dataset, they performed PCA to generate the shape space of dimension to d and used a linear model for shape and appearance modeling. In this case, a new model can be generated in the orthogonal basis using the shape and the texture eigenvector s_i and t_i , respectively: \mathbf{S}_{model} = \mathbf{\bar S} + \sum_{i=1}^m a_i \mathbf{s}_i \qquad \mathbf{T}_{model} = \mathbf{\bar T} + \sum_{i=1}^m b_i \mathbf{t}_i \qquad where \mathbf{\bar{S}} and \mathbf{\bar{T}} are the mean shape and texture of the dataset.

Text derived from https://en.wikipedia.org/w/index.php?oldid=1339674705 under CC BY-SA 4.0; modifications have been made.