【正文】 Fyffe et al. 20xx]. In contrast to the direct geometric puter vision approaches de scribed above, machine learning techniques have been successfully applied to f acial performance capture as well. A typical approach is to use radial basis functions (RBF) to infer blendshape weights from motioncapture data based on a set of training examples [Deng et al. 20xx]. More recently, restricted Boltzmann machines (RBM) and multilayer perceptrons (MLP) have also been used for simi lar tasks [Zeiler et al. 20xx。 Costigan et al. 20xx]. Furthermore, neural works have been used to infer facial animation data from audio [Hong et al. 20xx]. To our knowledge, none of the existing machine learning methods are, however, able to perform f acial per formance capture from video data in an endtoend fashion. As a consequence, they have to rely on other techniques to extract time varying feature vectors from the video. We base our work on deep convolutional neural works [Simard et al. 20xx], which have received significant attention in the re cent years and prov en particularly well suited for largescale image recognition tasks [Krizhevsky et al. 20xx。 Beeler et al. 20xx。 Simonyan and Zisser man 20xx]. Modern convolutional neural works employ var ious techniques to reduce training time and improv e generaliza tion over novel input data. These include piecewiselinear activa tion functions [Krizhevsky et al. 20xx], data augmentation [Simard et al. 20xx], dropout regularization [Hinton et al. 20xx。 Weise et al. 20xx], or by performing multiview stereo or photometric reconstructions of the individual input frames [Furukawa and Ponce 20xx。 Srivastav a et al. 20xx], and GPU acceleration [Krizhevsky et al. 20xx]. Fur thermore, it has been shown that stateof theart performance can be achieved with a very simple work architecture that mainly consists of small 3 3 convolution al layers [Simonyan and Zisser man 20xx] that employ strided output to reduce spatial resolution throughout the work [Springenberg et al. 20xx]. In a way, our method is a ―metaalgorithm‖ in the sense that it re lies on an existing technique for generating the training examples。Facial Performance Capture with Deep Neural Networks Abstract We present a deep learning technique for facial performance cap ture, ., the transfer of video footage into a motion sequence of a 3D mesh representing an actor’s face. Specifically, we build on a Conventional capture pipeline Target vertex positions conventional capture pipeline based on puter vision and multi view video, and use its results to train a deep neural work to produce similar output from a monocular video sequence. Once trained, our work produces highquality results for unseen inputs Training footage Neural work under training Gradients Loss function Predicted with greatly reduced effort pared to the conventional system. In practice, we have found that approximately 10 minutes worth of highquality data is sufficient for training a work that can then automatically process as much footage from video to 3D as needed. This yields major savings in the development of modern narrative driven video games involving digital doubles of actors and poten tially hours of animated dialogue per character. 1 Introduction Bulk of footage (a) Training Trained neural work vertex positions Inferred vertex positions Using digital doubles of human actors is a key ponent in mod ern video games’ strive for realism. Transferring the essence of a character into digital domain has many challenging technical prob lems, but the accurate capture of facial mov ement remains espe cially tricky. Due to humans’ innate sensitivity to the slightest f acial cues, it is difficult to surpass the uncanny valley, where an otherwise believable rendering of a character appears lifeless or otherwise un natural. Various tools are av ailable for building f acial capture pipelines that take video footage into 3D in one form or another, but their accu racy leaves room for improv ement. In practice, highquality results are generally achievable only with significant amount of manual polishing of output data. This can be a major cost in a large video game production. Furthermore, the animators doing the fixing need to be particularly skilled, or otherwise this editing may introduce distracting, unnatural motion. In this paper, we introduce a neural work based solution to f acial performance capture. Our goal is not to remove the need for man ual work entirely, but to dramatically reduce the extent to which it is required. In our approach, a conventional capture pipeline needs to be applied only to a small subset of input footage, in order to generate enough data for training a neural work. The bulk of the footage can then be processed using the trained work, skip ping the conventional laborintensive capture pipeline entirely. Our approach is outlined in Figure 1. Problem statement We assume that the input for the capture pipeline is one or more video streams of the actor’s head captured under controlled con ditions. The positions of the cameras remain fixed, the lighting and background are standardized, and the actor is to remain at ap proximately the same position relative to the cameras throughout all NVIDIA Technical Report NVR20xx004, September 20xx. Oc 20xx NVIDIA Corporation. All rights reserved. (b) Inference Figure 1: The goal of our system is to reduce the amount of footage that needs to be processed using a conventional, laborintensive capture pipeline. (a) In order to train a neural work, all training foot age