【導讀】Abstract. Conventional. capturepipeline. Target. vertexpositions. Training. footage. Predicted. 1Introduction. Bulkof. footage. (a)Training. Trainedneuralwork. vertexpositions. Inferred. vertex. positions. natural.distracting,unnaturalmotion.Problemstatement. (b)Inference. thatneedsto

　　

【正文】 earns the consistent features of the inputoutput mapping, and its output therefore does not fluctuate in the same way. We believe this explains some of the discrepancies between our output and the validation data. Figure 7 demonstrates a problem with our decision to use only one input image as the input to the work: The single frontal image apparently does not contain enough information for the work to detect when the jaw is pushed to the front. We did not observe this issue until late in the project, and thus these errors need to be manually corrected when using our current system. We however expect that adding a secondary camera to the side of the actor and using both images as inputs to the work should correct this issue. Figure 8 shows the convergenc e of the work that was used for creating the o utputs in Figures 4, 5 and 7. As previously explained, our loss function is the MS E betwee n work outputs and target positions from the training/ validatio n set. The vertex coordinates (a) Input video frame (b) Target (c) Our result (d) Difference RMSE = 0.99 mm RMSE = 1.33 mm RMSE = 1.74 mm RMSE = 1.06 mm 0 1 2 3 4 5 mm Figure 4: A selection of interesting frames f rom a validation shot that was not used as part of training set when training the work. (a) Crop of the original fullresolution input image. (b) The target positions were created by capture artists using the existing capture pipeline at Remedy Entertainment. (c) Our result, inferred by the neural work based solely on the input image (a). (d) Difference between target and inferred positions. The RMSE is calculated over the Euclidean distances between target and inferred vertex positions, with only the animated vertices taken into account. The RMSEs of these frames are higher than average, because the validation data mostly consist of more neutral material than the frames shown here. (a) Input video frame (b) Target (c) Our result (d) Difference RMSE = 1.00 mm RMSE = 1.19 mm RMSE = 1.37 mm RMSE = 1.18 mm 0 1 2 3 4 5 mm Figure 5: A selection of frames f rom a nother validation shot. Columns are as in Figure 4. Large differences are visible near the hairline, and these regions also appear distorted in the target meshes in colum n (b). It is not clear whether the man ually tracked target positions or the inferred positions are closer to the grou nd truth. Because t hese regio ns have traditionally bee n problematic for the conve ntional capture pipeline, they are automatically smoothed with the stationary vertices of the he ad later in the animation pipeline, which corrects these issues. There are also two regions above the eyeb rows that consistently show differe nces between inferred an d target positions. It is likely that these are similarly due to a systematic tracking error in the target positions. RMSE 0 1 585 1 Valida? on shot 1 Valida? on shot 2 339 Figure 6: Perframe RMSE in the two validation shots used for Figures 4 and 5. Frame index advances f rom left to right. The green dots indicate frames that were used in Figure 4, with leftmost dot corresponding to top row in Figure 4, etc. The orange dots indicate frames used in Figure 5, and the red dots indicate the problematic frames shown in Figure 7. are measure d in centimeters in our data, so the final validation loss of corresponds to RM SE of millimeters. With longer training the training loss could be pushed arbitrarily close to zero, but this did not improve the validation loss or the subjective quality of the results. The most unexpected result was that the work output was tempo rally perfectly stable, despite the f act that we do not employ recur rent works or smooth the generated vertex positions temporally in any way. Perceptual quality and production aspects The original goal of t he project was to cover the bulk of the fa cial tracking work specifically in facial performance heavy produc tions involving digital doubles of real actors, by inferring facial mo tion for noncinematic parts of the game. The system is however currently being evaluated at Remedy Entertainment also for high quality cinematic use, given the encouraging results. The convolutional neural work was tested in the wild using footage from a later session. The lighting conditions and facial fea tures exhibited in the training set were carefully preserved. The inference was evaluated numerically and perceptually in relation to a manually tracked ground truth. For extreme closeups manual tracking still results in slightly better quality despite being almost imperceptible at times in a blind experiment. The results exhibit extremely plausible motion and are temporally stable. Subtle lip contact and stickiness is an issue but this affects manual tracking as well, hence the aforementioned additional deformations are applied after inference. The work produces perfectly stabilized vert ex positions whereas human operators inevitably vary in their work between sequences. The results can also be improv ed by adding data to the training set if deemed necessary. The 10 minute dataset requirement for high quality output means that the actor needs a big enough role in the game to justify the cost. Building the dataset for the neural work however enables tracking work to start prior to having a locked screenplay for the project. This means that late stage pickup shoots and