Medium-Scale Wrinkle Reconstruction Using One RGB Image.

brief introduction;

The whole wrinkle extraction framework consists of several parts.

• The input of wrinkle extraction is an RGB face picture, the picture is preferably full of light, and the face wrinkles are distinct in the picture, all subsequent steps will be generated based on this picture.
• All subsequent steps will be generated based on this image. A 3D face fitting will be performed on the RGB face image, and the fitted model is best chosen as a 3D face model with reasonable uv map stretching and compression.
• After getting the texture, we will do a differential Gaussian operation on the texture and do average filtering to extract the possibility distribution of wrinkles in a certain part, and we will select a fixed size pixel block from high to low according to the result of the Gaussian differential.
• The pixel blocks are extracted and put into the already trained regressors to get the single channel pixel blocks in the replacement mapping. By combining these replacement mapping code blocks, a replacement mapping map can be obtained.
• After that, the blendshape is transformed into a normal map to get the wrinkle map of the image, and then the wrinkle map of the specified part of the image is obtained by mask, and a complete wrinkle map is obtained by combining the wrinkle maps of multiple images.
• Finally, a motion map of blendshape can be calculated by blendshape, using motion map to know which areas of the face are activated. map, which improves the storage efficiency.

One. Extract texture using 3d face model fitting：

https://github.com/LansburyCH/eos-expression-aware-proxy.git
We used google eos to do the 3D face fitting.
The model chosen is the bfm2017 model.
The last thing we need to use is the texture of the face.
After the face reconstruction step, the result output is like files in 'exmaple_data/' directory.

Prerequisites

• Visual Studio 2017 (>=15.5)
• Boost (>=1.50.0)
• OpenCV (>=2.4.3)
• Ceres (if compiling fit-model-ceres)

There is an example in wrinkle-py/example_data named result.isomap.png.

Two. Regression

The regressor uses pybind11 to call the C++ code
The pyd file is already generated in the file for the python file to call
If there is a problem calling pyd, you need to configure the C++ regressor environment to regenerate the pyd file

C++ source code directory: introduce/Wrinkle

Prerequisites

• opencv 3.4.16
• eigen 3.3.9
• pybind11 2.8.1
• python 3.6

These libraries has been downloaded and compiled and use Cmake to generate the visualstudio project.

The include and library directories are already configured.
If there is a problem with these libraries, you may need to download and compile them again.

Configure x64 Release and Right click on the Wrinkle project and click on "Generate".
You need to change the output file format from output configuration from exe to dll and change the extension name from .exe to .pyd.

Generate 'Wrinkle.pyd' file and move it to the /wrinkle-py after changing the name to 'regressor.pyd'.

Three. Single image wrinkle reconstruction：python configuration

Run wrin_dis.py directly.

• dir_tex： input texture directory(filename is result.isomap.png)
• dis2normal_coff: define how to tranfer displacement map to normal map:(default is 150)
• dog_thres: define the threshold the wrinkle is detected:(default is 60)
• Target: generate normal_2048.jpg —— the normal map with 2048*2048 size generated.

The final wrinkle map can be generated by overlaying the wrinkles of different input images with attach_mask

python wrin_dis.py --dir_tex="example_data\result.isomap.png" --dis2normal_coff=150 --dog_thres=60

Prerequisites

• python-opencv:4.5.5.62
• numpy:1.13.3

Four. Training data

Directory：introduce/metahuman_trainpatch

• data is the texture map and replacement map data
• data_train is the patch data used to train
• DOG is the confidence score map of the wrinkles after Gaussian differencing
• patch is the annotation of the patch in the texture