Try to get your own cutie portrait using MMGEN-FaceStylor

We attach the github link below at the beginning of the article. Please star our project if you like it ~

GitHub - open-mmlab/MMGEN-FaceStylor

When you are watching cartoon movies or comic books, have you ever wondered about your figure in the anime world?

Now, getting your own cartoon image can be realized through deep learning technology. OpenMMLab has opened source a face stylization project: MMGEN-FaceStylor ,which can not only change the characters into images of various styles, but also controls the stylization intensity.

Which one do you like best after seeing the above styles？

If you are not satisfied with the style provided and want to use your own data for training, we also provide a complete model training code. Feel free to try it.

Getting started

First, we need to build a conda virtual environment and activate it.

conda create -n facestylor python=3.7 -y
conda activate facestylor
conda install pytorch=1.6.0 cudatoolkit=10.1 torchvision -c pytorch

Since our project relys on MMCV and MMGEN, we need to install them.

pip install mmcv-full -f https://download.openmmlab.com/mmcv/dist/cu101/torch1.6.0/index.html
# Clone the MMGeneration repository.
git clone https://github.com/open-mmlab/mmgeneration.git
cd mmgeneration
# Install build requirements and then install MMGeneration.
pip install -r requirements.txt
pip install -v -e . # or “python setup.py develop”
cd ..

Finally, clone the repo and prepare the data and weights.

git clone https://github.com/open-mmlab/MMGEN-FaceStylor.git
cd MMGEN-FaceStylor
pip install requirements.txt
mkdir data
wget — no-check-certificate ‘https://docs.google.com/uc?export=download&id=1AavRxpZJYeCrAOghgtthYqVB06y9QJd3' -O data/shape_predictor_68_face_landmarks.dat
mkdir work_dirs
mkdir work_dirs/experiments
mkdir work_dirs/pre-trained

To try our project quickly, please run the command below

python demo/quick_try.py demo/src.png — style toonify

Then, you can check the result in work_dirs/demos/agile_result.png.

• If you want to play with your own photos, you can replace demo/src.png with your photo.
• If you want to switch to another style, change toonify with other styles. Now, supported styles include toonify, oil, sketch, bitmoji, cartoon, comic.

We show the style example below

Toonify:

Oil:

Cartoon:

Comic:

Bitmoji:

Sketch:

Introduction

In fact, you can play with it instantly by clicking the demo link in repo. (Don’t forget to star our repo!)

Mechanism

What is the mechanism behind such a facial stylization project? We mainly use two algorithms, Toonify and AgileGAN.

In AgileGAN algorithm, We first need to pre-train a StyleGAN2 model on the FFHQ dataset, and then train an encoder to obtain image embedding.

Then we need to fine-tune a cartoon StyleGAN2 model based on FFHQ-StyleGAN2.In order to make the cartoon face have a better similarity with the original face, we add the perceptual similarity loss between FFHQ-StyleGAN2 and cartoon StyleGAN2 outputs under the same noise input.

In the inference stage, the embedding of the original image is input to the cartoon StyleGAN to get the animated face. But the images output in this way do not seem satisfactory in terms of similarity.

So we combined Toonify’s layer swapping method and replace the generator shallow weights with the face generator to better preserve the original image features.

The four effects, from left to right, are input, toonify, AgileGAN and our MMGEN-FaceStylor.

The project will continue to be supported in the future ~ we will add more style options, more models (like AnimeGAN), and launch on the open platform, so stay tuned! If you want any data or features, please feel free to PR and issue!

Please give us STAR, FORK and WATCH，if you like it❤

By the way, you’d better upload a full-face photo, otherwise you might get strange results 😂

Reference link.

[1] Toonify

[2] Song Guoxian, AgileGAN

[3] https://github.com/NVlabs/metfaces-dataset

[4] https://www.kaggle.com/defileroff/comic-faces-paired-synthetic-v2

[5] https://www.kaggle.com/arnaud58/photo2cartoon

[6] https://www.kaggle.com/mostafamozafari/bitmoji-faces

[7] https://unsplash.com/