Shake Your Booty: AI Deepfakes Dance Moves From a Single Picture

Navin Bondade 1 Comment Artificial Intelligence, Data Science, Deep Learning, Machine Learning

Have you ever wished to dance like Michael Jackson, like the way they move there body in the air, the way that their feet kiss the stage and their face matches every expression of the song? Well, it doesn’t matter how shy you are or how bad you are at the dance, now with AI, you can dance like them at least in the digital world.

Researchers have created a new AI-powered 3D body mesh recovery module called Liquid Warping GAN. This tech is an advancement over old 2D keypoints techniques, with this 3D framework, we can now not only estimate body structure but can also preserve the information, such as texture, style, color, and face identity.

Liquid Warping GAN is a part of Liquid Warping Block that propagates the source information in both image and feature spaces and help for the identification of clothing in different styles, colors, and textures; the large spatial and geometric changes of the human body; and multiple source inputs.

The Liquid Warping GAN preserve the source information such as details of clothes and face identity, by first denoising convolutional auto-encoder is used to extract useful features that preserve source information, including texture, color, style, and face identity.

After that, the source features of each local part are blended into a global feature and then it supports multiple-source warping, such as in appearance transfer, warping the features of the head from one source and those of body from another, and aggregating into a global feature stream.

The team also uses a parametric statistical human body model, SMPL to capture the personalized shape and limbs (joints) rotations. it disentangles the n body into a pose and shapes.

The LWGAN works in three-part, 1) synthesizes the background image; 2) predicts the color of invisible parts based on the visible parts; 3) generates pixels of clothes, hairs, & others out of the reconstruction of SMPL.

The one thing that impresses me most about LWB, is that it addresses multiple sources, such as inhuman appearance transfer, preserving the head of source one, and wearing the upper outer garment from the source two, while wearing the lower outer garment from the source three.

In order to train the system, the researchers follow the network architecture and loss functions of HMR, the team used a pre-trained model of HMR.

Researchers said: “In the training phase, we randomly sample a pair of images from each video and set one of them as the source, and another as reference. Our proposed method is a unified framework for motion imitation, appearance transfer, and novel view synthesis.”

“Therefore once the model has been trained, it is capable to be applied to other tasks and does not need to train from scratch. In our experiments, we train a model for motion imitation and then apply it to other tasks, including appearance transfer and novel view synthesis.”

“The whole loss function contains four terms and they are a perceptual loss, face identity loss, attention regularization loss and adversarial loss.”

To evaluate Liquid Warping GAN performance, researchers had 30 subjects with diverse body shapes, heights, genders, and clothing demonstrate random movements to build a new dataset called Impersonator (iPER) with 206 video sequences and 241,564 frames.

Trained on the iPER dataset, Liquid Warping GAN outperformed existing motion imitation methods such as PG2, DSC, and SHUP.

The method, proposed in a new paper from ShanghaiTech University and Tencent AI Lab that’s been accepted by ICCV 2019, requires only a single photo and a video clip of the target dance.

Credit: Liquid Warping GAN: A Unified Framework for Human Motion Imitation, Appearance Transfer, and Novel View Synthesis. Github: Click here.