New Face Tracking Technology Manipulates Videos in Real-Time

NEW HAVEN, Conn. (WTNH) — The days of using a picture as proof that something really happened are long gone. That’s because a new technology is taking things one step further than Photoshop. With Face2Face, you can make anyone in a video say whatever you want them to say, including Donald Trump.

“The idea is essentially you can tag a video with a web camera like for instance the interview we are doing right now and we can transfer the animation of my facial motion to a different video,” explained Matthias Niessner, a visiting assistant professor at Stanford University and a member of the team developing Face2Face.

Computer generated graphics are nothing new. After all, Hollywood’s been using them for a long time, but what is new is being able to manipulate existing videos.
“It basically works for every video,” Niessner said. “We just use YouTube videos because they’re widely available.”

Face2Face works on news footage of public figures like former President George W. Bush, or even Vladimir Putin.

“The other part that’s pretty real is we can actually do it in real time,” Niessner said. “Basically while talking I could just animate another video.”

So how does it work? Face2face uses a webcam to capture an actor’s mouth, cheek and eyebrow movements. Those movements are then translated onto the target person’s face.

“The main goal we had in mind is video dubbing,” Niessner explained.

That means no more subtitles in foreign films or lips that are out of sync. Niessner says using the technology to dub films will work especially well for certain languages.

“So I’m German for instance and between German and English it’s relatively easy to do dubbing,” Niessner said. “Between Mandarin and other Chinese dialects it’s pretty hard to translate to English … the text will be definitely long. The audio will sound completely different so it’s very hard to match the lip motion there.”

Another cool feature about Face2Face is that it can be used to detect if a video has been edited, which may come in handy considering 300 hours of video are uploaded to YouTube every minute.

“If it becomes more common to edit videos and [easier], then I think it’s very important that we also have the right tools to detect these changes,” Niessner explained. “So I don’t want to leave the world with no ability to figure out if the video is real or not.”

The next goal for Face2Face is to be able to manipulate eye movements. The technology isn’t available to the public yet but if and when it is, it will most likely be the next big thing.
Face2Face is being developed by researchers at the Max Planck Institute for Informatics, Stanford University and The University of Erlangen-Nuremberg.

WTNH NEWS8 provides commenting to allow for constructive discussion on the stories we cover. In order to comment here, you acknowledge you have read and agreed to our Terms of Service. Commenters who violate these terms, including use of vulgar language or racial slurs, will be banned. Please be respectful of the opinions of others. If you see an inappropriate comment, please flag it for our moderators to review. Also, you can now block any inappropriate user by simple selecting the drop down menu on the right of any comment and selection "Block User" from there.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s