NEW HAVEN, Conn. (WTNH) — The days of using a picture as proof that something really happened are long gone. That’s because a new technology is taking things one step further than Photoshop. With Face2Face, you can make anyone in a video say whatever you want them to say, including Donald Trump.
“The idea is essentially you can tag a video with a web camera like for instance the interview we are doing right now and we can transfer the animation of my facial motion to a different video,” explained Matthias Niessner, a visiting assistant professor at Stanford University and a member of the team developing Face2Face.
Computer generated graphics are nothing new. After all, Hollywood’s been using them for a long time, but what is new is being able to manipulate existing videos.
“It basically works for every video,” Niessner said. “We just use YouTube videos because they’re widely available.”
Face2Face works on news footage of public figures like former President George W. Bush, or even Vladimir Putin.
“The other part that’s pretty real is we can actually do it in real time,” Niessner said. “Basically while talking I could just animate another video.”
So how does it work? Face2face uses a webcam to capture an actor’s mouth, cheek and eyebrow movements. Those movements are then translated onto the target person’s face.
“The main goal we had in mind is video dubbing,” Niessner explained.
That means no more subtitles in foreign films or lips that are out of sync. Niessner says using the technology to dub films will work especially well for certain languages.
“So I’m German for instance and between German and English it’s relatively easy to do dubbing,” Niessner said. “Between Mandarin and other Chinese dialects it’s pretty hard to translate to English … the text will be definitely long. The audio will sound completely different so it’s very hard to match the lip motion there.”
Another cool feature about Face2Face is that it can be used to detect if a video has been edited, which may come in handy considering 300 hours of video are uploaded to YouTube every minute.
“If it becomes more common to edit videos and [easier], then I think it’s very important that we also have the right tools to detect these changes,” Niessner explained. “So I don’t want to leave the world with no ability to figure out if the video is real or not.”
The next goal for Face2Face is to be able to manipulate eye movements. The technology isn’t available to the public yet but if and when it is, it will most likely be the next big thing.
Face2Face is being developed by researchers at the Max Planck Institute for Informatics, Stanford University and The University of Erlangen-Nuremberg.