When Diplo and The Strokes need a deepfake, they go to this guy

The Strokes look like they’re 20 years old again in their latest music video. That’d be impressive makeup work on a group of roughly 40-year-olds, but the band doesn’t appear in the music video at all: every one of their faces is a deepfake.

The deepfakes were created by Paul Shales, whose meme pages have blown up over the past year. Under the moniker The Fakening, Shales has put Sen. Bernie Sanders into a dance troupe, turned President Donald Trump into a young girl, and added Joe Rogan to Star Wars. More recently, he went viral after inserting Amazon CEO Jeff Bezos and Tesla CEO Elon Musk into an episode of Star Trek — a creation he called “an easy deepfake” after he got frustrated working on some harder projects.

“Sometimes I like to go for a bigger mainstream face that’ll be more recognizable,” Shales told The Verge in a phone call. “Generally, it’s something that I want to make you laugh.”

In the two years that deepfakes have been popping up, there’s been growing concern around how they could be used maliciously to shame women or manipulate politics. But they’ve also, increasingly, been used in more lighthearted ways. The technique has put actors into movies they weren’t really in, created ridiculous face swaps, and shown promise as an alternative to much more time-consuming manual visual effects work — like, say, removing Superman’s unwanted mustache.

Meme accounts have filled in some of those funnier gaps, and Shales got in early, sensing an opportunity. “I was like … ‘I think I could grow a social following quite quick if we start turning these away from porn toward funny memes and entertainment,” he said. Shales started The Fakening at the beginning of 2019, and he’s developed a small following in the year since. His Instagram account now has 170,000 followers, and he has another 16,000 on YouTube and almost 9,000 on Twitter. To help his accounts grow, he made deepfakes of comedians he thought would get a kick out of them. When one finally reposted him, Shales said it sent 10,000 followers his way.

There’s no professional industry around deepfakes yet, but Shales said musicians have steadily been among the first professional customers. He picked up a small project for The Chainsmokers, putting the band members’ faces onto the bodies of Gordon Ramsay and other famous people to promote a remix contest. But he never saw where they ended up using it. Eventually, Diplo reached out saying he wanted to do something together, too. He first had Shales deepfake his head onto Mark Wahlberg’s body from a classic Calvin Klein shoot. Later, when he did a remix of Niall Horan’s “Nice To Meet Ya,” Shales remade Horan’s music video with Diplo’s face swapped in.

The Strokes’ music video for “Bad Decisions” is Shales’ biggest project yet. “That took me a whole month,” Shales said. “All my computers were tied up working on The Strokes.” (His main system is a repurposed cryptocurrency mining rig with nine graphics cards and “a whole bunch of fans.”) The video is supposed to be an advertisement offering clones of The Strokes, so they decided to clone the band with deepfakes.

Because they intended to use deepfakes from the start, Shales was able to give the production advice on how to film in ways that would make the effects convincing. If you get too close or too far away from the person, the effect starts to fall apart, he said. Profile shots are also hard because there’s less information for the computer to work with, and any scene where there’s too much motion — especially if something passes in front of someone’s face — can get tricky. So for the most part, the video is shot with clean movements, a comfortable distance away from any of the fake Strokes.

Another big issue is face shapes. Shales said deepfaking essentially amounts to drawing one person’s face onto another’s, and that means the resemblance can start to fall apart if the overall shape of the new face is too different. (When we spoke, he was trying to deepfake Scarlett Johansson’s face onto a large body-builder type, and he said her sharp chin was disappearing into the model’s square jaw, losing the resemblance.) To solve that for the music video, the production cast actors with a resemblance to the band. The actor playing frontman Julian Casablancas, for instance, has somewhat of a longer, rounder face, like Casablancas himself.

Then there was the trick of making them look younger. When creating a deepfake, Shales has to hunt down footage of the person he’s trying to swap in — and it generally helps when the footage matches the angles and expressions he’s trying to re-create. For The Strokes, he had to plug in enough period-specific footage of the band. He said their label sent him “a dump of interviews” from at least a decade ago to work with. That was enough for prominent members of the band, but he was missing “a lot of the angles” on less prominent band members who “didn’t move very much” during the interviews. He ended up going on YouTube and searching for videos specifically of them in order to get enough visual data.

Calvin Klein

Here are some cool behind the scenes from the Calvin Klein shoot

Posted by Diplo on Wednesday, August 21, 2019

Even with all of that work, some footage still had to be tossed. In some cases, an actor was too far away or their face was too blurry to deepfake. “We just edited around that,” Shales said.

Shales said he’s now making enough off of deepfakes that it’s become a full-time job. So far, money has come entirely through commissions — his social pages aren’t monetized, and he’s “apprehensive” about posting ads — but he tends to get commissions every time one of his posts goes viral. Some he turns down for ethical reasons, like people requesting nonconsensual porn videos; another he might turn down over concerns that it could be misused to manipulate the stock market. Others just don’t pan out.

There’s still not a lot of professional work. “That’s gonna change,” Shales said.

Read More