‘The Irishman’ Gets De-Aging Right—No Tracking Dots Necessary

Martin Scorsese needed a favor. Well, not a favor, exactly—it’d cost quite a bit of money. He needed to make Robert De Niro young again. It was fall of 2015, and Scorsese, knee-deep in production on Silence, was having Thanksgiving dinner in Taiwan with Pablo Helman, one of Industrial Light & Magic’s veteran visual effects supervisors. At the time, Scorsese had been prepping a Frank Sinatra biopic, and he and Helman were talking shop about how to have one actor play a person at many phases of their life.

“He’s a very curious person, and he said to me, ‘Talk to me about that,’” Helman says, reflecting on that first meeting in one of ILM’s San Francisco offices overlooking the Presidio. “So we talked about how to make somebody younger. He said, ‘You know what? I’m not going to do Sinatra, but I’m going to do this other thing.”

That other thing turned out to be The Irishman, Netflix’s 3.5-hour, decades-spanning adaptation of I Heard You Paint Houses, Charles Brandt’s book about the alleged mafia hit man Frank Sheeran. Scorsese sent Helman the script, and he read it overnight, all 170-ish pages. “We were shooting [Silence] the next morning,” Helman recalls, “and I said, ‘I’m in.’”

What he was in for was a four-year quest to reinvent the way Hollywood makes its stars (not) act their age. In recent years, the race for viable de-aging technology has ramped up. Studios used to just ask VFX houses to do a little facial touch-up—thinning out wrinkles or fixing bad makeup—but as computers got faster and software got better, they started realizing fuller-scale de-aging was possible. Back in 2008, for David Fincher’s The Curious Case of Benjamin Button, Brad Pitt’s facial expressions were captured using the Mova camera system, which places a series of cameras around an actor to gather all of their facial movements and then uses that data to digitally build someone older/younger. (Since Benjamin Button ages backward, Pitt didn’t need to be de-aged so much as he had to be re-aged.) For this year’s Gemini Man, directed by Ang Lee, the VFX team cut Will Smith’s age in half by scanning his face, building a database of his expressions, and merging that with data from earlier performances—Fresh Prince of Bel-Air, Bad Boys. The 49-year-old Smith did his acting on a mo-cap stage, wearing headgear and face-tracking dots while cameras captured his every move; that performance was then combined with the information in the database to re-create a 23-year-old version of the actor.

To make Irishman, though, Helman couldn’t replicate Weta Digital’s process on Gemini Man. Scorsese wanted to be able to shoot his movie the way he would any other. No mo-cap. No actors walking around in headgear. “The film takes place from 1949 to 2000, and it goes back and forth in time continuously,” Scorsese says. “The problem is, by the time I was ready to make the film, Bob De Niro, Al Pacino, and Joe Pesci can no longer play these characters younger in makeup.” So, when Helman said he could do it more au naturel, the director was intrigued. “I said, ‘I don’t know. I can’t have the actors talking to each other with golf balls on their faces. It gets in the way of the actors, and the kind of film this is, they need to play off of each other. If you can find a way to lessen the technical aspects of it, it could work.’”

It was a huge hurdle, but one Helman was pretty sure he could jump. After Silence wrapped, he went to New York to test his theory. He and Scorsese asked De Niro to come in for a day and reshoot a scene from Goodfellas. If Helman could make the actor, then in his early seventies, look the way he did 25 years earlier as Jimmy Conway, then ILM could pull off de-aging the film’s principal cast. Helman spent 10 weeks working on the test scene. In the end, De Niro looked like he did in 1990. “That green-lit the movie,” Helman says.

Remaking one scene from Goodfellas was one thing; making a 3.5-hour movie full of actors of varying ages was another. Helman had a proof of concept, but he still had to develop the technology to allow Scorsese to film his movie the way he wanted to. That development process would take two years. Finishing the whole film would take twice that long. “For four years it was building a Ferrari,” he says, “while you’re running the grand prix.”

On the left, Robert De Niro’s original performance in The Irishman. On the right, his de-aged face as it appears in the film. Photograph: Netflix

What Helman came up with was unprecedented. Essentially, he created a new type of camera rig that would allow Scorsese to shoot as he normally would while capturing all of the data the ILM team would need to make Frank Sheeran (De Niro), Jimmy Hoffa (Pacino), and Russell Bufalino (Pesci) whatever age they needed them to be. The rig had a standard director’s camera in the center, and on either side were two film-grade Alexa Mini cameras outfitted to shoot infrared images, capturing all the volumetric information normally picked up by those dots Scorsese didn’t want to use. The rig was big, initially about 84 pounds, but working with the camera company Arri, Helman was able to cut it down to 64 pounds while also making it about 30 inches wide (two inches narrower than standard door frames).

That was the first two years. The second two years was all of the postproduction work. Helman’s cameras captured hours of footage and gigabytes of data, and now they needed to turn it into a movie. To do that, they created a piece of software called Flux that combines the infrared information with the images from the main camera to create masks on each actor’s face. Sitting in an ILM conference room, Helman demonstrates this by showing the scene of the now-infamous phone call between Sheeran and Hoffa (“I heard you paint houses … ”) from The Irishman. There they are as their (relatively) current-day selves, then they’re decades younger, with a few clicks. The face is smoother; the chin and neck replaced. De Niro had had a wisdom tooth removed the day before. It doesn’t even show.

Helman showing off the conversion with a few keystrokes makes it look easy, but it required hours upon hours of work. To create the various ages for someone like Sheeran, who goes from his early forties to his eighties in the movie, the team cataloged thousands of frames from movies ranging from Goodfellas to Casino. (Once Sheeran’s character hits his seventies, mostly everything was done with makeup, save for a few details.) They then made catalogs of images of noses, eyes, and mouths that they would use to make de-aged faces that could be placed into each scene. ILM also developed an artificial intelligence system that would take any frame they made and scour the full image library in an instant to give them reference images of what the actor should look like so that the team could check their work.

The camera rig that Pablo Helman and Industrial Light and Magic created with Arri for The Irishman. Photograph: Netflix

The result is a movie that spans five decades and has Pesci, De Niro, and Pacino playing their characters throughout, in performances that look, if not 100 percent real, then just about emerging from the uncanny valley. (The New York Film Critics Circle just named The Irishman its best picture.) “The most important thing about the technology we developed has to do with the nuances in the performances,” Helman says. “If we had that been doing this with markers, we couldn’t have gotten that.”

Offering the ability to film actors naturally and alter their ages at will could also make ILM’s de-aging system—the camera rig, the Flux software—a very appealing option for future filmmakers. Though, notes Mike Chambers, chair of the Visual Effects Society, whether directors or studios will choose to let their actors roam free or cover them in facial tracking dots will largely depend on what kind of movie they’re making. Mob films and sci-fi flicks about cloned hit men are very different beasts, after all—even if they do both involve assassins.

Regardless, de-aging is here to stay. What’s more interesting is where it goes from here. Asked about the future, Chambers notes the recent dustup over the forthcoming movie Finding Jack, for which the filmmakers are creating an entirely digital performance from long-deceased actor James Dean. Making an actor look younger is one thing; resurrecting them is another. “Philosophically and ethically, I think that’s fascinating,” he says. “If you give people the tools, somebody’s going to figure out a way and have a desire to use it. Then the question is, will people use it wisely or will it just be exploitative?”

What Chambers means, of course, is that de-aging is invaluable when you have three actors trying to cover five decades’ worth of time. It’s less good when a director could hire one person and change their appearance when they could have just hired someone who fit the role. Why re-create James Dean when you can hire a young actor who needs a job? Why fake it if you don’t have to? Finding the fountain of youth doesn’t require jumping in.

More Great WIRED Stories

Read More