FRANK ROSE

A silver-haired figure in worn jeans and a New Zealand Stunt Guild T-shirt, he’s holding a small, flat-panel computer screen with handles and knobs attached. This is the “virtual camera” he’s been dancing around with on the set—his window onto a performance that can’t be seen with the naked eye. The soundstage is set up for performance capture, so instead of recording images with the stereoscopic 3-D camera rig he spent years developing after Titanic, he’s gathering data. Strung across the ceiling are infrared cameras that register the actors’ movements from markers on their bodies; the same cameras pick up the motion of his screen from markers on its frame. Holding it in his hands and pointing it toward his two stars, Sam Worthington (from Terminator Salvation) and Zoe Saldana (Uhura in J.J. Abrams’s Star Trek reboot), he can tilt and pan as if it were an actual camera.

This in itself is surprising enough—but what’s really remarkable is what he sees on the screen. Not Sam Worthington and Zoe Saldana, standing a few feet away in their body-hugging black mocap suits; not the hollow gray risers on the soundstage; not the overarching blackness all around us. What the screen displays is the blue, catlike, ten-foot-tall avatar of Worthington’s character, Jake Sully, standing next to Saldana’s equally blue, catlike alien in the deep jungle of the rainforest planet Pandora, where most of this movie takes place. Not reality but virtual reality, in real time.

Cameron calls to an assistant, who starts feeding him a scene they recorded yesterday. Suddenly the screen image switches from Jake and his alien friend to what appears to be a toad on a log. This is a fan lizard—a Cameron-invented species unique to Pandora. Ugly at rest, they turn strangely beautiful when they lift up their whirlygig wings and fly. As this one comes to life on his screen, Cameron begins to gyre across the soundstage, tilting the screen this way and that. “I can zoom out a little bit. I can follow him around the landscape as he's flying. Come underneath, come up on top if I want, see how he flies over the terrain.”

This is what he does on “camera days,” when no one else is around and he can perform the camera moves on a previously captured performance. But today is a capture day, not a camera day. In the scene coming up, Jake and Neytiri, the alien love interest played by Saldana, are playfully chasing fan lizards through the jungle.

Jake is a paraplegic ex-Marine who inhabits the body of a Na’vi, as Pandora’s indigenous humanoids are called. Earth covets Pandora for its natural resources, in particular a critical mineral called “unobtanium,” so it has sent mechanized corporate armies to strip-mine the place. Avatars are Earth’s goodwill ambassadors—but the Na’vi aren’t buying it. “They take one look at what we're about and say, no—they tear the place up, they wreck the forest, they are poison,” Cameron explains. “So our guy Jake arrives in a very sour situation.”

In the scene Cameron is about to capture, Jake has become lost and separated from his unit. He met Neytiri a short time before, when she saved him from an enormous, charging thanator—another Cameron species, this one named after the Greek god of death—and a vicious pack of razor-toothed viperwolves. Now he and Neytiri are getting to know one another. Soon he will face a fundamental test of loyalties—but that lies ahead. For the moment, he’s just having a blast chasing fan lizards.

For Worthington and Saldana, that means leaping across the flimsy plywood risers, wearing not only standard-issue mocap tights but large fake ears (so they’ll know where their ears are if they have to touch their heads) and head rigs that keep a tiny high-definition camera pointed back at their faces. This is another Cameron innovation: By getting video close-ups of the actors’ eyes and features and mapping that to their characters’ faces, he hopes to avoid the “uncanny valley” that made Robert Zemeckis’s mocap figures in Beowulf and The Polar Express so unnerving.

Zemeckis, Cameron explains, “used a completely different kind of facial capture. They used the same type of marker system that we use for the body, and then they captured the facial performance with mocap cameras. Really a bonehead idea.”

The script calls for Jake and Neytiri to stare in wonderment at some reflective dots, each dot representing a fan lizard. The dots are stuck to the ends of a half-dozen skinny wooden poles that crew members are waving about. Setting aside his virtual camera, Cameron grabs a pole and joins the fun. For him as for Jake, the fan lizards offer a brief interlude between ordeals.

Eighteen months earlier, in September 2006, I’d stood next to Cameron on another soundstage as he discussed the challenge he was about to take on with Avatar. We were in Montreal, on the set of Journey 3-D, the Journey to the Center of the Earth remake that was the first picture to employ his stereoscopic camera system. He was on hand to see how his camera performed. I caught him in an oddly reflective mood.

“I’m 52,” he said. “I’ve made a bunch of movies, won a bunch of awards, made a bunch of money—none of those are interesting to me. They never were. It was only about the film. But what’s challenging? I’ve worked with actors for 25 years. I love actors—but there are no real challenges there. I look for the new thing. And with this film”—he ticked off the tools they’d been developing: the 3-D camera, the virtual camera, others barely beyond the concept stage—“we’ve kind of loaded it up.”

AT THE FAR end of the Playa Vista soundstage, three tiers of computer monitors rise like bleachers toward the ceiling. This is Avatar’s “brain bar,” the domain of its digital production team. Each tier is manned by five or six people—mocap specialists from Giant Studios on the lower rung, animation and visual effects supervisors from Cameron’s Lightstorm Entertainment and Peter Jackson’s Weta Digital up above. It looks like mission control, only scruffier.

On their monitors, the brain bar people are watching different iterations of the fan lizard scene as it’s being captured. One screen maps skeletal data points on a grid. Others toggle from Worthington and Zaldana on the risers to Jake and Neytiri in the jungle.

The brain bar is command central for another major advance, a nifty bit of software they’ve dubbed the simulcam. Using the simulcam, Cameron’s crew can mix performance capture with footage from the high-definition 3-D video shoot he recently completed in New Zealand. Most of that shoot was done against a greenscreen, with the background to be filled in by computer. The simulcam generates a real-time composite of performance-capture data, live-action photography, and computer-generated backdrops, compressing into seconds a job that would normally take weeks. On the fly, the denizens of the brain bar can transmute terabytes of raw data into a rough approximation of the fantastical universe that for the past fifteen years has existed only in Cameron’s mind.

“You can call it what you want,” say Glenn Derry from his perch on the second tier, “but it’s really a big database.”

Derry is Avatar’s virtual production supervisor—in effect, its CTO. Unlike the 3-D camera, which Cameron himself put together with Vince Pace, the virtual production system was assembled by Derry and a handful of others. “It always started with Jim going, ‘What if I could do this?’” says Richie Baneham, the production’s animation supervisor. But beyond that, there was no master plan.

“It just kind of evolved,” Derry explains. “We kept adding features and adding features, and then we got to the point where we couldn’t keep track of them all.” He laughs, a little too wildly. “We’ve been doing this for two years! But we’re so far ahead of the curve. We bring in people from outside and they go, ‘Whoa!’ On Beowulf”—still nine months away from release at this point—“they do work and you won’t see anything for months.”

Between them, Derry and Baneham have worked on some of the top effects-driven films of the decade. Derry apprenticed with the late Stan Winston, Cameron’s long-time effects man, and earned his chops working with Steven Spielberg on Artificial Intelligence: AI and Minority Report. Baneham led the effort to bring Gollum to life for Peter Jackson in The Lord of the Rings—the feat that convinced Cameron that Avatar could finally be made, years after he’d originally proposed it. But those pictures were just a run-up to Avatar.

“A movie like this had not been done,” Derry maintains—a live-action/performance-capture hybrid, in 3-D, with a scale differential that makes six-foot-tall actors look ten feet, and not just in one or two scenes but for almost every character in the picture. To make it all work, they had to create an entire virtual production system—the virtual camera, the simulcam, and a digital pipeline to feed it all together.

They’ve ended up with a form of pre-viz that functions in real time. Pre-viz is f/x-speak for previsualization—a computer animation designed to show the director what a scene will look like after it’s been shot and all the effects put in. When Derry started on Avatar in 2005, Spielberg and ILM were pushing pre-viz to its limits on War of the Worlds. Pre-viz enabled Spielberg to shoot an action scene knowing where the alien war machine would go when ILM rendered it many months in the future. But it didn’t show him the war machine crashing through the streets—which is what he’d have seen with Cameron’s system.

To build it, Derry and his team started with MotionBuilder, a 3-D character animation program that’s been used on everything from Assassin’s Creed to Beowulf. Giant Studios was brought in for its real-time “solve”—its ability to provide an instant solution to the mathematical problem of how to interpret performance-capture data as objects in time and space. The simulcam takes this information, feeds it together with 3-D video, and composites it on the fly. That’s how Cameron can hold a computer screen in his hands and see Jake and Neytiri with the fan lizards.

Baneham’s task was to replicate Weta’s feat with Gollum. Before Gollum, digital animators seeking to create a smile would just dial the mouth a little wider. Inevitably, it looked fake. So for The Lord of the Rings, Baneham and his team went back to basics: What muscles fire to produce a smile, and in what order? How much light does the skin absorb, and how much does it reflect? How deep is the pupil, the opening at the center of the eye? “We as human beings will process an image to see if there’s something we need to look at,” he explains, “with an emphasis on finding eyes. And all those things we unconsciously notice about other people, we notice immediately when they’re wrong.”

This, of course, is the definition of the uncanny valley—the pitfall Cameron wants so desperately to avoid. But the technique that worked for a single character in The Lord of the Rings would require serious streamlining to work at the scale Avatar requires. That’s because the details that made Gollum so convincing were provided not by performance-capture data, as on Beowulf, but by digital animators working frame-by-frame to Baneham’s specifications. Cameron wants this done for multiple characters—so much more information is required.

The solution was the head rigs. Plaster-casting each actor’s face, they fabricated custom-fitted harnesses to hold a tiny video camera a few inches away. The camera’s wide-angle lens records every twitch, every blink, every frown, so instead of having to imagine how an actor’s face is moving, the animators have a complete visual record they can map onto the face of the CG character. Taking no chances, Cameron has had reflective dots painted on the actors’ faces so the animators also have performance-capture data to work with. Reference cameras—HD videocams positioned around the soundstage—provide yet more information. That’s why, when Jake seems about to be eaten alive by the thanator and the viperwolves, you can follow his eyes as they dart wildly across the jungle and see Neytiri about to save him.