As most of you have seen by now, the results of the character animation are very impressive. They’ve already begun talking about the creepy Davy Jones as the next great CG peformance beyond Weta’s Gollum and King Kong. Lord of the Deep and commander of the mysterious Flying Dutchman ghost ship, Jones is a delicious mutation: part human and part squid, with a beard full of wiggly tentacles, and crab-like claws.
Unable to rely on traditional MoCap or hand animation, ILM created an innovative new system called Imocap that allowed onset and on location motion capture to elicit the most believable look and performance possible out of actor Bill Nighy.
“The characters required a lot of careful examination of human performances and then trying to combine that with the animation,” explains animation supervisor Hal Hickel. “We knew that there were going to be actors cast to play Davy Jones and his crew, and that those actors would be on set in the plates that we were going to be put those CG characters into and that somehow we had to extract the motion of the performances without having to reshoot later. We didn’t want to bring the mocap stage onto the set. So the R&D and MoCap groups came up with a solution: special [sensor-studded] suits that would be worn by Bill Nighy and other actors playing his crew. We would take reference cameras onto the sets and untethered cameras out on location with lightweight tripods and position them at angles off of what the main taking camera was seeing. This allowed us to track the movements and provided great data from the hero plates with the actors in them, casting their real shadows and making good eye contact with the live actors, and then we were able to extract their motion and apply it to our CG characters and put those characters right on top of the actors. There’s still a lot of animation artistry in there because there’s a lot of interpretation. This is just about getting the skeletal motion of the character; we still did all of the facial animation by hand [in Zeno].”
But more about Imocap in part two. Suffice it to say, Davy Jones is the most complex and human looking CG character in Dead Man’s Chest — and he’s all-CG. The animators incorporated as much of Nighy’s face as possible. However, the eyes proved to be an interesting test case. “As a backup, [director] Gore [Verbinski] asked us to put some makeup in a T-zone around his eyes and mouth, in case he wanted to do a blend for an extreme close-up,” Hickel continues. “But we never used it. We knew it would be difficult, but we figured we could get there pretty quickly. What was just as difficult was the whole spark of life. There’s always that last percent of realism that’s hardest to capture. The closer you get to the goal of it being real enough that people will stop worrying about it and thinking about it, the more glaring the omissions are. On top of which, there’s the gray area of his performance. The thing about Bill was he wasn’t a stone-faced villain. It was a very mercurial performance — he was constantly changing his expression and delivery. Nobody expected it. Every scene we’d stare at it and study it. I know there are animators that are leery of any technique that takes away some of their authorship. I totally understand that. Pure animation is wonderful. But I also think the collaboration between an animator and a live actor is an exciting thing too. I imagine it’s what makeup artists feel.”
Zeno added an additional challenge. ILM came into Dead Man’s Chest with only a small portion of its creature pipeline function intact. War of the Worlds and The Island, the two previous projects done with Zeno, were primarily hard surface works. The creature work on those didn’t need cloth, sim, flesh or hair. A large part of the effort was re-enabling the pipeline, particularly the facial animation.
Meanwhile, the second R&D project involved the tentacle beard itself. “Our R&D folks worked with James Tooley, our sim guru [creature development supervisor], and Karin Derlich [creature technical director], who came up with behaviors,” Hickel adds. “And we’d do tests and I’d say, ‘This one is too tentacly and this one feels too much like an elephant’s trunk and this one feels too much like a snake.’ We would look at tons of octopus references. After we got it, then those behaviors were added to the solver through what we call ‘Joint Motors,’ so all the tentacles were divided into little joint segments and each segment was essentially a little motor that was directed to move this way or that way. So those joint motor impulses were sent out at the same time the tentacles were receiving force information: I need to swing this way, I need to swing that way… and so it would all happen together.
|
|
The crew of The Flying Dutchmen features a cast of characters with visual references to the ocean and its creatures, including coral, sea sponges, barnacles, mussels, hammerheads and puffer fish.
|
|
“Once we added those behaviors to our sim engine, the last thing we needed was something called ‘Sticktion,’ which is a combination of friction and stickiness. The problem was that without Sticktion, the tentacles would just slide onto each other. We really wanted them to be this heap of viscous tubes that would stick to each other and stick to his chest. And the ones at the bottom of the stack would stick there in a big matte. The biggest ones out in front that hang from his chin and moustache-like tentacles could really swing around.”
“Imagine a piece of spaghetti sticking to a leather jacket,” suggests visual effects supervisor John Knoll. “That was the effect I wanted to get. R&D added this subtle stickiness to the engine.”
“The great thing,” Hickel continues, “is that as complicated as it was, once Karin came up with basic settings for all of the controls, the sim artists got up and running very quickly. I’m pretty amazed by that, actually, because this was very stressful for me. Back in December, when we really didn’t have this working yet, there was no plan B. We couldn’t animate it by hand and we looked at other sim possibilities, but they didn’t achieve what we had in mind. There are more than 200 shots and 15 minutes of screen time of Davy and we had only one artist who knows how to do this.”
With as many as 50 animators working together on a total of 18 CG characters, there were plenty of technical and artistic challenges. “What makes these characters so complicated is that they are encrusted with sea life and we had to figure out ways to cover them with barnacles and such,” Hickel observes. “We wrote tools that the modelers used where they had a sea life picker, where they could pick a mussel or a barnacle. As our model supervisor, Jeff Campbell, said, it was a little like flower arranging. And they also used ZBrush for displacement textures for the sea life and for the characters themselves and our usual suite of modeling and paint tools.”
The crew of The Flying Dutchmen include Ogilvey, who has a sea sponge head; Palafico, whose head is a red fan coral and very translucent; Koleniko, in which one side of his face is a puffer fish and can puff up with spines; and Knoll’s favorite: a crab-like creature whose head rotates in and out of the shell.
|
|
The Kraken’s tentacles modeled in Maya. The creature was keyframe animated with some flesh sim enhancements in Zeno, courtesy of the new creature pipeline.
|
|
Then there’s the Kraken, the mythological squid monster that most are familiar with from 20,000 Leagues Under the Sea, which plays a prominent role in Dead Man’s Chest as the instrument of Jones’ destruction. Tentacles were crucial here as well. Modeled in Maya, the Kraken was keyframe animated with some flesh sim enhancements in Zeno, courtesy of the new creature pipeline. They even had to procedurally tweak the suckers on each row of tentacles because they were too clean looking, so they randomly replaced suckers that were more rough and worn looking.
Utilizing ILM’s new fluid dynamics engine, developed in cooperation with the Stanford University research program, Dead Man’s Chest, like Poseidon, contains improved CG water, in which nifty algorithms are put through multiple processors. And thanks to Zeno, which has been described as “Maya on steroids,” you can introduce particle controls, Soft Body, Rigid Body controls and other techniques.
“We started out in parallel with Poseidon, but they got into a bit of a crisis and we loaned them my entire water crew,” Knoll admits. “They wrapped in April and I got them back to finish my shots. They really pushed the envelope. The development they did at the end of Poseidon really paid off here. We did a lot of difficult water shots right up to the last day. The crew really knew what nobs to turn to get it to look good. We used CG water around the bases of the tentacles when they’re sloshing back and forth underwater. The Flying Dutchman travels underwater and reaches the surface like a submarine, so those shots were done with CG water as well, and the Dutchman is 380 feet long. We got realistic droplet size and realistic dispersion of particles.”
The scenes on Cannibal Island, where Captain Jack Sparrow (Johnny Depp) narrowly escapes, contain a large number of shots where you see different variations of the same view under different lighting conditions, so Knoll and matte supervisor Susumu Yukuhiro needed to think about a 3D solution. “We saw ads for a product called Vue. It’s designed for organic landscapes and getting realistic renders. We started playing around with it and it became our primary tool for big, exotic landscapes.”
Overall, Knoll believes Dead Man’s Chest takes character animation another step forward at ILM, especially considering Nighy’s performance. “There are not as many shots numerically as on Sith, but it’s [a greater accomplishment] in terms of the amount of shots in the time that we had. Sith had 2,400 shots in about two years and this had 1,000 shots in about five months, but the average shot complexity was higher than on Star Wars.”
Looting CG Treasure From Dead Man’s Chest — Part 2
Bill Desowitz concludes our two-part coverage of Pirates of the Caribbean: Dead Man’s Chest with a report on ILM’s innovative Imocap system.
By Bill Desowitz
[ Posted on July 17, 2006 ]
|
|
ILM’s new Imocap was used to create the CG characters of Davy Jones and his crew in Dead Man’s Chest. All images © 2006 Disney Enterprises Inc and Jerry Bruckheimer Inc. Photo credit: ILM. |
|
Tasked by director Gore Verbinski to come up with more complex and authentic-looking CG characters in Dead Man’s Chest, since Davy Jones and the crew of The Flying Dutchman would be interacting closely with the live actors, Industrial Light & Magic put its R&D team to work on a new incarnation of its proprietary motion capture system, dubbed Imocap. The results of Jones are so impressive, in fact, that people have already begun talking about the sea-encrusted villain with his creepy tentacle beard as the next great CG performance breakthrough.
|
|
|
Early concept art for Davy Jones. |
|
“We’ve done a lot of computer vision work here in R&D for the last several years and we were hoping to apply that to motion capture work outside of the MoCap studio some day,” remarks Steve Sullivan, director of R&D at ILM. “Dead Man’s Chest provided an opportunity for [remote MoCap] and a clear case of [requiring] that same quality on set where we needed those actors together in a scene for those hero performances. So we worked with the production team to nail down constraints of what we could get away with and what’s off-limits.”
Imocap became a new protocol for measuring the actors and obtaining data during the actual shoot for the creation of skeletal motion in the computer. The software contained added functionality and new ways of tracking data. Special sensor-studded suits for the actors playing CG characters were created, which were more comfortable than typical MoCap outfits, as the actors were required to wear them in a variety of simple and treacherous conditions. “…On set, I wore a gray suit, which had reference points comprised of white bubbles and strips of black-and-white material, so that when they come to interpret your physical performance, they’re better placed to do so,” adds Bill Nighy, who plays Davy Jones.
According to Sullivan, “the suits needed to be ‘dignified.’ They had to be comfortable and not look ‘stupid.’ There were a few iterations of the material itself, which started out as a cotton blend but ended up being a stretchy, semi formfitting material. And we arrived at a neutral gray to help with our lighting calculations... and we used some markers and bands to help with the capture process itself. Those needed to be comfortable as well. Cameras were based on location and shooting conditions.”
“For shots where we used reference cameras, Kevin Wooley, our Imocap lead, housed some cameras in watertight enclosures and wired them to a computer for storing the images,” explains animation supervisor Hal Hickel. “This was great for the onset stuff. For beaches and jungles, we used untethered cameras with lightweight tripods. They were a little more trouble on the backend because they weren’t synchronized to each other, but both solutions worked well, and will continue to be used on the third Pirates movie [At World’s End].”
|
|
Davy Jones and the crew of The Flying Dutchman interact closely with live actors throughout the film. |
|
Thus, by integrating the MoCap process with the actual shoot — providing the animators with hero plates with the actors in them, casting their real shadows and making good eye contact with the live actors — they were able to create, for instance, a more expressive, nuanced performance out of the maniacal Davy Jones, with the help, of course, of Nighy.
“We had new ways for the computer to analyze the images,” Sullivan continues. “The software piggy backed on MARS, the matchmoving [and tracking] solver. It understood what the actors could and couldn’t do. Our process is more holistic than traditional MoCap. We try to capture the whole body at once from different kinds of information, and that allows the flexibility to use many kinds of cameras and to work with partial information sometimes.
“The product of Imocap comes out as an animated skeleton, just like regular MoCap, and the animators do with that whatever they want, with artists in the middle running the post process. Sometimes they’ll need to cheat the body to get a better composition of the image. But the advantage is that the animators are overriding things and animating for performance reasons rather than just getting the basic physics and timing down. That all comes from the actor.”
Although ILM is currently developing its own facial performance capture system, Hickel determined this wasn’t the time to introduce yet another R&D component. “We have a lot of confidence in our facial animation, so we decided to do it by hand. The creature pipeline was being moved over to Zeno and most of the faces were different enough from the actors anyway.”
|
|
During the actual shoot on beaches and jungles, ILM used untethered cameras with lightweight tripods to measure the actors and obtain data that was then used to create the characters’ skeletal motions in the computer. |
|
Hickel adds that there’s still a lot of animation artistry at work. “The CG characters weren’t 1:1 proportional copies of the actors, so there’s a lot of reinterpreting their motion and figuring out how to get a good performance out of a guy who’s head is made of coral. We had a little more freedom with some of the background actors because their faces are so different, such as Ogilvy, whose head is basically a giant sea sponge and he has one eye in some weird orifice. Davy Jones is the most complex and human-looking CG character. He’s 100% CG — even his eyes. We knew it would be difficult, but we figured we could get there pretty quickly. What was just as difficult was the whole spark of life. The thing about Bill was he wasn’t a stone-faced villain. It was a very mercurial performance — he was constantly changing his expression and delivery. Nobody expected it. Every scene we’d stare at it and study it.”
Concludes visual effects supervisor John Knoll: “For us, it’s taken character animation another step forward with Davy Jones and how nice Bill Nighy’s performance comes through.”
Bill Desowitz is editor of VFXWorld.
Note: Readers may contact any VFXWorld contributor by sending an e-mail to editor@vfxworld.com.