The prologue battle

The prologue battle armies were comprised of not only live-action warriors wearing prosthetics, but also hordes of computer generated soldiers, created through 'Massive' -Multiple Agent Simulation System in Virtual Environment -the culmination of five years' work by software developer and crowd supervisor Stephen Regelous. Massive proved invaluable to The Fellowship of the Ring, particularly in the epic-scaled prologue battle. "Very early in the project," said Weta chief technical officer Jon Labrie, "Peter was saying how he'd like to have 50,000, 100,000 people in this battle scene. Steve Regelous had been interested in artificial life for some time; and he thought the best way to give Peter what he wanted was to build intelligent models, or agents, that have smarts-you actually build brains for them."

An agent's 'brain' was actually a network of rules constructed by a technical director. "The brains didn't have a physical location in the body," noted Regelous, "but they looked like the schematic of an actual brain -a network of 7000 to 8000 logic nodes that enabled us to create appropriate responses to environmental stimuli. There were little sliders in the brain that we could use to switch on everything from simple walking to the order to look for an enemy on the battlefield."

Simulated senses of sight and hearing determined the behavior of any given agent. "An agent could see, essentially," explained Regelous. "On every frame, he'd get a scanline rendered image of the scene around him.

It was a very simple image-distant people showed up as just squares or dots- but it was an image; and the agent was set up to respond to particular images in particular ways. Having an image was important, because it meant that we got around all the limitations of searching a huge database for every frame, which would have brought the system to its knees. The agent would have been reacting to a guy a mile away just as much as he would a guy right in front of him. With vision simulation, the whole database was easy to traverse because we just made a picture of it. We didn't have to worry if something was relevant to the agent- if he could see it, it was relevant. It was that simple."

The sense of hearing was an even more simple simulation. Agents had the ability to transmit frequencies to other agents in audible range. "It was sort of symbolic in that they were output values that represented sound, without the actual sound waves," said Regelous. "It was a physical simulation of the way real sound propagates throughout an environment. All of the different agents would 'send out' different frequencies -one sound for an orc, another for an elf-so that the agents 'receiving' these frequencies would know if the agents around them were good guys or bad guys. All the good guys were in one frequency range, and all of the bad guys were in another.

"Agents modulated the sounds they were making according to what they were doing, and some of them could respond to that. We ended up using the sound most for walking, because it was more efficient than using vision cues. When they were fighting, they needed to clearly see the other guy, because they needed to make decisions on how to respond based on how the other agent's body was in relation to them, if a weapon was coming at them, what angle the weapon was coming at them, et cetera. But those kinds of decisions didn't need to be made when they were just walking around. They all basically hummed as they walked, so they could hear the other guys around them and know not to bump into them."

Finally, a crude sense of touch was simulated through collision detection -which enabled an agent to respond to collisions with other agents-and information gathered from the terrain of the digital environment. That information told the agent whether or not he needed to adjust his posture to climb up a hill, or sidestep a rock, or jump off or over an obstacle.

The ultimate benefit of the simulated senses of the brain system was that they made for autonomous, rather than assigned, agent behavior. "By simulating the way individuals react to their environment and each other," explained Regelous, "we got realistic crowd behavior for free. That was more interesting to me than the traditional particle-animation approach of applying forces and fixed animation cycles to little points moving through space. Massive is moving towards creating virtual or artificial ecologies that can evolve and develop on their own."

Geometries for prototype models, called 'master agents,' were crafted by the modeling department, with textures derived from scans of costumed performers. Those master agents were then used to generate tens of thousands of individual agents through instancing.

A dozen master agents were created for the prologue battle, with designations based on what each agent was carrying. such as 'orc sword: or 'elf spear: "When the bodies of agents were put together," observed Regelous, "we specified the variables and their allowable ranges-everything from physical attributes such as height and arm thickness to aggression and resistance to injury. When we populated a terrain with the instances of the agents, the values for those variables were randomly assigned. These copies had their own unique properties. We had more than 20,000 instances in the prologue battle; and, depending on the complexity of the agent, we were able to get up to 50,000 agents into one gigabyte of RAM. For all three movies, we're planning on building one hundred agents we can instance into thousands."

Motion capture data was imported in Massive to produce a library of motion cycles for the agents to use. For the prologue battle, specifically, the motion capture team captured the combatants running, attacking, stabbing, falling, dying -every conceivable action that might occur in a battle. Once that data was imported into Massive, the agents would determine, frame by frame, which actions to perform.

Typically, more than 200 actions were motion captured per master agent; and with thirty different master agents, that translated to literally thousands of actions imported into Massive, In addition to larger actions, such as running, swinging an ax and falling, each agent had about 350 smaller, individual motions from which to choose. "These smaller actions enabled the agent to react appropriately in many different contexts," Regelous explained. "For example, he may have a certain sword strike action, but we would capture that six different ways so he would have six slightly different sword strikes available to choose from, depending on whether he was stepping forward at the time he was swinging, or standing still, or raising his shield. So there was this huge motion library, and each of the agents was choosing actions from that entire motion library all of the time. In fact, 24 times per second they were choosing an appropriate action."

Separate actions were linked into a semblance of fluid motion through a motion blending engine. That engine blended the action of swinging a sword up, for example, with the following striking action. In addition to blending actions sequentially, the engine layered actions atop one another. If an agent needed to walk uphill, the engine would blend the basic walking motion with an uphill motion, where the character leaned forward and moved his legs a bit differently. Similarly, the walking motion could be blended with a sideways move as a means of having an agent walk around an obstacle in its path.

Once the agents had been placed in a scene and the Massive simulation run, the Weta Digital team finessed the actions provided by the computer. "We would see if the agents went in the right direction," said Jon Labrie, "if they were fighting when they needed to be, dying when they needed to be. We'd check to see if the whole thing was visually compelling. Then we'd twist and push and shape the action a bit using tool sets. We couldn't always get something compelling by just putting a bunch of figures in there and saying, 'fight.' Choreography tools that allowed us to create more subtle placement and action were very important."

In some cases, a technical director would choose to override the autonomous behaviors of the agents, if man dated by a particular shot. "You might have one group of guys walking along," explained Regelous, "and they might decide to pick a fight with another group of guys coming the other way. If we didn't want that, we could set up the brain in a way that didn't allow it to happen. We had a fight switch in the brain, and that was one of the variables we could control when we were placing these thousands of guys. We could also switch on the 'fight' halfway through a scene, if we wanted them to get to a certain point before they started fighting."

The TD's level of control extended to being able to designate the precise point on the terrain at which the agents would engage in combat. "Say we wanted the agents running through a certain gully, then going over the brow of the hill, looking for an enemy and engaging him," Regelous said. "We could achieve that by putting a flow field into the gully, which is just encoded direction information. The agents respond to those flow fields, as part of the information they receive from the terrain. We could also paint a spot on the ground to indicate to them that this is where they were going to start fighting. Just by painting and putting flow fields onto the terrain, we could give the agents some indication as to the appropriate behavior. We could control all of that."

Still, the element of surprise inherent in each new run of the simulation was one of the joys of Massive. "We literally didn't know what these guys were going to do," Peter Jackson observed. "I still get a chuckle when I think about one of the first tests, where we had about 2000 foreground guys desperately trying to kill each other; but in the background, about fifty of them had thought the better of it and had turned around to flee the battle! I thought, 'Those are the smart guys.' It was extraordinarily spooky."

After animation was approved, agents were rendered as elements, with the performance motion files sent to 'Grunt' -Guaranteed Rendering of Unlimited Numbers of Things-a renderer designed by technical director Jon Allitt to render many thousands of agents on a single processor. "Grunt is basically an A-buffer renderer," explained Allitt, "which means that the information it is rendering can arrive at any time in the process. It doesn't need to start at the top and work its way down. RenderMan needs to know a lot about a scene before it starts to render it, and then it works from the top left corner down to the bottom right. Grunt can throw the guys in there in any order it finds them, which means that we don't have to store information about the agents any longer than we need it. We only maintain the picture as it is being pulled up.

"Basically, the renderer has a list of all the agents that we want to render for a scene, and the motion data of the frame that it wants to render, and it says: 'Okay, I want this guy now.' So it figures out who he is, what he's made out of, builds him up on the fly as it is rendering, and says, 'Okay, he goes here.' It renders him and then moves onto the next one, who might be standing on the other side. We build up the image from the guys that are closest to the camera to the ones that are furthest away: so any pixels covered by closeup guys don't need to be rendered for the further-away guys."

The individual appearance of each agent was also determined at render stage. Massive would generate a random number for each of several physical characteristics -such as height, weight and costuming -and assign those numbers to each individual agent. "Theoretically," said Allitt, "every agent could look different. It wasn't practical to do that, but we did try to put as much variation into the geometry as we could, so everyone wouldn't be a clone of the guy next to him. The Massive system generated the random numbers and scaled the character inside the simulator, which modified its motion. The same information then went through the renderer, so it could build a guy that was exactly the same size."

A set of sliders served to morph various characteristics from one appearance to another. There were sliders to create different armor and helmets, sliders to control height and girth. Colors, in contrast, were controlled by shaders incorporated in Grunt. A procedural quirk of some of the shaders enabled the character's costume to appear dirtier as a scene progressed.

To contain render times, the prologue battle scenes featuring 80,000 agents were split into groups of 15,000 to 20,000, and each group was rendered separately. Renders consumed between two and four hours for each group, with the entire prologue sequence rendered in just over a week.

Finally, the Massive elements were composited into live-action, miniature or virtual backgrounds. "We found there was a certain scale of screen size for the Massive agents that worked quite well in compositing," stated John Nugent. "We might have a row of live-action warriors four layers deep, shot on bluescreen, and behind them, on into infinity, would be the agents, who looked no different from the costumed performers. We put them together with all the normal layers of atmosphere, such as smoke from the battle. We also rendered shadow passes that we could integrate with the ground and the live-action performers."

Nothing Real's Shake formed the backbone of compositing for Weta, in part because of its graphical user interface that enabled compositors to accomplish in minutes what would have taken hours. "We were able to access all the processors in the building and kick our renders off," said Nugent, "and within minutes we would have full, 2K-resolution comps completely rendered and available to load up and look at. If a supervisor asked for some changes, the compositor could kick off a render and have a new version available for review up to three times a day. With that, of course, comes a double edged sword of infinite changeability. But everybody realized they needed to lock things down and move on because there were so many components to the composites -layers of miniatures, CG elements, matte paintings and environmental components."