PDA

View Full Version : Processing AVATAR; Weta Digital datacenter



Iannis Holwech
01-24-2010, 05:05 AM
For those that are interested in the rendering power behind the Avatar production;

Link;Processing AVATAR (http://www.information-management.com/newsletters/avatar_data_processing-10016774-1.html)



Information Management Newsletters, December 21, 2009

A thoroughbred of a data center at Weta Digital sits behind intense visual effects in the blockbuster movie and the creative artists who put it together.


Weta Digital is really a visual effects job shop that manages thousands of work orders of intense amounts of data. That preselects most of the fast, constant capacity equipment required. The data center used to process the effects for AVATAR is Weta’s 10,000 square foot facility, rebuilt and stocked with HP BL2x220c blades in the summer of 2008.

The computing core - 34 racks, each with four chassis of 32 machines each - adds up to some 40,000 processors and 104 terabytes of RAM. The blades read and write against 3 petabytes of fast fiber channel disk network area storage from BluArc and NetApp.

All the gear sits tightly packed and connected by multiple 10-gigabit network links. “We need to stack the gear closely to get the bandwidth we need for our visual effects, and, because the data flows are so great, the storage has to be local,” says Paul Gunn, Weta’s data center systems administrator.

Bruce Allen
01-24-2010, 05:31 AM
Interesting! So about 4352 BL2x220c blades, 2 CPUs each - if they bought them in 2008 they'd be Xeon 5400 series processors. EG - equivalent to 4352 Mac Pros (8-core, circa 2008).

I think if everyone on RedUser teamed up, we could out-render them :)

Not mentioned in the article: The NVIDIA Tesla S1070 Server they used as well. They did some GPU preprocessing on it before sending over to Renderman.
http://www.nvidia.com/object/wetadigital_avatar.html

Bruce Allen
www.boacinema.com

Tom Lowe
01-24-2010, 10:34 AM
"3 petabytes"

Hehe, I remember reading an article in WIRED about a decade ago (11 years?) when Star Wars: Phantom Menace was coming out, and it said something about "3 terabytes" of networked storage. It was one of the first times I had heard the word terabyte!

Gavin Greenwalt
01-24-2010, 11:44 AM
If Caustic can deliver on the upper end of their claims then even our Brazil render farm will be almost that fast. And I've heard that they used Brazil at ILM for a number of the wide forest shots that ILM handled. But I don't know if that was just for projected matte paintings or the midground forest.

http://caustic.com/

Exciting times we live in all around. Caustic, RenderAnts (Proof of concept GPU accellerated Renderman), VRayRT, iRay. Everyone has hardware accelerated rendering in their pipeline.

Won't be too long before we'll be able to get photo realistic real-time virtual cameras for performance capture.

Matt W.
01-24-2010, 12:15 PM
The intel larabee was set to be a very interesting bridge between CPU and GPU as it was essentially a 48-core x68 processor that could emulate DirectX or OpenGL but also scale linearly if you added more cores. The problem, of course, is that GPUs exist specifically because they process special purpose code much faster than x86 processors, so it took a REALLY powerful larabee to match a mid-end nVidia chip.

On the other hand, nVidia is pushing CUDA, which allows more applications than games access to hardware acceleration. So there's definitely a convergence between CPU and GPU but where exactly they'll meet is anyone's guess. I really like the larabee model because it can be scaled linearly and because you can write not just a custom engine for it but custom shaders, etc., too. So it wins by far in terms of flexibility and scalability, but specialized hardware is much faster and more elegant that I wonder if intel will just abandon the project entirely pretty soon.

And yet After Effects still only uses one core on a Mac Pro (at least mental ray scales well)....