Chroma Key and Compositing

[Rocket]

Hi Guys,

I finally got to see Pirates of the Carribean 3 last night on the big screen, and I could notice almost every scene where they had obviously shot blue screen and chroma keyed the actors over the background scene.

Why in this day and technological age is this still so hard to get right. I couldn't put my finger on it. There was no halo to speak of, but for some reason it just didn't blend, lighting maybe?

Will digital aquisition at 4K make this any easier to perfect?

Rich

[Richard Andrewski]

Lighting in my opinion. Its so hard to duplicate the look of really being outdoors with the look of being in a studio with artificial lighting. In the fifties and sixties, even with elaborate sets that duplicated an outdoor scene, somehow it never looked real and you could just tell. While I'm sure 4K will make keying easier and cleaner, I doubt it will help any with the lighting issues and the "feel" of really being outdoors in an all digital CGI matte painting background, even with accurate tracking of camera position in relation to the matte painting.

[Ace]

There really is no excuse.. With all the tools available today, if it didn't look 100% real then a bad job was done. No excuses when you have budgets of that size.

[Richard Andrewski]

Well look at any of the bigger budget movies where much of the action is on a green screen stage. Sky Captain, 300, Star Wars I, II and III. Did any of them really look like the outdoor scenes were outdoors? I don't think so personally. It's not that the keys weren't good either because they're very good. It's just that it doesn't really look like someone is outdoors yet.

[GlennChan]

Check out some CG/VFX-generated lighting for chroma key sets:
http://gl.ict.usc.edu/research/LS5/

Paul Debevec does academic research and if you read all of it, you realize what he's doing goes way beyond the original problem (making a single composite look real). But what that webpage is showing is that you can get stuff looking very real if you surround that person with the correct lighting. In his tests, the subject is lit from almost every angle.

2- In non-chromakey situations, we know that it's possible to get away with certain cheats. e.g. a softlight placed close (angularly) to a practical light will look like light coming from the practical. So the full-blown approach that Debevec takes may not be necessary.

I played around with lighting a chroma key realistically, but it didn't quite work because the light sources I used were not big/powerful enough. I think for things like sky illumination, you have to bounce it off large surfaces (which obviously reduces output significantly).

3- IMO, the robots in Transformers looks great throughout, indoors and outdoors. Compositing them in is the same idea as chromakey (and you don't have to deal with spill; but if you wanted to shoot real subjects without spill, you can do that with a difference key and/or rotoscoping). Except a lot harder because you have to make the CG robots look right (including reflections of the scene).

Movies like LOTR and Harry Potter also show good chromakey work IMO; you don't notice it. IMO, movies like 300 don't look right because the backgrounds are stylistic and not realistic/real. For real backgrounds, the helicopter stuff in Charlie's Angels 1 didn't quite look right IMO.

[Jim Arthurs]

Today's "perfect" key work will look less than perfect in a few years as visual taste and the general audience's perception increases.

I remember seeing DragonSlayer and Poltergiest in my youth, thinking how great some of the bluescreen was and that it would never get better than that. It did. Film bluescreen probably was at its best with Willow or Hook, which both pushed the techniques to the limit.

Using the same source images, the most mediocre digital compositor in the world running a 500 dollar PC with After Effects will do cleaner matte work than the best trained optical techs in the world running a quarter million dollar optical printer fifteen years ago.

I'm humbled having worked in both worlds, optical and digital, and the power/speed we have now.

The Pirate movie has been well documented as a rush job and a nightmare to work on...

http://www.variety.com/article/VR1117965871.html?categoryid=2520&cs=1

[Jason Murphy]

The Pirate movie has been well documented as a rush job and a nightmare to work on...

I think this explains a lot of the less than perfect compositing and FX work on the movie. If you have the time, even if the lighting is off, you can certainly make things look more seamless (if not 'perfect'), assuming you have the time. Almost every FX heavy summer blockbuster these days doesn't have the time; some of them are locked into a release date before shooting begins, and the FX work is finished in a massive panicked crunch two weeks before the premiere. It shows, more often than not.

[Jim Arthurs]

Lighting in my opinion. Its so hard to duplicate the look of really being outdoors with the look of being in a studio with artificial lighting.

Oh yes, lighting is key (pun intended). Lighting a subject in front of a blue or greenscreen is fairly tricky for DP's without the background in it.

For starters, the blue or green spill is seen by the eye and evaluated as actual back or side lighting, when in fact, it is actually "negative lighting" and won't be represented in the composite. In order to see the lighting as it really is, it is sometimes helpful to fly in a large black behind the talent... as this is how the matting plugin views the foreground after the spill is removed. In almost all cases, extra white fill is needed to replace the areas where spill is happening, else the edges go dark on you.

Another factor, besides lighting, is the slight of hand of the shot itself... you see twenty copies of a talent side by side and know it's a comp. You see the hero standing in front of a vast alien cityscape and know it's a comp. It's the shots that you're not expecting to be an effect that always stand the best chance of slipping by the audience regardless of visual experience in the area...

An example of this trickery was with a bluescreen shot I did and posted back on DVXUser done with the HVX when testing it. By posting a clip of talent in front of a nature setting and asking for lighting advice on the nature of my talent "interview" lighting, I diverted attention away from the shot as a potential comp and put the attention on the talent lighting. It helps that the comp was good, but a bit of verbal slight of hand kept the true nature of the shoot as a bluescreen from being discovered for several pages of posts...

I did get very helpful advice on talent lighting, however... :)


http://www.imageshoppe.com/HVX_bluescreen2.html

[Charles Angus]

C

Movies like LOTR and Harry Potter also show good chromakey work IMO; you don't notice it. IMO, movies like 300 don't look right because the backgrounds are stylistic and not realistic/real. For real backgrounds, the helicopter stuff in Charlie's Angels 1 didn't quite look right IMO.

Really? I watched Lord of the Rings (the first one) again the other day, and I almost laughed at how unrealistic the lighting looked. Not that I could do better, but I don't get paid the big bucks those DPs do...

[Chris Nuzzaco]

Really? I watched Lord of the Rings (the first one) again the other day, and I almost laughed at how unrealistic the lighting looked. Not that I could do better, but I don't get paid the big bucks those DPs do...

The lighting was never supposed to look real... watch the documentaries, they wanted it to look like a story book of sorts. I think they pulled it off fantastically, course, I'm a big Andrew Lesnie fan....

[Jason Murphy]

Anyone interested in seeing some really excellent lighting/comp work in a recent film should check out David Fincher's Zodiac (out today). There's a surprising amount of it, and it's pretty much entirely seamless in the film. Probably the best example would be the investigation sequence on the corner of Washington and Cherry; it's entirely greenscreened since the location isn't the same as it was in 1969 (not to mention that the current residents weren't about to put up with a film crew moving onto the block for a week).

The effectiveness of these shots also goes to highlight what Jim Arthur rightly said a few posts above, when he mentioned that the shot itself is usually more effective when compositionally it does not draw attention to the fact that it's a composite. Very true.

[Jason Murphy]

Really? I watched Lord of the Rings (the first one) again the other day, and I almost laughed at how unrealistic the lighting looked. Not that I could do better, but I don't get paid the big bucks those DPs do...

The cinematography is unrealistic for sure (assuming that 'realism' should be held up as a criterion for a story about hobbits, elves, etc. trying to destroy an incredibly evil ring in a volcano in the south of a mythic historical England), but it is pretty effective nonetheless. There's a bit too much unmotivated lighting for my tastes (for example, the fact that in the middle of the night you have Black Riders silhouetted by an obscenely bright light coming from just over the horizon of a hill - gotta love HMIs), but then again, 'realism' would dictate that it would be more or less impossible to see a black-cloaked figure on a dark moonless night, especially at a time/place when electric lighting shouldn't exist. And it may just be me, but a totally black screen full of rustling and screaming and hobbits yelling for 10 minutes just doesn't strike me nearly as gripping as actually being able to see the little guys run... :)

[Gavin Greenwalt]

Hi Guys,

I finally got to see Pirates of the Carribean 3 last night on the big screen, and I could notice almost every scene where they had obviously shot blue screen and chroma keyed the actors over the background scene.

Why in this day and technological age is this still so hard to get right. I couldn't put my finger on it. There was no halo to speak of, but for some reason it just didn't blend, lighting maybe?

Rich

That's because there were some reallly badly made composites in Pirates 3. Some really really basic things like black levels were off. Also some shots the greenscreen footage was slightly out of focus while the plate was sharp... I was less than impressed with many shots.

That being said it was almost always the integration not the extraction that was the problem.

[Rocket]

Well, thanks for all your input. I guess I'll put it down to time in the case of Pirates. It's amazing how much this type of compositing is considered standard fare in almost any movie these days, not only in the case that the actual location can't be shot, or isn't accessible but that it could just be cheaper.

I'd imagine that if I was shooting a sequence in the middle of Time Square for instance, like at the beginning of Vanilla Sky, I'd probably consider modelling Time Square in all it's detail, and compositing the shot over the 3D rendered location. This probably wasn't a viable option when Vanilla Sky was shot.

I shot a corporate video earlier in the year (well, 4 x 30 min videos) that were entirely shot in a green screen studio. I was shooting with two DVX100's in 25p and had a really good lighting tech that came with the studio we rented. The lighting, for the interior scenes we were going to composite, was perfect, but the keying in the end was less than perfect, mostly due to the fact we were recording miniDV, and I really think the compression is a killer in overall quality of the job. I will never record a chroma shoot in DV again.

Now, I find myself doing another one, but this time things need to be right because the entire 10 minute video is filled with Matrix style effects and compositing over CG background. So, this time I'm shooting HDCAM in the hopes I'll get a good result.

Anyone here managed to do it sucessfully with DV?

[Jim Arthurs]

Now, I find myself doing another one, but this time things need to be right because the entire 10 minute video is filled with Matrix style effects and compositing over CG background. So, this time I'm shooting HDCAM in the hopes I'll get a good result.

Anyone here managed to do it sucessfully with DV?

HDCAM is far better than DV of course, as it's 3:1:1 vrs 4:1:1 or 4:2:0, but needs a certain amount of care as if you push it too far you'll see the compression artifacts. Make sure you're right on with exposures for screen and subject with the flattest screen exposure possible is the best advice.

Asking someone to define success with keying in DV is a tricky question to ask. Some will jump in and say "yes", others will say "never"... realize that it's case dependent and camera dependent and people have different tolerance levels for what they consider "good" or "great".

Most work with any format is spoiled if the detail is turned up too high. I've not done work with DV or HDV that I consider excellent, though HDV can do nicely if the end product is SD and the source footage scaled down for the composite.

In regards to keying, HVX-200 is the best bang for the buck shy of its 2/3" bigger brother that just hit the market. It performs with both blue and green equally well. I make good money with it, but it has situations where the noise in the picture make for difficulties.

All semi-affordable cameras will pale next to the RED in matting quality. I will treat the RED the way I treat the HVX, in that the HVX makes for great SD final product when scaled down, and the RED will produce the best possible HD from the downscaled 4K...

[Bruce Allen]

The lighting, for the interior scenes we were going to composite, was perfect, but the keying in the end was less than perfect, mostly due to the fact we were recording miniDV, and I really think the compression is a killer in overall quality of the job. I will never record a chroma shoot in DV again.

Agreed. Shooting DV makes no sense when you can shoot HDV for the same price and get a better SD image out of it. What were you using to pull the key?



Now, I find myself doing another one, but this time things need to be right because the entire 10 minute video is filled with Matrix style effects and compositing over CG background. So, this time I'm shooting HDCAM in the hopes I'll get a good result.

If you're finishing HD or film, you'll need a lot of light wrap, etc. Even HDCAM has problems on the edges - once you take the green out, they tend to go a bit grey... so you have to do a few tricks to make it look good.

Bruce Allen
www.boacinema.com

[Chris Nuzzaco]

.........or you could hire me and I'll shoot your green screen with my Andromeda :)


http://www.reduser.net/forum/showthread.php?t=3098

[Michael Schrengohst]

The Pirate movie has been well documented as a rush job and a nightmare to work on...

http://www.variety.com/article/VR1117965871.html?categoryid=2520&cs=1

Shit rolls down hill, as the studio tells the producer, who yells at the director
who yells at the DP who yells at the gaffer and then the best boy down
to the grunts....the grunts could give a crap and they want to go home.
So everyone rushes it out the door because they are getting tired of
hearing the $15,000,000 actor whine about the hang nail he has....
Yada, Yada......

[Richard Andrewski]

Today's "perfect" key work will look less than perfect in a few years as visual taste and the general audience's perception increases.

I remember seeing DragonSlayer and Poltergiest in my youth, thinking how great some of the bluescreen was and that it would never get better than that. It did. Film bluescreen probably was at its best with Willow or Hook, which both pushed the techniques to the limit.

Using the same source images, the most mediocre digital compositor in the world running a 500 dollar PC with After Effects will do cleaner matte work than the best trained optical techs in the world running a quarter million dollar optical printer fifteen years ago.

I'm humbled having worked in both worlds, optical and digital, and the power/speed we have now.

The Pirate movie has been well documented as a rush job and a nightmare to work on...

http://www.variety.com/article/VR1117965871.html?categoryid=2520&cs=1

How about that Dragonslayer? That was really great blue screen work by Brian Johnson (Empire and Space 1999). I remember the article on that in American Cinematographer. They had a huge blue screen that was backlit with DC operated, blue tinted fluorescent bulbs. DC of course to eliminate any chance of flicker as electronic ballasts didn't really exist in 1981. Very clean keys. Look at the shot of the wizard on the mountain top at the end. With his white hair blowing in the wind. Not that real looking mind you (because of subject lighting and background too perfect) but very clean.

I think looking real is a subjective thing anyway. I look at a lot of older movies where they used rear projection on car shots (Casablanca for example) and its easy to forgive because the movie was so great and the cheesy rear projection is actually a part of the experience. In fact, you almost feel the movie wouldn't be the same without the stylized elements like that.

The 1950's War of the Worlds was mostly shot in stages at Paramount, even many of the "outdoor" scenes at the beginning of the movie, but its easy to overlook because the movie is such a classic and really set records for number and quality of effects shots in a movie.

Maybe something looking very clean (no obviously bad edges for example) but not real is okay as it lends to the overall fantasy of the movie. Everyone will have different opinions on this though.

[Alex Boothby]

There was an interesting discussion about over/under exposing green screens on fxguidetv, episode 3:

http://www.fxguide.com/fxguidetv.html

The Australian Cinematographer Society did a shoot with FX-PHD to test green screen lighting and keying for digital cameras such as the F-23. Ross Emmery (fx DOP for Superman Returns & The Matrix) recommended to OVER-EXPOSE the green screen by 2 stops - to much surprise all round.... And it turns out that he was right. The episode then goes on to demonstrate that a cleaner key is indeed pulled from an over-exposed green screen (for digital cameras at least, they say).

I found all this very interesting - but I think it's an over simplifications. First of all, the chances of unwanted green spill increase as you increase the light on the green screen (I think they assume that the subject is always far enough away from the screen to avoid spill, which is not always possible.) Second, an over-exposed green screen may indeed yield a better key, but at the cost of chroma and luma corruption of the subject's outer edge. This can produce the tell-tail white halo or 'perpetual rim light' look in composites.

I certainly plan to test this more. In the past I have always considered the background. If the subject is to be comped into a dark environment, then it is preferable to under-expose the green screen. Likewise if the subject is to be comped into a very bright environment, then over-expose the green screen. But never more than 1 stop. Personally I don't mind a bit more work pulling a perfect matte (or even garbage matting or rotoscoping) so long as the edge looks natural.

Thoughts anyone?

[Jason Murphy]

The 1950's War of the Worlds was mostly shot in stages at Paramount, even many of the "outdoor" scenes at the beginning of the movie, but its easy to overlook because the movie is such a classic and really set records for number and quality of effects shots in a movie.

Another example of this: the effects work and rotoscoping on 2001 still looks fantastic 40 years on. Helps that Kubrick/Alcott was shooting 65mm, and much of the compositing was done against the blackness of space; but it's still great work, and hold up really well projected in 70mm.

[Jim Arthurs]

How about that Dragonslayer? That was really great blue screen work by Brian Johnson (Empire and Space 1999). I remember the article on that in American Cinematographer. They had a huge blue screen that was backlit with DC operated, blue tinted fluorescent bulbs. DC of course to eliminate any chance of flicker as electronic ballasts didn't really exist in 1981. Very clean keys. Look at the shot of the wizard on the mountain top at the end. With his white hair blowing in the wind. Not that real looking mind you (because of subject lighting and background too perfect) but very clean.

I still own a small Stewart transmission blue screen, the same material they used. Now THAT is blue, at a very narrow 450 nanometers. Put a red gelled light behind it and see NO light transmission on the front side. Had to be pure, as front light blue paint or fabric just didn't provide the color separation needed for good film optical work. It produced a ton of blue spill compared to front lighted screens, but that was what you had to deal with. I think Willow was the first major film project to use front painted cyc walls and it KILLED them in optical getting good mattes. Green screen? Very rare in film before digital compositing... only a small handful of poor shots in B-movies...

ILM did the actual opticals on DragonSlayer, with their super duper VistaVision one-of-a-kind quad head printer. Used low contrast separation stock instead of high-con B&W for the mattes to get that nice edge characteristic. Trade-off was transparency in the body of the matte... it's always something...

[Gavin Greenwalt]

I like to underexpose half a stop just because it seems to seperate from skin tones nicer but not enough to be noisy which is probably why they recommend 2 stops over.

DV = Seperate-Rotoed-keys. Unless everyone is wearing a hat.

[Jim Arthurs]

From hard life lessons with the HVX I find that the perfect green is IRE 60-65 which is a bit much from a conventional perspective and hard to get with good quality chroma paint or fabric unless the screen is backed up and lit seperate from talent. The HVX will generate noise in fine hair and motion blur at lesser levels. Using blue, 45 IRE seems the perfect level, and surprisingly, a bit more refined than green.

These are the same levels I've found with HDCAM, at least with the Sony 700a and ilk... lesser green IRE will result in noise and roiling grain in hair... with talking heads, I've found my green at IRE 70 quite nice if proper seperation from talent is achieved...

[Mathew Mackereth]

was testing some side by side footage between 35mm camera and the JVC HD-250 yesterday, and found the lack of colour space in the hdv format to be a problem, just like DV. Although the big killer IMHO with all digital formats is the "detail control", I've keyed with HDCAM, Digibeta, and DVCPROHD100 and the biggest killer is not colourspace but the sharpening that happens in the head of the camera before it hits the tape - essentially this seems to act as an "unsharp mask" type of convolving filter - which basically creates more contrasty edges - maybe good for eng or quick turnaround broadcast - but bad for keying - so with anything i shoot or supervise now i ensure the detail is off (not just turned down to zero!)

I imagine with the raw format this should be less of an issue - depends what the wavelet compression does.

with DV/hdv or any bad chroma source you are never going to pull a great key - but i've found that a better key can be pulled from this material if you convert the rgb image back into YUV - blur the UV channels slightly and then recombine with the untouched Y (luma) signal - this removes the unsightly chunks of colour and can go a long way toward stopping the noisy edges seen on images with a poor colour space.

[Rocket]

Man, I hate the fact that I have to stay up all night to be a part of the conversation. Damn time zones.

Well, there is one other option I can look at with this shoot that may still fit in the budget. A company here called Atomic VFX built thier own 2K camera system and I can get RGB 4:4:4 out of that straight to hard disk.

That might be the best bet end the end. Or hire a Genesis and record 1080p RGB 4:4:4 to HDCAM-SR, but there is no budget for that.

[Gavin Greenwalt]

I've also found something called "Neat Image" Run your footage through that and it'll remove a LOT of the DCT compression artifacts. It can be a life saver. But once you get your matte throw away the RGB and refer back to the original, otherwise you can get some de-noise artifacts in your plate.

[Rocket]

I'll look into that, thanks Gavin!

[Richard Andrewski]

Another example of this: the effects work and rotoscoping on 2001 still looks fantastic 40 years on. Helps that Kubrick/Alcott was shooting 65mm, and much of the compositing was done against the blackness of space; but it's still great work, and hold up really well projected in 70mm.

I had the book about the Making of 2001 when I was a kid. Did you know that many of those spaceship shots were produced on an animation stand with simple photo cutouts of the spaceships? The tracking shots were simply the animation stand head moving up and down toward or away from the spacecraft photo.

Also, I believe that was the first film where they successfully matted moving people into window shots on the outside of a spacecraft. Looked pretty darn good. Also I love that shot of the moon shuttle descending down and then landing on the pad. Then when you think it's over, the shuttle drops on an elevator into a huge cavernous miniature set with tons of optical work showing people moving around in windows. Great! Many of those people went on to work on Space 1999 which really was one of the first sci-fi tv shows with really great effects (for the time).

[Richard Andrewski]

ILM did the actual opticals on DragonSlayer, with their super duper VistaVision one-of-a-kind quad head printer. Used low contrast separation stock instead of high-con B&W for the mattes to get that nice edge characteristic. Trade-off was transparency in the body of the matte... it's always something...

Yup. A lot of that stuff built by Richard Edlund as I recall. I believe he built the EmpireFlex 65mm camera with motion control head too which may have been the first field oriented motion control camera to allow you to bring tracking data back to the studio and repeat shots for other elements. The printer was really advanced stuff for the time allowing super complex numbers of simultaneous mattes without so many generations. They had it for Empire but really showed it off in Return of the Jedi.

I wonder whatever happened to him after his company went out of business--I believe he did Air Force One and then quit?

And how about that sodium vapor process at Disney? Used a camera with special optics and a prism that was so special that it supposedly never could be duplicated after the original. Can you imagine how hot the stage must have been with those sodium vapor lights lighting the screen? Really worked great for some of the shots in Mary Poppins. I wonder if they still use that today sometimes? I remember they may have used it for The Black Hole which was 1980 or so but not sure after that.

[Jim Arthurs]

I imagine with the raw format this should be less of an issue - depends what the wavelet compression does.
The RED has no sharpening or detail feature... and nothing like sharpening in RED CINE when de-Bayering either... so it's a soft edge image perfect for keying. All indications are that the wavelet compression isn't a barrier to good matte work.

with DV/hdv or any bad chroma source you are never going to pull a great key - but i've found that a better key can be pulled from this material if you convert the rgb image back into YUV - blur the UV channels slightly and then recombine with the untouched Y (luma) signal - this removes the unsightly chunks of colour and can go a long way toward stopping the noisy edges seen on images with a poor colour space.
This is a great trick, and what should happen automatically at the codec stage, but usually doesn't. Codecs are either "generational" or "presentational". Generational codecs keep the color info as/is so that it will survive multiple generations. In the case of the DV and DVCRPRO codecs, this reveals itself in images where the color is blocky and literally "steps". Presentational codecs tend to blur out the color into a nice smear over the luma, which looks better to the eye and your keyer, but isn't ideal for going another generation. When keying, the analog outs of a DV deck usually key better than the data transfer via firewire for this very reason... the chroma is smoothed for you.