Pulling a Key from a Raw Bayer image

[Thom Steinhoff]

I had FAR too long on an airplane today (10.5 hours UK to LA) and had a lot of time to think. I got to thinking if it was possible to pull a key in the process of debayering in Red Cine.

Hear me out. Certainly I can only imagine the complexity of the process, Graeme, and you may have already tried this, but the Programmer in me thinks that, though it is likely far more complex, a debayer is essentially averaging all of the pixels around a pixel to come up with its color.

Since debayering does introduce a bit of noise around the hard edges (albeit it microscopic on a red) could a pixel be identified as within the green range from the sample during the averaging process, so then if you use that pixel to help you calculate others around it--you count it as alpha channel clear thereby ending up with a superior key?

Again, this may have been one of those things you tried early on, or doesn't make sense when you really know how things work, but I just had to ask if you've ever gone down that road as it preoccupied me most of the afternoon!

Thom

[David Fairbanks]

You'd have to make sure that there's fudge room because the green in your image can't be perfect color or tonality. And what about spill. Interesting idea though.

[Ace]

I was wondering weather the CMOS sensor has any special reactions to UV or other wavelengths of light, which can be used as a reflective keying surface which would otherwise be invisible to the human eye (but distinctly visable and keyable through filtering in digital).

[Evin Grant]

Most sensors are IR sensitive, but seems too close to skin.

[Andrew M.]

I was wondering weather the CMOS sensor has any special reactions to UV or other wavelengths of light, which can be used as a reflective keying surface which would otherwise be invisible to the human eye (but distinctly visable and keyable through filtering in digital).

Using heated dark red surfaces for keying instead of green?
Interesting idea. But how you make it disappear from the background? Make it black?
It may get hot in the studio:-)

[Graeme Nattress]

I think what would be best is to pull the key on a non-matrixed RAW RGB image, just after demosaic, perhaps with just a "normal" gamma curve on it. Then you'd be just using the raw green data with no cross contamination from the other channels, and it would be very clean and noise free. Later on, you'd mate that key with the full matrixed RGB image, and that might work better, not that pulling a key on the ordinarily converted image will be in any way bad.

Graeme

[Thom Steinhoff]

I think what would be best is to pull the key on a non-matrixed RAW RGB image, just after demosaic, perhaps with just a "normal" gamma curve on it. Then you'd be just using the raw green data with no cross contamination from the other channels, and it would be very clean and noise free. Later on, you'd mate that key with the full matrixed RGB image, and that might work better, not that pulling a key on the ordinarily converted image will be in any way bad.

Graeme

Is pulling a non-matrixed RAW RGB file something we're going to be able to do simply in Red-Cine? Maybe a checkbox that says "Use Matrix" and "Simple Gamma Curve" or even just "Best For Key"?

If, as I understand it, we are going to be able to take the Redcode files directly into Shake as Quicktime files, we could concievably have one node with the file fully matrixed with a proper curve setting and the same file in a key node with settings for "Best for Key" or Non Matrixed, Normal Gamma curve?

If we can truely drop a non processed Redcode file right into our systems with the ability to set features in it such as gamma curve and others, it is really going to be amazing.

[Graeme Nattress]

I think we're trying to put a "no matrix" setting in REDCINE that should help for this. I think the gamma curve would be best as the default 709 one, but you'd be able to tweak the curve on top of that if need be.

Graeme

[Thom Steinhoff]

Graeme, did I understand correctly that we will be able to do these settings on a redcode file without redcine processing right inside any quicktime enabled editor or program?

[Graeme Nattress]

That's an aspect we're working on - how to integrate the RAW development controls into a variety of environments that support Quicktime. More info when we can on this!

Graeme

[Thom Steinhoff]

That's an aspect we're working on - how to integrate the RAW development controls into a variety of environments that support Quicktime. More info when we can on this!

Graeme

That's really great--I really hope it works! I promise to stop with the stupid questions so you guys can get to work!

You really are going to change the world!

[Graeme Nattress]

Thanks. We're working on it.

Graeme

[Hrvoje Simic]

I was wondering if it would be possible to:

- light a subject with a IR or a UV light
- import the footage into editing/composting software
- isolate UV/IR light with a plugin filter with a specified cutoff frequencies or wavelengths
- and transfer that light into alpha channel

Any thoughts ?

[Nick Shaw]

That's an aspect we're working on - how to integrate the RAW development controls into a variety of environments that support Quicktime.

Might it be possible to have REDCINE run in a mode where it did not actually render a new file, but simply changed settings in the metadata so that any Quicktime application would read a REDCODE RAW file with customised LUTs etc? I'm thinking kind of the way Motion projects can be opened in other applications, and be seen as if they are ordinary Quicktime movies, but with the Motion engine running in the background to generate the frames dynamically. Could the REDCINE engine run in the background in the same way? This might also alow you to set metadata for proxy quality/size playback.

[Graeme Nattress]

We're certainly looking at that option Nick.

Graeme

[Nick Shaw]

Boy that was a quick respose Graeme!

[Gavin Greenwalt]

I was wondering if it would be possible to:

- light a subject with a IR or a UV light
- import the footage into editing/composting software
- isolate UV/IR light with a plugin filter with a specified cutoff frequencies or wavelengths
- and transfer that light into alpha channel

Any thoughts ?

Once you look at the world through IR or UV you'll find that a chroma key is far superior.

I think most of this "Keying inside of Redcine" talk is pointless. Why use a substandard keyer inside of REDCine? It's not designed to do that. As long as it can produce a nice linear output you can take it into one of a hundred applications which will do a far superior job.

Pulling a key is and shall be far from a one button operation. Not to mention you're still going to have to step into a compositing app to integrate the footage once it's been keyed.

[Nook Kim]

Once you look at the world through IR or UV you'll find that a chroma key is far superior.

I think most of this "Keying inside of Redcine" talk is pointless. Why use a substandard keyer inside of REDCine? It's not designed to do that. As long as it can produce a nice linear output you can take it into one of a hundred applications which will do a far superior job.

Pulling a key is and shall be far from a one button operation. Not to mention you're still going to have to step into a compositing app to integrate the footage once it's been keyed.

I don't agree with this even though I'm not anyone with any better idea than chroma keying method. But, if there is a better way to do this, why not try to find that way? After reading this thread, I happen to imagine getting "key-ed out" images right out of the camera, of course, nice and clean. All I have to do is to give it a nice background layer or two.
I appreciate all you guys innovative ideas. Cheers!

Nook

[Ivan D. Young]

Why not build a range finder into the camera and pick the range of pixels that you would like to knock out? Or is that too simplistic? if your data is all digital and you already know what is hapening in frame, isn't possible to assign a depth distance to those pixels and use that to your advantage? Perhaps that is RED 2.0

[Chris Kenny]

Why not build a range finder into the camera and pick the range of pixels that you would like to knock out? Or is that too simplistic? if your data is all digital and you already know what is hapening in frame, isn't possible to assign a depth distance to those pixels and use that to your advantage? Perhaps that is RED 2.0

Lining up the depth data with the pixel data would be very hard, and obtaining depth data of sufficient resolution to match up with a 4K image is probably close to impossible to do in real-time with current technology.

Five or ten years from now, machine vision algorithms will probably be good enough to do passable rotoscoping without human help. This is a rather more realistic approach.

[Thom Steinhoff]

I think most of this "Keying inside of Redcine" talk is pointless. Why use a substandard keyer inside of REDCine? It's not designed to do that. As long as it can produce a nice linear output you can take it into one of a hundred applications which will do a far superior job.


Gavin,

What I was looking for, and what Graeme suggested might be possible is a way to improve the base image that you are keying off of to continue to use high end keying but to put it a step farther back in the process.

A great deal of burdon is placed on keying software (expecially that designed for higher priced low end keyers) to make up for anomilies associated with debayering (and matrixing), interlace, DV compression, etc. It is forced to do all sorts of extra bluring and compensation for things that won't exist on RED.

I wondered if having the keying work off of non debayered images would provide a better key (and maybe stimulate Graeme to think down a line that may be good for all of us)

What Graeme suggested is you may be able to use your high end keying software and achieve a superior key by pulling the key on a pre-matrixed, standard curve, image then use this to key against a matrixed, potentially custom curved image.

If this works as he thinks, you would get the best key possible without forcing your software to "fake it" by blurring, etc. Though in Red keys will be amazing, perhaps this will make them super-amazing!

[Jeff Brue]

A default Rec 709 gamma curve graeme ? That seems almost criminal ;-)

[Graeme Nattress]

Criminal? It's a nice gamma curve really, well defined and industry standard. Of course, that's just the starting point to put the data in a gamma space. The tonal curve options allow you complete control.

Graeme

[Nick Shaw]

What Graeme is talking about is the RED equivalent of doing two passes in TK, one clean to pull the matte from, and one with the look applied, to apply that matte to. Very sensible approach. I have had my job made much harder in the past by a director who wanted a very de-saturated look, and got the colourist to apply such an extreme grade, it almost pulled all the green out of the image so I had nothing to key off. Obvious you would think, but no! I had to go back to the one-light to pull the keys!

[Graeme Nattress]

Yes, Nick, that's a very good way of describing what I propose. Pulling a key from the raw before any RGB matrix is applied should be much easier.

Graeme

[Gavin Greenwalt]

Why not build a range finder into the camera and pick the range of pixels that you would like to knock out? Or is that too simplistic?

MotionBlur. You can't rangefind the alpha of a motion blur without running at some rediculous scan rate like say... 400fps and interpolating. Also you have to sample from the exact same distortion as your lens... which means you need to somehow actually integrate the laser detector and the rgb circuitry into the same chip. There can still be distortions as this laser source will be different from the lens plane... and now your lens distortions have introduced errors in the range messing up your zchannel. Finally you won't get nice edges because you would need to super-sample and then anti-alias to get the proper alpha on super fine objects like hair.... so for a 4k image you need a 16k resolution depth accurate (to at least 3mm) scanner running at 400fps and can also detect RGB. Best of luck. Image algorithms are getting smarter every year. I'm sure within a few years we'll get a motion flow keyer which analyzes the entire shot (with garbage masks) and eleminates the moving foreground.

I guess I'm just confused about the definition of pre-matrixed. I'm assuming that just means "No LUT or color modification beyond compensating for the sensor response curve." Isn't this the default state of REDCine already? Or are you suggesting we key off of the bayer pattern?

[Graeme Nattress]

The image processing pipeline goes like this (simply speaking):

In camera, you get per-pixel offset and gain calibration -> RAW.

If you're doing RGB in camera, the RAW from above -> Colour Matrix from camera RGB space to say REC709 -> linear data to gamma corrected data

The default state of REDCINE will be whatever the tonal and colour controls were set in camera - ie Nice looking RGB. However, I think if you just take the RAW data and demosaic, and do nothing else perhaps a REC709 say gamma curve on the linear data, that will be easier to key than after the Camera RGB to REC709 transform has been done.

Graeme

[Mike Harrington]

I'm sure within a few years we'll get a motion flow keyer which analyzes the entire shot (with garbage masks) and eleminates the moving foreground.
?

I've done this already in a tracking software called Icarus....which was bought by pixelfarm and turned into PFTrack.
The caveat was the camera had to be moving( a pan or whatnot)I was able to cleanly extract a man running down a hill near the great wall of china...leaving just the background. The software also used optical flow to fill in the missing pixels where the man was...no painting involved...all automatic.Normally this would have involved heavy rotoscoping, and painting.Sadly this feature never made it into the PFtrack product.
An optical flow keyer seems a no brainer to me...why it hasn't been done already I'll never know...the algorithms are everywhere...I believe even used in mpeg2 to some degree.
Another interesting option is chromatte fabric keying http://www.reflecmedia.com/
which basically eliminates spill, but can be rather expensive if your on a budget

[Nick Shaw]

I think Icarus was not bought by Pixel Farm, but rather written bu the guys who started Pixel Farm back when they were students. At least that's how I heard the story!

Another app which can remove moving elements from a static background (even if the camera is moving) is Mokey from Imagineer Systems (http://www.imagineersystems.com/). It does it by 'planar tracking', which basically looks for a best fit corner pin of co-planar sections of the background. Nice software, but not cheap!

[Mike Harrington]

Make's sense Nick...all I know is it used to be free....which was amazing for what it could do.
all of this stuff coming out of the U.K. ,mokey as well...interesting

[Simon Blackledge]

Icarus guys were from up here at Manchester Uni I think, :-/ was a wee while ago!..

All theses apps are expensive, but at the end of the day you can charge, because they work, and get productions out of very sticky situations ;)

I hear you on the 2 pass stage guys. The thing I love about RED is no film weave!!

[Jance Allen]

One thing to keep in mind is that most keyers (Primatte, Keylight) can do a lot of their magic on the rgb/color/beauty, not just the alpha, in terms of spill suppression and edge blending. This 2 pass idea is a great idea though, and could be a huge benefit in a lot of situations, but you do have different comp work to do when your key is strictly generating a greyscale matte.

I've heard talk of the high-end color correctors/DI suites being able to output an xml file or 3D LUT or whatever that other comp apps could read in. Then you could use Primatte/Keylight/etc. to its full potential on an as unmolested source image as possible and drop the expensive color correction LUT on top of that.

flameop, I so totally agree with you about no film or gate weave - a huge advantage for vfx work in so many ways. I've spent countless hours stabilizing "locked off" shots.

[Gavin Greenwalt]

hehe no vistavision for you?

[Jance Allen]

Looks like not - none of the hollywood films I've comped on are on wikipedia's list of vistavision effects films. Sounds like it helps with grain, but is it also more stable in terms of gate weave?

For whatever reason I let myself stay pretty ignorant of film formats and film stocks. Every new show was a new experiment in grain matching, but I had a pretty scientific approach. I'm sure I will love Red images if the sensor is super clean, and I'll probably also be degrading my final comps with grain because I like the way it looks.

[Simon Blackledge]

I've tried to match grain so much from a spirit on drams shot 16mm dropped to digi. I always, always end up adding just black and white grain in flame at diff amount in rgb, it just never seems to contain any colour, really wierd.

tbh I prefer moving shots or a slight drift.. with a total lock of your track needs to be soooooo accurate.

god I can't wait for Red...

[Gavin Greenwalt]

Sounds like it helps with grain, but is it also more stable in terms of gate weave?


Yeah, it's an 8 perf feed mechanism so the drift is almost non-existant. Just not too economical.