10 Bit log gone?

[Øystein Mamen, FNF]

I hope not to be clogging up the board with redundant posting, but the search didn´t shed any light on this:

Is the 10 bit log REdcode optional in-camera format gone? It seems so from the format options:

"All formats recorded as 12 bit linear uncompressed or compressed RAW".

For theatrical film release, where our most common workflow yields best results going Log to digital film-out via Cinevator, this means one would have to make an additional lin-to-log 3D LUT for on-set monitoring.

Colour correction would be made on DPX files, so that´s a render from Redcode RAW anyway, whether Lin or Log.

Still, it does complicate matters a bit. It would be very neat to have it optionally.

[I Bloom]

Still, it does complicate matters a bit. It would be very neat to have it optionally.

I noticed 10bit had quietly disappeared. I don't think its necessary. The new trend seems to be to keep it simple in camera and keep it as RAW as possible. You need to render from REDCODE wavelet compression to any other format in the first place. At that point conversion from 12bit to 10bit log is a straightforward computation. Better to have control over that process in post.

IBloom

[Mark L. Pederson]

I hope not to be clogging up the board with redundant posting, but the search didn´t shed any light on this:

Is the 10 bit log REdcode optional in-camera format gone? It seems so from the format options:

"All formats recorded as 12 bit linear uncompressed or compressed RAW".

For theatrical film release, where our most common workflow yields best results going Log to digital film-out via Cinevator, this means one would have to make an additional lin-to-log 3D LUT for on-set monitoring.

Colour correction would be made on DPX files, so that´s a render from Redcode RAW anyway, whether Lin or Log.

Still, it does complicate matters a bit. It would be very neat to have it optionally.

If you color grade in SCRATCH - you can work DIRECTLY from the 12 bit (redcode raw) files, grade in float space WITHOUT RENDER and then export to whatever you want - log or lin - dpx or tiff - etc. etc.

[Jeff Kilgroe]

If you color grade in SCRATCH - you can work DIRECTLY from the 12 bit (redcode raw) files, grade in float space WITHOUT RENDER and then export to whatever you want - log or lin - dpx or tiff - etc. etc.

Exactly. The 10bit log modes were REDCODE RGB anyway and none of the RGB modes will be enabled with the first shipments... And depending on how scaled RAW develops and if it does happen, the RGB modes will probably not be enabled on-camera.

[Dominic Jones]

Were they? I had the impression that 10-bit log RAW was on the menu, but maybe I just made a wrong assumption.

I would have liked the option of 10-bit log Redcode RAW if it was on the table, but no real biggie if it's not (or never was!)...

[Chris Kenny]

My guess is there wasn't enough actual difference between 10-bit log and 12-bit linear to justify keeping both.

[Dominic Jones]

Sounds reasonable - no great loss if it is gone, at least in my books...

[I Bloom]

If you color grade in SCRATCH - you can work DIRECTLY from the 12 bit (redcode raw) files, grade in float space WITHOUT RENDER and then export to whatever you want - log or lin - dpx or tiff - etc. etc.

Mark what exactly do you mean when you say, 'without render'? Are you talking only about the 10bit aspect? Are you talking about working in realtime, directly from REDCODE?

[Mark L. Pederson]

Are you talking about working in realtime, directly from REDCODE?

Yes. That's what I am talking about.

[Rocco Schult]

For theatrical film release, where our most common workflow yields best results going Log to digital film-out via Cinevator, this means one would have to make an additional lin-to-log 3D LUT for on-set monitoring.

Err. Based on my experience, you need a log-lin-conversion to do a monitoring... why should you want to watch a log-picture ? To apply another 3D-LUT for film-simulation in a Cinetal then ?
I don't get the problem.
Anything I am missing ?

[Jeff Brue]

I don't think many people actually understand what redcode RAW acually means... guys its a bayer pattern sensor it capturing only 1 12 bit linear value per pixel. Whats interesting about scratch is it "debayers" ie interpolates the color using the surounding values based on the pixel matrix. In fact I'd almost have to say that you MUST grade your imagery in such a way as chopping off the values from a float operation (debayering) even to 16 bit can really reduce the images latitude in grading.

[Blair S. Paulsen]

I don't think many people actually understand what redcode RAW acually means... guys its a bayer pattern sensor it capturing only 1 12 bit linear value per pixel. Whats interesting about scratch is it "debayers" ie interpolates the color using the surounding values based on the pixel matrix. In fact I'd almost have to say that you MUST grade your imagery in such a way as chopping off the values from a float operation (debayering) even to 16 bit can really reduce the images latitude in grading.

It will be interesting to see just how sophisticated the color correction tools are right in RedCine. It begs the question, which matters more - the highly developed toolset of an application like Scratch, Baselight, etc. OR the presumably cruder control options in RedCine but with native, optimized "Graeme magic" algorithms?

[Graeme Nattress]

Sensor data is 12bit. For the Bayer demosaicig process we ensure that such precision is retained throughout the computations.

Graeme

[Øystein Mamen, FNF]

12 bit lin OK. Thank you for clarifying.

[Gavin Greenwalt]

For theatrical film release, where our most common workflow yields best results going Log to digital film-out via Cinevator, this means one would have to make an additional lin-to-log 3D LUT for on-set monitoring.


That's the third time now that I've heard this claim (images 'graded in log' produce better results.) And I still don't believe a word of it. I don't know of an application on the planet that actually does the color operations in log space. So I'll ask the same question I've asked the last two times: Am I missing something or is it just colorist superstitions?

[Graeme Nattress]

Gavin, I'd guess "colorist superstitions". The first task of any DI hardware or software is to linearize the log data....

For any RAW data needing a colour space transform, that MUST take place in fully linear data or else it will not work right.

[GlennChan]

Lustre has lin (video gamma linear) and log infrastructure modes. In log mode, the color corrections behave differently and you can emulate printer lights. Other systems may be like that (I believe Matrix is) in that you get different results if you work in log or lin.

You can apply LUTs to convert footage from log<-->lin
so you can work in either mode.

From the manual...

You colour grade all the shots in your project in either logarithmic or linear colour space. The colour grading toolsets are dependant on the colour space you select.

[Graeme Nattress]

Um, that's that touch mad.....

I can understand people wanting different modes to work in, but once it's data of a known linearity, be that linear, video or log, the mathematical conversion of one to the other is quite simple.

It's also just so wrong to refer to video gamma as "lin" and I wish people would stop doing that. It only confuses matters.

Graeme

[GlennChan]

It's two different toolsets and the output is different between them. Some algorithms/tools are not available in the other (e.g. printer lights).

2- Regarding the butchering of the word linear... "linear" is ambiguous but usually you can guess that they mean video gamma. "Video gamma" is a little better in that it's less ambiguous. "Rec. 601 gamma" and "Rec. 709 gamma" is better still, since the "gamma"/transfer function is different between them.

And you could use the phrase linear light to mean exactly that, when talking about signals/values directly proportional to units of light.

[Graeme Nattress]

I've taken to using linear light to emphasise linear. I shouldn't have to though - those who say video is linear are very wrong and to continue to allow them to use the word "linear" when we "know what they mean" just perpetuates confusion.

Indeed, I think "video gamma" is a generic term for a power law like function that encloses power law functions, and 601 and 709. However, as you point out, it's best to be specific and say "709" if that's what you're using as then there should be no ambiguity.

I can't see how an algorithm is not available in video gamma that is available in log. All that is needed are two luts or pow functions to convert the algorithm to work on the other data linearity type. Unless you can't be bothered to implement that I guess....

[GlennChan]

Unless you can't be bothered to implement that I guess....
Presumably the Lustre team has limited resources. For example, there is a GPU-accelerated mode except not all the features work GPU accelerated. So some colorists just forgo the speed and stick to CPU mode. (Granted, that's a more difficult task to achieve than getting log and lin toolsets aligned).

EDIT: By "lin" I'm talking about what Lustre refers to as lin. They mean video gamma (not sure if it's 601 or 709).

[Graeme Nattress]

That would be log and video gamma toolsets aligned, I hope! I had a quick glance through the manual and throughout it has confusing terminology with regards to both the word gamma (it's video and it's film usage) and linear.

[Dan Blanchett]

The whole lin, log, gamma discussion gets confusing. Glad I'm not a colorist, but I may be forced to use a poor man's version soon enough. I just read a two-year-old article (still relevant?) from the ASC and though it's very interesting from a historical standpoint, it doesn't shed as much light on the subject as I'd hoped.

http://www.theasc.com/magazine/april05/conundrum2/page4.html

[GlennChan]

Sorry Graeme, I should have specified that "Lustre lin" = video gamma. I think I've been corrupted by them.


It's also just so wrong to refer to video gamma as "lin" and I wish people would stop doing that. It only confuses matters.
We both know there's a special place in video engineering hell for these people. ;)

[M Most]

Gavin, I'd guess "colorist superstitions". The first task of any DI hardware or software is to linearize the log data....

For any RAW data needing a colour space transform, that MUST take place in fully linear data or else it will not work right.

You're thinking of this from your perspective as someone involved with electronic capture and electronic distribution. This has nothing to do with that, and it has nothing to do with "colorist superstition." It has everything to do with film being the primary deliverable in most DI environments today. To have an accurate preview of the film print, you use a 3D lookup table to convert a log image (i.e., Cineon print density) into a rendition of what the film print made from that print density image will be by targeting it to a profile of the negative and print stock being used, and monitoring on a profiled projector that is, hopefully, set up to DCI reference projector specs. This allows you to work with film colors, and derive all electronic deliverables from that. Now, there's nothing preventing you from using cascaded LUT's for this - i.e., convert the linear image to log out of the color corrector, then convert that to a film print preview based on the previously mentioned profiles, but it is simpler - not to mention more economical in terms of image storage space and processing - to use 10 bit log as the input. The "feel" of the color correction controls may be a bit different - i.e., the ranges change from the "traditional" video lift, gamma, and gain - but it is rather simple to get used to.

Of course, at some point in the future, film might no longer be the primary deliverable, in which case the more common way of working in DI might change, with the film being derived from the electronic version, rather than the other way around. But that's not the way it is now.

[Graeme Nattress]

But a 3d LUT can transform from practically anything to practically anything. I'd suggest that any sensible colour correction workflow would be both input and output agnostic (as long as you can define the colourimetry and linearity of the input and output, either in terms of a colour transform and luts or a 3d lut). Then the internals of the app can work in whatever artsy terms that the end user wishes to work in, while maintaining the ability to visualize in whatever output space they happen to be targetting for.

The beauty of all the stuff we're doing with RED is it's "just numbers". You want the data in a certain format and there's a precise mathematical transform that takes you there. I just need a definition of what that target is.

Graeme

[M Most]

But a 3d LUT can transform from practically anything to practically anything. I'd suggest that any sensible colour correction workflow would be both input and output agnostic (as long as you can define the colourimetry and linearity of the input and output, either in terms of a colour transform and luts or a 3d lut). Then the internals of the app can work in whatever artsy terms that the end user wishes to work in, while maintaining the ability to visualize in whatever output space they happen to be targetting for.


I read this statement about 20 times before responding...

What we do now in DI is basically what you just described. We start with a 10 bit log image and color correct it while viewing through a print preview LUT. For film output, the LUT is removed and the log files - with the color transforms baked in - are sent to the film recorder. For video deliverables, a video LUT - targeted to the same film print target - is swapped in and the files are rendered through that LUT directly to Rec 709 color space. For digital cinema, the original film LUT used for the DI preview is baked in (usually a higher resolution cube, but basically the same LUT) and 16 bit Tiff files are generated. The needs of color correction are not just that one output space is targeted, it is that multiple outputs - all in different color spaces - are required, but they all must ultimately look the same on the different devices they are targeted for. You must decide on a particular target for color correction decisions because multiple targets would require different compromises - for instance, there are many color combinations that video can produce, but film can't - highly saturated reds and greens, for instance. The colors of film can be rather tightly mimiced on video displays, but the reverse is not usually the case. So for the moment, film - the current dominant distribution medium for the first release of almost all motion pictures, not to mention the most likely archival medium - is considered the primary target for color correction, and all other deliverables are created to mimic that look on the various other mediums. The only difference between what you seem to be saying and what I'm saying is the originating file format, but going to 10 bit log is very economical and convenient because the primary delivery must ultimately be in that format anyway. It's yet to be determined whether it is ultimately the best way to post Red originated material, and whether there is anything to be gained by staying in linear space with the additional overhead of the larger files it requires, and with the additional need for more transforms along the color correction path. My guess is that in practical use, the 10 bit log format will still be the dominant format in DI work for the time being, even from Red originated material, with Redcine doing the original format conversion from the Redcode raw original to 10 bit log space. But I'm open to other possibilities (are you listening, Lucas?).

[Graeme Nattress]

Mike - I think we're saying the same things in different ways.

Targetting the final output with the largest use / smallest gamut makes some kind of sense. It sort of limits you when we get wider gamut displays (projectors) in the future though. I think some kind of wide gamut mastering coupled with very good gamut reduction algorithms / soft proofing is the way to go, but that's starting to get complex. Once film out of the picture for normal mainstream delivery, we can openly and easily move to a wide gamut mastering though.

I have no problems with 10bit log, but I do have issues with the precise definitions of that log curve and how the values are mapped. However, that can be sorted quite easily by working with the producers of DI software and giving them the precise RED Log spec so that they can get the largest dynamic range out of the RED RAW data in the best precision you can get out of 10bit log data. I have no problems there at all.

And you've latched onto the best way though - to just take the data in raw into the DI app and work with it there - should be no precision loss at all in that mode of operation.

[Antoine Fabi]

Hi Graeme,

I try to follow the tech discussions, and i try to learn, but with this camera, i think i'll have to learn new things very fast !
There are some very technically advanced people here.
Very high level of discussion. very very high.

i like to understand thing, at least the principles.
I am pretty sure i am not alone.

I am a little scared... to be honest.
Do i need to understand all this tech info to operate the camera ?

What would be your suggestion ?
What kind of readings ? support ?

thanks !

Antoine

[GlennChan]

Antoine, try
http://prolost.blogspot.com/2005/05/log-is-new-lin.html

It has pictures that should be helpful.

[Antoine Fabi]

Bookmarked !
Reading tomorrow morning.

thanks Glenn !

[GlennChan]

Also try "Digital Video and HDTV" by Charles Poynton.

http://poynton.com/DVAI/index.html

Heh...

(It is pretty hardcore, though relatively easy to read and understand for advanced material.)

[Graeme Nattress]

Antoinne, this is not about camera operation, but the "theory" of colour and image quality and precision through post production. There's a lot of things to discuss here, especially terminology, which is often hard when people persist in using terms that are meaningless or ambiguous.

Graeme

[Michael Morlan]

Graeme,

What two or three books on color theory and its application in digital post would you recommend this D.P. read? I have a general knowledge but would like to dig a little deeper so I can work more intelligently with my post personnel.

Michael

[Corrado Silveri]

I'm really interested too.
If only Graeme wilI give us the opportunity to read something that will turn on the light on this matter.
I'm thinking that I need to consolidate my knowledge.

Just to point out that this is a real revolution, not only a brand new camera body...

[Graeme Nattress]

The books I've gained from are:

Poynton's HDTV book - full of superb theory on how video works

And for colour science, there are a few I've been recommended and have used and found useful:

Wyszecki & Stiles, Color Scinece, Concepts and Methods, Quantitative Data and Formulae - basically, it's the colour science bible.

Billmeyer and Saltzman's Principles of Color Technology - is a more graphical guide to Color Science and talks about cameras and calibration.

Graeme

[Gavin Greenwalt]

But wouldn't even the film-light operations still be done in linear color space with an internal LUT node that is just invisible to the user?

And if it's not... it could be. Either way it would be a limitation of the feature set not the architecture of the grading application. There is no reason for example why every toolset in lustre couldn't be used on a lin or VidGamma input since it's all just mathematical operations.

[Graeme Nattress]

Exactly my point Gavin. Once you know the conversion numbers, the rest is math and that's what computers are for. Math should be transparent to the user, but I've got to know it in precise detail.

Graeme

[M Most]

But wouldn't even the film-light operations still be done in linear color space with an internal LUT node that is just invisible to the user?

Not quite. The color corrector performs linear operations, yes. But to say that it operates "in linear color space" is a misnomer. There is no color space conversion going on if you have log sources and color correct them. An offset operation is an offset operation, it's performed on the data regardless of what the data actually is. Same thing with contrast. You can easily see this if you watch a histogram when changing these settings.


There is no reason for example why every toolset in lustre couldn't be used on a lin or VidGamma input since it's all just mathematical operations.

It can. You have to put conversion LUTs in the slots provided in the processing chain. On a Lustre, there's an input LUT, an output LUT, and a viewing LUT. You can convert the source prior to the color corrector or after, and you can put a separate LUT in the viewing chain for proper preview of the intended target. But there's usually no real reason to do so if you're working from film scans, which are typically supplied in 10 bit Cineon Log format. They require only a viewing LUT for proper print preview. If you're coming from linear sources - i.e., electronic capture - you have a choice of converting to log for film preview accuracy prior to or after the color corrector. I've seen different people operate this in different ways, but part of the attraction of working in log is that the greyscale presents itself in more of a filmic way - so you don't wind up with prints that look like video transferred to film, i.e., blacks that are too dense, things that look too crushed, and in general, contrast that is too extreme. A "flatter than video" greyscale works well in large screen projection, regardless of the source, so it sometimes helps the colorist to have that be the natural tendency of the images.

[Antoine Fabi]

Antoinne, this is not about camera operation, but the "theory" of colour and image quality and precision through post production. There's a lot of things to discuss here, especially terminology, which is often hard when people persist in using terms that are meaningless or ambiguous.

Graeme

Graeme,

...yes...terminology...that's it ! ...@#@# !!!
The rest is logical.

I'll try to find the books you suggest, plus Glenn's suggestions.

Will we have access to basic litterature in the RED ONE user manual ?

thanks !

Antoine

[Graeme Nattress]

I'd not read Wyszecki & Stiles, for "light reading" though - it's just full of formulae and tables that relate to colour science. It's as heavy a tome as they come. But it's important that you know that I've read the relevant chapters.

Graeme

[GlennChan]

Antoine, some reading here:
http://tig.colorist.org/wiki3/index.php/Useful_reference_links

In particular:

Standard Color Spaces-
http://www.filmlight.ltd.uk/documents/FL-TL-TN-0101-StdColourSpaces.pdf
This explains some of the color science behind color management for film.

Visual illusions:
http://www.michaelbach.de/ot/
Fun!; informative.

Lightness illusions:
http://web.mit.edu/persci/gaz/
Academic. You need to understand "lightness constancy" first. Tries to explain certain visual illusions.

It's good to understand "Color constancy". I'm not sure what links are good.
Also metamerism. There stuff about both on Google.

If you understand why lightness and color constancy don't "work" (it's not perfect as visual illusions demonstrate), then I think that explains why we don't notice color inaccuracies. Accurate color doesn't quite exist in the first place- our perception of color seems to be a pretty good guess. Metamerism is another reason why color isn't that accurate.

Contrast sensitivity function demo:
http://www.usd.edu/psyc301/CSFIntro.htm

[Antoine Fabi]

Glenn, Graeme,

thanks a lot !

Very much appreciated !

[Corrado Silveri]

The books I've gained from are:

Poynton's HDTV book - full of superb theory on how video works

And for colour science, there are a few I've been recommended and have used and found useful:

Wyszecki & Stiles, Color Scinece, Concepts and Methods, Quantitative Data and Formulae - basically, it's the colour science bible.

Billmeyer and Saltzman's Principles of Color Technology - is a more graphical guide to Color Science and talks about cameras and calibration.

Graeme


Many thanks, Graeme, I'm going to make an Amazon search...

[Corrado Silveri]

Antoine, some reading here:
http://tig.colorist.org/wiki3/index.php/Useful_reference_links

In particular:

Standard Color Spaces-
http://www.filmlight.ltd.uk/documents/FL-TL-TN-0101-StdColourSpaces.pdf
This explains some of the color science behind color management for film.

Visual illusions:
http://www.michaelbach.de/ot/
Fun!; informative.

Lightness illusions:
http://web.mit.edu/persci/gaz/
Academic. You need to understand "lightness constancy" first. Tries to explain certain visual illusions.

It's good to understand "Color constancy". I'm not sure what links are good.
Also metamerism. There stuff about both on Google.

If you understand why lightness and color constancy don't "work" (it's not perfect as visual illusions demonstrate), then I think that explains why we don't notice color inaccuracies. Accurate color doesn't quite exist in the first place- our perception of color seems to be a pretty good guess. Metamerism is another reason why color isn't that accurate.

Contrast sensitivity function demo:
http://www.usd.edu/psyc301/CSFIntro.htm


Great links, Glenn.

Thanks!

Corrado.