RAW or DPX

[Nick Shaw]

Do you have examples of images that are superior because someone clipped off the highlight detail?

Who said anything about clipping off highlight detail?

Using log DPX does the exact opposite – it preserves highlight detail through the workflow, allowing the colourist to compress it into the viewable range (or clip it as a creative choice) during the final grade.

Using 16 bit log DPX is of course preferable to 10 bit, and I use that as my intermediate format of choice much of the time, as I have said elsewhere on this forum.

[Dan Hudgins]

Using log DPX does the exact opposite – it preserves highlight detail through the workflow, allowing the colourist to compress it into the viewable range (or clip it as a creative choice) during the final grade.

There is the issue of what is going on with the data "outside the viewable range" with various tools that minipulate the curves, if you cannot "see" the data above the upper clip point, its hard to adjust it, or at least be sure what effect each program has on it.

Yes you can A/B monitor look-up-tables, maybe, in some programs, but its not a system that is "fool proof" as many calibrations are needed and that could limit the freedom to pass the data around.

My view is that its better to have ALL the data visable on screen ALL the time, and to use a data format that has enough bits that you can adjust the relative contrast between the shadow, mid-tone, and highlight areas up and down several times without scaling errors being visable above the 9th bit from the top. With 16bits it takes quite a bit of up and down before you see histogram gaps above the 9th bit even if you use an S-curve.

With 10bits you can see histogram gaps above the 9th bit with just a small amount of adjustment, more so with the gain in the dark areas going from Log gamma 1.7 to monitor gamma 2.2, making tone banding in the dark areas show up even with minor contrast gains in grading.

That's why Kodak came up with their "hidden data" system in the Cineon, they did not have enough bits to show all the data (full range on a heavy S-curve) and still be able to minipulate it, so they left the highlights expanded, and use their "softclip" option to bring that expanded highlight information back into viewable range of the film recorder.

Its not a good or modern way of working with the data in software, using 16 bits or even 32bits or real numbers, lets you see the data all the time, just at different contrast levels in the highlights and shadows, so you get a more film-like multi-S-curve transfer from one generation of grading to another without histogram gaps.

Storage today is so much lower cost than when 10bpc (30bit) Cineon came in that there is little reason to use that obsolete format with todays Digital Cinema Cameras.

Why did they not use 32bits anyway, there are 2 bits extra in the 32bit storage, you can double the tones in the RED and GREEN to reduce the banding, seems off to through away almost half the tonal data that you could have in exactly the same disk space? (The eye sees fewer tones in the blue so you don't notice 10 vs. 11 bits in the blue as much).

BTW, what are the clip points for 16bpc files now, the range is 0 to 65535 not 0 to 1023, so the standard clip points must be different?

[Nick Shaw]

First of all, Dan, I must be clear that I am in full agreement with you that 16 bits are better than 10. I don't think anybody would dispute that. The choice of 10 bits dates back to a time when saving 4 bits made a huge difference in terms of storage and processing. Now it is much less of an issue.

Where I do disagree with you is over the best use of the available bits.

My view is that its better to have ALL the data visable on screen ALL the time

I do not deny that there are potential risks associated with some image data being "invisible", but the same applies to EXR files, and I think that most people would agree that were it not for the processing speed overhead of working in floating point, then EXR would be the best option. There should not be an issue with using log DPXs if all software used interprets them correctly.

Since both Rec709 and sRGB encoding can only code about 2.5 stops above 18% grey in the 0-1 range, you have two options when dealing with imagery with more than 2.5 stops of headroom (that's all Digital Cinema cameras). You either compress the highlights to fit into the 0-1 range, or you use a high dynamic range format like EXR. A log curve is really just a different way of coding HDR without compressing it.

If you need to work with a real time system then integer formats like DPX are generally faster, and not a problem if interpreted correctly. A Cineon curve can code about 6.2 stops above 18% grey (still not enough for HDRx) so is a useful integer approximation to un-clamped float when used with 16 bit precision.

BTW, what are the clip points for 16bpc files now, the range is 0 to 65535 not 0 to 1023, so the standard clip points must be different?

Normally the range is treated the same way, so the extra six bits just give extra precision within that range. So just multiply 95 and 685 by 2^6.

[Graeme Nattress]

Half Float, like used in EXR can be thought of as 15bit log + a sign bit.

Graeme

[Peter Moretti]

...
Since both Rec709 and sRGB encoding can only code about 2.5 stops above 18% grey in the 0-1 range, you have two options when dealing with imagery with more than 2.5 stops of headroom (that's all Digital Cinema cameras). You either compress the highlights to fit into the 0-1 range, or you use a high dynamic range format like EXR. A log curve is really just a different way of coding HDR without compressing it.
...

Nick, I don't understand where you're getting the above from. FWIU, Rec 709 and sRBG do not specify latitude/dynamic range either explicitly or implicitly.

Doesn't sRBG specify color space, white point, and gamma only? And Rec 709 do all these plus include color coefficients for RGB to YUV conversion? I don't see how the preceding clamps latitude above 18% grey to 2.5 stops.

[Graeme Nattress]

REC709 maps 0.18 to ~0.46, putting mid grey 18% at ~46%. So 36% maps to 63%, that's one stop above mid grey. 72% maps to 86%, that's two stops, and 100% maps to 100% and 100% is 5.55 times greater than 18%. Log2(5.55) = 2.47 stops. Hence the defined REC709 curve as standard, placing 18% mid grey from the scene at 46% in the image has only ~2.47 stops above mid grey.

However, the REC709 curve is defined as a decoding curve, rather than an encoding curve..... And nobody on a camera uses REC709 as defined - they all put a "knee" in to extend the highlight range.

Graeme

[Evangelos Achillopoulos]

I haven't read all the thread...

Answering the original question, REDfilm RAW is the best option...

A second is DPX Log directly derived from flat REDfilm conversion.

[Dan Hudgins]

However, the REC709 curve is defined as a decoding curve, rather than an encoding curve..... And nobody on a camera uses REC709 as defined - they all put a "knee" in to extend the highlight range.

Graeme

Exactly!

Unless you are using a very cheep old camcorder or surveillance camera, there are IN CAMERA or IN DE-BAYER curves BEFORE the output curve. Those curves adjust for the sensors un-balanced white balance, the K value of the light, and reduce the contrast in the deep shadow areas, as well as the highlights above midtone.

You can display 14-18 stops or more of subject dynamic range in a 8bit Gamma 2.2 image, as is done all the time with scans of good film negatives, how is that possable, the negative if not over processed has a curve at its foot and sholder that "compress" the image tones.

In the end you can get just ONE count of the ADC in the film scanner for a 1 or two stop change in the subject brightness in both the highlights and shadows of the image.

None the less, for LARGE changes in subject brightness you will end up seeing some detail, although lower contrast, in both the shadows and highlights.

To simulate the "traditional" film look (when there was enough silver in the film to give rounded foot and sholder) in a Digital Cinema Camera, S curves are used internally or internally in the de-Bayer to include MORE subject brightness range into the limited brightness range of the display or print.

It has always been the case that paper prints of photographs have a smaller contrast range then well made film negatives, and various methods were developed by good printers to "compress" the subject contrast range on the negative into the paper print, without reducting the overall slope angle of the negative to give a very low contrast result.

What 4 generation movie film printing does is to increase the mid-tone contrast to somewhat above the subject contrast, and to reduce the shadow and highlight contrast to below the subject contrast, when you view the projection vs. te original subject. That is because its a negative-positive process so both the highlights and shadows end up at both the foot and sholder of the film steps in printing.

About REC709, even the old Orthicon cameras were operated so that the output was non-linear, I read that in a lighting for TV book, it was specific that the cameras were operated out of their linear region.

Thinking about the needs of FM video tape recording is not relevant to what you see on a TV set or a movie screen. Its not important that the display reporduce the brightness range of "life", in fact it is better that you do not do that, you want to "edit" life and compress it into the comfortable range that you can watch for two hours in a dark room, just as you use audio compression to level the VU level so the audio volume is constant, it makes it easer to watch and hear.

Film scanners were calibrated to show Log (~gamma 1.7) steps in relation to density, that is so, BUT the density steps were UNEQUAL subject brightness values over much of the film's S-curve. There was NEVER a direct relationship of the DPX/CIN data counts and the f/ stops on the subject in front of the camera, the number of counts would vary depending on the lab used, the day of the week the film was processed, if it was push or pull process, and if it was over or under exposed, if the K value was of etc. etc.

People keep saying things like Super16mm is not dead, people still shoot it because they like the way it "looks", or rather transferes the subject brightness into 8-9bit values for the final viewing device at gamma 2.2 etc.

If you want the "Video Look" I don't know why you would be using a Digital Cinema Camera since the goal of Digital Cinema Design is to not get that look, its to get something like "film" only without the cost, dust, scratches, and grain. That's what RAW is about, its closer to a film negative, except that it IS linear, so you need to run the LINEAR sensor data through LUT to make it NON-LINEAR so its like "FILM" and ISO curve before it gets to the output format. The ISO curve of film also changes depending on over or under exposure or just exposure because you are having some parts of the subject closer to the sholder with higher exposure and those same parts closer to the foot of the films S-curve with less exposure, that changes the relative "gamma" contrast in different parts of the frame, and is one reason people like film it gives them in camera control over the rendering from soft to hard contrast depending on how the expose, the grade the shot to reverse compensate for the change in density, but timming the lights up or down does not fully compensate for the changes in CONTRAST that exposure changes make. Since sensors are linear, the only way to get that same control over the image is to de-linearize the sensor data.

==

I looked at both the LSB and MSB versions of 16bpc DPX from REDCINE-X in XNVIEW and there seems to be a RED-BLUE reversal, has anyone found other programs that show RED-BLUE reversal (i.e. blue people)?

[Peter Moretti]

REC709 maps 0.18 to ~0.46, putting mid grey 18% at ~46%. So 36% maps to 63%, that's one stop above mid grey. 72% maps to 86%, that's two stops, and 100% maps to 100% and 100% is 5.55 times greater than 18%. Log2(5.55) = 2.47 stops. Hence the defined REC709 curve as standard, placing 18% mid grey from the scene at 46% in the image has only ~2.47 stops above mid grey.

However, the REC709 curve is defined as a decoding curve, rather than an encoding curve..... And nobody on a camera uses REC709 as defined - they all put a "knee" in to extend the highlight range.

Graeme

Graeme,

Thanks very much for clarifying this. May I ask a follow up? A lot of HD cameras record in the super areas (0 to 15 and 236 to 255, using an eight bit scale) to allow for more dynamic range. Does/can the Red One/Epic/Scarlet record into these these values or not?

Usually when working on files in RedCine-X it looks like these areas a vacant, but I didn't shoot the material and can't tell for sure.

Thanks much!

[Dan Hudgins]

Normally the range is treated the same way, so the extra six bits just give extra precision within that range. So just multiply 95 and 685 by 2^6.

Using the factor of 2^6 or 64 does not give the full range,

you get 65472 rather than 65535,

95 * 64 = 6080

685 * 64 = 43840

if you take the ratio of full scale to find the clip points you get,

(65535/1023) * 95 = 6085.85 or 6086 (or 6 higher)

(65535/1023) * 685 = 43882.18 or 43882 (or 42 higher)

So I was wondering when you scale lower bit image to higher bit levels, or these clip points, is it right to adjust so that full scale becomes full scale, or do you always scale so that full scale in the higher bit format is never obtainable?