RED test in Van Nuys: RedOne #30

[Martin Drew]

I thought a lot of people here might be interested in this:

Quote from Adam Wilts website:

"On 15 September 2007, James Mathers of the Digital Cinema Society invited a crew to Upstage Studios in Van Nuys to participate in a test shoot with RED ONE serial # 30."

http://www.adamwilt.com/ProductionStills/RED%20test%201/

they shot a bunch of tests including:

Stouffer T4110 density wedge test
DSC Labs ChromaDuMonde

test images are available from the CML ftp

Host: ftp.cinematography.net (or 89.31.209.180)
User ID: upload-8z67c2
Pwd: B!ll-Brandt2

Watch out for the exclaimation mark in B!ll-Brandt2 and please be economical with what you download so you don't hammer CMLs bandwidth.

M

[erbp]

Thanks for posting this. The wedge test is very helpful.

I'm counting about 7.5-8 usable stops which matches the footage we've seen so far.

[Michael Schrengohst]

I wonder if they will show us the people they shot?
I would be curious to see what a Cooke 25-250 looks
like shot with the RED.

[Graeme Nattress]

Measuring Adam's chart (without calibration data as none provided)

Gray Value (Mean)
Measurement 1 6959.27
Measurement 2 7008.45
Measurement 3 7155.88
Measurement 4 7417.98
Measurement 5 8007.96
Measurement 6 9071.96
Measurement 7 10770.32
Measurement 8 13392.08
Measurement 9 17232.27
Measurement 10 21641.52
Measurement 11 26732.83
Measurement 12 31536.6
Measurement 13 32767

This really should be done on linear light data, but it does show that there's information in each and every box on the chart up to clipping. It's certainly not the easiest to see in the image, but it's there in the data. BTW, this was just done rough - I need calibration data from the chart and preferably the numbers to see the exact bounds of each wedge for accurate measurement. Adam's chart is different than the one we've been using on our linearity experiments.

Graeme

[donatello b]

"Adam's chart is different than the one we've been using on our linearity experiments."

what chart does Red use for their test ?

[Graeme Nattress]

We use a Stouffer too, but it's a different shape, and has numbers printed into it to mark each wedge, and comes with a calibration sheet.

Graeme

[Álex Montoya]

that there's information in each and every box on the chart up to clipping

Yeah, I see what you mean. If you play with levels you can see that it gets gradually brightened until the end. Is there any chance that you released your tests?

[Martin Drew]

This really should be done on linear light data

I presume this is something Adam could do "after the fact" going back to the R3d file and re-exporting through RedAlert.

M

[erbp]

Yeah, I see what you mean. If you play with levels you can see that it gets gradually brightened until the end. Is there any chance that you released your tests?

Okay, can you guys help me out with this then?

I tweaked the levels and there is information there (noise really), but I don't see any delineation between blocks in the entire left third of the image. In fact, it looks like the same noise that is on the right quarter of the image above clipping and isn't this the "safety override" black?

I don't know how this comes across in a forum post so I want to clearly state that I'm just trying to learn, and I don't think for a second that I'm going to disprove anything Graeme says.

[Graeme Nattress]

Digital cameras dynamic ranges are defined by two things - clipping and noise. All cameras clip, and below that point, you get progressively more noisy. From a digital sensor there should be a linear relationship between light in and signal out.

A backlit test chart is probably the best way to measure this. Stouffer make good ones.

What I generally do is export linear light, 16bit tiffs, and measure the average brightness for each wedge, and plot on a log / log graph of calibrated Stouffer optical density v measured value from the sensor. I also subtract a black offset to make the graph go through zero.

What I see is that the curve extends over the 12bit range of the A-to-D, from highlight clipping to black, the signal getting more noisy the darker it gets. This would be the expected result.

Dynamic range, in terms of what you see is therefore dependent totally on this measurement, and how noisy an image you can withstand. The more noise you're tolerant to, the more dynamic range you have. It's somewhat subjective. That's why I like to plot my graphs.

This also leads to notions of micro dynamic range, and macro dynamic range. Micro dynamic range gets swamped by noise quite quickly, whereas if you had the left and right only of the image as filled with two adjacent wedges, you'd be able to see more information down into the shadows. The full screen Stouffer is a compromised between micro and macro.

The final consideration is the character of the noise itself. This is not revealed easily by measurements and must be eyeballed.

Graeme

[Karl H]

I find it difficult to judge. One one hand, with photoshop level adjusting I can only see 23 steps on the chart, and thats being generous 'eyeball' speaking. So is this what you mean by usable DR? If so its looking around 7.5 stops.

On the other hand the number means nothing to me unless I compare this to other cameras or a DSLR test. Perhaps in theory all these cameras have 'data' of 11 stops, but we can relisitically see a difference of about 8? Perhaps this is consistent throughout the range of all high end cameras. Is that a fair comment?

to put it into perspective on DPreview i have looked at charts from the top DSLR cameras and can only count maybe 26 steps on these charts - yet they claim 11 stops from these tests. So there must be something else going on than just eyeballing the charts.

Unless I see this chart directly compared to those from the F23/f900/D20 etc then the number doesnt mean much only that it's a little short of the top of the line DSLRs, which is not bad.

[Graeme Nattress]

Another thing to look at is what Jim did with his ISO tests, which went up to 32,000 if I remember rightly and you could still see a noisy, but recognisable image. http://www.reduser.net/forum/showthread.php?t=3228

Interesting, that it contains a macbeth which has a 5 stop spread on it's greyscale, or thereabouts.

There's plenty of ways to look at dynamic range, and nobody really agrees on it.

DPreview tests are about the best, and they test JPEG and RAW, and they use a graph to determine the true range, rather than eyeballing it.

Graeme

[Martin Drew]

http://www.reduser.net/forum/uploaded/59_1190651280.jpg

I took the file into AE as 16 bit and used the WFM in "Color Finesse". You can clearly see the delineation between the steps at the right side, it gets more tricky as you go further left but you can still make out that there is information in the black from the trend of the plot, even when it gets too noisy to see the steps. I have superimposed the wedge and indicated where the 3rd stop markings should be as orange blobs.

Now I am assuming the WFM in Color Finesse is 8 bit but the trend it shows should still be valid, seems to indicate there is quite a bit of information in the blacks.

M

[David Mullen ASC]

Ultimately, I'm more interested (as a DP) in real-world shooting scenarios -- like when I'm shooting a close-up of an actor in a cafe and a white van in full sunlight suddenly parks in the background, or if I'm shooting in a room with a single shaft of sunlight, no fill, and I want to expose for the shadows, not the sun, etc. Or something as simple as someone being lit by a lampshade in the shot. And I'm not saying I expect or require an insane level of dynamic range, just a usable, naturalistic range that I can work within while handling the uncontrollable & unpredictable now & then.

[Álex Montoya]

http://www.dpreview.com/reviews/NikonD40X/page18.asp

Dynamic range is certainly tricky.

[Rick Darge]

I agree David. I'm interested in these same shooting scenarios as well. These are the real-world tests we will all find beneficial.

[jbeale]

Here's the photo of the setup for the T4110 chart.
http://www.adamwilt.com/ProductionStills/RED%20test%201/large-1.html

I did a quick analysis using Imatest on the tiff images for dynamic range and got several plots, which I put here: http://www.bealecorner.com/video/RED/Step-Sept15/index.html

This plot has most of the interesting stuff:
http://www.bealecorner.com/video/RED/Step-Sept15/REDONE30-20070915-T4110-Log-TRIM_Step_2.png

Based on this test, it looks like the RED dynamic range is similar to current DSLRs, at over 8 stops, that is quite a good result! So for those looking for real-world comparisons- if you can shoot any given scene with your Canon or Nikon DSLR and still hold the highlights and the darks, then Red will be able to as well.

These guys are pros and I'm sure they know their business...it seems to me they are not using a light box, rather just bouncing the light off a foamcore sheet. Personally I would wonder whether there wasn't some stray reflected light from the front of the chart, on the darker regions, unless they later enclosed the entire chart-camera light path.

[Graeme Nattress]

Yup, we backlight and control light from entering from the front when we use the Stouffer. I also test linear light too, rather than a curve. That helps get accurate numbers.

Graeme

[Evan Owen]

Yup, we backlight and control light from entering from the front when we use the Stouffer. I also test linear light too, rather than a curve. That helps get accurate numbers.

Graeme

I remember seeing a chart showing just over 11 stops a while back. Does the difference in range have something to do with uncompressed vs. REDCode RAW?

8 stops is great, I'm just wondering what's changed...

[Graeme Nattress]

No, compression does not adversely effect the DR, but how you raw develop does, and how you set up the wedge test does. When you're measuring into the noise floor, you've got to take an awful lot of care of what's going on.

Graeme

[jbeale]

There are many different criteria you can use for determining dynamic range so simply stating a number like "8 stops" or "11 stops" is not too useful, without the full explanation of the test you did and the interpretation you are making of it. For me, the takeaway is that the Red dynamic range is similar to the current DSLRs (when measured in the same way). That is actually a very good result.

[Sanjin Jukic]

Ultimately, I'm more interested (as a DP) in real-world shooting scenarios -- like when I'm shooting a close-up of an actor in a cafe and a white van in full sunlight suddenly parks in the background, or if I'm shooting in a room with a single shaft of sunlight, no fill, and I want to expose for the shadows, not the sun, etc. Or something as simple as someone being lit by a lampshade in the shot. And I'm not saying I expect or require an insane level of dynamic range, just a usable, naturalistic range that I can work within while handling the uncontrollable & unpredictable now & then.

Agree with David.

For a cinematographer or DP the real-world shooting scenario does matter only.
The graph and measurement numbers are for scientists to work out.

[Rick Darge]

What kind of mattebox is that? What model of Chroziel ?

[Evan Owen]

No, compression does not adversely effect the DR, but how you raw develop does, and how you set up the wedge test does. When you're measuring into the noise floor, you've got to take an awful lot of care of what's going on.

Graeme

That's what I figured.

I guess it's still possible to see even better results if the test is done carefully, but I'm not sure how important it is in the end. I appreciate the comment David made earlier. Sometimes it's more important how you use the dynamic range you do have...

[jbeale]

But they have claim to be equivalent to standard S35mm stock though!
Red has a quote on their homepage from Jon Farhat, VFX Supervisor of Wanted:
"...most people picked the RED footage as film because of a greater dynamic range in the highlights than in the 5218."
http://www.red.com/

To test this quantitatively, you could shoot the T4110 step wedge using Kodak 5218, develop and scan it as they did for the "Wanted" test and measure as I did with the Red data. If you haven't done that, you can't say whether the comparison to film is confirmed by the data or not. I've seen all kinds of numbers for dynamic range of film but there are many variables in this measurement.

Anyway, to some extent all of this is an academic excercise. If the Red output looks better than the competition when the result is projected, then it wins the game.

...well, perhaps that is too simplistic. The various kinds of film are good and Red is good also- so let me rephrase it: if the Red "look" works best for you with any given project, then I'd say the Red is the best tool for that job. And even if some particular film looks slightly better to you than Red, but some other important factors, like stock and processing and delays and lab mistakes are important- maybe Red is still a good option to consider. And if some film process looks better and you're comfortable with all that film entails- then you should shoot film.

[Brook Willard]

It all comes down to semantics when there is no direct comparison between mediums. Some claim film has 8 stops of DR and some claim it has 16. Are they using dratically different stocks? Nope - they just have different definitions of DR. In the end, unless this data can be compared to an accepted control camera that shot the same chart on the same day in the same conditions, there's not much to draw from it. Saying that it "only" scored roughly 8 stops when we were once told "11" doesn't really mean anything... It's the same sensor - that hasn't changed - its just a different interpretation of the data.

What I personally find interesting is that it holds up to and potentially beats modern SLRs DR. That says a lot.

[Roberto B]

footage?

[Michael Brennan]

Graeme,

Is there a way to use (a sensors) S/N ratio figure in the evaluation of dynamic range? Perhaps to create a formula that if used in conjunction with test charts avoids subjective appraisal of noise vs detail in blacks?

We would have to agree the S/N of the various sensors as well as a typical figure for scanned film for this idea to work.

The difficulty in the past six years with testing TV cameras (f900) has been frequent but modest improvements in dynamic range due to signal processing (ie gamma curves) every six months or so, making comparisons short lived.

RED has no gamma or knee to adjust which takes out one variable.

Is S/N the starting point for dynamic range evaluation?



Mike Brennan

[jbeale]

I'm not Graeme, but my $0.02: SNR has to be a factor in a meaningful discussion of DR. If you look at the Imatest output you'll notice four different dynamic range figures calculated from four specific image noise levels (SNR), in addition to a "total dynamic range" estimate. http://www.bealecorner.com/video/RED/Step-Sept15/index.html

That is one program, it's not the only way to measure DR. By the way, if you're including SNR (as you should) you need to consider sharpening and output resolution, since both affect it. Testing a raw 4k against any different size format pixel-for-pixel is not valid, you have to upsample or downsample to match the resolution first.

Another consideration is color fidelity, eg. do you use the R,G,B channels cleverly to extend the overall response in areas where not all channels are clipped. The Imatest program, as far as I can tell, throws that highlight data away, but a smart Raw converter would make use of it.

[Frank Mirbach]

Mate, i've been saying this for weeeeks. :-) Less about charts and more about what the cameras practical limits are and how that is directly related to lighting scenarios, gear and STORY choices. Leave the marketing to Red and the empirical (which does has its place) to others. Sure, it's early in the cameras release but no ones mentioning STORY/SET choices with regards to the cameras digital charactersitics and (not talked about much) challenges.

I don´t care so much about all the tech stuff as long as the images look great and they do. Would love a more practical approach here, too...

[Graeme Nattress]

S/N = DR, or S/N in db / 6 = stops.

You can adjust knee and gamma, but you do it all in post, which is nice. Recording RAW means you always record the max DR.

Graeme

[David Mullen ASC]

From a real-world shooting perspective, I think the challenge (for us film shooters) is that we essentially don't worry about the highlights when shooting color negative -- it's always the shadow end where we worry. Too much underexposure and we don't have enough density to get a good black or keep the grain down, etc. Whereas we often don't worry (too much) about blown-out windows, lampshades, shiny metal, etc. -- it seems that film just rolls off into white so gradually that there's almost always some detail to pull-out in post if needed.

For example, I shot DSLR stills with my Nikon D40x (in JPEG mode) of our main location of my recent feature to create a digital storyboard, but also to test how the place looked in available light. I then shot some available light tests on Fuji Eterna 500T of the same rooms and went into a D.I. suite. When I set-up the image to match the grey scale I shot, the image looked remarkably like my DSLR photos. But when I asked the colorist to go into the white areas, like the burned-out windows in the far background, and see what detail was there, he could pull-up a perfectly exposed exterior view! I couldn't believe how much information was hidden in the highlights on the negative despite being several stops overexposed.

Now I'm not saying that this sort of information can't be captured by a digital camera, but what I am saying is that as a DP shooting on a camera like the RED, I'd have to actually look to see where the highlights are falling and decide whether to bias the exposure to hold information there so that I have that detail in post. Whereas with film, I tend to not worry about it. Conversely, with film, you worry about low-level detail and whether to bias an exposure for that, whereas with digital, I'd probably worry less about that end. So it's the reverse of the old Ansel Adam's rule of "expose for the shadows, print for the highlights".

This requires a shift in the way cinematographers think about exposure. However, the flip-side to this is whether there will be noise problems from adjusting scene exposure based on overexposed highlight information, as opposed to always exposing mainly for the key subject and letting the camera handle the extremes at either end, so as to maintain the same noise level shot by shot. This is where you need to test and gain some experience with the particular camera, because conversely, I also believe that a common mistake people make when shooting digitally is believing that general underexposure is the solution to the dynamic range problem, although a high-rez, low-noise RAW camera like the RED (as opposed to an 8-bit HDCAM camera) gives you more wiggle-room in color-correction.

[mezmo]

Hi David,
Great post, could not agree more.
The need to compress the dynamic range of digital
with lighting,filtering and choice of shooting locations
would improve with a camera like Red. DR of 8+stops.
But film still has that great set & forget flexibity on it's DR side.
Having said that I'm more than happy with 8+stops for Red.
Cheers Mezmo

[david farland]

I'm not....

I'd still like to see a film comparision of the test Red Inc did to achieve 11+ stops.

Plus another test:

Side by side comparision of Red & film camera with and without NDs for the one setup (which would include good highlight & shadow sections)

I like what David says... I wanna see if Red's strenghts in the shadows whilst still holding the highlights....matches film's strenght (stock?) in the highlights whilst keeping information in the shadows.

No biases just a full comparison...which is the line that all these 'neutral' test groups should really take anyway!

Obviously if I know these strenghts/weaknesses it will help me whenever i need to compromise on exposure or lighting for the scene.

Cheers,

ps: without cherry!

[Michael Brennan]

S/N = DR, or S/N in db / 6 = stops.

You can adjust knee and gamma, but you do it all in post, which is nice. Recording RAW means you always record the max DR.

Graeme

The question is how reliable are published SNR figures?
or
Is their a bullet proof/standardised way of measuring it.


I think this is a better starting point with subject assesment of picture aesthetics on top rather than applying film methodology of shooting and measuring over/under exposure to a digital sensor.



Mike Brennan

[Floris Liesker]

http://www.reduser.net/forum/uploaded/59_1190651280.jpg
M

I think this graph shows more than 8 stops of DR. In the low end you're not able to make out the steps, but we can still see the curve slide down to the left, meaning there is luminance data in there.

I'm not sure about the R3D to TIFF conversion they did, by the way.
To be sure that the picture shows a clipped sensor and not a clipped RAW to TIFF conversion they should have toned down exposure in Red Alert (not on the lens) until the white clipped ceiling becomes something less than 255,255,255. Or 65535,65535,65535 for 16 bit.

[Karl H]

I'm confused. Can I ask the Red team, does this chart look visibly in the ballpark to your tests, or 'drastically' different to the wedge you shot to measure over 11?

I ask because I want to know if the chart is wrong or its fine and we're arguing over interpretation of the same data. Or in other words, usable DR over mathematical DR. If there is indeed data in the lower part of this graph to get to 11, then that answers my question - i know what im getting. I'm getting 8 that i can see and use, but more on paper.

On the other hand, if Red are saying they had 11 stops of visible DR and exposed much more of these squares correctly in their test, then that would be a very different graph from what we're seeing here and a more usable range of DR.

If it looks the same, then there is not much to discuss except how you arrived at your mathematical conclusion (which aesthetically isn't that important). If however you're saying, 'no we could see much more visible difference on our tests' then that makes a big difference and we should not use this wedge when comparing to other cameras.

That's the key, to put this wedge up against ones from other cameras so we can make a visual judgement on the DR.

[Álex Montoya]

I'm almost certain that there was more highlight detail to be recovered there. If they didn't care to underexpose about 1 stop in REDAlert!, which they probably didn't, that's a stop they lost there.

[Graeme Nattress]

To me, it looks different to wedge tests we've shot. Looking at the digital photo of the setup, the conditions were not controlled in the way that we do, where we take extra care to make sure there's no front-lighting reaching the chart as they will grey your blacks and give you false results in the shadows. That's probably what's happening with that chart. That we don't see the "black" surround in the crop, makes it's hard to tell how black they managed to get "black".

Graeme

[Graeme Nattress]

http://www.nattress.com/GraemeStouffer.jpg

Stouffer T4110 taken at the RED office. We take care to ensure that no stray light enters from the front. Hard to avoid totally, but we do take care as best as we can. We have a nice flat illumination light table which we use to backlight the chart. Obviously, any discrepencies in the backlight will make the test void.

The screengrab from REDAlert! is to show the settings I've used to allow me to see all the dynamic range from the chart. The inset images in red borders show where detail starts and ends more easily. They're 4k 16bit Tiff exports, taken to Photoshop where I used levels to show you that:

You can see the transition from wedge #2 to wedge #1 (note, wedge #1 is not numbered).

At the dark end, (I boosted it about 8 stops) wedges all the way down #28, #29, #30, #31, #32 are clear, getting tricky to see #32 to #33 to #34. You can on a larger scale see the transition off to the right getting darker and darker out to 41, but it's hard to make out any transitions. Measuring the wedges tells the whole story....

#31 = 139
#32 = 133
#33 = 128
#34 = 125
#35 = 121
#36 = 120
#37 = 118
#38 = 115
#39 = 114
#40 = 112
#41 = 109

So, this related to micro (small scale) dynamic range, and macro (large scale) dynamic range. The measurements, being an average of the wedge, give us large scale dynamic range, which does indeed extend all the way down to #41 on the chart. This is great, as it means I can plot sensor linearity in a log / log graph over the range of the sensor and see that it's pretty darn linear all the way.

Where do we call it on the micro dynamic range though? Well, I can see the transition from #33 to #34. So, I can see the transition from #1 to #2 and #33 to #34, so that's 34/3 = 11.3 stops. If you can only see #32 to #33, you can call it 11 stops.

That's not the full story though. #1 has an OD (optical density) of 0.05, ie it's pretty clear. #34 has an OD of 3.37. So, that's an OD range of 3.32. (3.21 if you can only see #32 to #33). To convert OD to stops, divide by 0.3, so that gives 3.32 / 0.3 = 11.06 stops, and 3.21 / 0.3 = 10.7 stops.

To put that in perspective, the #41 wedge has an OD of 4.02, and therefore from #1 to #41 is a range of 3.97 OD, or 13.23 stops. Below the darkest wedges, you can see the black card that the Stouffer has been mounted on is actually darker than wedge #41. So in macro dynamic range terms, we are seeing pretty much the whole range that the Stouffer offers. In micro dynamic range terms, we can probably say 11+ stops, depending on how you accurately you measure things, how good and even we got the backlighting with the lightbox, and how well we stopped stray light getting in. The more you can ensure black is black, the more you can see into the shadows.

I hope that explains about using a Stouffer, why you need it calibrated, why you have to take care with illumination and stray light.

Graeme

[Sanjin Jukic]

Thanks Graeme, that's the story.

[erbp]

Thanks a lot for posting that, Graeme.

If I'm understanding, you exported two tiffs from RedAlert, each with a different exposure setting and then used levels in Photoshop to produce the two inserts. So, that should mean that with the right software (RedCine or Scratch or whatever winds up working with r3d files) you could produce one image with enough dynamic range to make out the #1 wedge and the #31 wedge, although they would still retain the noise we're seeing here. Is that right?

[Graeme Nattress]

No, I only exported one image, but I processed it in Photoshop twice, to show where the detail ran out at either end. To see both ends, you'd probably either have to have quite an extreme curve, or to use some kind of shadow / highlight processing mimicing an HDR type effect.

Graeme

[Karl H]

Graeme, that helps enormously.

While I wont claim to understand all the values, it is clear to me that you can visably see more steps in the chart you shot.

Thanks for this.

[Graeme Nattress]

Nanhov, I'm sure you've got something interesting to say, but you're not saying it clearly. To say "whatever DR you get is not true DR" is a nonsensical statement - it reads like you're playing a definition game.

Graeme

[Graeme Nattress]

You seem to know all about the Mysterium, yet such specs as you quote have not been released. I obviously cannot comment on the nature of the specs one way or another.

The thing that I pointed out above is each and every person has their own DR test and methodology. I've published mine above.

You can doubt all you want, but I define dynamic range to start and stop where you can no longer distinguish between steps on the stouffer chart. I can see the difference between #1 and #2, and, #33 and #34.

If you wish to define DR differently, then that's your freedom to do so.

Graeme

[jbeale]

Stouffer T4110 taken at the RED office. [...]
That is a truly impressive exhibit Graeme. The photo and the description demonstrate both the remarkable performance of the camera and the care you have taken to measure it properly.

The inset crop showing the dark range detail is remarkable. Evidently I should experiment more with different curves after conversion to 16-bit TIFF, in order to appreciate all the dynamic range that is available.

[Graeme Nattress]

Thanks. I think we could get even better results with more strict calibration of wedge, lightsource and avoiding front lighting and any reflections. Also, it would be interesting to look more at macro-dynamic range by either zooming in so three wedges fill the screen or constructing larger wedges that could be slid infront of the lightbox. Ideally you'd measure dynamic range over a number of different scales and get a more complete picture of what the camera is capable of.

Graeme

[Kevin Halverson]

I think that there is a pretty clear definition of what constitutes dynamic range (the difference between the noise floor and the clipping point). Where this becomes difficult is when subjective terminologies get exchanged with engineering ones. Usable stops and dynamic range are clearly different quantities.

Graeme has stated his testing regiment and therefore makes reproducing his results possible. The inability of others to make front lite chart grabs match a controlled engineering test should not be considered a slight of the camera, but rather of the lack of following a methodology that allows for quantified results.

[Álex Montoya]

Graeme, according to THIS (http://www.dpreview.com/reviews/NikonD40X/page18.asp) terminology, what would you say is the highlight range of the RED?

I am asking this because I don't know if I can read it directly from your JPG since it is linear.

[jbeale]

Just had another thought. Note: academic discussion follows- perhaps little or no relevance to real cinematography.

For a very accurate measure at the extremes of DR there are some other considerations. Eg: any veiling glare in the lens will tend to lower "true" scene contrast- but this works equally as much for real scenes, so "true" contrast may be a moot point.

The other thought: is the Stouffer wedge a perfect ND filter even in the IR? If the camera has any measurable IR sensitivity, that may influence chart calibration in the darkest regions, unless the chart densitometer measurement had exactly as much IR sensitivity as the camera, or the light table illumination has no IR component.

Long ago I tried to make my own test wedge "on the cheap" using various tinted car window films (Tap Plastics). They look like a ND filter, but it turns out even the dark 10% transmission film was almost perfectly transparent in the IR.

I also discovered my Sekonic L-508 lightmeter is very sensitive to IR, even though Sekonic claimed it was not. For example, it will measure a dark room to be as bright as a daytime sky if you point a Sony Nightshot camera at it. Those use an LED illuminator around 800 nm I think, which looks very dim to the eye.

[Kevin Halverson]

Long ago I tried to make my own test wedge "on the cheap" using various tinted car window films (Tap Plastics). They look like a ND filter, but it turns out even the dark 10% transmission film was almost perfectly transparent in the IR.

This is very true. A company that I design for uses a dark blue cast lexan material on its front panel. The plastic is used as a window for both the front panel display (VFD) and for the IR detector. Despite the dark blue visible nature of the window, its IR transmission was incredible (well over 94%).

When I was first evaluating materials I went to a distributor of materials with a night vision device so that I could get a quick idea as to which of their materials was worth considering. I took the samples in a dark room and switched off the lights. When I fired up the night vision system, I assumed that I dropped the sample on the floor as I couldn't see it on the table surface. When I started to search the floor feeling around I still couldn't locate it. That is when I put my hand on the table top and felt the material sitting right where I placed it. I was shocked by the fact that I couldn't see the material in the IR region at all.

This all confirms that a good camera needs not only a pretty aggressive OLPF but an equally good IR rejection one, otherwise, a lot of what you end up capturing will be outside of the visible spectrum.

[Graeme Nattress]

KHMuse, indeed usable dynamic range is a subjective criterior. That's why I try to stick to the figures, and let the cinematographers debate on the subjective stuff. I've had cinematographers basically say RED has x stops, just by looking at footage, which to me is bonkers, as it's unbelievably hard to tell what the DR is just by looking! To me, it's something that is measured, not guessed. And usable stops comes down to noise tolerance in the end.

JBeale, We do have an IR filter, but I'm sure it must let some IR though. Good points, and it shows how hard it is to measure DR as the number of factors you have to take into account just keeps growing!

Conrete, all digital cameras have zero highlight range. All their range is in the shadows below clipping. By biassing you exposure differently, you can pick a mid grey anywhere you want in the range and hence get a different balance of stops above and below. DPreview's methodology is hampered by them having to use some tools to develop the raw image, whereas, as I've written the REDAlert code, I can tell it to outut fully linear data, whereas, as you can see they have a hard job getting a straight line out of the data, even though it was a straight line at the sensor. That does give us an unfair advantage :-)

Graeme

[Miltos Pilalitos]

What's the methodology in testing the dynamic range of film and how does it compare to the RED testing Graeme?

[Kevin Halverson]

What's the methodology in testing the dynamic range of film and how does it compare to the RED testing Graeme?

As I spent a number of years designing instruments that measured film, I will jump in and give you a really quick answer as I am sure Graeme's time would be better spent working on real problems that effect the camera's development.

As for your question, the concept isn't really applicable as film is such a non linear beast. It is the distribution of the characteristics that gives negative film its latitude. You have to integrate over an area to discover any "dynamic range" at all in film. If you try an measure just a single site, it would yield amazingly poor results.

[Martin Drew]

This has turned into a very interesting thread

M

[Miltos Pilalitos]

As for your question, the concept isn't really applicable as film is such a non linear beast.

So let's say we digitized a frame of film that contained the Stouffer test and we exported that as a 10-bit log DPX file. Would it make sense to compare this with a DPX frame from RED?

[jbeale]

You have to integrate over an area to discover any "dynamic range" at all in film. If you try an measure just a single site, it would yield amazingly poor results.

It's an interesting point, that people tend to forget about. You could set up a "for example" case though, like look at an area on the negative 5 microns on a side (roughly the size of a Red pixel?). Kodak publishes some measurements of their films, but I'm not familiar enough with their methodology to know exactly what their plots mean.

http://www.kodak.com/US/en/motion/products/negative/tech5218.jhtml?id=0.1.4.6.4.12.4&lc=en#imgstr


So let's say we digitized a frame of film that contained the Stouffer test and we exported that as a 10-bit log DPX file. Would it make sense to compare this with a DPX frame from RED?

This seems like a reasonable test. You might want to specify some additional parameters, though. Different development parameters can change dynamic range but along with it overall sensitivity curve, color response, granularity, sharpness etc.

[david farland]

Graeme,

Thank you very much for the clear and through description of your DR measurement procedure.
So the crux is when measuring 12bits of dynamic range on an 8 bit image file (and monitor) you need to expose the 8bit image of the Stouffer graph for shadows and highlights, like you would for an HDR effect on high dynamic ranged scene…as you mentioned. In this case the high dynamic scene is the Red data file of the Stouffer wedge.

Cheers,

[Graeme Nattress]

Yes, if you were extracting to an 8bit linear file, you'd need to double expose, which is why I used a 16bit tiff. Even so, I need to do extreme levels to show what's going on.

Graeme

[Finner]

I'm a little confused about all this. It seems to me that how a DR test is shot becomes irrelevant for comparison to any other digital camera system or film stock that would have a DR test done in a different way. From what I am understanding Graeme has found a way to shoot a DR test that will give the highest DR result possible. So technically to truely test what digital cameras and film stocks would do competing against each other the only way to do it is to have all cameras shoot the exact same test under the exact same cirumstances.

On the basis of this I would suggest that Graemes red test is meaningless because there is no other cameras that have tested that exact same way to compare the over 11 stops evaluation Graemes test shows. I mean absolutely no disrespect to you or your test system Mr. Nattress as your work has shown that you are extremely precise. It just seems to me that with your test not having any competitor comparison evaluations that the kodak planned test would actually be more accurate in comparing cameras as it will be the same test with many different cameras there.

I could be wrong with my thoughts on this and if so would like to hear where I am mistaken.

[Graeme Nattress]

The Stouffer test as we performed it is not designed in any way to give a better number than the camera should get under any other circumstance, but is designed to accurately measure that number in a repeatable way.

To measure dynamic range, you need something that has a DR greater than that which you are trying to measure. Reflective charts have ~5 stop range or so - you don't get much more from a printed piece of paper. To go beyond that you have to go to a transmissive chart. Stouffer make good ones.

To see the full 13 stop range from the backlit chart, you've got to make sure you can see that entire range accurately, which means even backlight and no foreground lighting landing on the chart (which would make blacks grey and hide the shadow detail you're looking for).

It's ONLY by making sure that the test subject is fully displaying it's > 13 stop range can we measure how much of that our camera sees. The problem with the earlier test shown is that I doubt that the subject was showing the full > 13 stop DR, and hence our camera couldn't sense it's full DR as the chart just didn't get black enough. You've got to simultaneously get a very black black in the same shot as a very bright white (and everything in between) and accurate measurements for them all. Don't do that and all you have is guesswork.

What other way is there to accurately measure DR than to have a subject that displays a greater DR than the device under test, and to display that in 41 calibrated steps, and then to point that device under test at it and see which of those steps it records???

What we've done is to show how to accurately record DR, something that's tricky to do. We can do it repeatedly with similar results each time, which is good, or how else could we improve on what we're doing without a good reference?

This type of test is best as it's repeatable, and you can put numbers to the results. Any other kind of test is just guesswork and mucking around. The DR tests that CML do are "nice" but they're not measuring the DR that the camera can take in a single shot, so you've got to analyse multiple frames, and there's no calibration process - you're relying on the lens aperture markings and ND markings, and the accuracy of the user using them. This kind of approach is just not good enough for us as it's inherently not repeatable.

Trying to capture a "real world" scene for DR testing, especially when comparing film to digital cinema is totally meaningless though. Film has very different exposure characteristics and you can develop in a multitude of ways. Although we can "develop" our RAW in many ways, there's only ONE way to show unprocessed (but demosaiced) RAW data in linear light, and that's what I did as that shows what the sensor recorded, precisely. You can always creatively choose to use less of the DR than this shows, but you cannot ever use more. With digital, you have absolutely NO headroom at all. None. All the dynamic range is in the shadows. So you must expose and rate very differently than for film. But with the Stouffer, as long as you can see clipping at one end, you can mark the "top" of your range, and then count how far you go into the shadows. You can point film or RED at the target and it doesn't matter what you set your exposure to, as long as you can get it to clip at the top, you can count how far down you can go.

Graeme

[jbeale]

On the basis of this I would suggest that Graemes red test is meaningless because there is no other cameras that have tested that exact same way to compare the over 11 stops evaluation Graemes test shows.

In my understanding, Graeme is doing engineering evaluations of Red's DR with an eye towards Red QC and development- so his test is designed accordingly. Since he has described the conditions of his test, anyone else who buys that Stouffer step wedge (they cost about $30, or $50 in the calibrated version) and follows his procedure can perform the same test on any camera they choose and get a result that can then be compared.

In the DSLR world they do use this 41-step wedge test with specific (different!) methodology and they get a different range figure. I am planning to do some tests including my own DR measurements on a Red under specific and well-controlled conditions and compare it as fairly as possible to my own Canon 20D DSLR, but that won't happen for more than a month yet, and I'm sure that others will do similar comparisons before that.

For the working cinematographer, I don't know if this particular test and DR definition has particular relevance. It seems there are a number of different ways to describe dynamic range. The film way of thinking about "highlights" doesn't fit digital sensors, just like the analog tape way of thinking about "headroom" doesn't fit digital audio.

[david farland]

I don't agree with Finner's assumptions or the way he is putting it.
i do agree with some of the essence he's stating.

I agree that people need a relative comparison of DR between Red vs other cameras.
Graeme has an accurate repeatable test for measuring DR.
Obviously we need results from an accurate repeatable comparative test asap!

The blindly obvious conclusion is without this accurate, repeatable & comparative test we're going round in circles!

Cheers,

[Finner]

I don't agree with Finner's assumptions or the way he is putting it.
i do agree with some of the essence he's stating.

I agree that people need a relative comparison of DR between Red vs other cameras.
Graeme has an accurate repeatable test for measuring DR.
Obviously we need results from an accurate repeatable comparative test asap!

The blindly obvious conclusion is without this accurate, repeatable & comparative test we're going round in circles!

Cheers,

You say you don't agree with what I stated but you sumed up exactly all the points I was making. Also I do not believe I assumed anything but only stated facts from tests I have done in the past.

[david farland]

You're assuming Graeme's red test is meanless.
I haven't (maybe you have) seen test results of other digitial camera's using Graeme's DR technique.

Cheers,

[Finner]

I do not assume Graemes tests are meaningless. I feel they are meaningless if you want to try and use them to compare the reds DR to any other cameras DR.

I feel I did not do a good job of making the post as I was trying to do it with the most respect as possible. Honestly I feel Graemes test like other things I have seen him do as being well thought out and executed very well. All I am saying is you can not use that test to compare DR's of any other camera out there as all the variables need to be the same.

Graeme I agree totally with you that the DR chart needs to be set up at a greater DR then the camera has to get accurate results. All I am saying is in comparing red to other cameras that have their DR tested we have no idea how good or poorly the tester has set them up. It would be really nice to see a well done DR test like yours with a bunch of the top digital motion picture cameras in a DR shoot out. Until then I don't think anyone can say how reds cameras DR truely stacks up against a viper or ARRI D20 or others.

Hope things are going well Graeme. From what I heard today you are very busy.

[Martin Drew]

Finner. Do you mean that you can't take Graemes figure of 11+ stops and use that in a comparison with figures published elsewhere for other cameras? If so I agree, but the issues discussed in this thread illustrate how problematic comparisons are in principal anyway, especially when comparing disimilar technologies.

M

[david farland]

As stated, the only real comparative lab test is to test different cameras under the same conditions.

We’d all like to see how Graeme’s test showing Red with a dynamic range of 11.3 stops compares with other digital cameras.

Graeme, have you performed Red dynamic range comparison tests with other cameras?

It was felt that a lot of digital still cameras only had a DR of ~8 stops.
It was also assumed that Red 11.3 stops was measured on a similar scale to these other camera’s 8 stops. I do not know if this assumption is wrong or right because to date there's been no published comparative lab tests between a fully working Red and any other camera except the CML test last year revealing Red at 8.3 stops. However Red stated the camera being tested had severe PCB problems reducing the dynamic range of the camera.

Before going on please read this excellent dynamic range test overview…. here (http://www.dpreview.com/reviews/nikond200/page22.asp)

It outlines the process used to test dynamic range of a Nikon D200 using a Stouffer 13 stop wedge chart.

It states dynamic range values of 8.3 stops for jpeg output files and 11 stops for corrected raw files. See graph here……


http://www.reduser.net/forum/uploaded/48_1190803889.jpg


The next comparison I would like to see would be similar to the graph below except with Red as one of the cameras.....!


http://www.reduser.net/forum/uploaded/48_1190804545.jpg


my last question is what are the limitations in this comparative test approach when testing Red with other digital cameras?

Cheers

[Graeme Nattress]

JBeale, the major problem DPRreview has, is that they don't have access to the RAW camera data in the way that we, a camera developer has. If you look at their graphs, they've got curves on them, and they've got to play around in Adobe Camera Raw to see if they can get as good a line from the RAW as possible. We can just go for the best straight line, as I coded that mode into REDAlert. Makes life much easier and more simple.

Graeme

[jbeale]

note, I believe Graeme's post above may be in reply to David Farland.

In my own opinion, the most accurate camera comparison is to take shots of the same stepchart on the same lightbox on the same day, minimizing any variables, and analyze the frames in the same way. Different raw converters complicate the task. I have not seen any such test results published so far. I plan to do a Red vs. Canon 20D test myself in roughly a month from now. Subject to change.

Based on what I have seen so far, I do not expect the Red to be very significantly different from the current crop of DSLRs in terms of dynamic range, for any consistent definition of that property. As I recall, that would match Jim's original goal a few years ago, which was to make a movie camera that had the image quality of the DSLRs that he was using at the time. It seemed like a crazy goal for a small company and many people doubted he could do it.

[Graeme Nattress]

Yes, I think I directed my comment at the wrong Person. Sorry about that.... Must remember to post only when awake in the future....

Graeme

[david farland]

..... We can just go for the best straight line, as I coded that mode into REDAlert. Makes life much easier and more simple.
Graeme
Nice to hear Graeme!

So you're saying Red /RedAlert out of the box achieves 11 stops (at a more linear response) without having to go thru
the hoops the reviewer in the Nikon D200 test setup needed to with his setup...as stated here...

"The best we could achieve (with some pretty extreme ACR settings) was just over 11 stops total dynamic range,
and more importantly about a stop more highlight range (although with no guarantee of color accuracy)."

Cheers,

[Graeme Nattress]

The problem of dynamic range is displaying it. That means you've, essentially, got to brighten up your shadows a lot for you to be able to see it all in one image. That doesn't tend to look totally nice if you just apply willy-nilly. Applied with care, with secondaries, can look superb. Really, capturing a wide dynamic range is so you can decide how to deal with that problem later, rather than in camera. And to give you room to change your mind :-)

To make things pretty, usually a contrast curve is applied, and this works great, but reduces dynamic range. Basically, if you're not careful, doing anything to the image to make it pretty will reduce the dynamic range. The linear light mode does nothing to the image, so you can see all of it, but not easily all at once as we don't have displays with 13 stop DR. If we did, we could get away with doing less, but images might be more fatiguing to view.

As for colour accuracy in highlights, this can be an issue, but we address that with the DRX control to a vast extent.

Graeme

[jbeale]

If you enjoy technical discussions, this one about raw conversion, dynamic range et.al. has a lot of good information and food for thought.
http://www.normankoren.com/digital_tonality.html

[donatello b]

"Graeme, have you performed Red dynamic range comparison tests with other cameras?"

no.... Red should NOT be testing other camera's for any comparisions ( or perhaps that should be show/publish the results ) ... they should only be testing RED ... anybody with a Red will be able to test it against other camera's ...

"We’d all like to see how Graeme’s test showing Red with a dynamic range of 11.3 stops compares with other digital cameras."

Graeme explained ( showed graph) how he did his test ... anybody here can perform the same test with other camera's ... ......or you can test RED the same way DP reviews test their camera's ... again IMO RED should not be testing/showing us results from other camera's no matter how they perform the test ....

[Floris Liesker]

Just a thought: this Stouffer test is quite unaccurate as there is the chance of light of the bright part of the chart affecting the dark side of the chart. This could happen by light reflections, or in the lens or wherever.
This risk will be there as long as the test involves the brightest and the darkest value to be in one image.
So why not expand it over time? Make a Time stretched Stouffer test that lights the sensor totally, maybe even without a lens in front of it: just a plastic cap that will diffuse the light.
I would preferrably want to have a R3D file consisting of 41 exposures, 1/3 stop apart, of which I could easily compose a lineair image without any bleeding and telling me what the sensor is really capable of.

[Graeme Nattress]

Floris, that would certainly give you a more accurate figure of sensor dynamic range, but we don't use sensors, we use cameras, and as such, the result could be academic and not achievable in real world use. I think the stouffer test as outlined above represents what you could get in real shooting.

Graeme

[Graeme Nattress]

Posted a new analysis of the chart here: http://www.reduser.net/forum/showthread.php?t=4904

Graeme