Working with RAW

[Rudi Herbert]

After seeing many of the samples posted on the forum, and reading a lot of the comments by many posters, I came to the realization that RAW is a very misunderstood beast for the majority of people, including talented and experienced DP's. Mostly, and this is good, I realize people don't know how much you can flex and push a RAW image without "breaking" it. You can be VERY agressive with RAW and, as long as you watch your histogram, you can get away with murder really. So, with that in mind, I wanted to start a thread where many of the talented people on this forum post examples of RAW images and explain how they handled them, as I think this would be invaluable to many others.

So here are some examples shot with a Nikon D-100, which is one of the oldest RAW cameras, and the one with probably the smallest sensor, at 5 mp. I chose the D-100 because it has the lowest dynamic range of these cameras and the files it produces are very similar in size to RED's, so by way of over simplification, it would be the most similar image I can find. Also, I find the D-100's dynamic range slightly better than RED's, but very, very similar. I would have used some r3d samples, but I really can't make Redcine work fast enough for it to justify the time investment.

http://www.reduser.net/forum/uploaded/2969_1206481871.jpg

This was shot in Santorini on a misserable, overcast winter day, but I wanted to see how much I could make the original image on the left resemble the magazine shots of the island you see so often. Histogram on the left shows very little information on both shadows and highlights. All processing was done on Photoshop CS3 on its RAW app. I increased exposure by 0.5 stop, increased brightness by 30%, recovery by 5%, and then played with the hue, saturation and brightness of the individual colors one by one, reds, yellows, oranges, and the blues. That's all. The image very closely resembles the true look on a nice, sunny day, and looking at the histogram, you can see now that we've spread the values all over the register, making the image "better" instead of degrading it.

http://www.reduser.net/forum/uploaded/2969_1206482239.jpg

Another example at the same location, a typical nightmare of bright whites and dark blacks and blues, but handled easily. Original image at left, again, historam bunched up right at the middle, but with no clipping. Increased exposure by 1.25 stops, brightness by 40%, recovery 10%, fill 5%, set white point at the left wall, and manipulate the blues, whites and blacks separetly. Again, the histogram on the right shows a wider distribution of values, with some intentional clipping on the shadows, as I wanted to crush the black gravel stones on the upper right a bit for punch. Image looks very bright and punchy, but notice there's no clipping on the highlights. There's a lot more information on the whites, unfortunately, it gets lost on the JPEG.

http://www.reduser.net/forum/uploaded/2969_1206482792.jpg

Last example. Original on top. Set white point on the clock's face, increased brightness by 40%, contrast by15%, saturation by 15%, fill by 5%, recovery by 5%, and individual hues and luminosity of whites, yellows, blues and greens. Done. Again, the modified histogram shows a much more balanced image without any clipping and still plenty of room to play with.

Please excuse the dust on the sensor, these images were on my "reject" bin so I never cleaned them up. I wanted to show how much you can do with just the simple RAW adjustments in Photoshop, before you even get into serious color correction of any kind. Hopefully this will encourage others to post examples and share their expertise with the rest of us, and the technical minds on the forum will make an effort and not start analyzing the mathematical differences between sensors and formats, etc. Let this be a "hands on" thread and nothing else.

[Jim Logan]

Great info - THANKS!

[Cüneyt Kaya]

asfaik redcine cant set individual hue for each channel-----but thank you really nice examples

[Rudi Herbert]

asfaik redcine cant set individual hue for each channel-----but thank you really nice examples

The ability to manipulate the hue, saturation and luminosity of channels separately WITHOUT exiting the RAW confines is very important IMO. You can do this with Cineform in Premiere and AE, so perhaps at some point when they release their RED codec this will be implemented? It is so much easier, intuitive and conducive to real time operations to work while in RAW, that I for one, would do as much possible still there and only jump into true color correction for the final pass and the secondary corrections.

[Radoslav Karapetkov]

Subscribing to this thread.

[Cüneyt Kaya]

The ability to manipulate the hue, saturation and luminosity of channels separately WITHOUT exiting the RAW confines is very important IMO. You can do this with Cineform in Premiere and AE, so perhaps at some point when they release their RED codec this will be implemented? It is so much easier, intuitive and conducive to real time operations to work while in RAW, that I for one, would do as much possible still there and only jump into true color correction for the final pass and the secondary corrections.

SCRATCH can do, methinks this is the line where 1st light CC ends.
Hence redcine wasnt designed for that......BUT IT WOULD BE COOL!!!!!!!!!

( I hope somebody chimes in and tells me you can do it in redcine, you missed it, you can do it this way------hope finishs :) )

[Tom Lowe]

Rudi, this thread is going to be awesome. What do you mean by "Set white point on the clock's face"? Can you tell me how you do this?

Also, would you mind if I posted or PM'd you a Canon RAW image from my 350D to see how you would handle it?

thanks!

[Rudi Herbert]

Tom,

To set the white point in the RAW app within Photoshop, there is a tool called, appropriately, "White Balance Tool" :-) which you then click onto whatever area of the image you want to set as absolute white, and the whole color temperature and hues get shifted accordingly. I remember the clock's face being very neutrally white, so I chose it as my reference point.

It is important to remember that the beauty of making corrections in RAW is that it's all happening in metadata, nothing is getting flattened, or imprinted onto the image, therefore you can work in real time and make countless changes without the need for proxies, rendering, etc. This can be VERY cool for RED originated video in that, for example, Cineform does something within Premiere with their SI-2K footage where you can adjust not just frames, but whole video clips in RAW form, without the need to render them out to TIFF, DPX or any other huge format that requires tremendous horsepower to work with AND time, and where by default, you won't feel as free to make as many adjustments as you would with RAW until you find that sweet spot you're looking for. If, through Cineform or anybody else, besides Scratch which, though a great app is definitely nowhere near as affordable as the Cineform/Premiere/AE combo, you could do this with the RED footage when it is still in RAW form, then you don't need a monstrously powerful computer. That is the beauty of RAW, as an "undeveloped" digital negative, a sort of "zipped" image file.

I'll be glad to take a look at your photo, I'm a Nikon user, but Photoshop can also work with Canon files, so e-mail me whatever files you want and tell me what look or effect you want me to try and achieve.

[Dj Joofa]

Tom,

It is important to remember that the beauty of making corrections in RAW is that it's all happening in metadata, nothing is getting flattened, or imprinted onto the image, therefore you can work in real time and make countless changes without the need for proxies, rendering, etc.

...

That is the beauty of RAW, as an "undeveloped" digital negative, a sort of "zipped" image file.


Actually, raw is no more beautiful than "processed" video as long as you knew which processing you did and you can always invert "video" to get to raw. There is nothing magical about it. And, you don't even need to invert the it directly, as long as you collapse multiple transformations into one -- effectively, what the LUTs do.

The advantage of raw is that since it is supposedly linear, simple operations such as matrix multiplications (white point is changed by simple matrix multiplications most of the times) have equal variation in everywhere in the raw space, where as in the non-linear video world linear transformations (such as matrix multiplication) will have different effect in different areas of the space.

[Jason Ing]

Nice PS work Rudi. :biggrin:
What I love about RAW is all that fantastic information in the file! :love:

I can do all kinds of things before it begins to degrade.

Here is some before and after work. Some of them are curve moves in LAB color space.

[Jason Ing]

last one is just a curve move.

[Philip Powell]

Rudi,

Thanks for you work.

Could you please explain the difference between the use of exposure vs brightness on the use of raw files. When would you want to use one versus another?

With most of your examples you spread the histogram "out" as it were, in Redcine, what would be the best tools to do this with and maintain maximum quality?

Thanks.

[Rudi Herbert]

Nice stuff Jing,

Working with curves is a bit of an art, and a bit intimidating for many people, so I purposely posted examples with simple, easy to follow adjustments to make the point of how flexible the RAW files are, but curves are the way to go for more sophisticated work.

Joofa, I do think that RAW is MAGIC in that, as Jing points, there's a huge amount of information in the file, which helps you become better at your craft since you can quantify every change you make as opposed to just "seeing the difference". Also, what other format allows you to perform complex operations at maximum depth bit without the size of TIFF or DPX sequences? Anyway, I'm not a technical person and I'm sure you and many others can explain and debate, as has been done before ad nauseum, every aspect on the image acquisition path, I just wanted to start a simple, "hands on" thread to actually show how to work with images.

Sir Pancake, again, in layman terms, exposure affects the image uniformly, leading to clipping in highlights or shadows if you get trigger happy with the slider in any onee direction. Brightness works similarly, but when you increase it, it actually compresses the highlights and expands the shadows, sort of pushing all the values more towards the middle. Which is why too much brightness can "muddy up" and flatten the image as you start loosing details at the ends. Always adjust the exposure first, THEN adjust the brightness for more subtle touches. Often, you'll find yourself going back to the exposure and reducing it, after adding some brightness. I don't know exactly how REDCINE works yet, but I would assume that it should produce similar results in the histogram. For what is worth, Photoshop's RAW interface is VERY easy to use and I would recommend playing with it and getting comfortable with RAW manipulation before tackling other softwares. But that's just me.

Any other samples?

[Tom Lowe]

Rudi, my biggest problem is trying to RAW process and CC night timelapse clips, which tend to be very noisy and often underexposed. Here is one frame from my recent shoot at Joshua Tree. I'm also attaching a processed frame I ran through photoshop and converted to a 1920x1080 PSD, then saved for the web as jpeg to give an example of how I handled it when I rendered this timelapse a couple weeks ago. I'm never really happy with the results of my RAW processing efforts, though.

Original RAW frame:

http://www.timescapes.org/arch.CR2

My attempt at CCing it:

http://www.timescapes.org/arch.jpg

If you have any luck RAW processing it and CCIng, could you post a couple samples of how you would do it. I think you can just attach 1920x1080 JPGs, as I have below.

Thanks, man, I appreciate your help very much.

[Rudi Herbert]

Guys,

I copied this directly from the Photoshop RAW help section, a simple explanation of the most important and most used adjustments you'll ever need to get any RAW image to look its best. Again, I'm staying away from curves and more complex stuff, but if you master these 6 adjustments and keep your eyes on the histogram at all times, you'll be producing great images in no time. 90% of all adjustments I ever make are with these 6 tools:

Exposure Adjusts the overall image brightness, with a greater effect in the high values. Decrease Exposure to darken the image; increase Exposure to brighten the image. The values are in increments equivalent to f‑stops. An adjustment of +1.50 is similar to widening the aperture 1‑1/2 stops. Similarly, an adjustment of ‑1.50 is similar to reducing the aperture 1‑1/2 stops. (Use Recovery to bring highlight values down.)

Recovery Attempts to recover details from highlights. Camera Raw can reconstruct some details from areas in which one or two color channels are clipped to white.

Fill Light Attempts to recover details from shadows, without brightening blacks. Camera Raw can reconstruct some details from areas in which one or two color channels are clipped to black. Using Fill Light is similar to using the shadows portion of the Photoshop Shadow/Highlight filter or the After Effects Shadow/Highlight effect.

Blacks Specifies which input levels are mapped to black in the final image. Increasing Blacks expands the areas that are mapped to black. This sometimes creates the impression of increased contrast in the image. The greatest change is in the shadows, with much less change in the midtones and highlights. Using the Blacks slider is similar to using the black point slider for input levels when using the Photoshop Levels command or the After Effects Levels effect.

Brightness Adjusts the brightness or darkness of the image, much as the Exposure property does. However, instead of clipping the image in the highlights or shadows, Brightness compresses the highlights and expands the shadows when you move the slider to the right. Often, the best way to use this control is to set the overall tonal scale by first setting Exposure, Recovery, and Blacks; then set Brightness. Large Brightness adjustments can affect shadow or highlight clipping, so you may want to readjust the Exposure, Recovery, or Blacks property after adjusting Brightness.

Contrast Increases or decreases image contrast, mainly affecting midtones. When you increase contrast, the middle-to-dark image areas become darker, and the middle-to-light image areas become lighter. Generally, you use the Contrast property to adjust the contrast of the midtones after setting the Exposure, Blacks, and Brightness values.[/I]

Now, all we need is to find the corresponding adjustments in REDCINE and we're set. I've tried opening the r3d files I got from Silverado to figure it out, but truth is, it is too unstable and too slow on my computers, so I haven't gotten around it yet, but this is where, hopefully, the many people that have been making great work with RED files for a while now can enlighten us?

[Graeme Nattress]

Joofa, it's very quick and easy to do some transforms from RAW to video that are not reversible though. Most of the time, the colour gamut is reduced when converting to video and code values dropped from the top and tail of the image. Sharpening is added which may be theoretically reversible, to an extent, but not practically.

Yes, a simple matrix transform is reversible, but it would be very common for out of range values to be removed after such a transformation.

If Video -> RAW was always reversible, simply and easily, there'd be no need for RAW.... But I don't think that's anywhere near the case.

Of course, raw bayer data -> RGB is not really reversible unless you use a over-simplistic algorithm in the first place....

Graeme

[Dj Joofa]

Greame, you are right about certain transformations that are not reversible. We were talking about mostly white point conversion. Other transformations have issues not necessarily because of linearity, but because in the transformed space they have:

(1) negative values, (typically YCbCr->RGB), and hence, not physically realizable,
(2) have reduced information becasue of chroma subsampling.
(3) Bayer->RGB is not an issue as long as we don't need to go back to bayer pattern. We can live in RGB space, which is linear light based -- though one penalty will be increased data size.

But, I agree with the gist of what you are saying.

[Graeme Nattress]

Yes, agreed there! Increasing saturation is another one that can lead to negative or too big values that tend to get clipped.

I would be ever so happy to see a move to linear light RGB for data transfer around and inbetween apps, leaving a gamma encode for display purposes only.

Graeme

[Rudi Herbert]

Tom,

Here's the camera information on your untouched RAW:

http://www.reduser.net/forum/uploaded/2969_1206552096.jpg

Wow! All I can say is that I dumbfounded to say the least, with such a wide lens wide open and an exposure of 58 seconds, there's no way this photo should be so dark and noisy! I mean, looking at the histogram, you can see there's basically no information on the mid tones or highlights, and even what you have in the shadows is badly clipped. I will give it a go to see what I can come up with, but this shot looks definitely like it could need another 3-4 stops of exposure. Right now, there's not much you can do with this, so it is not you being unable to make it work, there's not much to work with.

[Graeme Nattress]

There's usually a "long exposure" noise reduction feature on DSLRs that does two exposures - one normal and one with the shutter closed to get a good dark frame to subtract to remove any noise caused by the long exposure itself. Have you got that feature turned on?

Graeme

[Tom Lowe]

Yeah, my DLSR has that noise reduction feature, but it takes as long as the exposure. So, for example, if you take a 45-second shot, it takes the camera 45 seconds to process the noise removal. That won't work for timelapse, because the 45 pause between frames would make the sequence very choppy rather than smooth. Anything more than about a 10-second interval between frames looks choppy.

Rudi, is it simply a matter of exposure time? I could always just set for longer exposures.

[Rudi Herbert]

Tom,

First, there are a few color artifacts all over the sky area, which are the result of having such a wide lens open wide, so I would try to set the lens at 5.6 and that should decrease the noise/artifacting in the lit areas considerably. And yes, it looks like this particular shot could have used as long as a 3 minute exposure, at the very least 2. A friend of mine that was doing timelapses for video was having the same problems and he ended up going for very bright exposures, where the image basically looked like it was shot at dusk, but he had a nice, strong histogram to start with. Then he would just pull back the exposure, which is always easier and less destructive than increasing it, and he was fine. But his rule of thumb was to expose as if it was for an overcast daylight image, and pull back from there.

[Jason Ing]

Tom, what "look" did you want to achieve with the shot? What was important? Obviously the night sky and arch. But what I mean is did you want more detail in the arch? Or have it silhouette against the night sky? If the former, then a longer exposure would have helped. Raising dark/shadows in post will just reveal the noise there. So it's better to lift it during exposure (while being careful to not blow out any detail that you want to preserve in the highlights) and then pull it back down in post. (Of course, there's always noise removal software).

But obviously it's also good to know how much you can save a shot too. 20/20 hindsight advice is helpful for the future, but it doesn't help the shot. :) But that's what I love about RAW.

Maybe I missed your intent (photography is so subjective and personal to me), but here's some quick RAW and PS work. No noise reduction. My noise reduction software is on my other pc at home. Not that different then your PS work. (The third pic is just an attempt to lift the arch without introducing too much noise).

I like your composition, by the way. :)

[Jason Ing]

By the way, you can do so much more with this shot once you get into secondary/masking work. :)

[Graeme Nattress]

Yup, that wouldn't work for timelapse, but it would help with the noise. Maybe if you could take a black frame yourself ahead of the timelapse, it might be good enough to subtract it from all frames of the same shutter speed afterwards, but I don't know of any raw software that would allow you to do that....

Graeme

[Jason Ing]

Graeme, I've never heard of this black frame/timelapse/noise reduction thingy. What is this black frame? I'm assuming it must be more then just a simple frame of black.

[Rudi Herbert]

Very nice samples Jing!

I had one go to try and "fix" the image as much as possible without going for a look but for a better histogram. I decreased saturation on all the colors that, to me, were not essential to the shot and played with Exposure et al, and this is what I got, which, though not a bad looking image, is not as creative or pleasant as Jing's examples. The shot is just too badly underexposed, though for something like TV, you could get away with any of the looks Jing proposes, or many others. On a big screen, or at HD1080, you would get noticeable noise though.

http://www.reduser.net/forum/uploaded/2969_1206558384.jpg

It is still too dark, but these are the highest noise levels I could work with while still having hope to be able to sell it for NTSC broadcast. Your composition is nice!

[Jason Ing]

Thanks.

Personally, I don't think any of the samples are "too dark", tom's or rudi's. (the original RAW is too dark, of course).

I consider it just a "low key" shot. I guess that's what I meant by photography being subjective and personal. I can see this shot (even the darker versions), in a quick establishing shot in a movie. "Too dark" to me is when I can't get the information or understand what's going on. But in all the shots I can see that there's a beautiful starry sky and a stone arch that would be fun to walk on... :)

FYI: There was no vibrancy/saturation bump. Just temperature/tint moves in RAW. Instead of going neutral, I opted for more blue in the highs. One pic had a color dodge layer applied in PS. But that's about it for color.

[Tom Lowe]

Yeah the shot is waaay underexposed. The trouble is, there was no moon at all, so it was super dark out there. Plus I was about 5 beers into a six pack, waiting the stupid airplanes to clear out of the sky. :) I really have to try to guess at exposure times, and in this case, my LCD was probably set too bright, because it showed a decent exposure. (I think I forgot to turn down the LCD brightness :)). Next time I'm shooting with no moon I will try a much longer exposure and see how that works out.

Thanks for the advice and everything. I really need to learn more about how to read histograms.

[Mike Prevette]

Keep in mind Rudi that its a night shot. IE 80% black, there really shouldn't be much more meat to the histogram, and if there was it would no longer "look" like a night shot.