Experimenting with Wavelet Compression

[I Bloom]

I recently started experimenting with wavelet compression by taking RAW images off my DSLR and compressing them with JPEG2000 at roughly the ratio that RED advertises. Then I've taken both the raw image and the compressed one into photoshop to do transformations such as adjusting the levels on an underexposed shot or boosting the saturation way up. Then I do a side by side comparison.

The reason I'm interested in this is to get a better understanding of how a camera like the RED ONE might perform when we do some serious grading after the image has been compressed. I have to say the results I got made me very optomistic about shooting compressed 4K. Though I don't think my methods were very scientific, at least not yet.

I'm wondering if anyone else has explored this area. Or would like to propose a specific set of tests that would open our eyes to just how much can be done to the 4K compressed images in post.

Ian Bloom
http://www.ianbloom.com

[Brook Willard]

You could just use the RED stills that have already been released [there are TIFFs...].

[J. Bernard Vallon]

The results out of REDCODE should be theoretically better, because unlike JPG2000, redcode compression keeps the information in a 12 bit color space (jpeg and jpeg2000 transform to an 8 bit color space)

Adjustments like levels and curves preform much better on larger color spaces. Thats the primary advantage.

[MikeCurtis]

Also keep in mind that JPEG2000 is compressing an RGB image, whereas Redcode RAW will be single channel compressed - a 3:1 bandwidth advantage right there.

....and as PerfectOptics mentions, there is the bit depth advantage too - but that bit depth takes up more file space...

-mike

[David Newman]

JPEG2000 can support 4:2:2, 4:4:4, 12 bit or whatever. It is a nice format, but the features depend mainly on the implementation. Even if you have only 8-bit RGB there are still things to learn. Mike is correct, you will only be able to model REDCODE RGB as RAW is not supported in the JPEG2000 standard.

[I Bloom]

You could just use the RED stills that have already been released [there are TIFFs...].

Right but how can you tell without a precompressed original raw image in 12 bit to compare it too. Plus not seeing the compression "grains" in motion is really important to understanding exactly how the image is holding up.

[I Bloom]

The results out of REDCODE should be theoretically better, because unlike JPG2000, redcode compression keeps the information in a 12 bit color space (jpeg and jpeg2000 transform to an 8 bit color space)

Adjustments like levels and curves preform much better on larger color spaces. Thats the primary advantage.

Wait isn't it 10bit log when you compress it? I thought twelve bit was only straight raw.

[Brook Willard]

REDCODE RAW = 12-bit lin, 10-bit lin or 10-bit log... your choice.

The TIFFs that were released are available in uncompressed or REDCODE compressed. If memory serves, it's the older 10-bit REDCODE and the older/narrower dynamic range in camera. So things have changed so much since then that... well...

[Hans von Sonntag]

Hi ibloom,

Thought you might be interested in a comparison David Newman (posted above) made recently. On his blog cineform.blogspot.com there is an entertaining comparison of uncompressed, HDCAM SR and Cineform 444 badly performed greenscreen footage you will never be shooting your self (hopefullly). Take the challenge, try by your self and find out which is which. I could not tell. Great stuff!

Thanks David!

Hans

[David Newman]

Hans,

thanks for the plug for http://cineform.blogspot.com, and thanks for trying my Green Screen Challenge. There have been many hundreds on downloads on the DPX Viper frames, but not massive number of guesses as I think many like yourself will not be able to pick the uncompressed form the compressed. Those who have tried, in general have failed to guess correctly. I would also like that feedback from users how couldn't tell, help present the case that good compression is an alternative to uncompressed. I'll put you down for "c) I can't tell the difference; are you sure these aren't all uncompressed?"

[GlennChan]

At the risk of putting myself at risk of foot-in-mouth disease... and to throw a wild idea out there...

It might be that the cineform version (the original one, not the "greenscreen" version) makes the image slightly easier to key- not harder. This is what I've seen with Vegas/Cineform on HDV footage... the cineform wavelet compression cleans up the noise slightly. Wavelet compression tends to smooth out the image and smooth out the noise in an image.

In practice, this likely makes negligible difference for greenscreen since you would apply noise reduction to the key anyways... i.e. via matte choking. For color grading apps, there might be a small advantage to wavelet compression; with most of them, you won't be applying matte choking or noise reduction on the keys. At the same time, it's not that big a difference (and with the Viper footage, virtually no difference).

2- To me, having run a difference composite on all four images, it seems that it's somewhat easy to tell which is HDCAM and which is cineform optimized for greenscreen. Although you'd have to think about the artifacts to figure out which is what (i.e. figure out how DCT artifacts might look like when you run a difference composite, and figure out what "optimization" Cineform is doing for their greenscreen version).

3- In practice, there are much worse greenscreens. Shadows in the green (this happens easily on puppet shows, where the puppeteers dress in fabric suits), hair, footage shot on worse formats, reflections, etc. etc.

[Tom Lowe]

hey david, totally off topic, but since there is not a cineform thread here yet, is there any chance that Aspect HD will ever be able to work in true 1080p? i love your software so much, i have become addicted to using it in Premiere. I encode all my DLSR timelapse stuff to cineform AVIs, then drop them into my Premiere Pro timeline. The problem is, I want to CC and output 1080, rather than be confined to 720.

Or will I have to wait and upgrade to Prospect when your 4K solution gets worked in? I'm assuming that will be a couple thousand bucks, right?

[Hans von Sonntag]

At the risk of putting myself at risk of foot-in-mouth disease... and to throw a wild idea out there...

It might be that the cineform version (the original one, not the "greenscreen" version) makes the image slightly easier to key- not harder. This is what I've seen with Vegas/Cineform on HDV footage... the cineform wavelet compression cleans up the noise slightly. Wavelet compression tends to smooth out the image and smooth out the noise in an image.

3- In practice, there are much worse greenscreens. Shadows in the green (this happens easily on puppet shows, where the puppeteers dress in fabric suits), hair, footage shot on worse formats, reflections, etc. etc.

Hi Glenn,

this is interesting and I would like to know David Newmans comment on this.

Met Patrick Plamer from Iridas just some days ago. We had a short chat on wavelet compression and he said exactly what you are saying: Wavelet compression does slightly smooth out the noise of an image. Wether this is better for keying is questionable, since I find a better to de-noise the stuff by my self if it makes sense. Anyway, wavelet compression is much better than any DCT kind of compression. Different worlds.

Sure there are much worse and more difficult shots to tackle. But would you wear a green shirt in a greenbox? Found it funny that David put on a green shirt on purpose. Kind of nerd terrorism.

Hans

[SalaTar]

I think David may suprise us at NAB

[David Newman]

Glenn,

In theory wavelets can help in some image processing needs as it is a good noise filter, yet keying is one of those cases where compression filtering may be undesirable. Also the bit-rate which these tests were run doesn't result in any visible noise suppression, otherwise would be much better at determining compression from uncompressed.

Plus if the green screen challenge is easy please submit your guesses -- others are no finding it so. It is best to be on record when the results are revealed.


Tom,

Aspect HD does 1080p, you are not forced to do 720p. You may be confusing that Aspect HD is focused on HDV/DVCPRO-HD which has a maximum resolution of 1440x1080 (with a 1.33:1 pixel ratio.) 1440x1080 is perfectly fine for the range of sub-$10k sources, you can export out uncompressed at 1920x1080 if needed for film-out or HD-DVD mastering. Yes, Prospect HD can edit natively at 1920x1080.


Hans,

Wavelets at heavy compression have a softening effect with noise suppression, with light compression the soften and noise reduction is effectively none, i.e. looking very like uncompressed even under magnification.

Re: "Kind of nerd terrorism."

I like it.

[GlennChan]

1- I think I've probably confused the Cineform image with the uncompressed image. Now looking back on my guess, I'd like to change it. (I believe David mentioned that he would like results submitted privately so that you don't influence what others guess; I hope I haven't influenced anyone unduly with my comments.)

2- With Sony Vegas, encoding clips into Cineform does seem to clean up the noise. I didn't do a rigorous test, but just encoded a HDV clip into cineform then took a whack at it with the secondary CC. Vegas' signal path would be different than what's presented here though; HDV --> 8-bit R'G'B' (16-235) --> Cineform / Y'CbCr (4:2:2?) --> RGB.

3- Red's footage looks a lot cleaner than the Viper footage. It seems to have a lot less noise.


Of course, I could definitely be wrong!!

[David Newman]

1. The result as now up on the blog : http://cineform.blogspot.com and yes Glenn your second guess was correct.

2. The vegas version of the CineForm codec share little in common with the codec we test, Vegas 8-bit 4:2:2 YUV, vs 12-bit 4:4:4 RGB.

3. Read the blog to see why. It is to do with the curves applied more than one sensor being noiser than the other -- although Red could very well be clearer.

[GlennChan]

Glenn your second guess was correct.
I guess I'm not that smart, and not that not-smart either. Thanks for putting the test up David, it was a ball-buster (!).

Originally I assumed that Cineform would reduce noise (because wavelets reduce noise right?)... i.e. differencing Cineform and uncompressed to HDCAM SR, Cineform would have less noise. But that wasn't really the right way to look at it. The wavelet compression artifacts are "noise" (and increase the peak difference between HDCAM and CF/uncompressed), and reduce the visual appearance of noise. Hence me changing my mind. And in a way, it was cheating.

A numerical example:
Suppose you take all the pixels where the original has a value of 5.
Suppose CF adds variation (vs the original), so the compressed pixels have values of 4, 5, or 6.
Suppose HDCAM adds variation, so the compressed pixels have values of 3, 4, 5, 6, or 7.
If you find the difference between HDCAM versus CF, you get values ranging from 3 to -3.
If you find the difference between HDCAM and uncompressed, you get values ranging from 2 to -2.
3 to -3 is a bigger range than 2 to -2, so that's how I guessed which one was CF.

If the test had been between CF and uncompressed, you couldn't do this numerical trick. You'd have to guess back on keying cleanness or some other factor. My personal guesses/opinions for keying were:
Uncompressed was best
Then cineform (all channels equal)
Then cineform (R and B quantized more; I thought this was the version of cineform optimized for greenscreen)
Then HDCAM.

So what I'm saying is that I still would've guessed incorrectly, and then correctly (after a lot more pondering and squinting).

2- David: The version of Cineform with all channels with equal quantization is the version optimized for greenscreen right?

3- It is to do with the curves applied more than one sensor being noiser than the other
I've always assumed that Photoshop (and Imagemagick) perform a log to gamma(2.2?) conversion. (The Imagemagick documentation states that it converts log to linear, where linear implicitly means gamma corrected.) So most people using Photoshop are looking at the pictures (red and viper) on similar footing??
*Not sure what image processing Red used for their images and I think I may be missing important details... but bottom line is they look cleaner.

4- My (perhaps uninformed) opinion on keying quality isn't to rag on Cineform's image quality. Ranking the image really distorts the reality, which is that there is no practical difference in image quality between uncompressed and Cineform (IMO). HDCAM SR is damn good too, although if you are to split hairs it is a tad inferior.

5- In practice, there are way worse greenscreens.

One example would be Lord of the Rings... their production dairies talk about shots where the greenscreen didn't extend far enough. Which isn't even really a greenscreen at all. And therefore the roto artist has to roto the talent, her hair, and leaves/foliage (!).

[David Newman]

Very clever. Of course initially I didn't care whether people could pick which clip was which, wanting to know if there any impacts on keying; by not telling which is which I wanted to remove any bias people do have. But it came very clear that the keying was the same, and people prefered to discover the unknown.

2 - Yes, all channels of equal equalization is better for keying, although the 20% increase in data rate has no impact on the visual quality.

3 - The curves are amplifing the shadows, preserving their details (the Viper is know for seeing into the night), yet this also amplifies the noise. To normalize the images, you can reverse the Filmstream curves, whitebalance, add a saturation matrix, then apply a display LUT (like 2.2.) Most of the noise will vanish.

My new stuff on my blog : http://cineform.blogspot.com