"HDR MODE" - Possible on Red?

[Andrew M.]

We did few experiments with two cameras side by side, locked in sync.
We posted some early results of this experiment here on this forum.
Well, for some shots in wide open spaces the 3D effect was negligible.
For anything filmed from less than 150’ the spatial shift was not acceptable.
We couldn’t combine two frames with enough of edge accuracy.
One camera was running +2 stops the other -2 stops.

I am still thinking that two subsequent frames with the different shutter speed but double the frame rate for some less dynamic shots would be acceptable and strobeing effect would be negligible.

[Joe G.]

"I am still thinking that two subsequent frames with the different shutter speed but double the frame rate for some less dynamic shots would be acceptable and strobeing effect would be negligible."

That was my original pitch. The first frame takes the entire 360 degrees, and the next frame takes the continuation of time, for another 90 degrees or so. The motion is continuous across the two frames with no jump.

In a lot of situations that would be perfectly fine.

[David Mullen ASC]

I didn't understand what you were describing, but anytime you combine two sequential shots into one shot, then a moving object occupies two different positions in each element you are combining -- there's no way to say that the motion would be continuous since it basically can't be -- they are separate exposures, which is why you are getting the HDR effect. Saying that with minimal motion, you can get away with it ignores the fact that movies have all sorts of motions in it. You can think your actor is hardly moving but then they pick up a plate quickly or suddenly stand-up.

The only way it's going to work for motion pictures, other than static shots of static subjects, is a technique that involves simultaneous exposures, not sequential exposures.

[Eki Halkka]

I didn't understand what you were describing, but anytime you combine two sequential shots into one shot, then a moving object occupies two different positions in each element you are combining -- there's no way to say that the motion would be continuous since it basically can't be

You forget that digital cameras can have a 360 degree shutter. The motion continues without breaks (read-reset may cause a slight break, but that should be unnoticeable).

I actually wrote down a pretty detailed proposition of the technique last year - i posted it somewhere here, but can't find the thread right now, so here's my essay again:



Timeslice subframes: SNR / DR improvement with firmware and software modification for temporal supersampling

By Eki Halkka, 17. Nov. 2008 (Edited 20. Nov, 23. Nov)




Intro

The idea is to use "exposure timeslice subframes" to store more information for each frame, using temporal supersampling - a bit like adding more cores to a processor instead of making it clock faster, if you may.

Dynamic range can already be expanded with temporal supersampling using current tools - by shooting at higher frame rate and combining the frames in post. Simply averaging two frames together reduces noise in half, or in other words adds 6dB to signal-noise ratio (SNR). Adding the frames together adds one stop of dynamic range (DR).

Unfortunately, this is a rather complicated procedure in practice right now.

What i suggest, is to make temporal supersampling an integrated part of RED work flow. Ideally, this should be done with the raw linear sensor data, before debayer.




Example case

For the sake of simplicity, i'll use a target of 25 fps and 1/50 shutter (nice round numbers) on the following example of a practical implementation:

The camera sensor is set to run at 100 fps, 1/100 (360 degree) shutter speed. The exposure is set properly for this, not the targeted 25 fps, 1/50 shutter speed, thus preventing overexposure by one stop.

Out of these originally captured 100 fps frames, or exposure "timeslices" as i like to call them, each group of four frames would be treated as “subframes” of a final 25 fps frame.

Captured frames 1 and 2 of each group would be saved, but with a flag that describes them as partial frame 1a and 1b. Captured frames 3 and 4 would be discarded.

The result would be a 50 fps stream, but the extra frame rate would be used to store twice more of intra-frame precision, not more temporal information.

If possible, the unused subframes (3 and 4) would actually never be captured at all – the sensor would have a bit of time to cool down, maybe it could help to implement this at higher target frame rates.

Supersampled subframes give a lot of new flexibility in post production:

A) If subframes 1a and 1b are averaged together, the result is an 1/50 shutter speed frame which has 6dB lower noise, but the same dynamic range.

B) If subframes 1a and 1b are added together, we end up with an 1/50 shutter speed frame that has the same noise level, but one stop higher dynamic range – it's essentially a 13 bit accumulation of the original two 12 bit images, reaching sensor saturation one stop later. This will naturally require the post production software to be able to hold values that “go over 100%”, floating point style.

C) Any blend of the above is also possible - they're two sides of the same phenomenon.

D) One bonus possibility is to simply discard one of the two sub frames - user could modify shutter speed in post between 1/50 and 1/100 - at the cost of losing the improved SNR / DR.

E) If all four captured 100 fps timeslices were saved as subframes instead of just the two in the above example, user could have also 1/33 and 1/25 shutter speeds at his/her disposal, trading in more motion blur and capture storage space for additional DR and SNR in the resulting image, making the blur-noise trade-in decision in post rather than on the field.

F) Timeslice subframes might also open up other interesting possibilities, i.e. isolating moving areas of the frame by difference matting etc.

G) As a bonus, this method would work as an "electronic ND filter", the aperture could be opened by a stop without overexposing.

Summing it up

+ At least 6dB less noise (12 dB with four subframes)
+ At least one f-stop better dynamic range at original noise level (two f-stops with four subframes)
+ Ability to control shutter speed in post (i.e. 1/100, 1/50, 1/33, 1/25)
+ Capture proper motion blur for video and faster shutter speed for stills simultaneously.
+ Can also be used as an "electronic ND filter"
+ Should help to minimize compression artifacts by averaging them over frames
+ Should be doable with current hardware, with minor changes in firm- and software.
+ The technique scales up as technology evolves

- With Red One, the technique probably only works in windowed modes
- The technique only works at relatively slow target frame rates
- The shutter speed variation is limited
- The storage requirements increase
- The post processing burden increases
- The technique improves the dynamic range "from the top up" by preventing overexposure - there's no low light sensitivity gain.

Doable with current technology

Red one can shoot 2K up to 120 fps, which is enough for the above 6dB / one f-stop improvement with frame rates up to 30 fps. Depending on what the frame rate bottleneck is (sensor, the electronics, redcode, other..), it might be possible to use three or more subframes instead of two in a "burst mode", for an additional 3dB / half stop improvement, especially as the sensor has time to cool down between bursts.

Using this method would require adding a new timeslice/subframe recording mode to the camera's firmware, with the flags i described above. As there's probably not enough processing power in the camera to do proper frame blending etc., the real time preview should be modified to account for this method - possibly simply by adjusting display gain if needed.

The post production software would also need to be modified to read the flags and combine frames from the new recording mode, in the ways i described above. If individual frames cannot be flagged in redcode, it would probably be enough to simply to have "footage uses two subframes" flag in the clip's meta data.

But that's it - that's all that's needed to make this work, as far as i can see.




But wait, there's more

In the future cameras, faster sub frame rates and even bigger improvements could be obtained. If RED ever comes up with a 1000 fps high speed camera, the high temporal precision could be used to store many, many additional stops as timeslice subframes - the camera would in fact classify as a true high dynamic range motion camera.

Another highly desirable option would be to use time slices with varying shutter speeds - the same way as HDRI is currently captured with still cameras. I assume this would require much bigger modifications though - but the potential benefits would also be big. In addition to capturing HDRI, the same footage could be used for video with a (relatively) slow shutter speed partial frame and for stills with (relatively) high shutter speed partial frame - the trade off is differences in exposure, (still frame would be underexposed), but the benefit of usable, sharp stills of moving subjects might be worth it.

[David Mullen ASC]

I'm not sure why adding up frames all shot at the same exposure would expand the dynamic range captured. The RAW file of each frame in this case would have the same information in it, so how does adding it up increase that range? Unless all you really mean is that doing this allows you to underexpose one-stop but then reduce the noise in the shadows by adding the exposure from the other frames.

Still, all of this for one more stop of DR, at the cost of losing a stop of speed? So your 320 ASA Red now is a 160 ASA camera effectively?

I can understand the notion of two shots with no shutter lapse being treated as a continuous exposure, but this precludes the notion of shooting those two shots at different exposures, which some people are suggesting, because as the object moved from A to B in that combined frame, it would get darker or brighter halfway there

Also, I don't think many people want to drop to the lower resolution of 2K RAW on the Red unless absolutely necessary. And sticking to 4K at 100 fps is quite a lot of data to manage in post.

[Eki Halkka]

I'm not sure why adding up frames all shot at the same exposure would expand the dynamic range captured.

The sensor is just a counter, keeping track of how many photons hit any given photosite on the sensor. It's like a bucket that starts to fill, and at some point it's full - the sensor is fully saturated, clipping to white.

A dark part of an example image may have the bucket 10% full, while the bright part is 90% full, and highlights clip to pure white at 100% (they would have gone to 150%):

..........
.................................................. ........................................
.................................................. .................................................. |clipping


If you take two frames instead, using half the shutter speed, you end up with two 5% buckets in the dark parts, two 45% buckets in the bright part and two 75% buckets in the highlights:

.....
.....
.............................................
.............................................
.................................................. .........................________________|no clipping
.................................................. .........................________________|no clipping

When you add these two frames together, you get the same 10% in shadows, the same 90% in the bright parts, but the highlights don't clip at 100%, they go to 150% (the clipping would happen at 200%):

..........
.................................................. ........................................
.................................................. .................................................. .................................................. ____________________________|no clipping

In other words, you have one extra stop of dynamic range.

Still, all of this for one more stop of DR, at the cost of losing a stop of speed? So your 320 ASA Red now is a 160 ASA camera effectively?

No. It would still be a 320 ASA camera, but you would have the highlights clipping one stop later.

The combined shutter speed for i.e. 25 fps target would be the normal 180 degree 1/50 (1/48 for 24 fps). It would just be added together from two 1/100 (1/96) shutter speed frames.

Even if dynamic range is not a problem, an added benefit would be less noise, in situations that have enough light: as long as you can get proper exposure at 1/100, you would have two properly exposed frames instead of one. Averaging these two together also averages the noise to half - this could be very useful i.e. for greenscreen shoots, especially in tungsten light.

Also, I don't think many people want to drop to the lower resolution of 2K RAW on the Red unless absolutely necessary. And sticking to 4K at 100 fps is quite a lot of data to manage in post.

In my suggestion, two frames from each group of four would be discarded, resulting in 50 fps (48 fps) stream. It's twice the data rate, but there are cases where that extra stop of DR would come to critical use and that would be acceptable.

And of course, the new cameras are very soon available, with the ability to do higher data rates at full resolution.

The method i suggest can be put to use with the current camera, simply by shooting at four times the target speed, 360 degree shutter, and combining the images in post. It requires a few extra post steps but it seemed to work fine when i (briefly) tested it.

[Noah Yuan-Vogel]

I posted a thread with a similar idea a while ago. So I agree that this would be a great feature to have or at least for RED to test in terms of increasing dynamic range/reducing noise and easily allowing variable framerates and shutter speeds in post. I think perhaps a 3k native camera would be better suited for this, and it might also be worth implementing a lower bit-rate codec to make this option more attractive.

Another idea that may (or may not) be more feasible than creating HDR from two frames that do not match temporally or from an additional chip installed with a beamsplitter might be to just replace the Bayer and olpf filters. Seems possible that one could, make a new filter pattern that would put a green and a green+nd filter next to each other and red and red+nd filter next to each other, etc. or next to each block of grgb (or whatever it is) put a grgb with ND filters over them. in either case I'm sure both would require a stronger OLPF to avoid increased aliasing. These pixels could be combined in camera (no debayer, just simple adding of pixels, but higher bit depth codec might be required) or done in post. The benefit of doing it in camera is that you might be able to get away with combining two 12bit pixels and recording to 16bit values, which would save space, since there would be half as many pixels to store and only 33% more bit depth needed to store the HDR info adequately. Also, itd probably be much easier on the codec to store half the pixels at higher bit depth than a full-raster image with a checker pattern all over it. Of course the light and dark frames could be stored separately in the codec as well and combined during decoding. I'd gladly sacrifice some sharpness and half of the 4k sensor's pixel information to add 4stops of dynamic range. But then I'm also someone who doesnt feel that strong a need for shooting 12mp video as 6mp is more than plenty for me.

Anyway just came up with that idea 5min ago so any thoughts?

[Eki Halkka]

Anyway just came up with that idea 5min ago so any thoughts?

I recall reading that Panavision had something a bit like that under developement a few years ago. To me it sounds like a good idea, i see no reason why it wouldn't work.

If the two pixels were read at 12 bits, and combined in-camera, it would only require 13 bits to save all the data (each new bit doubles storable info). Oh, BTW, i don't know for sure, but i've always assumed that also the raw sensor data is actually saved to file for each color separately (as a big green chunk, blue chunk, red chunk), not as a checkerboard (RGGB, RGGB, RGGB, RGGB), then reconstructed when playing back. That should be more efficient for compression.

[J. Eric Camp]

The only ways a single chip will deliver HDR is:

Pixel binning of varying sensitivity pixels.

or

if the controller board could pull multiple frames out of one exposure.
ie separate images are complied simultaneously from different exposure times.
So at 24fps
Frame 1 starts.
Frame 1a is pulled at 1/192
Frame 1b is pulled at 1/96
Frame 1c is pulled at 1/48
Frame 1d is the full 1/24

However you would still have motion artifacts with this, albeit less than a high frame rate combination. Also this would actually give you DR gains versus over clocking the sensor for no extra sensitivity.

As David says. They only truly acceptable way for motion to gain HDR would be through simultaneous exposures (and exposure times). This can be combined on one sensor or if you use an existing 3d rig, or make a new device similar to a 3ccd camera with a beam splitter but with 3 fully contained CMOS bayers.

[Andrew M.]

Properly done downers from 4K to 2K theoretically should give you one stop extra.
Also proper combining of 4 frames in to one should give you another stop extra.
Here temporal movement between frames may help or may not help to average down the background noise.

However I am not sure how ground noise level will combine after going through A/D converter.

Remember, we are not combining analogue values from the sensor, we are processing it as a digital zeros and ones in post and as such you will need not 12 bit A/Ds but 14 bits A/D to start with and the noise level of 4 buckets combined digitally may not be adding much of dynamic range here.
Nothing will replace just 4 times bigger surface of the pixel.

The best will be to try it out.

Andrew