4k Tri-Sensor... Epic?

[Patrick Tresch]

I've just read an article of InCamera from Kodak speaking about 4k single sensor VS tri-sensor. Not much infos, but I thought they wanted us to come to some conclusions about why 4k single sensore are not so good...

"What is the take-away lesson for filmmakers? One lesson is that we shouldn't assume that the widespread acceptance of Bayer CFA sensor solutions for consumer still photography means that this approach will work well in cinema applications..." InCamera/Kodak/October2008/p.32

Funny to think that Kodak has the CMO bayer patent.

Here you can read the full paper of the study led by Senior Scientists at the Eastman Kodak Company :

http://motion.kodak.com/motion/uploadedFiles/Wheeler_Rodriguez_SMPTE_J-Oct07_Web.pdf

Have a nice read.

Patrick

[Graeme Nattress]

NOTE:
4K Tri-Sensor camera (versus a Single
Sensor 4K type) would require
considerable
--higher Bandwidth/Storage and
--larger size/weight, as well as being
--incompatible with Film Camera Lenses

[Patrick Tresch]

I like your note Graeme!
And I believe RED will design the most apropriate tool for film making.

Patrick

[Dan Hudgins]

NOTE:
4K Tri-Sensor camera (versus a Single
Sensor 4K type) would require
considerable
--higher Bandwidth/Storage and
--larger size/weight, as well as being
--incompatible with Film Camera Lenses

In a three sensor camera you can control the bandpass for the colors better, not that three chip cameras do since they want the most speed, but if you were willing to live with EI 25 you could get spectacular color.

The problem is, now people are running good film scans through 3D LUT to muck up the color, so it would be insane to make a nice 3 chip camera if all that work and cost would end up looking like "budget" lab after a bleach bypass hangover...

[Graeme Nattress]

The note was copied from the article.

Graeme

[Pawel Achtel]

What I found interesting is that, with a pixel size below 6 microns, the resolution is diffraction limited at around f/5.6. This means, that if the pixel size decreases (like in the 5k and beyond) there is no improvement in sharpness (MTF) even with a perfect lens stepped down to f/5.6 or more!

Considering that most lenses peak their MTF at around f/5.6 - f/8, there is no significant benefit in increasing resolution beyond 4k-5k using S35 format. We are maxing out the resolving power of even perfect optics, basically.

The only way to achieve higher resolving power with 4k+ S35 sized sensor would be to use (fast and sharp) lenses that produce higher MTF at apertures wider than f/5.6 diffraction limit.

There is a good article from Bob Atkins explaining this limitation:

http://www.bobatkins.com/photography/technical/diffraction.html

[Charles Angus]

I think there's a thread on here explaining the importance of increasing pixel density even if you are not gaining sharpness any more.

If I understand correctly, the main benefit is it means more dynamic range.

[Graeme Nattress]

Higher pixel density would lead to lower dynamic range, not greater. But you'd get less aliassing.

Graeme

[Daniel Browning]

What I found interesting is that, with a pixel size below 6 microns, the resolution is diffraction limited at around f/5.6. This means, that if the pixel size decreases (like in the 5k and beyond) there is no improvement in sharpness (MTF) even with a perfect lens stepped down to f/5.6 or more!


At 5K and f/5.6 the improvement is still big. Huge, in fact. Get a 50D (which is almost exactly 5K) and see for yourself.

Diffraction limited doesn't mean there is *no* improvement from more resolution, it just means the improvement will be *smaller* than if it would have been if the aperture weren't diffraction limited. I.e., instead of a 50% improvement in resolution, it's only 40%.

Even when the lens significantly limits the MTF of the system, the extra resolution of the sensor is very useful. For example, many artifacts such as aliasing (moire, jaggies, sparkles, etc.), debayer artifacts, hot pixels, dead pixels, and more all occur at high frequencies, close to the pixel. Resampling the image automatically removes all of those artifacts for you in your final image.

Read noise is less noticeable when it is made finer through more resolution (like smaller grains), which improves the quality of the final image. That's especially true if the per-area read noise level goes down, as it has with every new sensor with higher resolution. It might not be very significant in lower resolution outputs.

[Daniel Browning]

Higher pixel density would lead to lower dynamic range, not greater.

Finally! The opportunity for me to disagree with you. It has been boring to have so much in agreement (probably because I learned a lot if it from you).

I think higher pixel densities does not reduce dynamic range as long as read noise, quantum efficiency (QE), and full-well capacity (FWC) all decrease by the same amount as the size of the pixel. Furthermore, I observe that DSLR cameras have almost always followed that trend or surpassed it (increasing dynamic range at the same time density is increased). However, I acknowledge that in the future, at some point, technological challenges such as trapped carriers (causing RTS noise) will cause this trend to slow.

Here's an example of what I'm talking about.

Camera A is 2K, with FWC of 200,000 photons and read noise of 60 electrons/pixel (or equivalent millivolts).

Camera B is 4K, with FWC of 50,000 photons (1/4th) and read noise of 15 electrons (1/4th).

On the surface of it, it might seem like the 2K camera has more dynamic range. After all, each pixel can capture four times more light before it clips. But each pixel in the 4K camera takes the space of one pixel in the 2K camera, so they're only going to get one quarter of the light anyway, so it wont clip.

200,000 photons / 60 electrons read noise = about 11.5 stops of dynamic range.
50,000 / 15 = the same.

Another important factor is resampling. Random noise adds in quadrature. So when you resample 4K down to 2K so you can compare dynamic range with the 2K camera, the signal-to-noise ratio doubles, increasing dynamic range by a full stop. (Going 3K to 2K would only increase dynamic range by 1.4X.)

So in this example, camera B actually has one more stop of dynamic range than camera A, but only after resampling it down to the lower 2K resolution.

[Pawel Achtel]

At 5K and f/5.6 the improvement is still big. Huge, in fact. Get a 50D (which is almost exactly 5K) and see for yourself.

Diffraction limited doesn't mean there is *no* improvement from more resolution, it just means the improvement will be *smaller* than if it would have been if the aperture weren't diffraction limited. I.e., instead of a 50% improvement in resolution, it's only 40%.


Big? Huge? Is, what 2%?

Diffraction limit is diffraction limit - there is basically insignificant or no improvement in MTF by reducing pixel size below 6 microns when shooting at f/5.6 and beyond. I haven't tried (and don't want to try) 50D, but laws of physics are unavoidable. No point spending $1.5k to attempt to show otherwise.

This is not to say you can't gain sharpness out of 5k+ sensor by using faster and sharper lenses. In fact, when projecting Master Primes, I was nicely suprised they held up 120 lp/mm corner-to-corner open wide at f/1.2 (T1.3).

But then, 120lp/mm is about ~5k anyway, so if not diffraction limit of the lens, it is the limit of other aberrations of the glass that we hit at this resolution.

I'd hate to pull focus with CoC of 4 microns at f/1.2.

[Joseph Ward]

I tried finding if it mentioned if rolling/global shutter and other differ read-reset times, price for tech, ect., that would contribute to the final results? Ohh wait its by Kodak!

[Stephen Williams]

Ohh wait its by Kodak!

Hi,

I use a great deal of film but not much Kodak as I prefer Fuji. I have always found Kodak analysis to be accurate, remember they invented the Bayer chip so have no axe to grind, just trying to get the best out of any technology.

Stephen

[Graeme Nattress]

Daniel, as you know I'm always open to a good discussion!

Sure, if you scale things, the DR of the 2k camera will be the same as the 4k one. However, and I'm not a sensor guru remember, I don't think pixel design quite scales linearly like that. And that's where we'd have to have a guru educate me further.

In the end though, it's total area of sensor used to capture light that really defines the maximal dynamic range of a sensor. What higher resolution does is allow you more flexibility to do some trading.

Graeme

[Dj Joofa]

Another important factor is resampling. Random noise adds in quadrature. So when you resample 4K down to 2K so you can compare dynamic range with the 2K camera, the signal-to-noise ratio doubles, increasing dynamic range by a full stop. (Going 3K to 2K would only increase dynamic range by 1.4X.)


Hi Daniel,

That is just one side of the picture, and not a full analysis. When you do such resampling as you described, then actually the SNR is determined by two factors:

SNR = (SNR_b decrease from blurring) + (SNR_n increase from denoising)

I have mostly seen people talking about the SNR increase from denoising (viz., random noise adds in quadrature if considered uncorrelated, etc.). However, unfortunately, many times the effect of SNR decrease from blurring of data because of the window size is not taken into account.

Since, SNR_b is decreasing and SNR_n is increasing then there must an optimum point where the sum is maximum, for max SNR. Therefore, it becomes an optimization problem to find the appropriate parameters for max SNR.

Given a particular window type (when you added noise in quadrature you actually assumed rectangular window), explicit close form solutions may be derived that relate the optimum resampling parameters.

[Daniel Browning]

Diffraction limit is diffraction limit - there is basically insignificant or no improvement in MTF by reducing pixel size below 6 microns when shooting at f/5.6 and beyond. I haven't tried (and don't want to try) 50D, but laws of physics are unavoidable. No point spending $1.5k to attempt to show otherwise.


I wrote a lengthy response in a new thread, because it seemed a little off topic for this one.

[Daniel Browning]

I'm not a sensor guru either, I just measure RAW files. :-)



I don't think pixel design quite scales linearly like that.


You're right. That was a bad example because real life isn't like that.

The point I should have made was that read noise doesn't have to scale linearly in order for dynamic range to stay the same: it only has to scale with the square root of the decrease in area. That is, change camera B in the above example from "15 electrons" of read noise to "30 electrons" and the 1 stop dynamic range advantage goes away and both cameras become the same. So for every fourfold reduction in size, the read noise per pixel only has to be reduced by half to keep dynamic range the same.

What I've seen is that for every new sensor with smaller pixels, read noise per area (not per pixel) has stayed the same or decreased (sometimes only at high analog gain, but still it's there), with few exceptions. The next question is: why?

Shrinking the pixel is a leap in technology. I think that just the act of miniaturizing the pixel by a factor of four has the natural effect of reducing read noise by almost half. So, dynamic range is the same (or slightly less). Then, you add some other technology improvements such as a new DCS circuits, better ADC, microlenses, etc. and the dynamic range actually goes up.

If my guess is wrong, then it means that read noise and dynamic range have been improving only because of new and separate technology and not in part as a natural consequence of shrinkage.

Furthermore, I think the natural read noise reduction will become less effective as sizes get smaller (again, due to things that don't scale, such as trapped carrier RTS noise).

On a related note, a paper presented by G.Agranov entitled Super Small, Sub 2μm Pixels for Novel CMOS Image Sensors (http://www.imagesensors.org/Past%20Workshops/2007%20Workshop/2007%20Papers/079%20Agranov%20et%20al.pdf) at the 2007 International Image Sensor Workshop prompted Eric Fossum (inventor of CMOS APS) to coin Agranov's Law: "More pixels still improve image quality in the presence of noise".



In the end though, it's total area of sensor used to capture light that really defines the maximal dynamic range of a sensor. What higher resolution does is allow you more flexibility to do some trading.


Well put!

[Graeme Nattress]

Theory is fun, but from measuring we get learning.

Graeme

[Daniel Browning]

However, unfortunately, many times the effect of SNR decrease from blurring of data because of the window size is not taken into account.


Hi Joofa! Thanks for the response. I'm afraid that I don't understand what you've said, exactly, or how/why it is occurs. Is it possible to break it down in simpler terms?

[Dj Joofa]

Hi Joofa! Thanks for the response. I'm afraid that I don't understand what you've said, exactly, or how/why it is occurs. Is it possible to break it down in simpler terms?

When you are downsampling, then at each pixel location you would assemble a bunch of close pixels derived from the original image and determine the value of the dowsampled pixel from this set. I meant the collection of this set as the "window". One simple way is that you average together all pixel values. However, better methods exist than simple average -- though, which ever linear method you pick, it is some sort of weighted averaging of these pixel values in the set/window.

So we note two things. A larger window/set size will:

(1) reduce noise better, but,
(2) blur image more

Therefore (1) and (2) are two contradictory goals and one can be increased on the expense of the other. Therefore, the optimization problem involves finding a good window size and its shape -- i.e, its height as it varies over the set.

Some close form results exist for certain window types. In typical downsampling implementations you would see specifying the type of window say, sinc, lanczos, etc., but not always the optimality is derived considering what should be its size? Normally, after selecting a particular window type, a fixed window length (typically small, a few pixels in each direction) is chosen heuristically. That approach works well in practise. However, at least, in theory, an optimal window size exists considering the SNR formulation.

[Daniel Browning]

Thanks again. Do you consider resolution to be a part of "signal"? So that a four-fold reduction in resolution, though it reduces noise by half (depending on the resampling algorithm), reduces the resolution four-fold, so there is less noise but also less "signal"? If yes, then I agree, although I haven't heard of that definition of SNR before. If no, and it has nothing to do with what you're saying, then I'm afraid I still don't understand. :)

[Dj Joofa]

Thanks again. Do you consider resolution to be a part of "signal"? So that a four-fold reduction in resolution, though it reduces noise by half (depending on the resampling algorithm), reduces the resolution four-fold, so there is less noise but also less "signal"? If yes, then I agree, although I haven't heard of that definition of SNR before. If no, and it has nothing to do with what you're saying, then I'm afraid I still don't understand. :)

You are welcome Daniel.

Firstly, the scenario normally quoted where SNR varies as some relation of the square root of the pixel size is valid only if neighboring pixels have a fixed value, but are corrupted by different amount of noise resulting with some assumptions on the noise process. Since, the signal is not constant in its neighborhood in general, even without noise, the result is approximately valid. Secondly, this scenario will correspond to simple averaging, which does not have good data smoothing properties of good downsampling.

Some sort of data smoothing is needed in downsampling to avoid aliasing due to expansion of signal spectrum after downsampling. So indeed, yes the signal degradation due to blurring inherent in data smoothing is part of SNR estimation.

As I mentioned in earlier message that some parameters are heuristically chosen. There is always an implied assumption that noise is small. But how small? Not many bother to do an analysis.

As an example, for the simple case of averaging pixels, if the smoothed signal after averaging is a certain order of the window size, then it follows that the explicit formula for the calculation of window size is related to the differential rate at which signal is varying. It turns out that for optimum results, the signal degradation due to noise must equal 4 times the signal degradation due to the smoothing of the signal. Similar results could be derived for some other window types besides simple averaging.