Lens test with frame grabs?

[joshua csehak]

Hey guys, are there any lens test with frame grabs? Are they there, and I'm just stupid? The SALT thing is very valuable for sure, all those people are cinematographers I highly respect, but I want to be able to judge for myself.

Something like this (but with images that exist): http://finnerknowsbest.com/blogs/Matt-Uhrys-Lens-Fetish-Red-18-85-vs.-Cooke-18-100.html

or this:

http://www.cinematography.net/lens-comparisons.html

?

[Matthew Duclos]

Its difficult to judge a lens from a frame grab since there are so many variables that can be introduced. Of course you can judge certain things like distortion, but even then, changing the location or scene can add to or reduce the effects of distortion. The same thing can be said for illumination testing from frame grabs. Even when it comes down to simple things like resolution, sure you can shoot a res chart with your red one and pull a frame. But that doesn't mean that the lens and camera were calibrated properly and are producing an accurate result. While I don't discount the frame grab as a valuable source of real-world performance, it's just not a reliable source for in-depth testing. Think of it in terms of cars (again). Lets say I own a Ferrari and I am very good at driving it, a pro. And I take it down a nice smooth road and determine from the drive that it has about 462 horse power. That would be the equivalent of testing from a frame grab. Whereas the proper test would be to put it on a dyno and measure everything in a controlled environment. Simple as that.

[joshua csehak]

That makes sense, and I agree, you can't get a comprehensive, scientifically-accurate test without putting it on a dyno. If I knew what a dyno was, I'd probably agree more, but I get your gist ;)

But I feel like an unscientific test might even be more useful. If you shoot the same scene, say indoors, a couple lights on a model, something shiny, maybe something to provide high contrast to test the chromatic aberration; same setup with different lenses, I feel like that'd provide you with real-world information as to whether it's worth plopping down $500 extra a day for an Optimo and a sherpa to carry it, or if a Nikkor will be fine.

I remember seeing that Red Pro vs. Cooke page before he took off the images (why would he do that? Bandwidth is so freaking cheap these days. Switch to Dreamhost for crying out loud), and I could see exactly how the Cooke was better than the Red. IIRC, it was very slight, and hard to see, but the specular highlights on the Cooke were very organic and filmic, and on the Red they had a bit of haloing. There might've been differences in the breathing and stuff too, but I don't think much. I felt like I understood what you get when you pay ten times more. So if money grew on trees, I'd go with the Cooke, but budget being an issue, I'd be perfectly happy with the Red Pro.

Do you not find tests like that at all useful? Do you feel like you have to shoot with it yourself to sufficiently judge it's merits?

[Matthew Duclos]

I do find them very useful, as I said. Just not always accurate. Especially considering the variables. It becomes very useful when comparing two lenses and the tests are shot consecutively, with the exact same environment. But otherwise, it's just sort of arbitrary. Comparisons are always fun. :)
Personally, I would utilize an array of test equipment to determine image quality. But don't forget that there is more to a lens than image quality.

[joshua csehak]

Yeah, it's gotta be the exact same environment, for sure. Do you know of anything like that out there? I feel like it's about time I really get a feel for lens differences. If not, I'm thinking of asking my rental house if they'll let me set something like that up and post the results for everyone's benefit. I'm thinking the criteria would be sharpness (edge and center), distortion, chromatic aberration, breathing, focal throw, bokeh, flare, and obviously, speed, focal length, size, weight, and cost. Am I missing anything, you think? If there's anyone that's qualified to weigh in on it, it's you.

[Alexander Ibrahim]

Yeah, it's gotta be the exact same environment, for sure. Do you know of anything like that out there? I feel like it's about time I really get a feel for lens differences. If not, I'm thinking of asking my rental house if they'll let me set something like that up and post the results for everyone's benefit. I'm thinking the criteria would be sharpness (edge and center), distortion, chromatic aberration, breathing, focal throw, bokeh, flare, and obviously, speed, focal length, size, weight, and cost. Am I missing anything, you think? If there's anyone that's qualified to weigh in on it, it's you.

I actually think that is pretty good as a starting place for comparisons.

What I'd also like to do is bring an AC to the test to talk about using the lenses in practical situations. I'd set up a fake scene with a dolly with follow and a rack focus, a couple of other simple things, then see how the AC feels about real world use of that system.

[Alexander Ibrahim]

I do find them very useful, as I said. Just not always accurate. Especially considering the variables. It becomes very useful when comparing two lenses and the tests are shot consecutively, with the exact same environment. But otherwise, it's just sort of arbitrary. Comparisons are always fun. :)
Personally, I would utilize an array of test equipment to determine image quality. But don't forget that there is more to a lens than image quality.

I am thinking about doing a fairly comprehensive set of lens tests, and would like to hear more about your "array of test equipment."

Some post tools (scopes of various sorts in color correction and compositing applications) lens projector and some charts is what I've got on my list. What am I missing?

[Matthew Duclos]

I am thinking about doing a fairly comprehensive set of lens tests, and would like to hear more about your "array of test equipment."

Some post tools (scopes of various sorts in color correction and compositing applications) lens projector and some charts is what I've got on my list. What am I missing?

I'm not too savvy with the post stuff, but for lens testing there is a range of tools I use. I also have to mention that it's not all about image quality, as I've stated many times. For example, the Nikon 14-24mm f/2.8 is a great lens, one of the best Nikon has ever made. But it sucks for motion picture, plain and simple. There are more factors that make a lens good or bad than just image quality. That said, tools such as a projector, auto collimator, vertical collimator, MTF, T-stop bench, and several others that would apply to service but not so important to your goals would be used.