ProEXR for Premiere

[Bruce Allen]

Brendan,

So, which would be the preferable pipeline if you export your graded footage in Resolve as OpenEXR? That way, you take advantage of the bit depth starting from there...

Well, the easiest way to think of it is: use OpenEXR any place you would use DPX. Except it takes up a lot less space and can store more info if you need it to.

Resolve can export and import OpenEXR files with timecode.

So, for example, if you want to do a grade... but one of the shots is really noisy and you want to run Neat Video on it before grading...

You can run Neat Video on the shot (using Premiere or After Effects), then render out an EXR sequence and bring into Resolve. The nice thing is that you can even render out with handles - since it has the EXR has timecode, Resolve will be able to figure out what part of the EXR sequence is needed to be cut in.

Or if you want to add VFX to a shot, then grade... same thing. Does this help a little?

I guess I should put up a tutorial?

Bruce Allen
www.boacinema.com

[Brendan Bolles]

Hey everyone, I've found an issue with the plug-in. Actually, the plug-in is fine as far as I can tell, the issue is with Premiere. Since there aren't any other plug-in frame sequence exporters (that I know about), I was bound to run into an undiscovered bug or two.

When you export an EXR sequence, Premiere will set the start time to whatever frame the playhead is currently on. I'm talking about the playhead in the Export Settings window, which gets set initially to wherever the playhead is in your sequence. The work-around is to drag the Export Setting playhead back to the start (or wherever you want it).

If you send the job to Media Encoder you won't see the problem because the playhead will get reset. But if you go into the Export Settings dialog in Media Encoder you can move the playhead and get the same bug there as well.

I'm talking to people at Adobe to make sure it's nothing I'm doing on my end. I'll report back when I get a response.


Brendan

[Blair S. Paulsen]

Big shout out to Brendan for pushing OpenEXR options.

Very few post folks have dived into IIF/ACES so far, but I fully expect it to be SOP within 2 to 3 years at bigger facilities and, AFAIK, it's built around 16bit half float Open EXR file structure.

FWIW by 2015 I hope to be able to do 4K IIF/ACES compliant OpenEXR based DIs as easily as doing 2K 10 bit log DPX DIs today - similar cost, similar speed - but WAY more future proof masters.

Right now the disk I/O requirements are a big deal (read expensive) but PCIe 3.0, TB, SSD RAIDs and other emerging technologies should help.

Bruce mentioned compression options to manage data rates. I would love to know more about this starting with the overhead of decoding zip16...

Cheers - #19

[Bruce Allen]

So true, Blair!

Personally I am finding that OpenEXRs in some cases read and write faster than their DPX equivalents because I don't have the fastest disk array on earth and the compression doesn't take that long...

And 4K B44 4:2:2 files are similar to 2K full-aperture DPX files! So awesome...

And Brandon - thanks for finding the bug before we did!
I was always using the Media Encoder because I was sending like 10 clips out at a time, so I never found this bug.

Bruce Allen
www.boacinema.com

[Brendan Bolles]

The official IIF/ACES spec only allows None, Piz, and B44A compressions, although it's hard to imagine any software solutions not accepting the other ones. The spec does allow files to use Luminance/Chroma subsampling (which is actually 4:2:0). I think these requirements are really for the digital film camera makers who hopefully someday will use IIF/ACES as a universal raw format instead of the current Panalog, S-Log, and other things usually output as DPX.

ACES itself is just a color space—a very wide linear color space that contains all the visible colors. A given camera, of course, will only be able to capture some gamut within the realm of all visible colors, but the idea is that the manufacturer will know how their camera works and provide a way to convert their raw data into ACES. If all manufacturers were to do this perfectly, in theory different cameras shooting the same scene would deliver ACES files that all looked alike.

So anyway, if you start working with EXR sequences, the only difference with IIF/ACES is the color space. Right now I don't have a good way to do that conversion in Premiere, but hopefully we'll get there before too long. ACES LUTs are already available on the OpenColorIO site (in the sample configurations).


Brendan

[Bruce Allen]

BTW, just got back home - I was using DaVinci Resolve to grade in ACES a whole bunch of PIZ-compressed EXR clips exported from Premiere (brought in as After Effects comps into Premiere, then exported to ProEXR, btw).

Happy to say performance was realtime - using just an internal software RAID :)

That was 4:4:4 1080p - next up is seeing how close I can get things to realtime at higher resolutions if I go B44A and luma/chroma subsampling.

I think the limitations are more the GFX processing for the grading though than EXR decompression. Come on, new nVidia cards / new Xeons / new Ivy Bridge!

Bruce Allen
www.boacinema.com

[Subhadip Sen]

Off topic - I doubt the limitation is GPU for Resolve. An ancient 8800 GTX is enough for 3-4 nodes at 1080p! GTX 580 hardly breaks a sweat, I haven't been able to knock it out of real-time yet - including tracking, dynamics, blurs, keys, everything, at 15 nodes or so. Whilst I haven't tried 4K (using 8.2 Lite Beta for Windows), I would guess 2 x GTX 580 3GB will be enough for 4K processing. Or at most 3 if you are into complex node structures. The Mac OS X version is very limiting however, as the fastest GPU on OS - Quadro 4000 - is only 1/3rd as fast as GTX 580. Nvidia's big successor to the GTX 580 comes at the end of 2012. Up next is GK104 in April/May which succeeds GTX 560 Ti and targets AMD's HD 7970. No big performance boost, but it will be power efficient and cheap ($300 is the rumoured price).

As for EXR, at 4K+ my main bottleneck is always disk I/O. Before the Thailand floods I was thinking of making a storage server using Infiniband, nested RAID and a bunch of disks, but fast hard drives are just too expensive nowadays! Waiting for the supply to stabilize and also Windows 8's amazing Storage Spaces feature. All that said, I don't think I need it - but I think many here would benefit immensely.

[Brendan Bolles]

Waiting for the supply to stabilize and also Windows 8's amazing Storage Spaces feature. All that said, I don't think I need it - but I think many here would benefit immensely.

Hmmm, I had not heard of Storage Spaces before. Seems very similar to ZFS. There's now a commercial ZFS product for Mac, not to mention the free open source one:

http://tenscomplement.com/
http://code.google.com/p/maczfs/


Brendan

[Subhadip Sen]

Hmmm, I had not heard of Storage Spaces before. Seems very similar to ZFS. There's now a commercial ZFS product for Mac, not to mention the free open source one:

http://tenscomplement.com/
http://code.google.com/p/maczfs/

Brendan

Yes, it is also available for most OSes, including Linux distros and Windows. The general idea of having a pool of hard drives under a single virtual space is indeed similar to ZFS, but it has limited configurability and RAIDZ is slow and does not scale well. Storage Spaces offers great flexibility with hard drive configs and in the Developer Preview the performance numbers are close to Hardware RAID, scaling linearly with multiple hard drives and avoids erratic performance noted in some hardware RAID levels (such as write in RAID5). It will also scale with anything from SSDs, external hard drives to external storage servers and NASs - near unlimited flexibility. Also, simple things like being able to "reset" your array to a new config at any time without any data loss, or simply just dropping in a direct replacement in the case of disk failure which resyncs automatically, the possibility of asymmetric RAID (so to speak) etc. Of course, its still a work in progress and W8 beta is still 3 weeks away, so it can only get better. It is just a very simple, flexible and elegant solution - provided it performs like the early reports indicate. The new filesystem ReFS is also due soon. My concern is that by offering this much flexibility it could lead to anomalies. Not sure how a single storage pool containing 1 SSDs, 1 external hard drive, 1 NAS will scale...

Back on topic - I reckon 2 SSDs in RAID will be good for 4K EXR PIZ playback in real-time? 4K B44 4:2:2 works fine on 2 x Samsung F3 RAID.

[Blair S. Paulsen]

Not sure if the right people will ever see this thread but FWIW:

I call on every player in this industry to include robust OpenEXR support in their products.

We are inevitably moving to a future where sophisticated use of metadata (info that travels with the image and sound files) will become expected. I am as frustrated by some of the braindead omissions in FCPX as the next guy. That said, FCPX's embracing of a future where your tools can more effectively assist you in managing all the elements in a content creation project is simply leveraging the toolset. MXF, XML, ALE, etc have various issues with file structure integration and platform interoperability that have all too often sabotaged their promise. Perhaps someone who really knows the inner workings can educate us all, but from a users perspective it just seems like we're collateral damage in turf wars.

In any case, I believe OpenEXR has the combination of metadata payload capacity and flexibility, plus bit depth, plus IIF support to suit the next generation of post work. Would love to see some metrics for file sizes at various resolutions, chroma sub-sampling schemes, compression settings, etc to assess strategies for managing data rates. While we may eventually be able to move 2GBytes/sec on and off 10PetaByte storage devices the size of a cell phone, as Subhadip noted, right now there are some serious costs involved in high capacity, high bandwidth storage.

If anyone has any links to more info on the practical issues involved please post them to the forum.

Cheers - #19