check my math - data estimates for feature

[Michael Morlan]

Hi all,

I'm not officially an owner (yet) but I'm the DIT on a feature with two RED One's. Here's my estimate of the data footprint for the project.

This is for the A-camera only. Could you check my assumptions and math?

http://www.reduser.net/forum/uploaded/264_1199998777.jpg

Thanks.

Michael

[BASSAM MSSALATIE]

SCARY ...:cold:

[Jim Hoffman]

shooting with a RED... priceless

[chuck colburn]

14 to 1 ratio!

[David Didato]

will you be overcranking anything?

Dave

[Dane Brehm]

Wow!

I had a producer ask me the same question and I said 5TB for 23 days including redundancies. I'm using a Caldigit 3.7TB drive for all our "data loading"

so much data I love it!

[David Birdy]

Hi all,

I'm not officially an owner (yet) but I'm the DIT on a feature with two RED One's. Here's my estimate of the data footprint for the project.

This is for the A-camera only. Could you check my assumptions and math?

http://www.reduser.net/forum/uploaded/264_1199998777.jpg

Thanks.

Michale

Michael,

May I ask the reason you are shooting the project in 24 frames?


Dave

[Michael Morlan]

This is a feature starring a Hollywood B player. It's an ensemble piece with six friends often in the scene together hence the high shooting ratio. It is intended for theatrical release. This doesn't take into account any over/under cranking or the B camera.

So, no one answered the question. :shifty: Is the math correct?

M

[Mark Allen]

My math isn't going to help you - but I do want to remind you that collecting data is step one but at some point - when you're finishing - you're going to need more drives to which you convert your footage.

[Cüneyt Kaya]

quote:
mccarthy tech

Basically it is all a big math equation. I hear that 4K at 24fps in Redcode is supposed to be 27.5MB/s. That is 1.6GB/min, and 100GB/hr. 20 to 30 hours will require 2 to 3TB of storage space for the source footage.

Then you will have to select an offline format, unless you plan to use the Quicktime reference files, which doesn't seem like the best plan yet. I don't recommend using DV or HDV, but I will use those numbers for the same of estimates. 20 to 30 hours of footage would require 250-360GB at HDV. DVCPro-HD would require four times that much space, 1-1.5TB. ProRes or Cineform would be bit more than that. 1080p doesn't make much sense as an offline for a 2K conform since it is very close. 720p might give you a bit of a performance boost, and SD will work if you already have the tools available. I definitely recommend editing in a native 24p environment, instead of adding and removing 30 frame pullldown. Regardless, budgeting 2TB for the offline gives you the flexibilityto do whatever you need.

Once you have an offline, you will likely want to export uncompressed 2K versions of the shots you used. Uncompressed 2K 10bit RGB at 1.85 aspect ratio is 200MB/s (DPX or Blackmagic 2K) , which is 12GB/min, and 720GB/hr. For a two hour movie, with short handles, you are looking at 2TB. From there, ideally your color system can take those files directly, and export directly to DPX for filmout, which is another 1.5TB. If you need uncompressed versions of a texted and textless, that will be one more set of uncompressed files requiring 1.5TB.

That is 3-5TB for the source and online, and 5TB for the finishing at uncompressed 2K. Note not all of these files need exist at the same time, especially if you have LTO-3 backups. Using a compressed finishing codec for finishing would lower the requirements as well. A 14 disk array with 1TB spindles, split into two 6TB Raid-5 arrays, would give you 12TB. 10TB that we discussed, plus 20% to minimize fragmentation etc. You want to avoid filling any drive to more than 90% of capacity, unless you are just using it to transport data. Working on that data with no free space will cause excessive fragmentation.

The lowere end solution would be a single array of 8 1TB disks, 7TB usable after Raid redundancy. Delete your offline source after you move to 2K, delete your (BACKED UP) source footage after your 2K conform is done, before rendering colored versions.

I make a lot of assumptions, and this is only one of many ways to look at it, but I hope this helps, and gives you a few ideas.
__________________
For more information on PC based Post-Production, check out my tech website at www.hd4pc.com

[BASSAM MSSALATIE]

your english is fine and you post too. thanks KAYA:sarcasm:

[Wade McDonald]

Your math looks just perfect. I came to a similar conclusion for a film we're shooting with one camera later this spring (except we're 110 min x 20:1 to be on the safe side).

However, you might want to take into account how large the DPX's would need to be in case you need them as deliverables to the post house where you're doing DCI packaging -- that is, if you plan to use a pull-list and have REDCINE export them... Frankly, I think the path of least resistance is to take the raw R3D files to a finishing house with Scratch and do a native online and DI finish... but not every production can afford to rent a seat at a post house for an extended period of time.

Interestingly, I've talked with some IT guys I'm friends with regarding the data storage/protection issue and the solution presented was not something I expected... A CFO (who wants to remain anonymous) of a large (7 bil dollar) public company suggested that we keep a mobile RAID 5 of 4-5 TB on set with an XServe and an iMac to use for dailies/etc. -- plus an identical setup (minus the iMac) at "base". Each night the two are synced, and once a week copied to an online "Archive" RAID of 14+ TB that's meant to contain every R3D file shot by the production company, as no "offline" storage medium can really handle the amount of data we're talking about when shooting features... apparently (and correct me if I'm wrong) tape backups aren't really that reliable for multi-year storage, and to make an optical backup would be even more prohibitively expensive than constantly expanding a fibre RAID array. The two XServes connected to the "mobile" and "base" arrays + the archive's babysitter XServe (total of 3) can be connected via gigabit ethernet to be used as a mini-renderfarm for converting the R3Ds into ProRes proxies using Compressor. (I also do Shake work so this is very attractive for me, less so for the avg filmmaker)

The reason for the RAID arrays, he says, is because hard drives crash a lot more often than you'd think. For instance, he sent one of his guys to another state with a hard drive in carry-on to move a large amount of data -- dead on arrival. In an array, the system can identify a drive thats close to failure and switch to an online spare, or rebuild completely when one dies suddenly.

In addition, he claimed it was "bad form" to edit directly from something meant to be an archive, so we're looking at yet another RAID to be connected directly to the Mac Pro editing station for storage of the ProRes proxies. That's close to $55k extra for the production if we take his advice.

Myself, I like the idea of a redundant onset/base RAID 5 set, and I like the idea of an archive... but there's a couple of problems that I see: A) after a long 10-11 hour shoot, no one's going to be thrilled about schlepping a rackmount case back to the office and synching it. B) it seems a bit overkill for our purposes.

I'm also considering a lower-budget solution. Using $2.5k 4-TB firewire 4-drive RAID 5 enclosures (instead of the $12k-$17k XServe+RAID combos) I can manage just enough space to scrape by. That, plus an iMac on set would be enough to manage the shoot... provided we use two of the small RAIDs and maintain sync. Keep the Archive and editing arrays, so you still have one XServe to use with the Mac Pro as help for rendering. Definately have a dedicated couple of crew members for workflow.

Anyway, just wanted to share all that, since that's what I've been working on this week with my company as we await the arrival of #845. At this point it's a little academic as no real plans need to be made just yet... but my CFO friend made a good point: What amount of data loss is acceptable when you're shooting a feature?

[lance617]

My math isn't going to help you - but I do want to remind you that collecting data is step one but at some point - when you're finishing - you're going to need more drives to which you convert your footage.

Mark is quite right. Especially if your final render is 4K. 14:1 wow. That's high especially if you're filming out for a theatrical release and trying to manage all those shots. Probably need to do a lot more preproduction and rehearsal.

If you're going 2K DI to film and not finishing in-house you're probably okay if you running an XSAN or Unix SAN/Server.

We've got some really complex shots in some places but still looking at a 5:1 ratio with a large-Independent budget (small studio budget).

We calculated our mean negative frame size to be round 40Mb and will reserve about 54TB of the 110TB for this feature for edit, cgi, finishing, etc., but we will be rendering the Raw data and editing , etc. 2K proxies.

A few more TB might be wise, especially if you're not using a high performance OS and SAN.

Lance

[Michael Morlan]

Some excellent feedback guys; especially the articulation of other RAID choices and work flow through the entire post pipeline. Certainly, any transcoding tasks will dramatically increase the data footprint.

Right now, my task is to get .R3D files from set to post. Other persons/facilities will handle any transcoding (other than my prepping circle takes for an on-set FCP editor working in a standard-def format of choice.)

We'll be testing everything in the two weeks before photography.

I did note one error. I presumed 27MB/s not 27.5MB/s.

Thanks, again, for the time and effort to respond.

[Mark Allen]

Sometimes my 8 gig cards hold 4 minutes. Sometimes they hold 5 minutes. So - keep in mind there can be some variance.

[lance617]

Michael - we should be moving and setting up our facilities just down the road from you in a few weeks and should have our new SAN, editing, and DI up by Feb. I'll be in and out of Dallas, Austin, and San Antonio (where most of the film is being shot, SA & Chicago). We're just waiting on closing with our A List lead at the moment. Let us know if you need any more information.