Storage/Backup Workflow on the Road

[Steve Murray]

I have been shooting both digital stills and HD for many years and P2 since Jan 05. We have a 3 copy rule for images before we will reuse a P2 or CF card (yes, 2 copies should be enough but you know how stuff happens...) We currently carry 5-6 1TB Western Digital Pro drives. They are fast, made up of 2 500gb drives in a ready-to-use raid and have FW 800. They have a fan and you will see people online complaining about that but it's not a problem and the new ones are much better. Best thing is they are cheap ($360.00) and very fast. I love them. When we return we backup everything to cheaper, slow-ass USB drives and then go back out with the 1TB's for the next shoot. Of course you could save the time and keep buying more 1TB drives.

Steve


Canon 1Ds MKII - Varicam - HVX200 - HPX500

[David Battistella]

To me it makes sense to bring enough recording media on the road, (even though we have been spoiled by the low cost of tape) be it tape or digital media.

If you can trust RED drives then that might be the way to go. At 900 bucks each they are cheap and you can get about 3.5 hours on each drive. This works out to about $3.50 per minute shot. You can buy about four RED drives with the cost of a laptop and FW setup.

That's 1.2 TB of recordable media space. This is the equivalent of shooting about 28 1/2 hour beta tapes on one road trip, or before you need to unload the drives.

I think you have to look to a large format tape back up solution when you get back to the shop.

In the short term I figure I would get three drives anyway and add at least three as the projects pile up. Ultimately the production is going to have to pay to get the footage off the drives and store it long term. The take away will be media that THEY have to trust. (fw or LTO or what ever). You have to make sure it gets on the drives and it can get off the drives.

When the 32 gb cards are released the price will eventually drop to make it comparable to shooting tape now and that is not as far off as you think. I think that is how most people will end up recording on RED, just bring a hundred of these compact solid state cards on the road and let her rip.

David

[brain21]

So I am assuming that we have power at all of these locations? Even w/ a laptop, that external sata raid enclosure needs external power.

Also, everyone is looking to larger and larger drives, but yet they want speed. You guys talk about "fast" enclosures, but most enclosures are basically the same. Most run off of the same chipsets, etc. It's a question of how reliable or well constructed their circuit boards and connectors are, and thats about it. After that, it's about the brand/model drives they use. IOW, one enclosure itself is probably not going to be faster than another. Now one enclosure may come with higher RPM drives, or drives with larger caches, etc.

You gotta understand that you reach a margin of diminishing returns in terms of speed and storage space at 250Gig drives. Drives that are larger than that spend SO much time doing error correction that they actually become slower. This, and the fact that more drives often means a larger overall cache size is why LARGE storage companies (think Network Appliance, etc.) use a greater quantity of smaller drives. Almost ALL use drives of 250Gig or less, even when building 100TB storage centers.

When looking at an enclosure, look for reliability (look for reviews online), and get your own drives. Get faster RPM drives with lower volumje per drive and more drives. Find a balance between them. If you buy an enclosure from a vendor that sels enclosures with drives already in them, if they don't specify the drives and you don't know what they are or their specs are (try calling the company), then just don't buy. You are probably spending WAY too much money for an enclosure that basically LOOKS cooler but offers nothing over the competition.

I think the ultimate would be a small, portable NAS enlosure running something like Network Appliance's DataONTAP NAS software/OS.

It would mount via CFS and look like a 2nd drive on any host PC regardless of (host) OS, and you could connect to it via GigE (cables are cheap!). It can provide full backup and disk hot-swapability, and nor require double the sace (IOW 2 TB to have and backup 1TB of data).

Alas, such a thing does not exist (yet), but I am meeting with some guys from NetApp on friday, so I'll suggest it. :-)

Barring that, this seems like it would (ultimately) be the best solution:
- Get 2 red drives so 1 can record while the other dumps to backup
- Get one of those expresscard eSATA port multiplier encolsures w/ room for 6 drives & load it up with 6 250MB 10krpm drives
- format the (external) drives with zfs (don't have to and may not want to if you are doing this with a Windows PC or need to get the data to a windows PC)
- dump the Red Drive to your eSata external raid. You can do this ~4 times before it gets full (assuming no Raid striping). Go back to studio and dump to your DLT (or whatever) backup drives.

The reason for ZFS was this... NORMALLY you dump 320Gigs of red data to a drive and then back up the drive and you have 320Gigs of data backup (barring compression). Then you do a 2nd dump, and back up the drive. You now have a 320Gig first backup, and a 640Gig 2nd backup, unless you make it easy by putting stuff in separate directories, etc. w/ ZFS the 2nd backup will only back up the new stuff. IOW, it's just more idiot-proof. :-)

OR do what someone else suggested and get a Shuttle case, or maybe even a miniITX mb+case so you have something small and portable. With this you can:

Forget the external eSATA raid boxes mentioned. If the MB doesn't support it go out and buy a $50 SATA raid controller card and throw a bunch of disks in there and Raid them together. Get a sata to eSata cable and run it from the MB to out of the case or get an eSATA card and install it. Copy data from the reddrives to the ShuttlePC. Get a DLT drive (or whatever) and install it on the shuttle case, and then you can do a permanent backup right then and there. Get a cheap 8" VGA Lcd for the shuttle case

If you want permanent backup on the spot, go the shuttlePC route. It's a little bit more of a pain to lug around (but not much, really), but it's a slightly more powerful solution overall. If you can dump to permanent backup at a later time, and you plan on only filling up the reddrives twice each (assuming you have 2), then go the laptop/eSata external storage only because it's easier to carry around, and you only have 1 thing to plug into the wall (as opposed to 2 - Shuttle case, & small VGA LCD monitor). Also you can check footage on the laptop right there if you want.

Brain21

[Erik Greensmith]

This ain't my current worfklow but I've mulled it over a bit since I'll be needing to figure out a "batteries-only" backup solution in a couple months so this might have relevancy to a doc shoot. Keep in mind none of this is verified and I've never done it personally.

Plug CF cards into FW800 or express34 reader on the MB Pro and copy to 2 daisy-chained FW800 "rugged" LaCie 200GB externals that are config'd in a Disk Utility RAID 1 mirror. Then, I guess, use the ShuttlePC or similar setup to do a tape dump once you get back to the shop.

Batteries-only? The LaCie disks are bus-powered, but I'm skeptical as to how reliable they'd be, if functional at all, in a daisy-chain setup. I'm gonna test so I suppose we'll find out.

[J Clark]

I love the Buffalo II raids. They are cheap and reliable and have a built in gigabit ethernet connection and operate as a stand alone NAS so you can plug them into a small ethernet hub(good for hotels too) and transfer all shot data to the raid from your laptop, They also have raid 0-5 for data safety. Just protect in a sturdy case. 2tb will cost you less than $900.

When you return to the office your footage in on one drive and it's all accessible on the network for transferring to a file server or edit raid. We use a pair of them(2tb) for our primary and back-up file servers to handle complex, large-file VFX shots.

And I agree - you should always keep at least 2 copies of data to play it safe, but 3 makes you sleep better.

[david farland]

This 4 drive/esata cabinet works for me @ $360.
http://www.cfienclosure.com/4043ER.html

It's plug n play and I can configure the 4 drives any way I like.

I'll still need to do further tests to see if the a raid 1 array written in the CFI enclosure can be read as a raid 1 or single drive configuration by another intel or silicon image esata/raid controller.

Cheers,

[chocblu]

The reason for ZFS was this... NORMALLY you dump 320Gigs of red data to a drive and then back up the drive and you have 320Gigs of data backup (barring compression). Then you do a 2nd dump, and back up the drive. You now have a 320Gig first backup, and a 640Gig 2nd backup, unless you make it easy by putting stuff in separate directories, etc. w/ ZFS the 2nd backup will only back up the new stuff. IOW, it's just more idiot-proof. :-)

Im not sure i totally agree with you and ZFS certaintly isnt the only way to do this. such as rsync.

If your referring to snapshots/copy on write then whats really happening is the backup is just linking at file system level to data that has been copied to somewhere else.

My grasp on the issue is a bit hazy too, and i agree with the use of ZFS. It is a killer file system especially for the data conscious. And i think that everybody should have a look at it, if they're serious about having a sturdy file system that can be expanded as much as you like, is fast and has a GOOD version of software raid with more options than you can poke a stick at.

Downside, not available on windows as far as i know. OS X support is in beta, so i wouldn't say its rock solid (no personal experience, just a comment on beta) and its not available on linux either, so in terms of admin and use in the field its pretty restricted. But as far as a file system, it absolutley kills everything else out there.

Some guy has been quoted as saying someting along the lines of, given that a certain amount of matter can hold a certain amount of data, there is not enought matter on the earth to max out a ZFS file system (its a 128 bit file system).

I cant say enough of how much of a fan i am of this file system. Have a look at nexentstor http://www.nexenta.com/corp/ for a commercial of the shelf offering.

Mark

[sbaechler]

Hi

I will be using the following system on longer shoots:
I have a portable PC with a 4 or 5 disk Raid 5. Data from the camera gets offloaded to this drive first. The Raid is big enough to hold around 15 days of footage. So the director can watch a clip whenever he wants to.
At the end of every day there will be a copy of the day's footage either on a LTO tape or a removable hard disk that gets shipped to the editing room.

I've been shooting a lot of P2 footage for the last two years and I've had hard disks die on me on the set. Luckily I've never lost any data because I've always had a backup.

Also make sure you use at least a smanll uninterrupted power source (upc) if you plug in your computer and drives on a film set. Those electricians blow fuses all the time.