practical tapeless workflow [Archive]

sbaechler

05-24-2007, 03:02 AM

Hi

I know a few people who have an option for the Red camera and I hope to shoot the first short by the end of this year.
The one thing to consider is that it is not enough if you just rent the camera. You will need a proven solution to get all the footage safe to the editing room. Producers and camera assistants are used to a film and a tape workflow. Only very few have experience in a tapeless workflow. I have worked as a 1st AC on a feature film that was shot in the Panasonic P2 format and we have gained some valuable experience, although there was much less data than you will have with the red one. P2 is 1 GB/min.

I made a list of equipment for a feature length movie where you are on a location that has power and plan to deliver your material to the editor at the end of each day.

On Set:
1 MacBook Pro.
2-3 portable harddrives with 2 HDs each in a RAID-1 configuration. FW-800
1 solution to download footage "in the field". (Compare to P2 store)
Editing room:

1 X-Serve RAID 4TB. Raid 10 or 0+1 (Usable disk space 2TB)
1 Apple RAID emergency packet for immediate repairs
1 Fibre Channel PCI card
1 UPC power station
1 Power Book G4 with SCSI interface card
1 LTO or DLT streamer (alt. with Gigabit interface)

I spent a lot of resources on a redundant storage all the way through. I had discussions with another editor if this wasn't a waste of money. From my experience time is the most precious thing during production.
I think producers who have a due date for their film rather spend a little bit more money on equipment than to reserve an additional two days for emergency recovery and restoring data in case of a HD crash.

Regards
Simon

vanguy

05-26-2007, 10:09 AM

I would think that you could do it a lot cheaper and simpler.

On set:

You could get four RedRAIDs, taking two to the set (5-6 hours recording time).

The other two are at the edit suite being transferred/archived/backed up. When the transfer is done, swap with the set drives. If the edit suite is a MacBook and some large RAID drives (as below), it could be portable, as long as you've got a power supply for the drives.

In the Suite:

A MacBook Pro or 24" iMac and a pair of LaCie Bigger Disk Extreme 2 TB drives, mirrored for safety. When you're done, put the drives on the shelf. Buy new drives for each project (they're reasonably cheap ~ $750). After 3 years or so, cycle the media on the drives.

If you really want to go with LTO/DLT, fine, but it's a large initial investment and a lot of work on each project. I'd think the mirrored drives would give you almost bulletproof safety, and they go back online instantly.

Simon Blackledge

05-27-2007, 03:11 AM

I'd never use Lacie in a production enviroment.. but thats just me. Had to many fail in the past. If your data is important use enterprise drives in a raid.

s

Stephen Gentle

05-27-2007, 04:26 AM

You could get four RedRAIDs, taking two to the set (5-6 hours recording time).

I assume that you mean RED DRIVEs, here. RED RAIDs are supposted to be big raids for recording uncompressed data - although I'm not sure if RED plans to make them, as no details have been released about them yet.

vanguy

05-27-2007, 08:23 AM

I'd never use Lacie in a production enviroment.. but thats just me. Had to many fail in the past. If your data is important use enterprise drives in a raid.

You're probably right. I've had some iffy Lacies. It's never been the media, though, always the firewire connector. They're a lot less iffy with Firewire 800. And you can always pull the drive out and put it in another housing. I've had pretty good luck with G-RAIDs, although I wasn't as tough on them as the Lacies I've had.

But I like the single large drive you can store like a tape. Anybody know any decent, inexpensive modular enterprise RAID systems?

I assume that you mean RED DRIVEs, here.

Yup. Sorry. I was looking at the red.com Accessories page, which is a little unclear. Also, the RED DRIVE is a RAID, which confused me.

Simon Blackledge

05-27-2007, 08:50 AM

Have a Seagate FreeAgent Pro 500gig I'm playing with. Also has E-Sata so just waiting for a card. Giving it some grief on FW 400 and ok so far. Also we have MyBookPro's as one of the OSX backup drives and one at home also and there doing ok so far.

Only problem with enterprise class drives is they are meant to be in a raid.

s

vanguy

05-27-2007, 09:18 AM

RAID is not that big of a problem, seeing as editing Red footage will require a RAID anyway. But most enterprise solutions cost at least a couple of Gs for 2 TB, and the Lacie is $769 MSRP. And it's RAID 0.

Apart from the firewire 400 whackiness (which can be solved by not moving wires around unless you have to) Lacies have been pretty good to me. Much better than other externals (grrr Maxtor).

I've even had some luck buying empty enclosures and putting Seagates in them. But it's a bit of a crapshoot. Some enclosures are not that great...

Jeff Kilgroe

05-27-2007, 10:15 AM

Much better than other externals (grrr Maxtor).

Hehe, that's funny... Considering Maxtor was LaCie's primary supplier of hard drives for many years. Anyway I'm not a big fan of LaCie products. They're rarely any different or better than most of the consumer stuff on the market. But for some reason graphics and design professionals go for them -- because that's who LaCie primarily markets to. But just consider that LaCie makes very little of their own hardware, they're just an integrator that sticks their name on the products and charges a premium price.

vanguy

05-27-2007, 11:05 AM

I've just finished working on a project where the producer insisted on using a Maxtor external (he'd digitized all his media to it). It crashed the system twice a day. The Lacies never did that. Probably the platters were fine, but something about the Maxtor enclosure is major trouble.

Lacie may be crap, and they may use other people's internals but I've always found they work as good or better than anything in the price range. Except for the weakness of the Firewire connector, they've never failed for me. Sure, an XServe RAID would be better, but it costs $6K for one terabyte, plus the fibre channel card. The Lacie Bigger disk 2TB costs $769. I don't see how the Lacie is overpriced.

The question stands: what is a good, reliable storage system that doesn't cost an arm and a leg; a practical tapeless workflow? If I throw out my Lacies, what should I buy?

Chris Kenny

05-27-2007, 11:49 AM

Why not just get an eSATA card and port multiplier enclosure (like one of these (http://www.macgurus.com/productpages/sata/BurlyPortMultiEncl.php)). Add drives as needed. Fairly cheap, a lot faster than FW800, and probably a much better solution for scaling up your storage than having a bunch of FW drives kicking around. You'll also have a lot more flexibility than with an external FW RIAD (that shows up as a single drive to the host computer), particularly once ZFS shows up in Leopard.

Each slot on one of those enclosures is worth about $100 (an 8-drive enclosure costs $780), so loading the enclosure up with 750 GB drives is actually a little cheaper per GB than loading it with 500 GB drives. Loading up with 1 TB drives only costs a couple cents more per GB, so that's actually not a bad option. That'll get you 8 TB of raw storage for about $4000, once you fill the thing. (Less, if it takes you a while to fill it, since drives will keep coming down in price.)

Adrian T.

05-27-2007, 12:32 PM

Have you guys heard of Drobo?
http://www.drobo.com/products_demo.aspx

Sounds pretty cool, except for the USB interface.

Rob Lohman

05-27-2007, 03:26 PM

Looks interesting. No mention of speed though. Also prints in small letters that for over 2 TB you'll get a second volume. I'm wondering if they need USB to do the dynamic resizing of a volume thing.

sbaechler

05-29-2007, 12:19 AM

I have a La Cie bigger disk and it fails at me from time to time. The drive just dissapears from the system and you have to restart the unit to make it come back. The eSata enclosure would be an interesting option. Does it have hardware RAID support?

Chris Kenny

05-29-2007, 12:26 AM

The eSata enclosure would be an interesting option. Does it have hardware RAID support?

No, but with 4 and 8-core systems kicking around, software RAID isn't quite the burden it once was. And if you're on a Mac, when Leopard comes out you should be able to use ZFS, which is much more flexible and robust than any hardware RAID.

Antoine Baumann

05-30-2007, 04:11 AM

I would think that you could do it a lot cheaper and simpler.

On set:

You could get four RedRAIDs, taking two to the set (5-6 hours recording time).

The other two are at the edit suite being transferred/archived/backed up. When the transfer is done, swap with the set drives. If the edit suite is a MacBook and some large RAID drives (as below), it could be portable, as long as you've got a power supply for the drives.

RED DRIVE are set up in RAID 0 mode, so it is quite risky. I will copy on set the data from the RED DRIVE to a dual Hard drives RAID 1 system and then take out each drives of the enclosure to sent on to the edit suite, and keeping one as backup.

I think CallDigit has nice product, and not more expensive than RED DRIVE.

antoine.

Carlo Rho

05-30-2007, 08:07 AM

I was chatting with g-tech guys at NAB and their new G-SPEED eS (http://www.g-technology.com/Products/pdf/G-SPEED-eS-web.pdf), is up to 4TB, eSATA and apparently able to make 2 copies of the footage at the same time (RAID 0+1, 2 copies spletted in 2 drives each one).
It's the best solution I found at NAB for field back-up.
We've just to convince them to make a ruggerized version!

Carlo

sbaechler

06-27-2007, 03:31 PM

This RAID system sounds very interesting. It's SATA based and comes with a PCI-Express RAID adapter. They advertise especially for HD editing. Check it out:
http://www.sonnettech.com/product/fusiond800raid.html

Mark L. Pederson

06-28-2007, 02:36 AM

I'd never use Lacie in a production enviroment.. but thats just me. Had to many fail in the past. If your data is important use enterprise drives in a raid.

s

I never use Lacie. Period. Wost enclosures on the planet. We run 100% of G-Tech drives and also use Gspeed XL 4 gig raids.

Cail Young

06-28-2007, 05:04 AM

I never use Lacie. Period. Wost enclosures on the planet.

You ain't seen nothing, my friend. At least Lacie shells out for Oxford chipsets (right?). Some of the cheaper Prolific-based firewire drives will conflict with each other, so you can only use one at a time. Great!

Fergus Meiklejohn

06-28-2007, 05:24 AM

I'm giving up on lacie and going g-tech from now on.
Each to their own of course, but I think one can be ott about data backup on set etc (Just because you can..). You can't backup film on-set, most people filming EFG capture onto tape then put the tapes in the hotel safe and backup when they get back to base.
So why should RED be different?

Mark L. Pederson

06-28-2007, 09:02 AM

I'm wondering if they need USB to do the dynamic resizing of a volume thing.

That's what I am thinking. This thing has some NICE features - really nice to scale up - and you can use different size drives, etc. - and price point and flexibility on this make the USB 2 not so bad - I am getting one - I'll ping you after I beat the sh#* out of it - but it is getting stellar reviews -

Jeff Kilgroe

06-28-2007, 10:23 AM

I'm getting one to play with I think. Haven't ordered yet though. I'd really like to see one that holds 8 drives and connects via eSATA.

I'm also thinking they need the USB interface for the dynamic resizing and communication between the unit and the system. Although, dynamic volume sizing should be available through SATA as well. Maybe it's just a limitation because this is an initial product.

Jeff Kilgroe

06-28-2007, 10:32 AM

Gavin Greenwalt

06-28-2007, 11:48 AM

Going back to the earlier bit of the thread. I've heard that Enterprise HDDs are marketing junk and the failure rates are pretty much indistiguishable from 'consumer' HDDs.

Jeff Kilgroe

06-28-2007, 04:58 PM

Going back to the earlier bit of the thread. I've heard that Enterprise HDDs are marketing junk and the failure rates are pretty much indistiguishable from 'consumer' HDDs.

That comment is right on... Because if you look at the OEM components market, there really is no such thing as "enterprise" HDDs. These manufacturers only have a certain number of models and most, if not all, are available in consumer channels. About the closest you can get to anything "enterprise level" is the 15Krpm units, but they're no more reliable, just very fast at a premium price and a bit lower capacity. And they're very popular with a lot of the performance computing PC crowd that likes to have them for bragging rights and/or to stripe 2 or 3 together in a RAID-0 for near instant load times on their favorite games.

Some manufacturers do offer occasional ruggedized models that are intended to be more robust. Usually at the expense of performance and/or capacity. But these are intended for use in industrial applications or military, aerospace, etc.. We see that with the SSD market too. Look at the first crops of SSDs -- 16GB for $12000. Seriously, other than the military or highly-specialized niche markets, who is going to buy that? :matrix:

HARMLESSPictures

07-09-2007, 11:49 PM

Ahhhh, a tapeless world. I think that there will be some time until a complete tapeless workflow becomes a regular part of the production platform. At least down here in Melbourne.

Cail Young

07-10-2007, 02:54 AM

Ahhhh, a tapeless world. I think that there will be some time until a complete tapeless workflow becomes a regular part of the production platform. At least down here in Melbourne.

Prey Alice, a feature, is shooting right now on P2 exclusively. The tipping point is here, I think, given that we're currently data-managing on several student tapeless projects! Once these guys move into the industry, the cycle will be complete. Party time!

Gavin Greenwalt

07-10-2007, 01:41 PM

I'm also thinking that drobo could be a great solution for those looking to use hard drives as backup. Bare drives are easily added and removed from the system. You could put 4x500GB drives in there and fill up the RAID. Then shelve the RAID set and start with clean ones. It would be highly reliable HDD backup with a lot of fail-safe properties. Obviously lower capacity than just using individual or even duplicate raw data to drives, but very secure.

I thought about that but then a terrible thought crossed my mind: What if the Drobo is using some crazy proprietary device mounting system.

Can you pull a drive out of drobo and mount it like any other drive?

If the answer is "no" then you've just signed a 10 year+ contract with DRobo for their hardware. At least with RAID 1 I can take that drive and rebuild it in any Raid 1 enclosure anywhere in the world for the forseeable future.

Drobo just wreaks of the DAT nightmare that once was. Perfect for the home PC. Terrible for long term storage (assuming it works the way I think it does). If I'm wrong and the drives are mountable using generic hardware then it does look like a great archive solution.

Jeff Kilgroe

07-10-2007, 04:29 PM

I thought about that but then a terrible thought crossed my mind: What if the Drobo is using some crazy proprietary device mounting system.

Can you pull a drive out of drobo and mount it like any other drive?

Unfortunately, it appears to be a proprietary RAID style system. So yeah, you would be limited to using or re-mounting those drives in a drobo system to regain access to the data. But for this reason, I was looking at drobo to be an interim solution. It's also unclear if drobo allows for various configurations that may be more manageable for the future. At this point, I think it's all proprietary.

At least with RAID 1 I can take that drive and rebuild it in any Raid 1 enclosure anywhere in the world for the forseeable future.

In theory, yes... But not all RAID controllers mount drives just the same. RAID 1 should be just fine as it's just two mirrored drives. Porting over a RAID 3 or 5 set would be problematic though. I've tried to move a RAID 5 drive set from one server to another after system failures and unless the systems are using the same hardware, it can be problematic.

Drobo just wreaks of the DAT nightmare that once was. Perfect for the home PC. Terrible for long term storage (assuming it works the way I think it does). If I'm wrong and the drives are mountable using generic hardware then it does look like a great archive solution.

I'll see what drobo looks like here in a few days. I've got one being delivered either tomorrow or thursday. But I'm fully expecting it to fill the need of fault-tolerant desktop storage that isn't performance critical. It should serve itself well for that and should also provide a fairly secure means for short-term backups. For long-term though, I think the best is still RAID-1 HDDs and/or duplicate copies made to LTO or DLT. But I'm playing with some other options in the meantime.... Until I decide just which way I'm going... The Exabyte Magnum 224 with LTO-4 drive is sooo tempting, but I'd have to backup 20TB to duplicate tapes over the next year to justify its cost over using 500GB HDDs.

Christoffer Glans

07-10-2007, 05:47 PM

Define "tape".
If P2 or CF cards gets cheaper and larger, would they not take over as "tapes"? The future is not a tapeless workflow, I believe it to be a evolution of "tape". The future is a card workflow.
Picture a CF card able to record 30 minutes of 4K raw. That will happen some day, or at least, some kind of storage in that size will be able to.
I love flash storage...

Jeff Kilgroe

07-10-2007, 07:42 PM

When most people are looking at a "tapeless" workflow... It's the elimination of linear magnetic tape from their workflow. That's a big step for a lot of production pipelines, especially in the broadcast industry. P2 or other solid-state media are definitely a huge factor in a tapeless workflow. I think we'll be seeing other memory options transition to large-scale storage over the next few years. Write-once FLASH or SRAM / PROM however you want to look at it, is a strong possibility for archival storage. Think of a CF card or a P2 or Express Card that can only be written to once, but it holds 320GB. Now what if that card could be bought for $50 or less?

...Just a few more years, it will happen.

Christoffer Glans

07-11-2007, 04:18 AM

Wait, what happens within the card if it can only be written once?
And on another note, what is the difference between that and a blu-ray dual layer that can come up to 100gb? Can CF cards store data for close to an eternity compared to a disc?

Jeff Kilgroe

07-11-2007, 06:57 AM

Solid-state should be more robust than optical disc. It's being shown now that optical discs are good for 12 to 20 years or so. Originally manufacturers were claiming 100 year shelf-life. But now that the tech has been around about 20 years, we're seeing that isn't the case. OTOH, solid state options haven't been around for real-world tests that long either... Time will tell.

The write-once solid-state media isn't true FLASH. It's more of a neutral-state memory structure that can be written to once and each bit defined as an on or off / 1 or 0 as it is "written" or "burned". They haven't hit the market just yet, but Sandisk and Transcend are both supposed to have such products by early next year. Initial capacities are supposed to be about 128GB for a CF style card.

BluRay -- bad idea for archival. So far, shelf life is only claimed to be 15 years. The cost per gigabyte is very high -- it's $30 for a 50GB dual-layer disc. 100GB discs, which are currently two layers per side and you manually have to flip the disc go for about $70 each. 100GB discs that are 4 layers on one side should be available late next year, but Sony is still working on the drives and these will require more powerful (more expensive) laser diodes. We'll see what happens.... But prices on BluRay have to come down before it's a viable archival or storage format. For the cost of a $100GB BD disc, I can buy a 250GB hard drive and lunch.