Cheap eSATA RAID solution

[Rick Darge]

So this is what I'm thinking about doing...

Buying the 4bay Firmtek + 4host eSATA PCI-X card $650

http://firmtek.com/seritek/seritek-2eEN4/

Going to newegg and picking up 4 500GB SATA drives at $140 a pop, $560 for 4

http://www.newegg.com/Product/Produc...82E16822144016


Then I would stripe these 4 together as a Raid 5 - Roughly 1.5 Terabytes of space..

1.5TB = 15 HOURS of RAW 4K if I'm not mistaken

Nice thing about this setup is after I fill up 1.5 TB, I can shelve all 4 drives and drop another $560 for 4 more drives + the firmtek cages at $22 each, making the price roughly $650 for an extra easily swappable 1.5 TB of space.


Crap, this is going to get expensive fast shooting 4k

1.5tb=15hours= $43 per hour = $0.72 per minute +

Still cheaper than film :alien:

[Jeff Kilgroe]

So this is what I'm thinking about doing...

Buying the 4bay Firmtek + 4host eSATA PCI-X card $650

OK, sounds good. Why PCI-X? Considering REDCINE won't run on non-Intel Mac systems and will require a more current PC... Most everything has gone PCI-Express these days.

The 500GB Maxtors are good drives... So are the Seagate and Hitachi units. All perform about the same. Can't buy from a better place than NewEgg either... I love those guys.

RAID5 would be the way to go for reliability. I would consider a straight RAID-0 array for performance though if you plan to do any editing from this array. However, you mention shelving the drives for storage. IMO this is a bad idea. ...I know we've been over this in other threads, but hard drives (even if they are components of a RAID5 configuration) are not designed to sit idle on a shelf for any extended period of time.

Crap, this is going to get expensive fast shooting 4k

This is the other reason swapping out those drives for archival isn't a great idea. LTO-3, DLTS4, and other large format tape solutions are far more reliable for archival and are intended for that very purpose. They are also cheaper than hard drives by a good margin. The tape unit itself is expensive, but once it's cost is recovered, it's far more economical. If I had to buy an archival solution today, I would go for the Exabyte or Dell (ADIC) autoloader units. I can buy brand new Exabyte 12-tape LTO-3 autoloader systems off ebay for less than $3K. Figuring $3500 for the tape system with SCSI interface and cables and $0.15/GB tapes... If you compare that to hard drives ($0.33/GB plus swappable enclosure, interface, gables - $650). The tape system makes a lot of sense.

Compare 20TB of backup capacity -- why I chose 20TB will be immediately clear.

LTO-3: $3500 + $3000 (bulk buy of 50 tapes) = $6500

HDD: 40 * 500G drives = $6500

So the break-even point between the two solutions is about 20TB at about $6500. Lets look at the cost for the next 10TB. Oh, and we'll assume that the price of 500GB HDDs has dropped another 25% in the mean time.

LTO-3: $1500

HDD: $2435

With the amounts of data we'll be working with in 4K, 20TB isn't going to be archiving that much considering original "negatives" or RAW files, plus intermediate data, projects, finals, etc.. 1TB of drive storage is roughly 10.5 hours of REDCODE RAW 4K @ 27MB/s. The first 20TB (210 hours) of archived footage equates to about $31 per hour. From that point forward, LTO3 or DLTS4 would equate to about $7.15/hour.

We also have new tape formats coming up like DLTS5 and LTO-4 which will double capacities and keep tape prices roughly the same. So expect tape offerings to go from $0.15/GB to aout $0.09/GB within the next few months.

[istvanttt]

well you can also consider that the LTO-3 tapes occupy less physical space and have less weight when you have to ship them.

[Curran Giddens]

http://firmtek.com/seritek/seritek-1en2/

I got 2 of these FirmTek SeriTek/1EN2 Dual-Bay Hot-Swap External SATA Enclosures a while ago. My PCI-X card is a Sonnet Tempo 4+4. I use Seagate Barracuda 7200.8 400 GB drives. I would not buy this same setup knowing that Redcine won't run on my G5. I will wait until the new dual quad-core intel Mac Pro's are available and get a PCI Express card with RAID array.

I brought this up on the old dvxuser RED forum:

If the RED-RAID (20-40 drive mini-fridge) had drives that were user-replaceable (doesn't need to be hot-swappable), you could use it as your desktop RAID. When you need to use the RAID for 4k 60 fps or 2k 120 fps, you could just swap out the drives for another set. Just make sure you put the drives back the same order.

I hope the Holographic storage comes down in price quickly. It is supposed to have a long shelf-life, not to mention random-access, high throughput, and high-capacity.

[MikeCurtis]

1.) You'll actually get less than that - 1387GB formatted capacity

2.) Nowhere NEAR fast enough for uncompressed RAW

3.) It makes no sense for you to work in uncompressed RAW if you're on that tight a budget - you're shooting/transferring compressed, so roughly 100GB/hr, and THAT would be a datarate you could do.

4.) Are you Mac or PC? If you're Mac, you have no RAID 5 options with that card. There's no software based RAID 5 for Macs.

5.) Yeah - gotta be on a PCIe based Intel Mac system for Redcine

6.) and if you had 4K footage, how would you monitor and output it? Shoot 4K compressed RAW, then downconvert to HD for lovely results that you could potentially output and watch in a meaningful way. But your 4 drive RAID still too slow for that, probably even at RAID 0.

Not meaning to sound discouraging, just realistic.

-mike

[Rick Darge]

I should have been more clear. I'm not looking to edit in 4k by any means, even when I get my 8-core 8GB Intel-Mac at the end of the summer. I'm just looking for a practical solution of where to store all this data coming in. That's why I was thinking temporary storage would be on a RAID 5 1.38 TB - When that filled up after projects were finished, I would archive to tape or something else. The more I think about it though, the more this whole post idea scares me. we're talking HUGE amounts of data

What's your setup like or going to be like after you get the RED Mike?

[Sean Michael Johnston]

I'm looking at this:

http://www.macgurus.com/productpages...FirewireHS.php

or similar for backup. We'll be buying a few of these and lots of $20 sleds. HDs are about $100 for 250gig and $200 for 500gig.

I was ready to jump on the LTO path. But after talking to a few people who have been using these and researching the software required to run the hardware I realized that even though the capacity has grown, the limitations of a linear storage medium have not. These solutions are not much better than they were 10 years ago when we used DLT to backup our AVID.

A hard drive costs a little more per gigabyte, but when a client calls and asks[demands] that they want to revise that project you did last year and they need you to send a low res quicktime for them to review now. Do you really want to load up an LTO tape[or what if you striped a project across 4 tapes in an autoloader?], run Retrospect and spend an hour or more resurecting an entire project?
Wouldn't it be faster to pull a hard drive off a shelf, plug it into a hot swap drive bay and access that file immediately?
There are also a few good disc catalogging programs that are shareware or freeware that do a good job of keeping track of which hard drive I need to pull down off that shelf. We've been using one of these with DVDRs for years.

[Jeff Kilgroe]

I was ready to jump on the LTO path. But after talking to a few people who have been using these and researching the software required to run the hardware I realized that even though the capacity has grown, the limitations of a linear storage medium have not. These solutions are not much better than they were 10 years ago when we used DLT to backup our AVID.

Well, there's a few things about tape that are being ignored here. Autoloaders don't run striped volumes. They're spanned from one tape to the next. Firewire isn't all that fast and individual hard drives aren't going to be much (if any) faster than reading tape. If you have proper backup procedures in place, you can pull the proper tape (or tapes) from your archives and load them just as fast as you could from individual hard drives. Sure, tapes are linear, but a proper tape system with good backup software will tell you which tape you need and it will fast forward right to the data it needs.

I know that a lot of people have a dislike of tape systems... I know several IT guys that detest tape just because it's tape. Their opinions have nothing to do with how reliable or practical of a solution it is and they continuously go on and on about how other storage media is far superior. I've got a buddy that preaches holographic storage as if it's going to save the world. Fine, great, sounds good... Where can I buy it? But there's a real flaw with that logic. Other than hard drives, no other media provides equivalent capacity and nothing touches tape in terms of TCO and ROI numbers when you start considering large volumes of data. Personally, I would feel very comfortable if I have to pull data from an LTO-3 tape 5 years from now. I would not feel so good about pulling data from a 500GB hard drive that has been boxed up for that long. And in a year or two after I have been using my RED One and I've got over 100TB of archived data. I know that I'll be happy I don't have $30K invested in hard drives, but rather $12K in a format that is designed from the ground up to sit on shelves with fewer moving parts and less chance of magnetic dispersion and corruption of the data itself, which tends to happen over extended periods of time on hard drive platters. Not to mention the motors, internal lubricants, head mechanisms, etc.. that have just been sitting there for 5 years or more... Before the next great backup format comes along (holographic?) and I can revise and condense all the tapes to something more efficeint and hopefully something more reliable.

I do consider tape to be more reliable than HDD... I've been using both for years and have managed large SANs and helped run datacenters. While I love the idea of hammering that final nail into the proverbial coffin for all tape formats, that day just isn't here yet. When it comes to long-term archivial of tens and hundreds of terrabytes, even petabytes and zetabytes of data, the world still operates on tape. Large companies have various levels of backup solutions and ways to get at their data. But some of the biggest out there, like the library of congress for example, still uses tape as their final archival step and will do so for the forseeable future.

For people planning to shoot a lot of video with RED (200+ hours per year), they would be best served to have a SAN in place for short term backup, project workspace, etc.. Archival to a common tape format, perhaps consider secondary backups of the really important stuff to additional tapes and/or smaller projects to optical media as well... If the price of optical media ever comes down. Right now a 50GB BD+RE disc is the same price as a 400GB LTO-3 tape.

[Sean Michael Johnston]

Thanks Jeff, do you have an LTO system in house now?
I'm still on the fence about which solution we're going with. We'll probably be living with it for a long time. Any user feedback is much appreciated.

[Jeff Kilgroe]

No LTO system here now, but I have used them and some of the other sytsems like DLT-S3. Currently everything here is on RAID-5 storage accessible via SAN. My SAN consists of an XServe RAID and two other PC nodes each with a RAID-5 array. I'm using a VXA-2 tape system for archival (160GB uncompressed per tape) for archival of my animation and video - mostly DVCPROHD via HVX200. A single slot VXA-2 drive isn't going to cut it for RED and its data requirements and I plan to buy something new when my RED ships. I'm planning on either LTO-3 or DLT-S4, but hopefully LTO-4 or DLT-S5 will be available later this year in time for my RED. As for workstations, I have a couple G5 quads, a C2D Macbook Pro and a few PCs. My latest PC is an 8-core 2.33GHz system I just built about 6 weeks ago. I plan to buy a new 8-core Mac Pro and retire one of the G5 quads to render farm duty sometime later this year. Most of my workstations are configured with a system drive and two additional drives in a RAID-0 stripe for workspace, etc.. My primary G5 quad just has two 500GB drives in a RAID-0 stripe as its primary and only HDD volume. I keep a few external fw/usb2 drives handy as well.