Storage for RED. . . Merged Threads

[Manfred Lopez]

From today's NY Times:

********************************************
"Much worse, to keep the enormous swarm of data produced when a picture is “born digital” — that is, produced using all-electronic processes, rather than relying wholly or partially on film — pushes the cost of preservation to $208,569 a year, vastly higher than the $486 it costs to toss the equivalent camera negatives, audio recordings, on-set photographs and annotated scripts of an all-film production into the cold-storage vault."
********************************************

http://www.nytimes.com/2007/12/23/business/media/23steal.html?_r=1&oref=slogin


How on earth do they manage to spend all that money on preservation? The $486 for film seems realistic... but over 200K a year for digital? Are they hiring people to each memorize 3 number at the time from the data string? Or are they chiseling the ones and zeros on limestone tablets? Why so much?

[dino g]

I have stored neg at the producers film group and it is much more expensive than that for a features worth of neg, also, if there is any sort of release you have to add the prints too.

as far as storing data, are they quoting from the IRON MT rate card?

[Manfred Lopez]

The article talks about a report from the science and technology council of the Academy of Motion Picture Arts and Sciences that came out recently (but that was overshadowed by the writer's strike). The $486 is what it costs to ship everything to a vault in a salt mine in Kansas and forget about it. I don't know what the "IRON MT rate card" is (care to explain?), but I would guess that their figures are correct.

[Dan Hudgins]

From today's NY Times:

********************************************
"...$208,569 a year, vastly higher than the $486 it costs to toss the equivalent camera negatives,"
********************************************
Why so much?

Lets figure a 10:1 ratio for a 120 minute feature, plus 5:1 for work files and final frames at a resolution of 4096x2048x48bpp (6 bytes per pixel).

Each frame is about: 50.3MB (4096x2048x6)

120 minutes is: 172800 frames (24x60x120)

Total frames is: 15 * 172800 * 50.3MB = 130377600MB or 131TB

To store on 500GB HD would take,

131TB/0.5TB=262 500MB HD, 262*$120=$31400.00 USD

So it looks like HD backup would be about $50,000 maybe if you pack the disks in air tight boxes with Desiccants.

To back up on DVD-R would be about,

131TB/0.0047TB=27873 disks, 27873*$0.17=$4739 USD

So it looks like DVD-R backup would be about $5000 maybe if you pack the disks in air tight boxes with Desiccants.

Maybe DVD-R would last better than HD, DVD-R may last 100 years, like film, and could be copied every 80 years without data loss, where as film cannot be copied without 15% to 40% loss of data quality (film loss seems less in generations that are themselves copies).

If the files are ZIPPED there might be a reduction of 30% in the size of the data to archive. To keep the data on live servers would cost much more, but I do not see where that is needed since film can be put away, why could not optical disks? And optical disks can be copied to get 100% of the data in the copy.

Tapes start to demagnetize over about 10 years and need to be copied, so may require more upkeep than optical disks.

This is an issue for me since the file servers I am planning on setting up for the raw and working files will need to be emptied to work on the next project, which means taking the HD out or copying the data off to optical disk or tape. If Ubuntu or some other free OS is used on the file servers, then new HD can be put in without having to get more OS and application licenses. Also if the HD are taken out, they can be put back in if additional work is required on the project without the time delay of copying the data back in and organizing it.

If the files are compressed using a JPG2000 codec you might be able to store the 4k frame images and audio on $800 worth of DVD-R disks? If you store in lossy compressed formats you cannot make changes later without re-encoding losses of some kind.

Everybody working with Digital Intermediate will face some issues with what to save and what to erase, and what to archive in several backup copies.

I would be interested in hearing what others here plan to do to archive their feature projects for long term, and what format they will keep the final frames or streams in, and what format they will keep the raw and work files and streams in (if the camera footage and outtakes will be kept?)?

I have been asked before why I am interested in uncompressed 4k, today that is not important, but HD is going to be obsolete soon, and uncompressed 4k is about the limit of what people can see on a screen with a normal angle of view. The "Lucy show" was shot on 35mm film which was overkill for the TV of the 50's, but it still shows today, not only for its production value, but also because being shot on 35mm and not kept as 16mm kinescopes it is still fresh looking and marketable. So shooting with all the resolution you can get, or at least up to good 35mm standards, will allow some lasting viewability. 70mm is better than 35mm, and when watching 70mm shot 65mm under good conditions, you see a more lifelike image since your eyes can look around the frame and probe small details that would just be fuzz on 35mm, but since most people sit back in the theater they seem satisfied with 35mm resolution.

I bring this up since if you down res the 4k frames to 2k, to get "true" 2k frames, (then erase the 4k files) the costs to archive would be about 1/4 or $12500 for HD or $1250 for DVD-R, plus storage somewhere (safe)?

Anyone have experience archiving Digital Intermediates? If so, what were your costs for the media, and annual storage?

[dino g]

I don't know what the "IRON MT rate card" is (care to explain?), but I would guess that their figures are correct.

IRON MOUNTAIN.....http://www.ironmountain.com/index.asp (I was a partner in a company that sold 1080 & 3480 tape to DEC, WANG, MIT & others in the late 80's...i know a little about tape backup).


and the LUCY footage is stored 15 ft from my stereoscopic footage at Producer film center http://www.filmstorage.net/, the building is right by Target in the hollywood/west hollywood gateway by the intersection of la brea and santa monica.

from the PFC website..."Producers Film Center is centrally-located in the heart of the entertainment capital of the world - Hollywood, CA. We specialize in the storage of film and tape, positive/negative film elements, audio/video tapes (all formats), and much more. Our current client list includes Foto-Kem Industries, Lucille Ball Productions/Desilu Too, Danny Thomas/William Morris Agency, Jack La Lanne and Befit Enterprises, The Price is Right, LLC, and many others. We offer to these companies as well as to you....."

PFC charges a flat rate to put each of your elements (a single tape, a 1000 foot neg, a disc, a drive...whatever), into the system, (it was $.25 in 1995, not sure now), and then a reasonable rate per month for as long as you can pay your bill.

[Manfred Lopez]

Maybe DVD-R would last better than HD, DVD-R may last 100 years, like film, and could be copied every 80 years without data loss,


DVD-R lasts 100 years??? The NY Times article said that regularly pressed DVD's will have problems lasting 15 years... and those are made more permanent than DVD-R's. The article also said that NASA already has problems accessing data from an early Apollo mission from the 60's because all the formats used have become obsolete. What makes you think that DVD-R's will still exist 80 years from now?

From what I understand what lasts the longest (and is proven) is B&W separations on film (each color channel separated onto its own B&W stock). By the way, the article also said that hard drives have a tendency to freeze up if not put to use for 2 or 3 years. I didn't know this. :blink:

[Manfred Lopez]

IRON MOUNTAIN.....http://www.ironmountain.com/index.asp (I was a partner in a company that sold 1080 & 3480 tape to DEC, WANG, MIT & others in the late 80's...i know a little about tape backup).


Thanks... Good to know.

[Dan Hudgins]

DVD-R lasts 100 years??? The NY Times article said that regularly pressed DVD's will have problems lasting 15 years... and those are made more permanent than DVD-R's. The article also said that NASA already has problems accessing data from an early Apollo mission from the 60's because all the formats used have become obsolete. What makes you think that DVD-R's will still exist 80 years from now?

There may be issues with the polymers, but if you keep them cold and very dry they may last longer, how cold and how dry is best who knows, you would need to have two copies and check them from time to time, if you need to recopy every 10 years, then 100 years would cost $50,000 rather than $5,000... plus labor.

The "still exist" issues, means that you may need to stockpile computer equipment to be able to read the data disks or tapes, even 10 years from now. This is a big issue with Quadraplex 2" video tape since the day will come soon I guess when there will no longer be any decks that are able to play the tapes anymore, if there is still enough signal on the tapes to play...


From what I understand what lasts the longest (and is proven) is B&W separations on film (each color channel separated onto its own B&W stock). By the way, the article also said that hard drives have a tendency to freeze up if not put to use for 2 or 3 years. I didn't know this. :blink:

It could be possable to record Digital Cinema on fine grain black and white film like ortho-sound recording film, it has up to about 1000lp/mm. You would need about 1000' per minute to record uncompressed maybe, so the cost to archive Digital Cinema on film would be about $200-$500 per minute maybe, maybe Kodak (tm) should offer a low cost Microfilm type product, like they made Kinescope film in days gone by, for digital image archive storage, since it is known that film can last maybe 100 years or more in cold dry storage.

If film base is stable for 100 years, you might burn data dots directly into the film base with a laser rather than use emulsion, it might cost less and be more stable...

I do not favor HD for archive from my experience, maybe HD mfg could work on a drive without lube that would not stick or mold in storage...

A laser could cut glass disks (or ceramic) without coating, that would last thousands of years maybe, perhaps there is another business opportunity for the RED (tm) team to make a long term achive media. I favor media that have a visible recording, rather than a magnetic one...

Anyway, I was interested to hear what media people ARE using to archive their Digital Intermediates and how much their cost is for that media?

[Jeff Kilgroe]

It could be possable to record Digital Cinema on fine grain black and white film like ortho-sound recording film, it has up to about 1000lp/mm. You would need about 1000' per minute to record uncompressed maybe, so the cost to archive Digital Cinema on film would be about $200-$500 per minute maybe, maybe Kodak (tm) should offer a low cost Microfilm type product, like they made Kinescope film in days gone by, for digital image archive storage, since it is known that film can last maybe 100 years or more in cold dry storage.

Here's my take on the long-term archival... Of course, my point of view will probably change in the future as it has over the years due to changes in technology.

For large storage quantities like this. Conventional hard drives are NOT the answer. Hard drives do not have the lowest $ to GB ratio, nor are they designed to sit on shelves, harboring data for long periods of time. To pull from Dan's numbers, the 131TB, it's about as cut and dry for a 120min feature at that resolution. There will probably be 3 times as much storage involved in saving all the intermediate working data. But we'll just work with the 131TB.

500GB HDD's are the most logical for HDD storage as they provide the best $/GB ratio. I'll have to adjust the price -- they can be had for $95 now, and actually if you buy cartons of 100 units, the price can be $80 each after freight charges. However, 500GB drives are not 500GB -- It's more like 465GB formatted capacity for NTFS or journaled HPFS+.

131TB / 0.465TB = 281.72 == 282 hard drives

282 * $85 (somewhat conservative ballpark figure) = $23,970


Now look at LTO-3 tape:

12-slot auto loader unit with single tape mechanism - $4750 w/ interface card & cable + 1 tape.

400GB tapes (378GB usable uncompressed) - $32 each in large bulk.

131TB / 0.378TB = 346.5 == 347 tapes

347 x $32 = $11,104 + $4750 == $15,854 Hmmm...

And relative price from there continues to go down since the cost of that tape drive has obviously been paid off. Plan on doubling or even tripling that hard drive cost number if you go that route. I never trust a long-term backup of anything unless I have two copies stored in two different places. With conventional hard drives, I'd make a 3rd copy just to be safe.

LTO-4 will be a better option within the next year. For now, it's probably still more cost-effective to buy LTO-3 and then upgrade the tape drive to LTO-4 sometime in the next year or so. LTO-4 mechanisms are currently 50% more expensive than LTO-3 and the tapes hold 2X as much for 3X the price.

No matter which form of digital storage media is chosen, it would be wide to re-organize, consolidate and re-copy every 7 to 10 years. This would increase reliability and eventually the physical size of the storage media would diminish to something that will fit in your shirt pocket. This would also virtually eliminate concerns over format obsolescence as well as longevity concerns over media. CD-R was supposed to last 100 years, and it's been shown that real-world expectancies, even under favorable conditions are 12 to 20 years in most cases. Blu-Ray is now trying that 100 year claim again, but the primary difference between Blu-Ray, DVD-R, and CD-R is not the photochemical elements, but the plastic or metallic substrate on which these dyes are applied. I have a strong suspicion that Blu-Ray discs won't last 30 years, let alone 100 years. I most likely won't be around to see the results of any tests for the latter. But that's the only way to truly test it - store some for 100 years and see what happens. Accelerated testing in a lab can replicate a lot, but it can't replicate time. And accelerating biodegradation with heat, various light wavelengths, ultrasound, vibration, etc.. is not the same thing by a large margin.

You wrote above about B&W film to encode data. But that's essentially the same thing as recording to optical disc media. Most of the photo-emulsive chemicals used on disc media is the same as what's used in film. There's obviously no developer process, but the principals are the same. Encoding data to film could become problematic due to the sheer size of the film reels needed compared to other data storage formats. There is no standard for such or film type that provides accurate enough registration for the consistent alignment of read/write instruments. There would have to be a running registration pattern encoded to the side of the data stream so that a high-precision reading system could track that to keep in sync with the data. Film degrades over time. Sure, it lasts 100 years or more, but it still degrades. And what may be optically unnoticeable to the human eye, under magnification, for a photographic image, may turn out to be catastrophic data failure if only a handful of bits on that data film have started to bleed or fade. ...This is how optical disc media goes bad. The photochemicals change, bleed, fade, etc.. over time. Just like film.

Magnetic hard drives also suffer from similar changes in state over time as the data bits are not photochemically changed dyes, but magnetically charged bits of metal laminated on an insulating plate. Over time they can lose their charge, charges can migrate or become altered due to close proximity to one another or due to distortions or shifts in ambient magnetic fields. So I'm lest likely to trust a hard drive that has sat on a shelf for 10 years than I am to trust a DVD-R that was stored right next to it. Magnetic tape (like LTO-3) also suffers from this degradation too, but the write patterns, data block sizes, etc.. are designed with long-term shelf storage in mind.

[jbeale]

"$208,569 a year" ... I still can't work out how that's a yearly recurring cost. You buy new hard drives every year?

I like the idea of an optical laser-etched glass/ceramic disc that could last a thousand years but the market for that is probably small. As I recall, when the NIST working group on optical media longevity did an industry survey, they found few to no businesses wanted to pay anything above bare minimum price for DVD-R media. The rationale is they don't trust DVDs to last no matter what, so why pay more. I think the military was one of few groups indicating interest in an "archival" optical media type.

I saw a couple of papers on media stability in 2004, but nothing since. As far as I know currently there is no research happening on this.
http://nvl.nist.gov/pub/nistpubs/jres/109/5/j95sla.pdf
http://www.osta.org/technology/pdf/whitepapers/NIST-6-14-04.pdf

[Dan Hudgins]

Now look at LTO-3 tape:

12-slot auto loader unit with single tape mechanism - $4750 w/ interface card & cable + 1 tape.

400GB tapes (378GB usable uncompressed) - $32 each in large bulk.

131TB / 0.378TB = 346.5 == 347 tapes

347 x $32 = $11,104 + $4750 == $15,854 Hmmm...



Thanks for the tape backup estimate, there is some issue of the write time, but I am sure it would be less than for DVD-R and more than HD?

This site says that DVD-RAM are said to have a life of 30 years,

http://www.proactionmedia.com/dvd_media_formats.htm

So I do not know if DVD-RAM are better than DVD-R for archive?

About the film idea, it would be better to leave the emulsion off of the film and just use the film base, the IR laser would burn holes half way through the film base. Thin Polyester Film Base would make rolls smaller. The information would be scanned using a line sensor array like in a flat bed scanner that has about 16x resolution (16384 pixels) so that the pits in the base could be tracked by software processing rather than by mechanical means, it would work like a parallel track tape recorder, maybe recording 1024 parallel tracks at once. The size of the pits could be adjusted to use a base higher than 2 to decrease the area required, and the data could be written so that error correction could be used to extract the data better and the images of the film surface could be used to extract the data from surface durt and marks. The focal plane could be inside the film base, and not on the front or back, so that surface marks would be out of focus. 70mm or wider rolls could be used to get 4096 or more tracks. It would look something like the "tape" deck in "Brainstorm" (1983).

Here seems to be a LOT-3 tape in single for $34.87,

http://www.tape4backup.com/lto-3-tapes.php

Here seems to be an autoloder for $4,095.00, and a manual unit for, $1,955.00,

http://www.tape4backup.com/lto-3-drives.php

Is anyone using LOT-3 tapes under Linux like Ubuntu? I was wondering what software you would recommend for use with the LOT-3 drives to swap files off of and back onto HD under various OS?

[Stephen Pruitt]

Hi all. . .

I'm a true newbie on the RED, but I have been doing some calculations on the amount of storage my DIT will need for a feature we'll be shooting this summer.

Let's say that the film will run 120 minutes. Let's say we have a shooting ratio of 10 to 1, so that's 1200 total minutes of footage.

Since an 8 gig flash card holds 4 minutes of data, what means that I'd need the equivalent of 300 flash cards. 300 x 8 gigs means 2400 gigs of storage.

Let's say that I want each file stored three times. That's roughly 7500 gig.

Sooooooooooo. . .it seems like 8 1TB hard drives ought to do the trick.

Or did I grossly miscalculate something somewhere????

Just don't tell me I'll need $200,000 worth of storage!

:-)

Thanks much.

Stephen

[Darren Orange]

Yes your Logic works...8 - 1TB's should be far more than you need.

[Teague Kennedy]

Just use the same card or two and transfer to hard drive. More like a couple thousand bucks, that way.

[Steve Sherrick]

Let's say for the sake of argument that you'll shoot 45 minutes of footage a day (might be more, might be less but we'll use it as a benchmark). 12 CF card would have you covered for a normal day of shooting. During the shoot or at the end of the day that material is backed up to data DVDs (only the best of the best) and to hard drive (many possibilities here). In a perfect world you would have backup software that can do a reliable checksum so that you know your data has made it properly onto the backup medium. Okay, let's look at particulars.

1. CF Cards. Red is the only reliable one so far as far as we can tell. There cards are $200. Total investment - $2400

2. DVDs - Delkin archive quality DVD-Rs - $1854 (going by your estimates for footage)
http://www.delkin.com/products/archivalgold/dvdr.html

3. Hard Drives - Many options. Personally, I would be nervous with the 1TB drives for archive because it's a lot of eggs in one basket. What about something like this. The Wiebetech dock with OEM bare drives in 320GB form factors. $200 for dock, $75 per 320gb Hitachi drive so 8 of them costs $600. If you want to copy to yet another set, add $600. So let's say $1500 total.

This kind of solution would run you about $5800 to archive all of your data from the shoot. You would be re-using the CF cards in this scenario, but it will be infinitely cheaper. The DIT will have to be careful when using the bare drives to make sure they are wearing static bracelet (which they should be anyway), and that they make sure to keep the drives packaged well and labeled well.

Of course there are many other ways (tape drives, Raids, etc) But this is a budget way to do it.

Steve

[Stephen Pruitt]

This is great news and a lot of great suggestions, to boot!

Thanks so much!

Stephen

[Steve Sherrick]

For those who don't know about the Wiebetech products, here's the link.

http://www.wiebetech.com/products/ComboDock.php#pricingavailability

Steve

[Stephen Pruitt]

Hey there, Steve. . .

How many hard drives can one Wiebetech device run? Eight? It only looks like it works with one from their picture. Obviously I'm not seeing things properly.

So I could run THREE of these set ups into one Mac to get 7.5TB of storage???

Very interesting.

Stephen

[Jeff Kilgroe]

My one piece of advice, if you're figuring costs of drives for storing footage... Double your numbers, because you're going to need at least one back-up copy of what you're putting on those drives. And I wouldn't recommend trusting your luck and the fate of your project to a single set of hard drives. :)

[Jeff Kilgroe]

LTO-3 tape drives can usually sustain write speeds in excess of 60MB/s. So they're actually quite speedy. Just figure about 1 hour 45 min or 2 hours to fill an entire 400GB LTO-3 tape. LTO-4 is faster, but that goes with the higher capacity tapes and increased price.

The film storage idea is interesting. In the end, it may be far less efficient and cost effective because there is no standard to work with and it will take a fair amount of physical storage space. It seems more practical to just go with tried and true backup methodologies like LTO tape. A lot of people here seem to be skittish about data storage formats, many are set on using hard drive because of convenience and/or they're familiar with it. But LTO tape (as well as DLT-S formats) are proven technology used all over the world. Large insurance companies, the US Library of Congress, etc.. I don't see where making a proprietary backup system based on burning micro-dots into film base makes a whole lot of sense. But that's just me... I've seen similar film-based concepts come up here and elsewhere over the past year or two. I think once again it's a level of familiarity with what people know.

[Martin Drew]

This kind of solution would run you about $5800 to archive all of your data from the shoot. You would be re-using the CF cards in this scenario, but it will be infinitely cheaper.

Quite a bit cheaper, but not infinitely cheaper... :devil:

M

[I Bloom]

More and smaller drives are more secure but more expensive. I find I'm usually going with pairs of $500 gig drives.

IBloom

[Steve Sherrick]

Yes, that is a better way to put it Martin.
Quite a bit cheaper, but not infinitely cheaper... :devil:

M

[Steve Sherrick]

More and smaller drives are more secure but more expensive. I find I'm usually going with pairs of $500 gig drives.

IBloom

I think 500gb drives would be fine too. Spreading out data over multiple drives seems safer to me in a feature film scenario.

Steve

[Steve Sherrick]

I suppose you could run more than one. I use only one. Backup to that drive, check data, then off it goes into archive. This is something you would use to get the data on the drive, then pull that drive and hook up another one. Not meant to run a drive for long periods of time.

Steve


Hey there, Steve. . .

How many hard drives can one Wiebetech device run? Eight? It only looks like it works with one from their picture. Obviously I'm not seeing things properly.

So I could run THREE of these set ups into one Mac to get 7.5TB of storage???

Very interesting.

Stephen

[Seth Larney]

My one piece of advice, if you're figuring costs of drives for storing footage... Double your numbers, because you're going to need at least one back-up copy of what you're putting on those drives. And I wouldn't recommend trusting your luck and the fate of your project to a single set of hard drives. :)

I wouldn't trust my backups to two sets of hard drives either.

The only solution I would EVER use is one redundantly RAIDed hdd copy for online use, and two LTO or DLT bakups in seperate physical locations.

Do NOT trust our information to DVD's, Flash cards OR Hard drives, even if you have mutiple copies. DVD's and Hard Drives WILL fail after a few years, guaranteed and Flash media is unstable.

The problem with DVD's is that you will burn 10 backups on 10 different discs' then in 3 years you will take them out and EVERY SINGLE ONE will be unreadable. When they go, they tend to go together.

Data tape is a bit of an investment, but besides the camera, it may be the best one you ever make.

Cheers,
Seth.

[Chris Kenny]

When calculating $/GB for hard drives, remember to factor in the value of the slot the drive goes in. A 500 GB drive might have a lower cost per gigabyte than a 750 GB drive just looking at the bare drive... but if you stick it in a 5-bay enclosure that costs $500, each slot is costing you $100 no matter what size drive is in it.

Looking at some drive prices off of NewEgg, I see a 750 GB model for $190. Add the cost of the slot: $290/750 = $0.39/GB. With a 500 GB drive at $105, you get $205/500 = $0.41/GB. With 1000 GB drives, you get $410/1000 = $0.41. So, in terms of simple cost per gigabyte, 750 is the sweet spot here.

I'd probably just go for the 1000 GB drives in this instance, though. Spending an extra $20/TB vs. the 750 GB drives seems well worth it to have 1/3 fewer drives taking up space, making noise, and using power.

As far as the "more small drives means safer" argument... I don't really buy it. The more drives you have, the more likely you are to lose one. And losing, say, 320 GB of data will generally ruin a project just a completely as losing 1000 GB of data. If you have proper backups, you'll be fine either way, it'll just take slightly longer to restore with larger drives. But if that sort of downtime is a major issue, the proper solution is really fault-tolerant RAID, not storing data spread out on lots of small drives.

[Martin Ludwig]

take attention if looking for hard drives that they have firewire 800 port. thats a lot more speed - and you will enjoy it if the transfer is fast enough

[Gunleik Groven]

eSATA is pretty cool too, but the advice of redundant RAIDS is so far the best (hard-drive) based one.

Gunleik

[Michael Brennan]

Lets keep an eye on the write once read many flash cards in development too

And the new firewire standard S3200 that will run at 3.2gb/s


Mike Brennan

[Jonathan L. Bowen]

Are you guys saying that my DVDs, like my average studio film I bought at retail, won't work after 15 years? Because I have a lot of trouble believing that. I have plenty of discs from 1997 and 1998 and it's almost ten years later, they work fine, no reason they won't work fine another ten years from now as long as they're not scratched...

I don't expect anything to last forever, but I don't see any reason they'd stop working either. I have like 1,800 discs too, dang, that would suck if they did ;)

I think that article is crazy. It's not that hard backing up data, if you just have two backups and check them every once in a while it should be fine, it's not rocket science. Digital backup is far superior to film backup, that was an early reason for adoption, the Star Wars prints looked like crap after less than 20 years. They needed major restoration for the Special Edition re-releases.

[oldphart]

I think 500gb drives would be fine too. Spreading out data over multiple drives seems safer to me in a feature film scenario.

Steve

That depends on HOW you spread it. If you use striping, fewer drives will be much safer than many. In a striped (RAID-0) array, you lose ALL data if ONE drive goes bad. With RAID-5, you will have redundancy so that you can incrementally fix single-drive or even two-drive failures.

If you use a proprietary RAID-controller, you'll be screwed when the controller fails and the maker is long out of business. I will trust IBM ServeRAID and standard Linux software RAID.

There are 3, 4 and 5-platter Terabyte drives on the market. The fewer platters, the less heat buildup and the higher reliability you can expect. I would consider the new Samsung drives as likely to be reliable, while the early TB-drives seem a bit more risky.

You NEED linear tape for backup, though. It is the only archival quality digital medium available.

[Steve Sherrick]

I wouldn't trust my backups to two sets of hard drives either.

The only solution I would EVER use is one redundantly RAIDed hdd copy for online use, and two LTO or DLT bakups in seperate physical locations.

Do NOT trust our information to DVD's, Flash cards OR Hard drives, even if you have mutiple copies. DVD's and Hard Drives WILL fail after a few years, guaranteed and Flash media is unstable.

The problem with DVD's is that you will burn 10 backups on 10 different discs' then in 3 years you will take them out and EVERY SINGLE ONE will be unreadable. When they go, they tend to go together.

Data tape is a bit of an investment, but besides the camera, it may be the best one you ever make.

Cheers,
Seth.
Although in principle I agree with you about the tape backup, I was assuming he is after a budget solution. Of course Raids and tape backup will offer even more stability, but this won't be everyone's plan.

And about DVDs. Do they go bad, yes. Have I had very many go bad, no. If they are stored properly and you don't stick paper labels on them or anything else that can damage them, then they should hold up for a while. Delkin says 100 year life on their archive DVDs. Are they lying? Doubt they are lying, but the real world results may differ from their test laboratory. But even if you can get 30 years, that's good enough. You can always back up to another more reliable format in the future.

I guess my argument at this point is that all of the storage is to a degree volatile. So you have your redundant raided drive and everything is great and then it takes a huge power hit and the whole thing is destroyed. Is this likely to happen, no. The point is, I think you have to look at solutions that fit within your budget and get the best solution you can. For some, a network attached LTO drive is going to cost more than they can afford, so they'll look to other solutions.

We could argue all day whether smaller drives with media spread out or one large drive with everything on it is the best way to go. The bottom line is, they are different solutions, and people will have had good or bad experiences with either and that will lead to a decision one way or the other.

I have mentioned before how I will be backing up. It does involve Raids and LTO. In fact, I'm developing a backup system that can do a lot of things and be portable on set. But I know everyone's needs will be different.

Anyway, there have been some good suggestions, just wanted to clarify why I suggested the approach I did.

Steve

[Jeff Kilgroe]

Are you guys saying that my DVDs, like my average studio film I bought at retail, won't work after 15 years?

No, no, no... :) Just recordable optical media that works by phase-changing photochemical dyes with a laser. Over time that dye is going to alter or degrade. No one has come up with one that doesn't. Manufactured audio / video discs should last much longer, provided you don't bake them in direct sunlight or put ones with foil substrate in your microwave oven or that sort of thing.


I think that article is crazy. It's not that hard backing up data, if you just have two backups and check them every once in a while it should be fine, it's not rocket science. Digital backup is far superior to film backup, that was an early reason for adoption, the Star Wars prints looked like crap after less than 20 years. They needed major restoration for the Special Edition re-releases.

Agreed. That's basically what we were saying... Or at least many of us in this thread. The pricing for hard drives to store 120 minutes of uncompressed, 4K @ 16bits/channel is the 131TB number. For hard drives we were coming up with prices 1/5th what that article was talking about -- assuming dual-drive backups. Tape formats working out quite a bit cheaper than that.

Digital backup isn't rocket science, but for some reason people not familiar with it are scared of it or don't want to trust it. Those of us coming from a strong IT or tech background who are already familiar with the various options and how to approach massive data archival don't fear this stuff. The best approach is to know the capabilities and limitations of the digital realm. This isn't film where you need to keep your master prints and negs in an environmentally-controlled vault and hope it lasts 100 years or more. It's the digital era where digital media happily sits on a shelf in its plastic case and you make a duplicate copy or two just to be safe, keep them in different locations because you're more likely to lose them to flood or fire than anything else. Re-vist the backups every few years to consolidate and update onto more current media to physically down-size the archival collection. Always keep your current media backup plus the copies on the previous media format, purge anything older than that whenever you do a transfer to new media. Easy. And cheaper than paying for space in those film vaults.

[Nikolai Vavilov]

Why copy each file three times?
My way: WesternDigital My Book Pro 2TB external drives in RAID1 mode.
Each drive costs $650, so you need less than $2000 to safely store 2400 gigs.

[Steve Sherrick]

Nikolai, this is relative. First, Western Digital drives are not at the top of my list in reliability. Second, if we are talking safety, storing the files in separate locations is critical. You are not wrong at all. That is the point of my last post. One man's safety is another man's lack of sleep. It's all relative. This could work just fine, it's cost effective. The question is will it meet the requirements of your clients? If it does, then you are golden.

Steve

[Nikolai Vavilov]

Steve, using RAID 1 mode means storing in separate locations. There is two 1TB HDD's inside MyBook drive, each file has it's mirror on other drive. How high is chance to lose two drives at same time? Well, to feel absolutely safe, you can double the number of drives to gain four mirrors of your files, it will cost less than 4000, not so much for a full feature.

[Mark L. Pederson]

Well ... anyone who thinks it costs $208,569 a year to protect that digital media has got the wrong vendor and/or "expert" advising them, - or, it's Kodak propaganda ...

[Steve Sherrick]

Separate locations as in place they are stored. One in the office and one someplace else (home, second office, storage facility, etc). Again, this is if your productions need this kind of safety. Some will need this, others won't. I think your Raid workflow is fine.

Steve

[chuckt]

Hi all. . .

I'm a true newbie on the RED, but I have been doing some calculations on the amount of storage my DIT will need for a feature we'll be shooting this summer.

Let's say that the film will run 120 minutes. Let's say we have a shooting ratio of 10 to 1, so that's 1200 total minutes of footage.

Since an 8 gig flash card holds 4 minutes of data, what means that I'd need the equivalent of 300 flash cards. 300 x 8 gigs means 2400 gigs of storage.

Let's say that I want each file stored three times. That's roughly 7500 gig.

Sooooooooooo. . .it seems like 8 1TB hard drives ought to do the trick.

Or did I grossly miscalculate something somewhere????

Just don't tell me I'll need $200,000 worth of storage!

:-)

Thanks much.

Stephen

Now, you are talking about a problem inherent and pervasive in the digital information systems. How to Preserve the data ?
All my digitally scanned photos and videos were lost over the years. Formats and standards change. The old systems become obsolete and won't work. The systems cannot read old formats. 12" and 5 1/4 " and 3 1/2 " floppys died and took all their data with them. The electronic media dies.
I still have the old photos in albums and videos in films. The actual cost of preservation is very high with electronic media. 35 mm film can last up to 100 years. Electric outages and crashed heads won't affect them. Data can always be retrieved with naked eye or with a magnifying glass.
Even though the initial cost of recording on digital is low, if you intend to keep it, you will pay hundreds of times more than for 35 mm film. (And in the end , after all that expense, there is no guarantee that you will ever be able usably retrieve data. )

[SF Geek]

I thought an LTO deck was around $2k-$3k in price and 800gig tapes were around $50-$60. Is that that much more expensive than three backups on drives? I have to agree with one copy on a raid setup and two backups on LTO tape. Also that LTO deck should be a one time investment.

[Jeff Kilgroe]

chuckt,

Don't take this personally or in a negative way, but... Floppy disks were never meant for long-term storage. Some people never got that memo, though. Formats do change, but I'm not sure how that translates to lost data for you... I can still load image files I created 15 years ago, most formats are standard and still supported. Archiving in proprietary formats is always a bad idea, but I know, people do it.

Digital archiving is very cost effective, simple and reliable if handled properly. I still have all my data from 20 years ago, back when 3.5" floppy disks were the new kid on the block and 30MB hard drives were "huge". I've had my share of hard drive and system crashes over the years, flooded basements, etc.. It's all part of life. I haven't lost any data - such events were anticipated and when they happened, it was no big deal when a drive went down or a box of CDs went swimming due to frozen pipes.

What files did you lose due to formats changing? I see a lot of people bringing that up as a concern over digital archival systems. But I have yet to see any evidence that should be a concern as long as data is stored in accepted formats and on accepted media. But we also have to be realistic. If you stored images made on an Apple IIc with Print Shop Deluxe to a 5.25" floppy and shelved it for 25 years, well, then that's inconvenient for sure. But *if* the disk is still good, shouldn't be a problem recovering the data.

At the rate floppy disks would go bad during every day use, I'm amazed (and appalled) that anyone would consider using them for storage or back-up. Never understood that.

[Jeff Kilgroe]

I thought an LTO deck was around $2k-$3k in price and 800gig tapes were around $50-$60. Is that that much more expensive than three backups on drives? I have to agree with one copy on a raid setup and two backups on LTO tape. Also that LTO deck should be a one time investment.

LTO deck is basically a one-time investment, unless the unit dies. Eventually you will upgrade it anyway, perhaps to LTO-4 or LTO-5 at some point.

Single slot LTO-3 drives can be had for just under $2K with interface card and some generic software and an included 400GB tape. LTO-4 is new and is a bit more money for the drive, but the tapes hold 2X as much (800GB), but are 3X the price ($100 vs. $33).

Linear tape in the form of LTO or DLT is cheaper than hard drive back-up once you back up enough data to recover the costs of the tape drive. A simple price-out of using 500GB hard drives vs. LTO-3 tape with a 12-slot auto-loader drive is already posted here in another thread. The LTO format priced out around $6K cheaper vs. the hard drives to do a single back up 131TB (2 hours of 4K 2:1 uncompressed @ 16bpc) of data. Take a look at the price comparison... I've merged this thread with the other existing one on the same topic.

[Curran Giddens]

For those who don't know about the Wiebetech products, here's the link.

http://www.wiebetech.com/products/ComboDock.php#pricingavailability

Steve

I use the WiebeTech ComboDock since I like to buy bare 3.5" SATA drives for backup. I also have a Pelican HD-221812-24PACK to store/transport the drives. This is just a temporary backup solution for all the RAW files. I still need to get a high-performance online storage solution for editing.

http://www.solarsystemstudio.com/SolarSystem/Photos/Pages/New_Setup.html#3

[Chris Kenny]

The easiest way to make sure nothing bad happens to your data is -- somewhat counterintuitively -- to not attempt long-term archiving. If you stick media (or, worse yet, hard drives) on a shelf and come back in 15 years, odds are some of the media will have gone bad. Even if you have two copies of everything, it's possible they've both gone bad.

On the other hand, if you keep all your valuable data on active storage and you have a good backup system in place, your data will be significantly safer. Why? Because if you're keeping everything on your active storage, you know as soon as it goes bad. To lose data, your active storage would have to go bad, and then your backup would have to also turn out to be bad at the same time. In other words, to lose data, both copies has to go bad within minutes or hours of each other, which is much less likely than them both going bad sometime during a multi-year period. (To make this really safe, you'd want to do period checksumming to make sure there wasn't silent data corruption going on, but this isn't that hard to do. Hell, next generation file systems like ZFS will do it for you.)

As for Hollywood's problem with archiving 130 TB per movie, or whatever, well, keeping all that data on actively spinning drives would be rather crazy. But there are obvious solutions. For every movie, store two copies on LTO-4 tapes, in different locations. Each location is an automated facility with space for, say, a million tapes. At the first facility, when the database says a given tape is, say, four years old, the tape robot plucks it from the shelf, verifies the data is still good, copies it to a new tape, verifies the copy went off without a hitch, and tosses the original tape. At the second facility, the same thing happens at the eight year mark. Thereafter, the process occurs every eight years at each facility.

At any given time, there's always at least one copy of everything that's four years old or less, and a second copy that's eight years old or less, plus the parity information. In order to lose data with this system, you'd need to lose more than 25% of the data for a movie at both facilities, and there would have to be an overlap in the lost data.

Periodically the LTO-4 drives get replaced with newer tape formats (or whatever comes along), and tapes get copies to the new formats the next time they're due to be copied.

These facilities would probably cost a fair bit to build and run, but at a million tapes each, they'd have enough shelf space to store a couple of decades worth of Hollywood's output. In fact, since you could count on moving things to more dense storage long before the facility filled up, you'd essentially never run out of space.

[Manfred Lopez]

I've merged this thread with the other existing one on the same topic.

So you are the one who got rid of my snazzy thread title? :biggrin:




... At the first facility, when the database says a given tape is, say, four years old, the tape robot plucks it from the shelf, verifies the data is still good, copies it to a new tape, ...

At any given time, there's always at least one copy of everything that's four years old or less, ...


Okay, one of the biggest problems that I see here is that the digital data method requires constant shepherding and baby-sitting. (by constant I mean that every few years you have to think about it and do something). What happens if you make a masterpiece that is way ahead of it's time (just for argument's sake :biggrin: ), and then you die. If no one attends to the data your masterpiece is very likely to be lost in several years. For some reason I can't picture future Criterion editions of "...from newly discovered Hard Drives or LTO tapes". (besides, it just doesn't have a nice ring to it, does it? :biggrin: )

[Manfred Lopez]

By the way, I think that 131 Gigs estimate for an indie picture is off. This doesn't take into consideration of original production sound, music, sound design sessions (100 plus tracks, and then in DTS, ACT, etc, versions of each), color corrected version of footage, Behind-The-Scenes Footage, Etc...

I find that todays computer-based production methods generate tons of data. I think it WILL get expensive for anyone who thinks that they can get away with shooting a cheapy Red feature just because they bought a camera. Remember that a movie is MORE than just the picture. And backing up all the other stuff as well is what is going to throw people's calculations off.

[rod bradley]

Hi Jeff -- you seem to be very astute re LTO drives and I have no experience, complete newbie -- I just purchased an Dell LTO 4 drive with SAS interface and wondering if you know of or can recommend card/software that would allow this to operate on OSX? And/or what you'd recommend for Windows. (I plan on using LTO3 tapes until LTO media comes down, as you point out far less expensive.) And will it make any difference if you're downloading the Red raw files on a Mac in the field, if you back up the data via Windows? It's just data right? Platform neutral?

Any elucidation on the nuts and bolts side of how to do this, most welcome. My apologies if this is outside the bounds of this thread. I notice there is another that is addressing LTO drives.

Many thanks.

Rod Bradley

[Chris Kenny]

Okay, one of the biggest problems that I see here is that the digital data method requires constant shepherding and baby-sitting. (by constant I mean that every few years you have to think about it and do something). What happens if you make a masterpiece that is way ahead of it's time (just for argument's sake :biggrin: ), and then you die. If no one attends to the data your masterpiece is very likely to be lost in several years.

Sure, this is possible. The only solution is to either be successful during your lifetime, or try to live long enough for more reliable long-term digital storage to be invented. :biggrin:


By the way, I think that 131 Gigs estimate for an indie picture is off. This doesn't take into consideration of original production sound, music, sound design sessions (100 plus tracks, and then in DTS, ACT, etc, versions of each), color corrected version of footage, Behind-The-Scenes Footage, Etc...

The estimate isn't 131 GB, it's 131 TB. Bit of a difference. Combine uncompressed recording with crazy shooting ratios, and this isn't that hard to do. Fincher actually shot more than that on Zodiac, I think. Still, it's only $16K worth of LTO-4 tapes. If you keep two sets around and replace them every eight years, that's $4000/year to store that data, plus whatever it costs to run the facility. (Not that much on a per-movie basis, once it gets a decent number of movies in it.)

But the really horrible flaw with the numbers that the Times gives -- the flaw so big that it even makes the difference between $4K and $200K seem trivial -- is that the $200K is presented as an ongoing cost. The price of storing a gigabyte of data drops by (quick googling) a factor of 1.5-2.0 every year. Going with the more conservative figure, this means that in 10 years, it'll be less than 2% of what it is now. In 20 years it'll be 0.03% of what it is now. Even if it were as outrageously expensive to store these quantities of data as the article claims with today's technology, the cost to store the data for 100 years would still probably be under $1M (with almost all of it being spent in the first few years, and storage costs dropping to the practically-free level after the first couple of decades.)

All the data generated for indie feature with a compressed workflow and a reasonable shooting ratio will probably weigh in at somewhere between two and six terabytes, which even at the high side puts your storage costs at a few hundred bucks a year initially. (And after 10 years, storing your feature for a year will cost less than buying a cup of coffee at Starbucks.)

[Manfred Lopez]

Sure, this is possible. The only solution is to either be successful during your lifetime, or try to live long enough for more reliable long-term digital storage to be invented. :biggrin:


...Or shoot film. :innocent:




The estimate isn't 131 GB, it's 131 TB. Bit of a difference.


I know the difference. But some people in this thread were talking that one could get away with only 131 Gigabytes of data (not Tera). That part of the post was addressed to them.

Regarding the rest of your post I completely agree. Prices will fall. And LTO is a good solution right now. But what is a cash-strapped indie director to do if he can only afford to get the 'film in the can'? (A very frequent occurrence). With film there is the implied knowledge that this director could go several years without finding further financing and his negatives would be fine. With Red and all other digital imaging systems it is a different story. You really have to think about storage before you shoot anything. That's why I think that this thread is a really good service to all indies thinking about pulling the trigger on a Red production with partial financing in place.

[Anders Holck]

To back up on DVD-R would be about,
131TB/0.0047TB=27873 disks, 27873*$0.17=$4739 USD
So it looks like DVD-R backup would be about $5000 maybe if you pack the disks in air tight boxes with Desiccants.

Maybe DVD-R would last better than HD, DVD-R may last 100 years, like film, and could be copied every 80 years without data loss, where as film cannot be copied without 15% to 40% loss of data quality (film loss seems less in generations that are themselves copies).

Ok, and how long does it take you to burn those 27873 unique DVD-R disks :-)

[Manfred Lopez]

Ok, and how long does it take you to burn those 27873 unique DVD-R disks :-)

At an average of 30 minutes each...

...13,936.5 hours

...or 348.4125 weeks (at 40-hour work week schedule)

...or about 7 years if you give yourself a two week vacation a year. :pinch:

***

So the real question is... If the first DVD's you started to burn are up for renewal after 5 or 7 years... would you ever finish backing them up or would you fall into an infinite loop? :usd:

[Chris Kenny]

Regarding the rest of your post I completely agree. Prices will fall. And LTO is a good solution right now. But what is a cash-strapped indie director to do if he can only afford to get the 'film in the can'? (A very frequent occurrence).

I don't think this is a huge deal. If you stick to a reasonable shooting ratio (say 10:1 or less) you can store all the material for an indie feature on maybe $900 worth of hard drives. You have to have at least that much storage space, just to you have somewhere to dump your footage at the end of the day. Buy another $900 worth of drives for backup. For added safety, buy the second set from a different vendor in case it turns out there's a problem with the first drive model.

There aren't many productions that would be shooting on Red, but would be so cash-strapped that they couldn't afford this.

Is your data as safe as it could be in an ideal world? No. Is it safe enough that you're extremely unlikely to lose it over, say, a four year period, if you spin those drives up a couple of times a year? Yes. Assuming your drive sets are stored in physically separate locations, and you take into account that a place like DriveSavers is virtually certain to be able to recover your data if there's a mechanical problem, the chances of losing data this way are sufficiently small that it really wouldn't be worth losing sleep over.

At the end of four years or so, see what storage options are available on the market, and migrate your data to whatever seems best. Projecting out the price drops of the last four years, you should expect to pay 70-80% less for the same amount of storage at that point.

[Mitch Deoudes]

A question that's too small for its own thread - hopefully it'll get picked up here, as this thread seems fairly fresh:

Does anyone know of backup software that will write in a non-proprietary format (i.e. tar)? I've been considering LTO + Amanda for a non-Red related backup project, but I'd be interested to hear alternatives.

Bonus question: am I correct in understanding that there's compression built into the LTO format (i.e. done by the drive itself), or is this solely a function of the backup software? If so, I'm assuming that it's standard to the format, and switching drives doesn't affect readability, correct?

[Lachlan Ward]

The solution is to either buy a 110L LTO Autoloader: up to 8TB storage. Uses LTO1,2,3 tapes. Good for archiving.

These guys are in Australia. They have some stuff that maybe useful.

http://www.discovery.net.au/exabyte.html

PS Approximately fifteen petabytes of data will be generated each year in particle physics experiments using CERN’s Large Hadron Collider, due to be launched in May 2008. We should nick there storage system it can't weigh much more than 10 - 20 ton.
That means 15000hr of 4k .R3D, Screw new science I want storages space muhahahaha:holloween:

[Hrvoje Simic]

At an average of 30 minutes each...

...13,936.5 hours

...or 348.4125 weeks (at 40-hour work week schedule)

...or about 7 years if you give yourself a two week vacation a year. :pinch:

***

So the real question is... If the first DVD's you started to burn are up for renewal after 5 or 7 years... would you ever finish backing them up or would you fall into an infinite loop? :usd:


That's why you have DVD multiplication facilities.
It would take few weeks tops but it would raise the price to ~$1-1.5 per DVD (where I live). You'd even get full colour print onto disc :)

The real problems are:

- the fact that there are 28000 units (storage space & risk factor)
- when comes a time to get back and copy the data from those 28000 discs


So that's definitely a no-no.

[Manfred Lopez]

That's why you have DVD multiplication facilities.

yeah, but each DVD would be unique and different... so there can't be any 'multiplication' going on. It's almost the same as if you wanted to back up every single DVD title ever published...

[Hrvoje Simic]

My bad. Hasty posting. You're absolutely right.