Digital Storage Debacle

[Sevenout]

Variety has a good piece on how the MPAA has no idea how to archive and store the new digitally shot movies. A highlight from the article details how backup tapes have failed as little as nine months out and that in some cases a re-scan of the DI was necessary.

http://www.variety.com/article/VR1117963533.html?categoryid=1019&cs=1

[jbeale]

As I recall, NIST and the Library of Congress had a industry symposium on archival standards for optical media (eg. CD-R, DVD-R) a few years ago and the consensus they got from big storage users like banks, hospitals, etc. was no one wanted to spend more money for media that lasted more than a few years, because they would never trust it that long. They only wanted to budget for bargain-bin cheap media, and their plan was to just copy their entire archives over onto new media every few years.

[Álex Montoya]

Optical media should be optimal if well built and protected

[garageman]

Just use film to store the image, there'll be enough spare in a few years.

[Jeremiah McLamb]

but what about us small time non film users? How will I archive and backup my work? What if I shoot and have 9 hours of raw footage...that would be around a terabyte worth of archive space I would need...geez...I guess I'll have to start buying terabyte drives and hope they start coming down in price too...

but...I LOVE RED!!! just need some money for a reservation!

[david farland]

Knock yourselves out on this read here (http://www.uky.edu/~kiernan/DL/hedstrom.html).


The short answer is regular(ish) copying!

Cheers,

[garageman]

but what about us small time non film users? How will I archive and backup my work? What if I shoot and have 9 hours of raw footage...that would be around a terabyte worth of archive space I would need...geez...I guess I'll have to start buying terabyte drives and hope they start coming down in price too...

but...I LOVE RED!!! just need some money for a reservation!

Dump what you don't need and keep the final cut, let's put an end to recut re-releases and director's cuts. There used to be a time when trims and alternate takes were never kept. I think "Night of the Hunter" is the only classic film in existence were every single frame of shot footage still exists & that's down to Charles Laughton keeping it all at his own house for years.

[Zack Birlew]

Sheesh, I didn't know storage options were in such a bad state. Nine months out on tape? With my experience using external hard drives, I'm not so sure that they're quite the answer either.

[Fix]

but what about us small time non film users? How will I archive and backup my work? What if I shoot and have 9 hours of raw footage...that would be around a terabyte worth of archive space I would need...geez...I guess I'll have to start buying terabyte drives and hope they start coming down in price too...


Start copy alot of dvd's. But seriously, terabyte drives will go down in price for sure. Maybe blueray (400 gig version when it comes) is the way to go.

Another thought, think of it this way. Hollywood take care of theyr stock normaly 2k res (which is also the final print btw). So if you are going to keep everything in 4k we talk about cracking the big boys nuts while saying "geee hope I can get a new Hardrive for christmas.".

[vanguy]

What if I shoot and have 9 hours of raw footage...that would be around a terabyte worth of archive space I would need...geez...I guess I'll have to start buying terabyte drives and hope they start coming down in price too...

HDCam SR tapes are about $100 per hour of footage.

Current cost of a terabyte is around $800 for 9 hours of footage. Cheaper if you use raw drives (about $500). So even using terabyte drives is not insane.

But the archiving scares me a little.

[Thomas Mathai]

So far in the beginning of every new media so far there has been loss of historical data.

Early films, early tv, even early computer software, have been lost forever.

We shall see where the future of digital archiving takes us, it's a known problem with no real long term solution yet. Making copies is the workaround.

[Stuart English]

Interesting, but does this story actually align with our experience? Answers appreciated....

We've been shooting "data tape" for years now - i.e Digital Betacam, D1, D2, D3, DVCPRO, Betacam SX, D-5, HDCAM, HDCAM SR. And do we lose that data after 9 months?

I was told a while ago that data loss has little to do with the actual recording method, its the dyes and lubricatants that change over time causing oxide shedding, spindle stiction, clouding etc.

Perhaps all that can be said is long term archival has been proven on film, it works for most people on tape and there is too little experience to say much about the stability of optical media.

[Christian Ford]

From the Variety article: Color film can be turned into black-and-white color separations on polyester stock. Properly stored in cool vaults at low humidity, such film can last centuries.


There is a certain irony in the fact that the best archival system for digital cinema turns out to be the equivalent of three-strip technicolor.

[Jeff Kilgroe]

I find the article in Variety to be very, uh, uninformed. They are simply quoting individual sentences from industry spokesmen... Quite possibly out of context. They are reiterating common IT horror stories, that most in the IT industry have had to listen to for years... Some of these rank right up there with stories about new computer users 15 years ago thinking the CD tray was a drink holder and such nonsense.

The fact of the matter is, mass data storage and archival is a huge sector of the IT industry and there are plenty of viable solutions available today. Entertainment industry doomsayers who stress over how to store the data for their films are creating a false sense of panic. Studios like Sony and Disney have little to fear or concern themselves with as long as they invest in a proper IT infrastructure and don't skimp on the details.

Data tapes bad only 9 months out? Sure, it's possible, but what actually happened to cause that? What? They only made one copy? Who's the dumbass that did that? Point is, 99% of all such problems are user error and can be eliminated if proper redundancies and fail-safes are in place. Honestly, what major studios these days only store a single master film print? The petabytes of data stored by insurance companies and government agencies makes uncompressed digital archives of many film studios look small.

Out of all this, the ones who have the most to fear are the small independent players. Our resources are more limited, but our requirements are smaller too. We have to look at smaller scale, more affordable solutions like backing up to hard drive, common optical media and even smaller tape libraries. I strongly believe that anyone planning to archive their RED footage to a hard drive is asking for trouble... Now if they archived to three hard drives and stored each in different locations, hopefully under ideal conditions, it should be good for 3 to 5 years.

Same goes for optical media... It's not as bad as a lot of the industry articles make it out to be. Yes, the dyes degrade over time. Metallic substrate layers can oxidize and corrode.... But film degrades and corrodes too. Treat optical media with the same care that you would with film, I think people will be surprised at how long a quality piece of media can last. I've got CD-Rs that are 12+ years old that still function as new. I've got some that are 5 years old that are going bad. All depends on how they are used and stored. ...Most newer media doesn't have metallic substrate layers, but rather various forms of plastic or non-corroding materials. The dyes still degrade, but no more than film negs/prints and if stored properly (away from light / UV) they will last quite a while. Keep in mind that a laser not much more powerful than a presentation laser pointer is what altered the dyes on that disc in the first place to imprint the data.

Personally, I'll be using a combination of hard drive and tape solutions for backup and archival. Collections will need to be revisited, consolidated and re-archived every few years.

I don't see the logic in using film as an archival medium. You're eliminating many of the benefits of shooting digitally in the first place. Incurring additional costs to print to film. Once on film the product is in an analog medium of which we now have issues of generational loss and other analog factors if copies or new scans need to be made. Using film as a media to store data, as some on these boards have suggested also doesn't seem that great of an idea. Film degrades just like every other form of media... So how would bits that can change over time on a piece of film be any more reliable than magnetic shift of a data tape or degradation of an organic dye in an optical disc? I don't believe that film has been "proven". History of film has shown that film stored under ideal conditions can retain a very close copy of what was originally printed on it. Identical as far as the human eye is concerned in many cases if we're not talking too many years (25+). But film doesn't make a whole lot of sense for archival of digital source material. Final production release content, perhaps, but a proper digital master if maintained through the years would be far more desirable.

...For the best interest of their offspring, the wise turtle lays many eggs and is never foolish enough to place them all in one nest.

[C.H.Haskell]

This is a crucial discussion for us Indie players, keep it up! Any news on holographic solution that inphase has been working on?

http://www.inphase-technologies.com/products/default.asp?tnn=3

Best of luck

[Justin O'Neill]

HDCam SR tapes are about $100 per hour of footage.

Current cost of a terabyte is around $800 for 9 hours of footage. Cheaper if you use raw drives (about $500). So even using terabyte drives is not insane.

But the archiving scares me a little.

I regularly see 500gb hard drives for $129. That would be about $260 for a terabyte. Not bad at all.

[vanguy]

I regularly see 500gb hard drives for $129. That would be about $260 for a terabyte. Not bad at all.

Ho-ley cow! About a week and a half ago I priced a G-Raid 1 TB Firewire drive for $800. Now Lacie has them for $409!

[Sproketz]

Haskell,

Those InPhase holo recorders cost around $20,000 apiece. I don't think the price will come down very fast in the same way DVD's did.

Tough for most indie film makers unless some wanted to pool their money.

[Roger Singh]

I think right now LTO-3 (4 if you can afford it) is the best option of doing DI backups.

The initial cost of the drive is what's insanely high, but the disks aren't badly priced about 5 for $200. at 400gb per disk. About 2TB of storage for $200.

http://en.wikipedia.org/wiki/Linear_Tape-Open#LTO-3

LTO-4 is double the capacity.

The disks are rated at 30 year shelf life. Is it enough? I don't know.

[C.H.Haskell]

Interesting

I like the inphase holographic idea but they need to put out a prosumer cheaper version for us budget conscious folks.

I wonder how Quantum SDLT 600A solution could hold up.
http://www.quantum.com/Products/TapeDrives/DLT/SDLT600A/Index.aspx

we shall see.

[Roger Singh]

SDLT II holds 100gigs less than LTO-3 and the actual costs of the tapes cost more. LTO-3 also has a faster Read/Write Speed.

I'd only go with SDLT if you shoot DVCPRO HD or whatever formats are in the MXF video container.

Since RED and DPX are file based, you won't gain any advantage using SDLT.

[Stuart English]

Quantum announced that the 600A product line will be available with LT0-3 tape drives by July. 400GB per cassette, 68MB/s trasnfer rate, $8K for a network attached (Windows or Mac OSX) drive.

[Gavin Greenwalt]

Somehow I've managed to keep all of the captured footage from a stupid movie I shot in highshool all these years. I call bullshit. Keep a "live" copy on a server. If a hard drive fails, swap it out. That's what $500 for a terrabyte. Spread out over 30 disks on a RAID. In the event of corruption, swap it out, rebuild and keep on trucking. You could store 50 movies for less than $10,000

[Marcus Irvin]

Murphy's Law applies to digital backup equally.

I don't know how many stories I've heard from friends in IT about failed restores from "super-reliable" tape systems at all price levels. Unfortunately this is the worst failure of all, because it only becomes known right when you need it most.

Every CD writes the same data 3 times with interleaving, because it is an inherently lossy format. I've had cheap CDs fail after 2 years. They start with one or two file failures and degrade from there. That's not been a big deal with photo files, but huge video files would be more of an issue. Of course I no longer buy cheap CD's or DVD's but they don't last as advertised regardless of what you pay.

Consumer grade disk drives are promoted with ridiculously misleading MTBF (mean time between failure) numbers and you can verify that yourself with just a little research in IT publications.

I used to shoot sports pictures for a team photography company that kept all of their photo files online in a 9-drive SCSII rackmount Linux RAID. 500 Gigabytes of photos went poof one day when 2 drives failed simultaneously. The raid vendor said they had never seen it happen before. Back then small companies often couldn't afford to back up 500 Gig on tape, so the company just went poof as well when thousands of prepaid customers could no longer pick out the photos they had ordered and paid for. In hindsight, maybe they could have afforded tape backup, if it worked!

There is one backup strategy that inspires some confidence, but it is still expensive. That is off-site unattended backup in multiple locations. The best of these companies provide salt-mine level security with continual backup through subscriptions. They aren't cheap and require significant bandwidth speed, but the combination of physical security with redundant locations does offer more than any local solution. They are far less subject to fire, theft, vandalism, employee goofs, etc.

I love the idea of holographic storage, but in reading the methods involved I've seen no proof of archival permanence yet. Hopefully I'm too pessimistic on this as the potential speed and capacity is very exciting.

I have no financial interest in any of these, I'm just providing open-eyed feedback from 28 years of experience around IT people.

[dapperDitty]

I think the Egyptians had it figured out. Now if we can only get the data carved into stone.

There was a company a while back (hi-space I think) that had a 100 year guarantee (I think they were using 24k gold instead of silver) on their CD-Rs, not sure if they made DVD-Rs with the same guarantee. But it seems that is the way to do it, have redundant copies, and re-copy them every 4-5 years.

I don't think you would want to use DVD's (@ ~ 100 per TB) and definitely not CD's, but if they take that technology into the blue-ray realm then maybe it will become a viable option.

[Phil Becque]

Obviously all the people bleating about data loss on this thread must have very bad 'electronic storage' karma. I've been storing Tera bytes of data on HD's for over 20 years and I can only remember one drive (which had been backed up) going down. I got some 9 year old DV cassettes out the other day - it was all as good as the day I recorded it. OK, the occasional file you just open when the CPU crashes get's trashed but that's quite rare isn't it? What am I doing right???

A friend of mine has analog music tapes going back to the seventies that he re-masters professionally and even those are surprising good quality after a bit of electronic magic.

I agree that most DVD/CD's are flakey but then I don't use them to back anything up. Just buy a decent quality HD - IMO they are very reliable.

I suppose I'm tempting fate now aren't I?? I'll let you know should the worse happen.

[Jeff Kilgroe]

This is a crucial discussion for us Indie players, keep it up! Any news on holographic solution that inphase has been working on?

http://www.inphase-technologies.com/products/default.asp?tnn=3

Best of luck

I saw the InPhase drive a couple weeks ago. It's kinda neat, but they do have a few kinks to work out yet -- especially in the longevity and durability of the media and re-writable media is still in development. Very expensive. They wouldn't quote me a price directly and I had to contact a couple of their distributors to get a quote. Nowhere near worth it unless you just want to "support the cause" and be an alpha tester with cutting edge gadgets.

Holographic storage is one potential solution to cramming more data onto a piece of optical media. But I don't know if that's really where the storage industry is headed or not. There are some other emerging technologies that may hold some promise too, but nothing really coming to market just yet.

[Petr Dvorak]

Ikegamy also released one holodevice from inphase
http://www.ikegami.com/br/products/hdtv/hds300r.html

[Jeff Kilgroe]

The Ikegami is just the InPhase Tapestry drive rebranded... There's others slapping their names on it as well. Anyway, it's a cool unit and as soon as the issues with the media are ironed out and the costs are driven down, this could be the real beginning of something good. I hope I didn't paint too negative of a picture in my previous post. 300GB on a single piece of media (same size as a standard CD/DVD in a disc caddy) and with random access and write times at about 3X Blu-Ray, it's pretty cool. It's just not quite "done" and ready for the mass market, IMO. So close though... I'm only about a 30 minute drive from InPhase HQ and I'll have to check it out a gain in a few months and see how everything is progressing. After all, I'll need a good backup solution myself toward the end of this year. ;)

[Chris Kenny]

When comparing the reliability of digital archiving to, say, a color separation master, people need to keep cost in mind. If you're comparing tossing some data on tape and sticking it in a vault for 50 years with creating a separation master, OK, the edge probably goes to film. But for the price of creating that separation master, you could afford to store digital copies of your film in several different data formats, on several different types of media, spread across multiple geographical locations. I suspect if you do that, digital wins. And is probably still cheaper.

[Mark L. Pederson]

while waiting for holographic to be viable - I think Offhollywood go with two copies of LTO-3 off of the forthcoming LOT-3 version of 600A network drive -

[Mike Devlin]

Any thoughts on SAIT for archive rather than LTO-3? Sony now has a fiber channel SAIT drive for under $7000 (SCSI is cheaper). It is 500GB native and up to 1.3TB with their "Adaptive Lossless Compression". I doubt they will get much compression on a RECODE RAW file since it is already compressed. Only 30MB/sec uncompressed.

[Justin O'Neill]

Here is an external 500gb hard drive for $129. And shipping is free.

http://www.outpost.com/product/4963661

Quite a good deal.

[Alexander Black]

A DLT-S4 drive is about 3.5k rackmount, 800gb a tape. Really, not that bad.

[Frank Jonen]

Forgive my ignorance... but are people really still using tape to archive? I was totally unaware of that.

Here are some more reliable options: The Combo Dock (http://wiebetech.com/products/ComboDock.php) and I'm really intrigued by the REV Pro (http://www.iomega.com/rev/revpro/). I'm aware that iomega never put out a cartridge based format that didn't fail. But I'm looking at this more to send shots around. It's faster than file upload still.

With the Combo Dock you just plug in any old had drive without much hassle. Sure the stupid old power connectors (bleeding fingers) take more work than the more modern SATA drives, but still. They also make a read-only dock for forensics (and clients).

Hope at least 1 person finds this useful.

[Jeff Kilgroe]

Forgive my ignorance... but are people really still using tape to archive? I was totally unaware of that.

Unfortunately, yes. Because there really is no other solution that competes with tape in terms of cost per gigabyte. Most larger data centers these days, web farms, large companies, library of congress, etc.. Have massive, fault-tolerant and redundant hard drive systems. However archival or long-term storage is still done to tape. Even with 500GB hard drives now in the $125 price range, they're still significantly more than LTO or DLT tape when you start figuring tens or even hundreds of terrabytes or petabytes.

The problem that many in the RED community are facing is that they don't have a large scale backup system in place. Before many were using DV tape masters thrown in a shoe box. It worked great, but can't do that here. Hard drive backups are a good solution, but I wouldn't recommend any less than two copies to separate drives for every backup. Store the drives in different locations. Then again, I would recommend the same with tape, but with hard drive, a third copy would make me feel better. Heh. But as I was saying a lot of us around here are probably sitting on the fence about hard drive and tape. Hard drive is probably the most economical for anything less than 20TB (or duplicate copied 10TB) of backups. The cheap price of tape media will offset the cost of the tape drive and start to offer huge price advantages over HDD and other options once you start backing up 30TB or more... But tape has that nasty up-front expense of the tape drive. And it may take many here a year or more to need 20TB or more of backup space. So that's a tough call...


Here are some more reliable options:

Reliable is in the eye of the beholder. Not sure I'd go for the combo dock thing... It's expensive for what it is. I can buy nice aluminum enclosures for 3.5" IDE or SATA hard drives with FW800 connectors for about $45 each. OTOH, the combo dock is easier than swapping enclosures and cheaper than new enclosures for every drive. As for other solutions, REVs are expensive and not designed for shelf archival. For short term, reusable media, small backup and transit, makes sense to me. :) BluRay may become a good alternative to REV in the near future as prices drop. Right now, their cost per GB is way to high to be practical.


Hope at least 1 person finds this useful.

Sure, that combo dock thing could be real useful in some situations.

[Michael Schrengohst]

Forgive my ignorance... but are people really still using tape to archive? I was totally unaware of that.

Here are some more reliable options: The Combo Dock (http://wiebetech.com/products/ComboDock.php) and I'm really intrigued by the REV Pro (http://www.iomega.com/rev/revpro/). I'm aware that iomega never put out a cartridge based format that didn't fail. But I'm looking at this more to send shots around. It's faster than file upload still.

With the Combo Dock you just plug in any old had drive without much hassle. Sure the stupid old power connectors (bleeding fingers) take more work than the more modern SATA drives, but still. They also make a read-only dock for forensics (and clients).

Hope at least 1 person finds this useful.

For the price of combo dock you could just use an open firewire enclosure,
that is what I do right now. Not as elegant - but it will do until something better comes along.

[Bachman]

Obviously all the people bleating about data loss on this thread must have very bad 'electronic storage' karma. I've been storing Tera bytes of data on HD's for over 20 years and I can only remember one drive (which had been backed up) going down. I got some 9 year old DV cassettes out the other day - it was all as good as the day I recorded it. OK, the occasional file you just open when the CPU crashes get's trashed but that's quite rare isn't it? What am I doing right???

A friend of mine has analog music tapes going back to the seventies that he re-masters professionally and even those are surprising good quality after a bit of electronic magic.

I agree that most DVD/CD's are flakey but then I don't use them to back anything up. Just buy a decent quality HD - IMO they are very reliable.

I suppose I'm tempting fate now aren't I?? I'll let you know should the worse happen.

I totally agree. Ive said it before, but ive been archiving to HD for over 10 years and never had an issue and only once had a HD fail and that was a primary drive failure. If you look after your gear you shouldnt have any issues. Store them away in a controlled enviroment, I use a big old safe. Never backup to an exposed media, like CD or DVD, your asking for trouble. Personally I will be glad to see the back of CD/DVD/BlueRay when we all go solid state

PS. You can actually back up data to DV/DVCAM tape using a camera or deck

http://dvbackup.sourceforge.net/

[Rocco Schult]

I had dozens of HDD failures over the last 10 or so years, in a controlled environment that anybody at home can only dream of. Everybody not havong that is a very lucky boy.

Tape is much cheaper, but that will probably change when the 1TB drives fall in price. There will be a turn, when discs become cheaper per GB than the drives. Besides the tapes remain more reliable than adrive, from a simple mechanical point of view.

Archiving on DV ? It has been an interesting technique some year ago, but if you acquire 8-10 times the amount of data per time interval (DV or DVcam archives in 'realtime' of a DV-tape, don't forget) its gets annoying even when you shot only an hour of source footage. Good luck.


..Sony..SAIT..is 500GB native and up to 1.3TB with their "Adaptive Lossless Compression". I doubt they will get much compression on a RECODE RAW file since it is already compressed...

From my experience with LTO3 which was generally very positive (and much faster), compression there (no clue what type) brought onyl 10-max20%, mostly less. And that with uncompressed tiff and dpx. Dont expect too much from those compressions. You're probably very lucky if you should get 600GB out of a 500GB tape.