Archival Storage Questions

[Kim Frank]

After an external Backup-Harddrive of mine made some Problems I really need to work out my archiving workflow and I have some questions.

LTO still seems to be the best and cost effective way for long time archival. The question is LTO-4 or LTO-5. The tape deck price difference is just a few hundred. LTO-5 tapes store 1.500GB whats nearly double of LTO-4 tapes but at more than double the price. But as long as you can write LTO-4 tapes with an LTO-5 deck I assume the 5 is worth the price difference?

So this would be an archival sollution for RAW files and finished Projects. But that is 'just' one copy.

While working on a project, having a backup of the RAW Data on LTO-Tape and storing Project files Online, while working with a redundant Disc Array like RAID 5, seems fine to me, or what is your workflow.

I'm mainly working on Music-Videos. I always deliver standart Formats, for TV and Web, but after a project is finished, payed and distributed the one thing I'm asked most of the time is to render out specific file Formats. I then just need to grab the final uncompressed render of the Clip and encode it to their wishes - of course they could do it by their own, but most clients don't know how and don't even want the big uncompressed file, but consider me to archive it and deliver further formats, something that's fine for me to do.

So I'm thinking about a cost effective way to store just the final uncompressed clips and be able to grab them fast and easy. And as I dont trust harddrives I was thinking about Bluray. Does anyone of you use Blueray for storage? A final Music video in 1080p is about 30 GB so it would fit a 50GB Bluray. What if a file is larger, like a shortfilm I made, where the final size is 150 GB? Am I able to store this one file to several Blurays?

Or are there any other cost effective possibilties I'm not aware of?

[Matt Gottshalk]

I went with Cache-a and Cat-DV.

[Kim Frank]

Thanks Matt...

[Jarek Zabczynski]

A final Music video in 1080p is about 30 GB so it would fit a 50GB Bluray.

I archive my finals in Prores 4444. A 5 min video weighs in at just under 10 gigs.

[Kirk.Attard]

Just started using bluray. The double layer discs can store nearly 50gigs so a short film would fit on a couple of blu ray discs.

I'm in music videos as well and have the exact same requests. One thing a friend suggested me to do is copy to 1 TB hard drives and store in a good place. Perhaps more than one copy. Though this is more accesible I doubt it is more reliable than blu ray discs. I think Blu ray have a good life time as the discs are usually of good quality.. Dual layers are worth considering though a bit more pricey. I never used tapes to be honest and dont want to go in further expenses at the moment. Perhaps others have different views.

[Marc Wielage]

I try to operate on Peter Krogh's 3-2-1 Backup Rule (http://www.dpbestflow.org/backup/backup-overview): 3 copies of your data, stored on 2 separate kinds of media, with 1 stored off-site (out of your house or office).

There's an old expression: "if you only have your data in one place, you don't really own your data." I know of several awful stories involving record labels and film studios where many important films, videotapes, and sound recordings were destroyed because a) they stored them all in one place, and b) they had no real backups. In one case, they had backups... but they were stored in the same building that burned down.

Me personally, I'd put the projects on two brand-new hard drives, also make an LTO backup (LTO-4 is fine for short reels), and also a high-def videotape, assuming the final delivery was in HD. Long-term, so far, the only strategy is to migrate your files to new drives every few years.

[Jack Cooper]

Just started using bluray. The double layer discs can store nearly 50gigs so a short film would fit on a couple of blu ray discs.

I'm in music videos as well and have the exact same requests. One thing a friend suggested me to do is copy to 1 TB hard drives and store in a good place. Perhaps more than one copy. Though this is more accesible I doubt it is more reliable than blu ray discs. I think Blu ray have a good life time as the discs are usually of good quality.. Dual layers are worth considering though a bit more pricey. I never used tapes to be honest and dont want to go in further expenses at the moment. Perhaps others have different views.

Kirk.

We've had TERRIBLE experience with those portable harddrives. I almost have as many failed drives in the closet as I have working drives. FreeAgent's, WD myBooks, all total crap. Data recovery is expensive if you hit between the "cycle backups". We are talking to Rorke about their systems which include LTO-5. No decisions on our upgrade path just yet.

[conrad gaunt]

Kirk.

We've had TERRIBLE experience with those portable harddrives. I almost have as many failed drives in the closet as I have working drives. FreeAgent's, WD myBooks, all total crap. Data recovery is expensive if you hit between the "cycle backups". We are talking to Rorke about their systems which include LTO-5. No decisions on our upgrade path just yet.

i've had the same experience with harddrives, ive had them fail after an hour, a week, a month.. and yet i have old harddrives that must be nearly pushing 15years old, some older, that still power up and have NO surface defects.. drives got higher in capacity, and crap, in equal measure.. what happened, i had more luck recording data from my zx spectrun to an audio cassette in the 80s than using modern external drives recently, and that task rarely went smoothly (the c64 was much better, with its fancy tape recorder)

If your data is really important, I'd suggest converting it to binary then manually transcribing that to the inside surface of a cave using berry juice and mud.. don't forget the checksum, so future anthropologists can verify the data .. imagine their primitive faces when they realize we were once more advanced than them, before the asteroid came, and the irony that most records of our culture were ultimately lost by advances in technology, and because the last big paper manufacturer closed shop due to incompetent management..

You know, 1/4 and 1/2 tape, is still the best medium to capture music with, by a long long way.. its not limited to 44/48/96khz discrete samples, it can capture millions of transients a second, and all those harmonics.. digital sound devices can't.. but neither can it ever be backed up exactly.. food for thought

I imagine bluray discs are best hung from poles and left in fields swaying in the wind, to scare crows away from crops .. which is definately the best use I've seen for CDr so far

[Kirk.Attard]

Conrad - you moight be right about the BD but they are cheap and add to an easy extra medium which you can easily store in a different place at least. As of hard drives. I obviously had crashes and data lost but nowehere near as disastrous as some people suggest. To be honest I still have to find a viable cost effective solution, cos though occasionaly I have to retrieve something I did years before its not common and frankly not worth the investment. Though I am a big fan of backups and RAID etc.. You can never be too cautious while working on a project. But we are taking about archving here and 4k files especially can get pretty large quickly so its a different story.

[Ken Kosub]

We've been using LTO-4 and Bru software for several years with no isues. very reliable and safe storage. keep one copy in the studio and clone in the bank's safe deposit box of site.

[Frederick von Sulle]

Hi Kim. When thinking data backup don't think in film business terms but in IT terms. The thumb rule is 3 copies in 3 different locations. I am 25 years in IT and you won't believe how many times I have seen Murphy's law hitting. It will always hit when you never expect it. The solution we use is quite simple, we have a RAID system in the office. (There are quite economic ones around now, with Thunderbolt interface) - 2TB HD are very cost efficient. The prices are falling regularly and 4TB will soon be around 100 US... So our workflow is simple. We keep the project files on our RAID array and then once a week I make 2 incremental backups on a weekly base or whenever important material is coming in using to 2 external HD's with an ESATA interface. The 2 HD are than stored in 2 different locations, one at home and one in a safe location. You can crypt your drive too, but never forget your password. Every year or when the industry produces new drives with more capacity I merge them back onto a new drive. Actually going from 2TB to 4TB. Your entire archive will fit easily in a suitcase. And every one or 2 years you copy everything to a higher capacity drive. Drives are more secure than your tapes. Tapes are very sensible to magnetic fields, humidity and other environmental facts. In reality it does not really matter if Tape or HD important is to keep at least 2 better 2 + 1 copy of your important material.

[Kim Frank]

Thank you all for your input!


... And every one or 2 years you copy everything to a higher capacity drive. Drives are more secure than your tapes. Tapes are very sensible to magnetic fields, humidity and other environmental facts. In reality it does not really matter if Tape or HD important is to keep at least 2 better 2 + 1 copy of your important material.

I'm a bit confused as my information is, that you need to power up Hard drives at least every six months or you are risking physical errors. And also a tape doesnt include as many physical parts as hard drives - therefor less risk of failure I would assume. Also having to invest money every two years for jobs that are finished and I at this point don't earn money with, somehow sounds counter intuitive for me.


I try to operate on Peter Krogh's 3-2-1 Backup Rule (http://www.dpbestflow.org/backup/backup-overview): 3 copies of your data, stored on 2 separate kinds of media, with 1 stored off-site (out of your house or office).

That comes near to what I was thinking about. Archiving whole Projects on LTO-tapes and final renders on Bluray - cost effective and long lasting.


We've had TERRIBLE experience with those portable harddrives. I almost have as many failed drives in the closet as I have working drives.

Same with me.



If your data is really important, I'd suggest converting it to binary then manually transcribing that to the inside surface of a cave using berry juice and mud.. don't forget the checksum, so future anthropologists can verify the data .. imagine their primitive faces when they realize we were once more advanced than them, before the asteroid came, and the irony that most records of our culture were ultimately lost by advances in technology, and because the last big paper manufacturer closed shop due to incompetent management..


Thanks for the laugh.