Archive Wisdom

[Akcelik]

how will you be archiving?

[Stuart English]

Well REDCODE RAW is just data. So all the IT domain options are open to you.

But its a great question to poll everyone on.

Do you intend to use data tape - such as LTO, DLT, AIT-5 ? All 400GB / tape

Or to 3.5" hard drives 1 TByte 3.5" drives available soon from Hitachi.

Or Blue Ray optical? 50GB disk burners are available soon from LaCie

[Dominic Jones]

Other:
Ideally I'd like to be storing on self-contained RAID 5 SATA systems, with either Gbit LAN or FW800 connectivity.

For really, really old "I'll probably never use it" footage, maybe a lay-off to tape (or by that point most likely high-capacity optical or holographic storage) and re-use the RAID drives.

[Isaac Brody]

I was wondering this same question yesterday. What's the most cost effective and long term/durable method for archiving data?

Archiving is one area where I feel safer with film.

[Ignas K]

3.5" hard drives are easy deal, just get one naked, connect to firewire enclosure, archive and lay it on a shelf.
now tape, even it might be lower $/GB value, is a bit steep investment to start, as one needs a recorder..
so I go 3.5", yes it might be risky but it´s easy.

[Evan Owen]

3.5" hard drives are easy deal, just get one naked, connect to firewire enclosure, archive and lay it on a shelf.
now tape, even it might be lower $/GB value, is a bit steep investment to start, as one needs a recorder..
so I go 3.5", yes it might be risky but it´s easy.

But how risky is it really? My understanding is that a parked hard drive is actually quite reliable. Obviously you'd want to store it in a location away from vibrations and excessive temperature/humidity variations, but I don't think it would be significantly less reliable than tape.

[Akcelik]

Well REDCODE RAW is just data. So all the IT domain options are open to you.

But its a great question to poll everyone on.

Do you intend to use data tape - such as LTO, DLT, AIT-5 ? All 400GB / tape

Or to 3.5" hard drives 1 TByte 3.5" drives available soon from Hitachi.

Or Blue Ray optical? 50GB disk burners are available soon from LaCie

hi Stuart! thanks!

i am undecided. :confused: by the time im ready to buy a Red (before 2008) i hope to see a reliable medium that is also cost effective.

it seem that technology is moving faster than its safety net.

[Jeff Kilgroe]

3.5" hard drives are easy deal, just get one naked, connect to firewire enclosure, archive and lay it on a shelf.
now tape, even it might be lower $/GB value, is a bit steep investment to start, as one needs a recorder..
so I go 3.5", yes it might be risky but it´s easy.

No. Bad.

Hard drives are not designed to sit for long periods of time on a shelf. Sorry, it's true. Most large data centers these days with huge volumes of data to manage and archive have gone to "live backups". Think redundant RAID storage networks that can easily be expanded just by adding more RAID nodes. Backups are stored across the SAN and archives are still offloaded to tape for long-term archival and off-site storage. So when you read these articles on the net how so and so Fortune 500 company now uses hard drives for backup, this is what they're talking about, not placing data on drives and shelving them. I've built, maintained, managed, etc.. enough networks and storage systems over the years, that I will never trust a backup to a hard drive.

Unfortunately for long-term storage the most reliable and cost-effective method is still tape. I know tape systems are expensive, but you have to look at the total investment over time and the costs of volume tape purchases. Figure out your data requirements and then price out your various options... If tape is too expensive, then you obviously don't need it. Try to maintain redundant live backups on HDD (not saying you need a SAN, but a nice secondary RAID5 box that stores multiple daily or weekly backups may be in order). This should work until BluRay becomes more reasonable over the next year.

But there's lots of tape options out there... One of the most reasonable for smaller operations, and one I've recommended to the HVX200 crowd is VXA2 tape. Tape drives can be had with external Firewire interface and including a single tape for about $1K. Tapes hold uncompressed 260GB each. Blank tapes can be bought for less than $20... VXA2 becomes cheaper than hard drive storage (taking the upcoming Hitachi 1TB drives at an estimated $370 street price tag into account) at about the 4TB mark. So that's definitely something to consider. And VXA2 tape is indeed designed to sit in its plastic case, on a shelf for years and years, or at least until something better comes along and you can regroup and re-archive your collection.

[Jeff Kilgroe]

Oooops... Correction to that above. VXA2 tapes hold 160GB uncompressed, not 260GB. Prices seem to have really shot up too. Make that 160GB for under $40 in tape if you buy bulk. That changes the dynamics a good bit, but for large volumes of data, it still comes in cheaper than hard drive storage, starting about the 6.5 to 7 TB mark. With 4K RAW footage (even 2K RAW) to archive, a TB isn't all that much space.

[Floris Liesker]

I'll start using 3,5 inch harddrives.

The rest of what I'm saying here is nonsense. I miscalculated by a factor 100. I'll keep it in here for you and everyone to read for fun purposes and a reminder to myself not to post too quickly and do some better math. My father's a math teacher for crying out loud...

I calculated an approx. 1 hour per GB datarate for the REDCODE RAW at 24 fps (correct me if I miscalculated), so a normal 500 GB drive could hold 500 hours. Lowest HDD price I found in a quick websearch ($ 64,00 for 320 gig) would make the non-RAID storage about $0,20 per hour of REDCODE RAW 24 fps, so RAID 1 would cost about $0,40 per hour.

[Stuart English]

4K at 24fps REDCODE RAW is a bit less than 2 GB / minute.

28MB/s x 60 / 1024 = 1.6 GB/min (plus audio & metadata etc)

So 1 TB ~ 500 minutes

[Mark Thorpe]

I am yet undecided, wait and see what is available closer to the time of camera acquisition. However I can't understand when a lot of people here are on the verge of getting their hands on what will be a very expensive camera, with all the whistles and bells attached, even more so. People wanna shoot at the max rez and capability of the unit but want to archive out to the cheapest denominator. I, for one, will have to bite the bullet and opt for what may not be the cheapest archiving option but the option which at least guarantees the best security for my commodity, the stock footage. After all we are purchasing a means to obtaining great footage. To me it is more than obvious that I should be taking more care of the footage thasn I do of the camera in real terms.

Cameras can be replaced!!

My 2c worth.
Mark.

[Ignas K]

I´m planing opposite - start low end, as cheap as possible and build slowly from there. [my budget is deciding everything and I don´t think that if I can´t afford xraid or arri accessories I can´t shoot 4k..]

[Jaime Vallés]

Yeah, for archiving the raw footage, tape drive is really the best way to go until large capacity optical disks become a reality (read: inexpensive). 50GB per blu-ray disk is small for tons of footage, but think of it as a reel of film. 20-25 minutes of footage per disk.

I'm not quite sure why people feel safer with a film negative as an archival medium. You can't clone the negative and still keep the quality intact. And you still have to keep it in a climate-controlled sealed and secured vault somewhere, and then another copy someplace else.

With a blu-ray disk, I'll just burn 3 at a time, and they'll sit there, in separate geographic locations. If 2 of them mysteriously combust into flames at the same time, then I put the 3rd one in my computer's blu-ray drive, copy the contents to my HDD, and spit out another 3 perfect digital clones.

When bigger, better holographic storage solutions become available, thenn just transfer it to those. No biggie.

[Jared VanLeuven]

Here's to the price of Blu-Ray disks coming down *clink*.

I'll be using HDD's for the time being. I remember reading others' opinions on the former forum that a shelved hard disk in controlled conditions is pretty stable. RAID would always be better, of course. The power company's going to just love us!

[Evin Grant]

LTO 2 is still the best value/security balance option out there. Yes a drive and SCSI card will cost you $2K but $40 for 200GB uncompressed tapes that write as fast as a normal Harddrive is hard to beat. Plus I think all LTOs are backwards compatible, so most of the higher end post facilities used to working with Viper/F950 footage will be able to easily handle it.

[Akcelik]

would it be safe to make 2:1 compression on LTO for Redcode?

[Chris Kenny]

would it be safe to make 2:1 compression on LTO for Redcode?

Redcode is already compressed. You're not going to get any significant benefit from trying to compress it again.

[David Limpus]

other option is S-AIT1 until holographic storage solutions become available.

Storage 500 GB (native), 1.3 TB (compressed ) MAX. SUSTAINED TRANSFER RATE 30 MB/s (native), 78 MB/s (compressed1)

has anyone used a SAITe1300-F/S?

[Nicholas Shields]

We only use hard drives for archiving (shelved). We have had several tape backups fail, costing us thousands in losses. We keep duplicate drives off-site for insurance purposes. Everything is databased via Disktracker - a great tool for now (Apple has just bought out Proximity - a great media management solution). Speed is also a benefit of hard drives over tape. Hope this helps.

[Floris Liesker]

oooopsss...Did I miscalculate in my post earlier in this thread. I'll completely rewrite it.

So, at 24 fps, REDCODE RAW produces 28 MB/sec, which should be 98.4375 GB/hour.
So 1 hour is approx. 100 GB, a 500 GB drive will hold about 5 hours...
Still this isn't the same figure that Stuart is giving. What am I doing wrong this time?

With prices Ive found of a around $180 for a 500 GB drive, one hour of storage will cost about $36,00 and twice that for a RAID configuration.
That's OK for a project that is still running, but for permanent storage I would hope for something a little cheaper...

[_BK]

Im very curious about this topic as well. I recently got burned by trying to archive data to 3.5" drives... after only 3 years of sitting 2 out of 4 drives seized up on me. Cost me a fortune to recover the data. Right now Im leaning towards LTO 2...

[David Nardini]

LTO 2 is still the best value/security balance option out there. Yes a drive and SCSI card will cost you $2K but $40 for 200GB uncompressed tapes that write as fast as a normal Harddrive is hard to beat. Plus I think all LTOs are backwards compatible, so most of the higher end post facilities used to working with Viper/F950 footage will be able to easily handle it.

Hi Evin, thanks for the pointer. Could you let us (me) know the better manufactures for LTO devices (those with a good track record in this technology) ?

Like you all, I'm very interested in this topic; a critical subject I believe.

The RED (and the new digicams) can generate vasts amount of data, and storing this data long term is not a trivial task. Technology moves at such a rate, it's hard to decide the strategy for the medium to long terms !

My experience ...
- disk is nice : but backup strategy can be a real mare (ie : off side copies)
- burning based technologies (DVD, Blue-Ray, etc) : always been concerned about the long terms stability of the medium (have had a number of non-readable disks, even if they verified fine ok at write time)
- tape : not used it personally, but have not heard 100% success stories

David Nardini.

[Milan Nikolic]

I am going to use MACOLA 2T (with infiniband interface) HD array for storage and editing. Edited material in thematic chunks of 20 minutes will go to Blue Ray disks multi copies for other place storage.

[Sanjin Jukic]

Macola Infiniband interface 4TB and 2TB RAID link
http://www.mac.co.yu/ponuda/macola/macola.html

[David Limpus]

macola infiniband drive type, anyone translate for the english only impaired

http://www.infortrend.com/main/2_product/a16f-g2430.asp is looking ok if they do a 16 bay fc-sas will be better

[Jochen Schmidt-Hambrock]

2 years ago at the AES conference in SF: A panel on archiving media with a couple of representatives from Exabyte, Sony, etc..

Each one had his 5 minutes where he "proved" that his medium has the best reliability, cost and longevity. (You know those presentations: See a chart - and of course somebody will read it to you)

The last guy (a Frenchman) was unattached to any company. His first words were that until now we all only heard BS.

That got everyones attention.

His point was that only backing up to multiple harddrives makes sense.
And have everything online all the time.

Made sense to me. Over a year I transfered my whole stuff to disk. Threw away tons of Umatics, VHSs, CDs, DVD-ROMs, etc...

Lots of free space now. Never looked back.

And another thing: It took me 4 hours to get the files off my DVD-Roms (you know, the ones that came in a cartridge)
Dig out the Quadra 9500, a monitor with "sync on green", connect the SCSI (where´s a terminator for gods sake!), install some long forgotten OS 8 driver.....)

Jochen

[Clark Dunbar]

Off-line storage and archiving.

A major workflow and ASSET quesion. If your shooting a few hours a week, at any 4K to HD modes - your looking at a huge amount of data that needs to be backed up (remember the only original is the file you just transferred to your editing system/sever - so once you've cleared your RED-drive or flash NOTHING remains except the digital copy - no film, or tape to throw in the vault).

So for backup to the project, re-confirguring the final out-put, basic archiving and re-assest utilization (read stock footage) needs to be covered.

You can rack up a huge budget item with offline storage and archiving in your annual overhead on our jump into digital only acquistion. Copies can be made on to harddrives at about $1/gig or you can look into all the tape systems for data - ie: DLT and the like which prices out at about $0.25/gig, and then you have the DVD and BluRay options which are even slower and smaller storage per unit. With the RED-drives being 320gigs, you can start to see that this becomes a major item in your overhead

So, do you look to buying a lot of hardrives and storing them in your media vault (closet or storage room) or do you invest in a tape system - which is slower but more cost efficient? And as we should all consider 2 copies of everything for that on-site copy and an the extra protection of an off-site copy.

Your 'footage' is you future - both as re-purposing the original program to new media but also as stock footage and yet unforeseen new markets.

So - the storage and archiving of all digital acquired media becomes an budgetary item but you also have to consider the time and storage space for the media, just as you would for all the film and/or tape originals that we had before - with the caveat that this must be the first step in your workflow - BACKUP. Both when on the road and the office/studio/edit suite.

So far in reading the forum - I see this as a over-looked and an important part of everyone's business model and an item that needs to be budgeted in to our new workflows.

Long post - but always looking to make sure that my assets aren't lost, so as of this note - we're using a SDLT tape system at 300gigs a tape for everything coming in AND a another copy of the final edit...this adds about $1k a month to our overhead and with 4K (even using RED-Code) we are planning on doubling this.

Even with media and storage prices coming down - this will be an item that needs major consideration. And .. I haven't even gotten into the logging of all this off-line/archived data.

No answers just concerns that we all need to deal with

[Mark L. Pederson]

Right now, it appears that LTO3 is best bang for the buck - $51 per tape - 400 GIGs - and yes, BACKUP is KEY to shooting tape/film free -

[David Limpus]

http://www.ikegami.com/IKEGAMI_NAB_2007_FINAL.html
Looks like Ikegami doing the holographic archiving HDS-300R external holographic drive. Hope it brings the InPhase Technologies Tapestry HDS-300R prices down.

[Bruce Allen]

Hmm, you guys have some pretty weird figures for cost/gb for drives... Seagate 750s are $270 at newegg, making it 36c per gb, not $1. And that's not even the best bang for your buck. I have been looking out for deals and buying drives at roughly the 30c / gb range for at least a year now. So, I personally would vote for hard drives too.

The last piece of data I lost was when my 2 GB drive died. That was a while ago ;) Since then, I simply made sure to have several backup copies of everything spread across drives. I use a little shareware program called SyncBack, which just synchronizes files onto my external drives every night. And then every now and then I take my external drives and put them in a different physical location and connect a different batch up to my PCs for nightly backup.

RAID-5 is not enough - sometimes it even gives you a false sense of security. Just make sure you have your data in three places at the same time - on your online drive, on a spare drive, and preferably on a third drive that is in a different physical location.

Since areal density is very likely to increase at great guns for the next few years, it makes sense to simply buy drives as you go - adopt a JIT assembly line philosophy since the price is dropping every month.

The only time I or my friends have ever lost data is when they have it only stored in one place. The worst idea is to trust only one (usually expensive) backup system "investment". For example, I've seen 2 Xserve RAIDs lose WHOLE TERABYTE RAID-5 SHARES overnight at places I have worked as a VFX artist, and have another horror story from a close friend. One of those systems had a Retrospect tape backup system going, but it took them several days to restore, and they still lost things because a) an old tape had died and nobody had figured out how to get Retrospect to re-backup the data that had been on that tape and b) the crash happened on the weekend, when the tape backup system ran out of tapes. Luckily, however, some people had made a copy of project files to his local computer, and another guy had made a backup copy of some of the stock footage that got lost with the tape, so everyone scraped by.

Tape might be a good option, but I would personally feel a bit happier with multiple copies of my project stored on different drives (getting different manufacturers is good too, as then the chance of both having the same defect is minimal). Every 2 years or so, I'll buy bigger drives and transfer everything to those, making another backup in the process. The nice way is then you are actually verifying your data.

Bruce Allen
www.boacinema.com

[Jason Francois]

I'm going to back-up to hand-inscribed tablets of stone....at least until I can figure out a better and cheaper method.

Hey, it worked for Moses and that story has lasted 2000+ years. :)

[Jeff Kilgroe]

Bruce,

Pricing out hard drive solutions isn't as cut and dry as pricing tape, IMO. Unless you're just talking about copying data to a drive and placing it on a shelf (bad idea, IMO). As you said yourself, RAID-5 is not enough for critical storage. Taking that into account, I would be compelled to built RAID-7 arrays or store duplicate RAID-5 volumes. Either way, when you start building RAID-1/5/6/7 volumes, your $/GB changes.

Tape seems a lot more economical in this respect... Even figuring making duplicate copies. The beauty of tape is that as your archival quantities and number of tapes increase, the savings add up more and more vs. hard drives. One of the huge factors in tape archival costs is the tape drive itself. Because just figuring tapes themselves, LTO-3 is less than $0.12/GB when you start buying multi-packs or grabbing overstocks off ebay.

[Jeff Kilgroe]

Hey, it worked for Moses and that story has lasted 2000+ years. :)

Yeah, but how do we know the data didn't get corrupted along the way?

Ooops, I didn't want to turn this into a religious debate... Afterall, Moses was highly regarded by the Egyptians too, or at least by most accounts of him that have been found.

[Clark Dunbar]

the really scarcy reality - is that no matter what backup/archive solution you put into place now - in 5-10 years it'll be obsolete and everything on your current media will need to be backup'd onto the "next" generation. Just think floppies, syquest disk, optical disks......

[Chris Kenny]

the really scarcy reality - is that no matter what backup/archive solution you put into place now - in 5-10 years it'll be obsolete and everything on your current media will need to be backup'd onto the "next" generation. Just think floppies, syquest disk, optical disks......

Yes, but all the data you generate this year will probably fit on one holographic disc in 10 years. Or into a small corner of your online storage.

And online storage probably is the way to go. You're far better off keeping multiple copies of your data on actively monitored online storage than writing it out to tapes or discs and sticking them in a closet, with nobody paying attention as they slowly degrade over the years, or as technology moves on and it becomes difficult to find a drive that can read the media.

Between the price/GB of hard drives now, the availability of cheap external eSATA cases with port multiplication, and the software RAID features built into modern operating systems (and particularly with ZFS in Leopard), it doesn't have to be very expensive to run such a system. Could be done for around $1/GB of data (taking into account that you have two copies of everything).

[JD Holloway]

I have no intention of maintaining ALL of my shot material on tape or drive for all time. I will need to dump data fast to free up drives. Some of my projects shoot 300 min/day sustained for week/s. EDLs will be vital.

I believe tape drive decks can be rented/borrowed in Toronto, so only occasional back-up would be necessary for safety when working on projects. Therefore only the per/tape cost concerns me. Maybe RED user locals can coordinate this service and centralise it.

My concern is with 4k RGB Finished Masters. What is the best way to re-compress them? Any "lossless" methods? A re-mosaic back to REDCODE (jokes)? Of course there is a image loss cost... I'm guessing its close to 2 TB per feature film (assuming 90-100 min) at 350 MB/sec.

Those are big numbers...

[Jeff Kilgroe]

the really scarcy reality - is that no matter what backup/archive solution you put into place now - in 5-10 years it'll be obsolete and everything on your current media will need to be backup'd onto the "next" generation. Just think floppies, syquest disk, optical disks......

Of course... But storage continues to get bigger, faster and cheaper. No big deal to those of us who've already done the large scale backup thing on some level. That's why it's silly to sit around and debate which archival format has the best shelf-life or if DVD-R will really last the 100 years, oh wait.. 30 years, they claim. In the next 5 years or so I'll be wanting to re-sort and organize and re-archive my current stuff. If for no other reason becasue I don't want shelves full of boxes full of tapes.

[Jeff Kilgroe]

I believe tape drive decks can be rented/borrowed in Toronto, so only occasional back-up would be necessary for safety when working on projects. Therefore only the per/tape cost concerns me. Maybe RED user locals can coordinate this service and centralise it.

Yep. I'm hoping to do something like that... I'm going to be buying into some form of archival system. And I'll definitely offer its use to fellow REDusers if I think it would be practical or beneficial. But I suppose we'll have to see what system I ultimately go with.


My concern is with 4k RGB Finished Masters. What is the best way to re-compress them? Any "lossless" methods? A re-mosaic back to REDCODE (jokes)? Of course there is a image loss cost... I'm guessing its close to 2 TB per feature film (assuming 90-100 min) at 350 MB/sec.

I wouldn't recompress anything. Archive the 4K masters as-is, even if they're 5TB in size. For 5TB of archival you're looking at $650 for LTO-3 tape (13 tapes no redundancy) or 5TB of HDD space (~$1750 for straight drives, no redundancy). Choosing either and making a couple duplicates to ensure it survives for a while is a lot cheaper than creating a film print for archival. Or drastically shrink the storage requirements by remastering to 2K and archiving that, hoping it will be good enough down the road.

[Chris Kenny]

My concern is with 4k RGB Finished Masters. What is the best way to re-compress them? Any "lossless" methods? A re-mosaic back to REDCODE (jokes)? Of course there is a image loss cost... I'm guessing its close to 2 TB per feature film (assuming 90-100 min) at 350 MB/sec.


Remember, there's an RGB version of REDCODE as well. You can't shoot it at 4K in-camera, but I'd be surprised if you couldn't encode it at 4K from QuickTime. I'm sure it would have a higher data rate than the RAW, but I'd think it would be pretty manageable.

[S. Um]

http://www.ikegami.com/IKEGAMI_NAB_2007_FINAL.html
Looks like Ikegami doing the holographic archiving HDS-300R external holographic drive. Hope it brings the InPhase Technologies Tapestry HDS-300R prices down.

I think Ikegami is just re-branding In Phase's drive, so it probably would not bring the price down.

http://broadcastengineering.com/news/ikegami_holographic_storage_20060918/

[JD Holloway]

Remember, there's an RGB version of REDCODE as well. You can't shoot it at 4K in-camera, but I'd be surprised if you couldn't encode it at 4K from QuickTime. I'm sure it would have a higher data rate than the RAW, but I'd think it would be pretty manageable.


Any idea about numbers Kenny? Compression ratios? I can't imagine any 4k RGB data being particularly smallish (explaining why it's not offered in-camera). But I'm not a post pro guy. Any help would be appreciated.

J

[Chris Kenny]

Any idea about numbers Kenny? Compression ratios? I can't imagine any 4k RGB data being particularly smallish (explaining why it's not offered in-camera).


I'd guess it's not offered on camera primarily because the camera doesn't have a fast enough processor on board, rather than specifically because of data rate issues.

But as far as data rate goes... worst case scenario would seem to be 3x the RAW data rate, so about 90 MB/s. However, while RGB has three times as many channels, they should each have less spacial complexity. If I had to take a stab at a guess, I'd say somewhere in the range of 50-70 MB/s, at the same quality levels as the RAW.

It's also possible REDCODE will have selectable data rates; you pick your rate, it gives you the best image it can at that rate. Many modern codecs support this, though it's generally more common for distribution codecs than for acquisition codecs.

[martinnoweck]

I think I have to distinguish between two kind of projects:

- personal projects like own films or projects where I supply the whole production chain for clients from pre- to postproduction: for this kind of productions I will look for a tape (LTO or similar) based archive solution.

- for other projects where I only supply my craft as dop i have not found a solution yet, because I encounter two kinds of difficulties / dilemmas:

1. i am doing a lot of documentary work, for the last film we shot over 60 hours of tape. at the end of the shooting day, the ac labels the tapes and hands them over to the director, who will log them him-/herself at home and go to some editing place ... sometimes it is a special post house, sometimes a tv station, because they are co-producers -> often we don't know at the begin of production where the material will be edited.

2. many producers or not used to an IT-based workflow, either they still think in "tape"-terms or the insurance doesn't cover IT-based productions, so we are really thinking about establishing a "transfer shop", where RED pictures can be transfered to HDCAM SR for postproduction and archival purposes ...

Any thoughts about this - how will others approach / support their clients?

Martin