Jim, how about data backup?

[Peter Majtan]

Jim - You guys should spend some R&D resources on developing some kind of medium-to-long-term backup system for all the 3K, 4K and 5K footage people will shoot on their "redheads"...

Tape might be dead as a recording medium, but (for now) it can't be beaten in archiving situations. Shelf life of well over 12 years, affordable and minimum storage requirements (both space and ambient)...

It will be great if You could develop a "RED VAULT" system that would record RED RAW onto a MiniDV tapes - this should not be too hard...

RED RAY is out of question as an archival medium.
CD's are able to handle most common scratches relatively well.
DVD's are already quite sensitive to any scratching.
Any optical format that will use recording density (track width) equal or higher then DVD will suffer from scratching greatly.
I can still (but barely) access 5-year old CD's, but having troubles even with 2-year old DVD's...

With new digital camera's coming out from RED next year, You expect to revolutionize the art of storytelling through cinematography. You do have a little responsibility to look into archival solution... :innocent:

I can see the proud Scarlet (and RED ONE / EPIC) owners pulling their hair out after the HDD with all "original" footage ("digital negative") have died on them... :waaa:

RED ONE and EPIC owners will probably have TB's of RAID-protected storage as well as DLT, LTO, or some other form of tape backup for their data. but what about Scarlet owners? It sure will be nice to have some affordable backup solution and MiniDV is perfect for this!!!

Everyone who would like JIM and his team to develop some kind of affordable RED-RAW data backup solution (at least medium-term: 10+ Years) please vote in the poll...

[Peter Majtan]

If your R&D resources are spread too thin already (which is easy to imagine) I would love to develop this for RED myself!!! :w00t:

PS:
I have single-handedly designed from scratch an Apple/ARRI/Foundry/SONY based film restoration system in Tokyo for Hisao Kurosawa (Akira's son).

I have also designed an optical archival system for digital data (as a part of research for the above project), that allows one to record digital image data onto 35mm film (with 100+ year shelf life) for archival purposes...

Drop me an e-mail if You are interested, I'm certainly capable of doing this and can provide You with references...

[Jason Wingrove]

Probably wht Red Ray will be, they've already said you can fit hours of red data on it imagine for about a grand having the redray on your desktop, plug in your cards like a card reader, copy it to your computer or backup to Red Ray.. Anyway we've got a year ... all will be revealed. Red is working this stuff out as they go along too.

[Chris Parker]

what about lto-3a data tape?

[Peter Majtan]

Probably wht Red Ray will be, they've already said you can fit hours of red data on it imagine for about a grand having the redray on your desktop, plug in your cards like a card reader, copy it to your computer or backup to Red Ray.. Anyway we've got a year ... all will be revealed. Red is working this stuff out as they go along too.

In the main post above I have specifically mentioned why RED RAY is out of question for backup purposes. Dailies, presentation, even distribution - sure...
Backup? Count me out...

[Peter Majtan]

what about lto-3a data tape?

Sure one of the best options around (for now), but quite pricey for "home" user (drives starting at around $2K) and 400GB might be too much (mind You there is never too much) if You prefer to backup on shooting basis, but that is personal preference. MiniDV RED-CODE-RAW data recorder could cost bellow $500,- or even less...

[Allbird]

the cash per GB in hard disks are becoming cheaper.
what do u think guys, about 4TB 100$ in two or three years?

cash test: DVD_taiyo, DV_Exellence, HDD_Seagate, BR_MCC

DVD - 0.10 $/GB
MiniDV - 0.25 $/GB
HDD - 0.13 $/GB
BR - 0.60 $/GB

[michael zaletel]

Secondary backup (off-site) is crucial as well. Even if we had cheap 4TB drives, it just takes too long to make duplicate copies and they can't easily or quickly be made over the Internet to a remote location for Disaster Recovery.

Any ideas along these lines would be greatly appreciated. BTW, what is currently the fastest possible technology for transferring data from 1TB drive to another? What type of drive? What type of bus, connection, etc?

-shooter

[Allbird]

Any ideas along these lines would be greatly appreciated. BTW, what is currently the fastest possible technology for transferring data from 1TB drive to another? What type of drive? What type of bus, connection, etc?-shooter

e-SATA II runs at 2Gbit/sec. the fastest, just the half of dual link 4Gb fiber chanel.

and e-sata II is standard & afforadable.
only need a dual-built-in-harddrive in a RAID1 configuration (like WD) with e-sata connection and u´ll transfer at 100 MBytes/s sustained, from CF to 1TB e-sata RAID1 drives, & faster in a future 2TB drives.

if REDCODE-36 RAW data rate is 36MBytes/s, then, with a 100 MBytes/s cannection, u´ll transfer about 3 seconds of RAW data in a single second.

----- it means that u´ll back up your data in a 1/3 of the recorded footage, in a RAID1 mirror secure HDD. -----

[kmikami]

The benefit of a data based workflow is that you can use existing data backup technology. There's no need for a Red-specific solution. Long term archival of digital information is the billion dollar question that affects the future of every information based industry. There are plenty of other people working on the issue and while the team at Red is obviously brilliant I think they're busy enough making cutting edge cameras. We can't expect them to solve this problem as well!

[Dylan Reeve]

MiniDV archival is unrealistic. DV is 25mbit/s or less than a 10th of REDCODE. Which means at best you'd be looking at about 4 minutes of RED footage on a single MiniDV tape.

Worse than that, the solution would be entirely proprietary - there are no existing DV-data drives.

There are plenty of data backup systems around, and they should be the ones we look to. They have advantages of scale (ie. not limited to a small market of video users) and have substantial uptake (meaning there are lots of drives around if you need to get the data back later).

I don't think it's really in RED's interest to get into that area too much when there are plenty of providers who can provide options.

ShotWrangler will develop and market a backup system once the SolidStore is shipping (assuming there is a sufficient demand for such a product). And other data backup solutions are already being adapted/adopted to suit RED workflows.

[Peter Majtan]

e-SATA II runs at 2Gbit/sec. the fastest, just the half of dual link 4Gb fiber chanel.

and e-sata II is standard & afforadable.
only need a dual-built-in-harddrive in a RAID1 configuration (like WD) with e-sata connection and u´ll transfer at 100 MBytes/s sustained, from CF to 1TB e-sata RAID1 drives, & faster in a future 2TB drives.

if REDCODE-36 RAW data rate is 36MBytes/s, then, with a 100 MBytes/s cannection, u´ll transfer about 3 seconds of RAW data in a single second.

----- it means that u´ll back up your data in a 1/3 of the recorded footage, in a RAID1 mirror secure HDD. -----

Kind off...

A) In RAID-1 You don't benefit with speed - You get 100% security. In fact the speed may slightly decrease depending whether the RAID-1 is HW or SW-based. You most probably meant RAID-0 where the total transfer speed is almost double of a single drive...

B) E-SATA would be the best choice (especially on budget), but mind You - most interfaces are many times faster than the maximum transfer rate of the HDD itself. USB is a definitive "no-no" due to the large "overhead". Both FW-400 and FW-800 would be my second recommendation after E-SATA...

[Dylan Reeve]

Also, looking at that - I doubt you'll even come close to 100MB/s read from CF cards.

The limitation with these connections is more often the drive. Transferring from a single 1TB drive to another, the best option is SATA (or eSATA) however the drive won't achieve anywhere near 1.5Gb/s (or 3Gb/s) read speeds, and the target drive certainly won't reach those write speeds. And the OS is likely to slow things down too.

Copying ~400GB from a RAID-5 set to a RAID-1 the other day took something like an hour.

[Peter Majtan]

MiniDV archival is unrealistic. DV is 25mbit/s or less than a 10th of REDCODE. Which means at best you'd be looking at about 4 minutes of RED footage on a single MiniDV tape.

Which is about the equivalent of 400ft magazine...
Unless You shoot "video-style" (read hours per day), You don't shoot that many... Being able to backup day worth of filming on $25 worth of MiniDV tapes is definitely worth it. The 25mbit/s is in this case referring to density, rather then the speed of recording, which would have to be improved of course.
But You do have a good point.


Worse than that, the solution would be entirely proprietary - there are no existing DV-data drives.

So is going to be RED RAY. But at least You could use standard media...


There are plenty of data backup systems around, and they should be the ones we look to. They have advantages of scale (ie. not limited to a small market of video users) and have substantial uptake (meaning there are lots of drives around if you need to get the data back later).

Yes, however majority is relatively expensive and You might find it hard to get the specific tape on location, or at 2AM. MiniDV are sold in 7-11's...


I don't think it's really in RED's interest to get into that area too much when there are plenty of providers who can provide options.

So there is Blu-Ray and yet here comes RED RAY...
Looks like they are trying to offer most common tools needed by filmmakers. I can see coming RED LIGHT (4K projector), RED VIEW (4K display) - why not backup solution?


ShotWrangler will develop and market a backup system once the SolidStore is shipping (assuming there is a sufficient demand for such a product). And other data backup solutions are already being adapted/adopted to suit RED workflows.

The Kiwi solution looks interesting, but there is yet to be any proper study as to the longevity of solid media. Also I can see the cost of such a solution to be aimed at postproduction houses - hardly at home users...

[Peter Majtan]

The benefit of a data based workflow is that you can use existing data backup technology. There's no need for a Red-specific solution. Long term archival of digital information is the billion dollar question that affects the future of every information based industry. There are plenty of other people working on the issue and while the team at Red is obviously brilliant I think they're busy enough making cutting edge cameras. We can't expect them to solve this problem as well!

Yet they had (and have) the time to develop RED RAY...

[Allbird]

Kind off...

A) In RAID-1 You don't benefit with speed - You get 100% security. In fact the speed may slightly decrease depending whether the RAID-1 is HW or SW-based. You most probably meant RAID-0 where the total transfer speed is almost double of a single drive...

B) E-SATA would be the best choice (especially on budget), but mind You - most interfaces are many times faster than the maximum transfer rate of the HDD itself. USB is a definitive "no-no" due to the large "overhead". Both FW-400 and FW-800 would be my second recommendation after E-SATA...

B) yeahh! sure.

A) RAID 0 you have 30% more speed, not double.
RAID 1 you may have the same speed if there is a well designed controller.
Is in RAID 5 where you decrease the performance, because the calculation of parity bits. RAID 1 hasn't parity bits, is just a mirror.
and of course, it should be HW controlled because is an external drive connected to an eSATA bus.
either you can go with a diferent sector sizes when formating the HD, increasing from 128kB to 2048kB sector size for increase performance with large files.
its possible, of course, to get 100MBytes transfer from a 1TB drive, & increase the speed, increasing the density per rpm in a 2TB 7200rpm drive with 2 plates.

then, actually:

cash test: DVD_taiyo, DV_Exellence, HDD_Seagate, BR_MCC

DVD - 0.10 $/GB
MiniDV - 0.25 $/GB
HDD - 0.13 $/GB
BR - 0.60 $/GB

[Dylan Reeve]

Which is about the equivalent of 400ft magazine...
Unless You shoot "video-style" (read hours per day), You don't shoot that many... Being able to backup day worth of filming on $25 worth of MiniDV tapes is definitely worth it. The 25mbit/s is in this case referring to density, rather then the speed of recording, which would have to be improved of course.
But You do have a good point.


It's would be a painful and frustrating process to archive or recover with such small loads.

The density of the tape could probably be improved, but look at standard DV with a 15micron track pitch, it's pretty prone to drop outs from even minor mistreatment. Increasing the track pitch to 18micron (DVCAM) improves the reliability somewhat, but it's still a fairly fragile format.


So is going to be RED RAY. But at least You could use standard media...

Storing 2 Hours in 9GB doesn't seem much like a backup system. RED RAY looks likely to serve a different purpose. I don't think it's a backup soluton, and the investment in it's proprietary nature therefore is quite a different issue.


Yes, however majority is relatively expensive and You might find it hard to get the specific tape on location, or at 2AM. MiniDV are sold in 7-11's...

The drives (thinking LTO specifically) are expensive. But the tapes are very cost effective, and pretty widely available. While MiniDVs are readily available now, that may not be the case in a few years. Tried to buy a Hi8 tape recently?


Looks like they are trying to offer most common tools needed by filmmakers. I can see coming RED LIGHT (4K projector), RED VIEW (4K display) - why not backup solution?

No reason necessarily, but to some extent RED seems to be creating products to fit places where none exist already. There are many data storage options, which are, in general, suitable for RED footage and varied in budget.


The Kiwi solution looks interesting, but there is yet to be any proper study as to the longevity of solid media. Also I can see the cost of such a solution to be aimed at postproduction houses - hardly at home users...

The solution we're [in the very early stages of] planning will be an open-architecture system, able to be deployed at a range of different price levels with a range of different technologies (Blu-Ray, LTO, etc). We're not currently considering storage on solid state memory - to my mind the physical size of most options presents a significant problem, no point archiving to something that is really easy to misplace or loose.

I'm not a huge fan of reinventing wheels, so I think it's very much worth exploring the options with existing technologies before creating something new (especially something with no current precedent). Also, if RED (or someone) were to experiment with a new product and then abandon it there's not much chance of it continuing one - people risk having their backups on an orphaned format. One advantage with existing systems (especially with LTO) is that the large market penetration and multiple vendors pretty much assures that the technology will survive.

[WesVasher]

I wonder if you can re-code your Redcode RAW footage with a heavier pass of compression or even downscale it also? Get the datarate down to an acceptable level, say 25Mbps and back that up to DV tape using one of these DV tape backup programs for Mac and PC? At the lower datarate you could also burn 30+ minutes to a DVD DL burnable.

Sure you aren't going to have a backup of the absolute best quality source but you'll have something on a pretty good and cheap backup.

And then just keep your full res files on Drobo or a comparable product.

[JoakimZiegler]

We use LTO-4, 800GB tapes, 120MB/sec transfer rate, and tapes are about 95 dollars in the US now (around 12 cents per GB).

[Peter Majtan]

We use LTO-4, 800GB tapes, 120MB/sec transfer rate, and tapes are about 95 dollars in the US now (around 12 cents per GB).

But the price of the drive unit is way more then that of Scarlet...

There are plenty of options for good quality storage, but we need something in terms what Scarlet is doing for cinematography - something affordable that all indies (targeted Scarlet demography) can and will buy. It should be bellow $500,-

[Peter Majtan]

A) RAID 0 you have 30% more speed, not double.
RAID 1 you may have the same speed if there is a well designed controller.
Is in RAID 5 where you decrease the performance, because the calculation of parity bits. RAID 1 hasn't parity bits, is just a mirror.
and of course, it should be HW controlled because is an external drive connected to an eSATA bus.

Dude, dual drive RAID-0 gives You pretty close to double (maybe between 190~195%) rate of a single drive, not 130%...

RAID-1 can be HW, but it can also be SW (especially on mac). And I have clearly said that You would get almost the same speed (~100%) of a single drive.


cash test: DVD_taiyo, DV_Exellence, HDD_Seagate, BR_MCC
DVD - 0.10 $/GB
MiniDV - 0.25 $/GB
HDD - 0.13 $/GB
BR - 0.60 $/GB

Why people always think $$$$/GB as the only factor when it comes to ARCHIVE/BACKUP? By far the most important factor is longevity of media. How about:

DVD - 2 to 3 years at best
MiniDV - 12 years + (good tapes do 25+)
HDD - 5 years max, but drop it once and say goodbye to your data
BR - early to say, but would not expect more then 2 years

Also - with all optical media - scratch it and is almost the same as dropping HDD...

Tape - You can't scratch it by handling it, You can drop it from a roof of a building if You want and still be able to read the data...

THIS THREAD IS ABOUT MEDIUM to LONG-TERM ARCHIVE SOLUTION, not online, or near-line (short-term) ones...

[kmikami]

Yet they had (and have) the time to develop RED RAY...

Which is a 4k delivery and playback system that's totally unique, specific to the cinema market, and necessary for Red workflow. Nobody else was going to make it for them. Data backup is another issue.

[kmikami]

Why people always think $$$$/GB as the only factor when it comes to ARCHIVE/BACKUP? By far the most important factor is longevity of media.

You're the one asking for it cheaply! You said you want an indie solution that's under $500. So burn to multiple sets of DVDs or backup to multiple sets of hard drives and store them in separate locations. Verify the data once a year and replace any bad media. If you don't have a lot of money to put into it you're going to have to put a lot of sweat into it.

[Peter Majtan]

RED RAY looks likely to serve a different purpose. I don't think it's a backup soluton, and the investment in it's proprietary nature therefore is quite a different issue.


I have discounted RED REY as a backup solution right at the beginning of this thread. I was referring to You saying that "Worse than that, the solution would be entirely proprietary - there are no existing DV-data drives."
Meaning that RED RAY drive is also proprietary (as would be MiniDV-DATA drive), but unlike RED REY, Mini DV media is not... Therefore one could get the media anywhere (not just from RED)...

[MistahTibbs]

DVD - 2 to 3 years at best


Hey Peter,

That can't be correct.

Granted it's the corporation's perspective but Taiyo Yuden DVD-R media have an archival shelf life (when kept in appropriate conditions and that's pretty easy to do) of up to 50 years.

The downside is only 4.7 gigs per disc which means a TON of swapping while archiving but, still, for now, until larger media (blu-ray, etc) come out at viable prices for said archiving the DVD is a reasonable avenue to pursue for those on a budget. Hell, TY DVD-Rs are about .45 cents each.

It'll be tougher for those who shoot weddings and recitals d/t the amount of footage shot but for those doing shorts and films that shouldn't be an issue.

sincerely,


MT

[Allbird]

Dude, dual drive RAID-0 gives You pretty close to double (maybe between 190~195%) rate of a single drive, not 130%...

RAID-1 can be HW, but it can also be SW (especially on mac). And I have clearly said that You would get almost the same speed (~100%) of a single drive.

Why people always think $$$$/GB as the only factor when it comes to ARCHIVE/BACKUP? By far the most important factor is longevity of media. How about:

DVD - 2 to 3 years at best
MiniDV - 12 years + (good tapes do 25+)
HDD - 5 years max, but drop it once and say goodbye to your data
BR - early to say, but would not expect more then 2 years

miniDV tapes has 12 years of longegity, but you could have a drop out since the first second.

DVD has 3 years longevity over the shelf of my living room. in its best conditions for best brands (taiyo, not princo, or pringo [fu*ked], like we say in spanish), it should be 20 years.

I dont know how much lives a HDD. but for my, now, is, of course, the best solution.

This HDD has its own raid controller, & transfer across FW800.
This shelf show 7TB striped in RAID1. it means 3.5TB of data fully duplicated.
it means 28 hours of redcode36 raw.
an indi movie should not be more than 20 hour long of footage.

Then, the purchase of this shelf is now about $1300.

For me is not so much if you think that we are releasing our own 2 hours long pro movie.
If we want a cheaper solution, we have to wait 3 years until it cost $400, and remain working with a tape-based camera like sony or canon.

never before like now we were able to be part of the optical/digital cinema because the extreme cost of it.
If you want to be part of this revolution era of "ourselves" filmmakers you have to pay it.

[Dylan Reeve]

I've had a single SCSI hard drive (4GB I believe it was) running 24/7 for nearly 10 years before failure. Hard drives can and do have a good life-span, but that becomes something of a crap-shoot if they are left sitting unused an extensive period.

There's no reason that a RAID setup like the one above couldn't be kept in operation for many years without failure. Even if a drive fails, it can be replaced without data loss. This process could conceivable continue for as long as you're able to provide power and have a method of connectivity. Your biggest risk (and one of the drawbacks of HW vs. SW RAID) is controller failure, where the inability to replace it with an identical or compatible hardware controller can render the array unreadable.

Personally I think high-capacity data tapes (even with the prohibitive initial cost) is the best option. As demand for media data backup increases, it is increasingly likely that people will offer out-sourced services to do it. Something like 'you send me a SATA hard drive, I return it and a few LTO-4 tapes.'

And while LTO is very expensive (but also offers a fantastic capacity) there are other less expensive option. AIT drives are quite a lot cheaper (although offer lower capacity, still greater than Blu-Ray or DVD however). Also DLT and DAT/DDS tape drives are a possability.

[Peter Majtan]

How about remote locations?

Just imagine indie filmmaker shooting a doco for 3 months in Madagascar (like I did). Even if You could carry RAID storage with You, it will die on the first day from the vibration after driving on bumpy roads for the first hour. My back still hurts just remembering that... Even if it somehow managed to survive - how do You power it? A small portable digital tape recorder could easily run on batteries that You could recharge on solar-power, or even run it of solar directly. We had few Motorola's portable solar panels that charged our Iridium phones, radios and all camera gear...

Any optical media in this situation will also have trouble not getting scratched...

Digital tape is the best option by far and Sycophant is right there. However they all require special tapes that might be hard to come by in a remote regions. Of course some could argue You can stock up before Your trip, but You always end up shooting more then You thought thanks to Murphy...

I still doubt Allbird that You can buy 7TB with RAID controller for $ 1300,- USD, but HDD's will get there (and beyond) soon enough. Issues there are high power consumption and associated maintenance.

The Taiyo Yuden DVD-R are impossible to come by in Mexico, especially in our region. Heck - we even can't get hold of the Verbatim's Gold Archival DVD's here without ordering large quantities from USA...

And lastly - I am not trying to get things "cheap" for me. After all we have in our companies several 4Gb fiber RAID enclosures in 19" racks exceeding dozens TBs...

But I have not forgotten about the indies on extreme tight budgets (that I was once myself) and Scarlet is clearly targeted to them...

Few of my guys loves to do indie projects in their spare time (which I don't give them much :innocent: ) and I fully support them. They are all over their heads right now having to wait so long for Scarlet to come out...

[Allbird]

hi peter.

u are right with remote locations. i still remember a work i made in Menorca that became a simple holidays when i came & discovered that my hdd becomes to brick! i think it wasn't vibration, i thought about hot.

i was thinking about normal conditions for a short-movies or pictures in standard locations. sure that LTO is the best solution, but you need a deck.

yes, the WD HDD cost is here, in europe, €433. 2TB.
last month was €480. do u understand? if we give again the calculator:

1TB (2TB mirrored) = 8 hours of REDCODE36

100GB = 48min

1GB ≈ 30seg

or

€433 / 1024GB = 0.4 €/GB ≈ 1€/1 minute of redcode36

here are the calculations for other formats:

DVD ≈ $0.20/min
miniDV ≈ $1.5/min
BR ≈ $1.4/min

in the HDD solution, u has more speed than optical formats.
in the HDD solution, u has direct access from any computer w/o decks.
in the HDD solution, u has a fully duplicated backup, where the others don't.

yes peter, for u the price is not so important, but for me is.
and i am agree to develop another solution for remote locations.

thanks peter for this post, this question always make me eating my head!

_____
EDIT
------


Originally Posted by Sycophant
There's no reason that a RAID setup like the one above couldn't be kept in operation for many years without failure. Even if a drive fails, it can be replaced without data loss. This process could conceivable continue for as long as you're able to provide power and have a method of connectivity. Your biggest risk (and one of the drawbacks of HW vs. SW RAID) is controller failure, where the inability to replace it with an identical or compatible hardware controller can render the array unreadable.

RAID1 controller only works on write.
in our situation, once done the back up, we don't need the controller any more.
if u want, u has two HDD separately with the same data.

[Dylan Reeve]

RAID1 controller only works on write.
in our situation, once done the back up, we don't need the controller any more.
if u want, u has two HDD separately with the same data.

I don't believe that is correct. The drives are not stand-alone drives. They are part of a RAID set. They can't simply be removed and used independently as a standard drive. While it may be easier to access data from a RAID1 set with a different RAID controller than with other RAID setups, it still requires the RAID controller.

[Peter Majtan]

hi peter.thanks peter for this post, this question always make me eating my head!

Thanks Allbird, I think this is the question that is by far the most underestimated in our industry.

When I was developing the restoration system in Japan I had the pleasure to work with the actual original negative shot by Akira Kurosawa. Even after so many decades and despite the film being greatly damaged - I was able to see the "data" (image) and have enough information to restore it successfully.

I just can imagine any of the current formats going through what this film was and being able to recover anything...

I have developed an optical system to record digital data using film recorder onto archival Kodak stock that could easily survive hundreds (yes - there is "s" at the end of the word!) of years and protect the data. This is by far the best option right now. However is it way to expensive and only suitable for material of high cultural or other value...

What is missing is truly portable and affordable solution that could archive to readily available and also affordable media with reasonable shelf life (10~25 years), that is not prone to easy damage...

DVD's fill that quite well except for the part of resisting damage...
Ditto for HDD's...
LTO, DLT's and other tapes are not too affordable and certainly not readily available...

SSM (solid state media) could be the answer when they drop in price and rise in capacity. But no one knows yet what happens to data on SSM after it has been in the archive "box" for 10 years...