A solution to a serious problem

[Jay A. Kelley]

Imagine this:

A 4 -6 bay RAID 5 array using Solid state 2.5" drives.
On Top an LTO drive or simlar.
7 inch touch screen monitor.
Multi card reader
PCCard Reader
USB ports
Fibrechannel
Esata, ets
Portable, battery powered.

Called the "J-Box Field Media Safe"

User shoots with their "Tapeless Camera" then plugs their flash media, firestore, red drive, etc. Into the J-box where, it checks the drive and, with a push of a couple buttons on the screen, all media is dumped to the SSD drives quickly, verified, and then, while the shooter is back to work, all material is backed up to the LTO tape.

Cost: About 4 -6 grand.

Why: Cause the tapeless workflow is NOT safe enough right now and no one has built something that is designed around safty, and can be used in the field.

WE MUST HAVE A PLACE TO PUT ALL THIS FOOTAGE.

Jim, build this.. But remember it's my idea! So talk to me first!!

:)

Jay

[John Moores]

Why don't you build it Jay? I'd buy one ;)

Jm

[Jay A. Kelley]

Thanks.. If only I could.. I know HOW it's designed, but do not have the resources.

Jay

[Noah Kadner]

Most of us working tapeless have already built something similar for our own use. I wonder why people get scared that they don't have a tape. Tapes can be degaussed, burned and run over just as much as a hard drive can.

I've shot tapeless for over a year now and not lost one shot. But nevertheless, a nice rack mounted unit that does everything I do manually now could be an attractive product if the functionality is perfect and the price is right.

For example- I can build a P2 transfer station with a laptop, P2 drive and a raided set of FW800 drives for about 5 grand. Take away the drive and go right into PCMCIA on the laptop and we're at maybe 3, depending on the laptop.

Noah

[Darwin]

I see no real need for tape. Yes drives crash and fail, thing go wrong. And some kind of backup should be used. Also most data can be recovered no matter how bad the crash.

[Simon Blackledge]

RedSafe ?

[Jay A. Kelley]

I get you.. The goal here would be "One Box" self running.. It's NOT a playback device.. It's built on safely holding the data.

As for being as safe as tape, I do not agree. IMHO at least two backups should be present when shooting.

I am sometimes on shoots that cost up to $50,000 per day... Safe is what I am all about.

Jay

[Noah Kadner]

And 35mm film can be accidentally exposed, scratched or fogged on an airport x-ray scanner or the perfs ripped out in telecine. What else is new?

Show us a medium that is completely bullet proof and we'll find a way to destroy it. Unless we're talking about those new UnObtanium HD Cartridges®. Those are sweet. :construction:

Noah

[Mick van Rossum, NSC]

I see no real need for tape. Yes drives crash and fail, thing go wrong. And some kind of backup should be used. Also most data can be recovered no matter how bad the crash.

AFAIK there is NO insurance company that will insure a tapeless production.....Which is a very important consideration for big productions....
Maybe I am informed wrong (I hope) so anyone knows more about this ?

Mick van Rossum NSC
Netherlands

[Darwin]

AFAIK there is NO insurance company that will insure a tapeless production.....Which is a very important consideration for big productions....
Maybe I am informed wrong (I hope) so anyone knows more about this ?

Mick van Rossum NSC
Netherlands

No your right...Just thinking you could get by in a pinch with out tape.

[vanguy]

Imagine this:

A 4 -6 bay RAID 5 array using Solid state 2.5" drives.
On Top an LTO drive or simlar.
7 inch touch screen monitor.
Multi card reader
PCCard Reader
USB ports
Fibrechannel
Esata, ets
Portable, battery powered.

Cost: About 4 -6 grand.


Well, the parts costs are going to put it over the top.

I can see the computer parts (screen, card readers, fibre channel) being about a grand total, and the batteries a couple of hundred, but LTOs are about 4 grand for an internal LTO-4, and solid state 2.5" 32 GB drives are going for around $350 each, and I'd estimate you'd need the best part of a TB to do this (to allow for the RAID redundancy), which means 30 of the little buggers at a cost of about 11 grand.

Total parts costs: upwards of $16K.

Now if you were willing to accept 2.5" standard laptop drives (6 of them), it could be done in the price range. LOTS cheaper without the LTO.

[Jay A. Kelley]

Well, the parts costs are going to put it over the top.

I can see the computer parts (screen, card readers, fibre channel) being about a grand total, and the batteries a couple of hundred, but LTOs are about 4 grand for an internal LTO-4, and solid state 2.5" 32 GB drives are going for around $350 each, and I'd estimate you'd need the best part of a TB to do this (to allow for the RAID redundancy), which means 30 of the little buggers at a cost of about 11 grand.

Total parts costs: upwards of $16K.

Now if you were willing to accept 2.5" standard laptop drives (6 of them), it could be done in the price range. LOTS cheaper without the LTO.
I'm grateful for such specific feedback.. I think the LTO Drive is the most important part.. Don't you?
Jay

[Storyline]

This seems to be an interim problem that will plague field productions (especially documentary shoots) for the next few years until Flash-based recording media drop substantially in price. The problem should lessen when Flash becomes something that high-shooting ratio productions can buy at a reasonable enough multiple of tape price so that a shoot of several days to weeks can be kept on Flash while still in the field until a return to home base allows for proper transfer to RAID-protected HDDs and archival data tape storage. In the meantime, field HDDs are risky, Flash is high-priced, and few indy shooters want to end a long day of shooting then handling data backups in the field (exhausted, hungry, error-prone state of mind).

It'll get better over the coming years, but for now it's a major pain point economically and logistically. Meanwhile, I support the notion of a Red transfer system that would allow multiple copies to be made in the field without much fuss or management.

[Gavin Greenwalt]

Links:
UMPC (http://www.provantage.com/samsung-np-q1-v002~7SAMG001.htm) $1109.20
+
Multicard Reader/USB Hub (http://www.newegg.com/Product/Product.aspx?Item=N82E16817158010) $31.99
+
External USB2.0 RAID (http://www.tigerdirect.com/applications/searchtools/item-details.asp?EdpNo=2719707&Sku=S78-1002&SRCCODE=GOOGLEBASE&CMP=OTC-GOOGLEBASE) $244.99
+
2x 1TB Drives for 1TB of RAID 1 (Solid State just would be too freakin' expensive for the amount of data you would probably need to offload (http://www.newegg.com/Product/Product.aspx?Item=N82E16822145143) $798
+
External USB2.0 400GB Tape drive ~$1200
+
100' Extension Cord (http://www.amazon.com/Coleman-Cable-02509-Orange-Extension/dp/B00004SQF0) $49.93
+
UPS+Surge (http://www.tigerdirect.com/applications/SearchTools/item-details.asp?EdpNo=1522802&Sku=C910-1258) $64.99
+
Battery x 3 (http://www.sears.com/sr/javasr/product.do?cat=Batteries+%26+Chargers&pid=02827994000&vertical=AUTO&subcat=Optima+Batteries&BV_SessionID=@@@@1098667593.1181793435@@@@&BV_EngineID=cccjaddlejfiddlcefecemldffidfmg.0) $657
+
Inverter (http://www.dcacpowerinverters.com/itemdesc.asp?CartId={39AB538E-FBE7-4DFE-ADD0-FD0C2F33680BEVEREST}&ic=PW900-12) $74.99
+
Dedicated Generator... just in case. (http://duropower.com/item.asp?PID=57&FID=1&level=0) $99
+
Pelican Case (http://www.all-pelican-cases-4-less.com/detail_pelican_1560.html) $134.50

$4465

[vanguy]

Nice work, Gavin, sourcing all that stuff!

I can't find a 400 GB tape drive for much under $4k. What unit were you thinking of?

[Gavin Greenwalt]

Uhhh I only found a SCSI one for that much. So I sort of made it up. Hence no link. :D

http://www.tape4backup.com/dw017b.php

[vanguy]

Well, you could always add some kind of SCSI adapter. Not too expensive.

BTW that drive is only 200 GB. The 400 GB rating is for 2:1 compression, which you'll never get out of video.

But not too insane for a portable system. Now for the software!

I think there might be a better solution for the power supply. A battery/invertor/adapter chain is inefficient. Possibly some kind of custom power harness that supplies all the different voltages. Not too complicated to build.

But you might be onto something here. The poor man's Red Vault!

[Steven M. Bailey]

I like the Idea for the j-box and as a past IT guy I have been thinking of many different ways to build it. I'm not sure how practical it would be, but I think that a small portable satellite transceiver would be in order. Something that can be programmed to send an encrypted copy of your work straight to a multi-raid multi redundant server back at your home-base workstation. Maybe something like the j-box with an onstar-style antenna. If you have to have a tape, put it on the backside of your server.



This device would also come in handy when the theaters wise up and go to satellite download for distribution. You could upload your finished film to a satellite feed station. Maybe just distribute your encrypted daily's.

Google could start a storage management program for film management and indy-storage. Satellite feeds could go strait to a G-store drive at the G-film center.

I think a lot of your "j-boxes" will be popping up soon. :shifty:

[Gavin Greenwalt]

Well, you could always add some kind of SCSI adapter. Not too expensive.


It is if you're using an UMPC as your host!

Personally I really have to question the need for a field tape backup system. If you're shooting in a studio then the whole Jbox is pointless anyway. Use a workstation. But it just feels like a RAID 1 would be more than sufficient for field capture. What are the chances 2 drives are going to fail in the field simultaneously? Even better buy a stack of 8 200GB drives for $100 each. Fill 2. Place each in seperate pelican. Fill another 2 place each in pelican. (Assuming you're using something like CF that can only shoot 10-20 minutes.) This way you have a pair of backups in two seperate cases. Lose one, no problem. Drop one, no problem. If you're still paranoid. As soon as they've been cloned using the UMPC as a host, take the pelican cases (in seperate vehicles) to the nearest desktop workstation and then backup to large NAS and backup to LTO.

Edit: Whoops stand corrected:

USB to SCSI adapter (http://www.backupworks.com/index.asp?PageAction=VIEWPROD&ProdID=881) $93

[Axel Mertes]

AFAIK there is NO insurance company that will insure a tapeless production.....Which is a very important consideration for big productions....
Maybe I am informed wrong (I hope) so anyone knows more about this ?

Mick van Rossum NSC
Netherlands


I was told that DVS was forced to build a backup SATA RAID as compagnion for their CineReel (or how its named) field recorder, so that data is double protected by independed RAID5 systems. Apparently I am not sure if this stuff is still on sale at all.

Anyhow, I am quite sure that you must get insurance for this, because you could simply ask the insurance company how THEY insure their own data...

We for instance have an insurance for our company that just says certain amount of backup cycles per week. There is no specification if it has to be tape or disk based. We prefer disk based nearline storage and even for archival (then two disks with the very same data, independently).

We had a long history with tape based backup systems, starting with DDS-DAT, Exabyte, Exabyte Mammoth, DLT etc. also as libraries. While just running as a backup it was all neat, but beware of the moment you really need to get data back... IF you manage to get through its a big bottle neck at last - except you can afford a really huge library system. But the TCO of such a library has a break even point that is clearly far away from Indy use!

If you want a cheap backup I'd recommend to use something like "C-Enter" hard disk removable frames. This one is build into a standart CD-ROM slot, gives you hot-plug SATA drive change (with full SATA2 bandwidth - because it connects you pure drive) and you can use any standart 3.5" SATA drive without any screws, housing etc. just the naked drive. In germany I'd pay ~25 Euros for this and from then on can use any naked drive I want to, giving you less than 0,20 Euros/GByte.

See:
http://www.pearl.de/productchoice_redirect.jsp?pdid=PE4292&catid=1306&rate=1&query=c-enter SATA

So initial investment is ~25 Euros and cost per GByte is ~0,18 Euros currently for best price/GByte (300 GByte selling for 55 Euros here, including 19% VAT).

A tape could be cheaper, but its drastically slower, its inconvenient in use, its initial investment is drastically higher to the above. When do the curves cross? And did you include to calculate the time you need extra when you WANT DATA back?

If you don't care about this extra time, the lines cross around slightly less than 4000 Euros between the harddrive vs. LTO3, totaling in ~22 TByte as break even. How realistic is that for an Indy filmer?

And if you care about your time, knowing this LTO3 drive is the bottleneck, not affording reserve LTO3 drives, you could delay this break even way off.

Imagine you take at least twice as much time using the LTO3 rather than the harddisk drive, paying a hypothetic extra cost (for machine use and your time to do it) of like 15 Euros / hour of used time to backup or restore the data. With a SATA drive you get about 200 GByte/hour, with the LTO3 you may get 100 GByte/hour. Imagine you just do backup, no retreaval. The calculation tells me that your break even will be around 170 TBytes of total archived storage. If you charge 20 Euros/hour of use you already never match! OK, these parameters are symbolic, but the direction is right.

These removeable disks are hell fast and instantly available, on any system. Even if you don't have that neat removeable frame, it'll be easy to hook up the drive to any computer.

The only problem with harddrives for long term storage is that their mechanics is build in and may get dry and therefor fail over time. The tape has not much own mechanics, making it better in that perspective. But how long do you need that data anyway? What about encoding the data for compacter storage and using optical media for this kind of long term archival?

Cheers,
Axel

[Craig W. Bickerstaff]

AFAIK there is NO insurance company that will insure a tapeless production.....Which is a very important consideration for big productions....
Maybe I am informed wrong (I hope) so anyone knows more about this ?

Mick van Rossum NSC
Netherlands

Maybe the next Red announcement will be RED TAPE a cheap cost effective tape solution for 4:4:4 RGB along with an SRW type mountable mag,
You herd it here first :bleh:

[IAN SUN]

How about a solution that is flexible live, and cheaper to get into?

drobo.com (http://www.drobo.com/)
drobospace.com (http://www.drobospace.com/)

Is anyone familiar with this product?
It sounds great except for the USB connection.
But as a far more flexible alternative to RAID 1, I am very intrigued.

[Shane Betts]

What about a notebook mounted in a Pelican with a couple of TB firewire drives on set. Red drives are copied to those for safety (and on-set playback where needed) and then transported back to the edit station each night to be copied to the edit suite (RAID 5), which in turn is backed up to another RAID 5 array offsite (like down/across the road) via ethernet, iSCSI or FC network.

Once each days work has been copied to the edit suite/server, those RED drives and onset laptop array are erased and used for the next day's shooting.

You'd only need to have enough spare for a day's shooting while one set of drives is being transported/copied/erased. One set on set and another in transit.

As someone else has said, it's not as though film can't be destroyed or lost. Losing a day should be well within the insurance company's ability to cope - so long as the whole lot doesn't go down the toilet. The whole business world survives on dual RAID systems, a film should be able to do the same.

[Jay A. Kelley]

I have attached a VERY rough idea of what I am thinking.. To keep the cost down I went with normal SATA 400gb drives. They would be a RAID 5 configutation and be a little over a Terabyte.

Bottom left front is the LTO drive or something similar.

It's also possible to make the hard drives SCSI since the tape drive is.. Cut down on interfaces..

Bottom right front: All the readers and connections.

Top is a small LCD monitor, and the control panel is on the right on top.

Box is not that big (About the size of a pana field tape deck.

Since I am using "real" hard drives, power would be an issue. But I would still love to have one
Jay

[Joel Kaye]

How about a solution that is flexible live, and cheaper to get into?

drobo.com (http://www.drobo.com/)
drobospace.com (http://www.drobospace.com/)

Is anyone familiar with this product?
It sounds great except for the USB connection.
But as a far more flexible alternative to RAID 1, I am very intrigued.


Yeah - the demo is pretty cool.
http://www.drobo.com/products_demo.aspx

That and a laptop might be a pretty good way to go until we figure out something better.

I just bought a 300 gig drive for $60 a couple days ago. Maybe you get two of those for each RED Drive you need to dump.

I'm very curious how everyone is going to solve backup issues.

[Curran Giddens]

Didn't the S.two used to have an LTO backup system? You could use much of the same components of the shelf and just use a laptop for interface.

http://www.stwo-corp.com

[Gavin Greenwalt]

DROBO is pretty cool sounds a lot like windows home server: http://www.microsoft.com/windows/products/winfamily/windowshomeserver/default.mspx

The only problem with Drobo that I could possibly see, which is simultaneously its greatest stength, is the ability to interchange drives. If I backed up everything to two drives and then later sent them off (one to storage and one to post) in order to reconstruct the data would I need a Drobo on the other end since the intelligent balancing is assume proprietary?

If you used RAID 1 (which is really all you would ever want in a Master/Clone backup scenario) you could drop it into any RAID 1 array or simply reconstruct the partition table depending on your preference to get access to the data.

[tj williams]

The place where the hard disk crashed is before the backup so how do these tape solutions deal with that. My understanding of the Stwo system is that it records to two raid arrays at once so there are ALWAYS two sets of recordings. Then they copy that onto to other larger arrays and tape. Suspenders and Belt all the way...

[Aaron Burtle]

The place where the hard disk crashed is before the backup so how do these tape solutions deal with that. My understanding of the Stwo system is that it records to two raid arrays at once so there are ALWAYS two sets of recordings. Then they copy that onto to other larger arrays and tape. Suspenders and Belt all the way...


The Stwo work flow records to an array of drives in raid-0 which can then be used to make 2 LTO masters with their docking station.


http://www.stwo-corp.com/white-paps/cinegear.pdf

[iconoclast]

Memory engrams work well for around 70 years or so :) Except Dementia tends to be a problem in the longer term.

Tape or Tape less, makes little difference. As already mentioned, film stock can come to harm just as much as a tape can be magnetized.

I think the issue here is that the person responsible for product control have systems in place to ensure duplication of the 1's and 0's is done as quickly and efficiently as possible.

BIG Hard drives are cheap today. (My first 20 MB drive cost me $2500! I can't buy a Hard drive in that price proportional to data capacity today! 750G drives are around $300 already)

Tape to Tape copy is painfully slow, even at double and triple speed. And for some reason, people always keep them in pairs?

Flash cards are susceptible to damage as well.

So there is no "bullet proof" system. I keep an online version of working material on our server farm, I keep tapes safe, and I keep an extra copy on hard drive.

Blue-Ray/HD-DVD might help in the future, at least it's closer to a more useful size than a 4.7GB DVD, and should be faster to burn and retrieve, but it's still not optimal when your online material is in the terrabytes! (And don't worry, RAW and bigger image sizes will quickly fill all that up too!)

I like tapes, small errors can be fixed. Hard drives that crash are generally a total loss. Flash cards corrupt and you can loose the lot.

I haven't yet resorted to "dual" recording, although I have been known to record to tape and to hard drive at the same time :)

But only because I don't have to capture the tape back in the office :)

BTW does anyone actually CHECK their tapes before wrapping? I'm amazed as to how many people don't do random quality checks on location as they are filming.

[Cail Young]

BTW does anyone actually CHECK their tapes before wrapping? I'm amazed as to how many people don't do random quality checks on location as they are filming.

The problem with tape checking (unlike gate checking for film) is that if you're not very careful you can blow your timecode which causes many headaches.

[iconoclast]

What you mean you don't let your tape run an extra few seconds after you "cut" and you don't run 5 seconds of colour bars after pausing the camera for longer than about 90 seconds?

We got into the habit a LONG LONG time ago, even before I knew what TIME CODE BREAK was! (Form the VHS days of loosing sync I think was where I got into the habit!) of running an extra few seconds of tape after a CUT, or when you know you are going to leave the camera or turn it off, whack on 5 seconds of bars.

Then using your frame feeders, you can simply run back and forth to the middle of the unimportant part and pick up running time code form there.

In 7 years we've had very very few time code breaks or drops. So it's all really habit and good camera operatormanship that ensures you don't have broken time code, wrong time code or 2 minute "tape roll" breaks in your material!

In my books, one will drop focus more times than they will drop time code :)

[Cail Young]

I should have elaborated - I don't personally have any problems with this, but the fact that it requires such vigilance means that education these days tends to sidestep the issue and say 'never do it' rather than actually go through the procedures.

[Michael Schrengohst]

Tape? What's tape?

[iconoclast]

ROFL! Tape, is, a not quite antiquated method of storing vast quantities of data in a light portable and pocket sized format.

Mind you, a DVCAM PDV184 tape is 38 Gigs if data, and a 300 Gig hard drive can hold nearly ten of those and is about the same size. Hell of a lot easier than carrying ten tapes!

[Gavin Greenwalt]

The best digital backup policy is, has been and shall for all time be: Keep as many copies in as many places as humanely possible.

Easy for a few gigs. More problematic for terrabytes.

Like it was said: every solution fails. With multiple solutions, the probability that multiple solutions fail simultaneously goes down. Hence... RAIDs, Tape backups and JBODs. If I put a feature on 10 hard drives and put them in 10 closets, the probability that all 10 would fail just sitting in 10 years is suprisingly low. If you check their integrity anually the probability that all will fail is almost minimal. Add in a few tape copies in storage and it's practically foolproof.

Burn your data to a 10 Bluray Disks. Save it to 2 HDDs and store in seperate locations. When it comes to backing up I have always believed that quantity > quality. This is why I get reallllly nervous about storing physical mediums: paper, celluoid etc.. Even if it's in a vault... what if the vault floods? What if there's a fire?

Nature's ultimate survivor is bacteria. They aren't all that hearty by themselves but I'll be damned if you could track down all of them and kill them even if you wanted to. I think the Star Wars Christmas Special is living proof that as long as the data is stored in enough unique places, you can reach a critical mass of survival with any piece of data. Assuming someone wants to keep it around.

[James Liu]

I know this is an old thread, and I have no mean of advertising but since its related to this thread. CineRAID has recent made a debut of 'REDSafe' at the REDucation convention in LA last week. REDSafe is a portable RAID box (RAID 5 or 6 configurable) with 2-8TB of RAW storage and integrated CF card slot.
http://www.cineraid.com/4bayCF.htm

[Jay A. Kelley]

Zombie Thread....

[Jeff Kilgroe]

Dude.. James.. WTF? Start a new thread. Also not sure if this is spam or not. If you're just pimping their product on your own, I guess that's OK. I'd say the REDsafe name looks like a trademark violation to me too.