Hard drives fail 15 times more than advertised.

[Marcus Irvin]

The article(linked below) in Computerworld magazine summarizes how real world drive failures are up to 15 times higher than the manufacturers' published MTBF numbers.

Quote-Despite major efforts, both in industry and in academia, high reliability remains a major challenge in running large-scale IT systems, and disaster prevention and cost of actual disasters make up a large fraction of the total cost of ownership.-EndQuote

This study by Carnegie Mellon University was focused on larger systems probably because that's where Computer Science majors expect to work, but the information applies to all high-performance drive users.

I've been called Chicken Little in this forum for harping on the real world unreliability of current storage options. I love tech, but one of the reasons we don't have more reliable tape and drive storage solutions is because too many people believe the reliability numbers provided by the data storage industry. My friends in IT tell me stories about every type of storage including the hugely expensive "enterprise" solutions. Until we customers publicly recognize there is a reliability problem, the industry will not be motivated to fix it.

Another surprising item in current research is how storing an archive drive, rather than using it, can actually increase the failure rate.

Up till now we've all known the biggest enemies of hard drives are heat, vibration/shock and, indirectly, dirty electricity. Apparently storage on a shelf has become a risk as well.

RED users are going to need lot's of reliable storage. If anyone has a "Consumer Reports" type of database for storage technology, that could be very helpful to us all. If such a database becomes very popular industry wide, it could start a reliability race exactly as the auto insurance crash tests have created with automobile crash safety ratings.

http://www.computerworld.com/action/article.do?command=viewArticleBasic&articleId=9012066

[Kenn Christenson]

In the current state of storage devices, I think it comes down to the old adage: "Don't put all your eggs in one basket."

I don't think anyone but some sort of kamikaze filmmaker is going to put all their prized footage on one drive and call it good. Backups, backups, backups!

[Craig Schober]

I've been called Chicken Little in this forum for harping on the real world unreliability of current storage options. I love tech, but one of the reasons we don't have more reliable tape and drive storage solutions is because too many people believe the reliability numbers provided by the data storage industry. My friends in IT tell me stories about every type of storage including the hugely expensive "enterprise" solutions. Until we customers publicly recognize there is a reliability problem, the industry will not be motivated to fix it.


i don't think anyone believes the mtbf numbers or even cares about them. they are too abstract. 1000000 mtbf hours will outlive me so how can it be relevant?

what people do believe is their own experience. i've had one laptop drive die on me before i could save it. i've had about 3 other drives die just after i was able to save the contents. now compared to the hundreds of drives i've owned and worked with, that's not too alarming to me. no, it's not as reliable as flash media but i just bought a 500gb sata for $100 at fry's yesterday. i'm willing to sacrifice a little reliability for a lot more storage space and money left in my pocket.

[I Bloom]

Don't forget about recovery services.

Worse comes to worse, they can take a broken drive into a clean room, open it up and pull off the contents for about $500 bucks. Not something I want to go through. But if there is $100,000 worth of production value on the drive, and you weren't able to back it up in time then this option exists.

How recoverable will RED footage be if the file is corrupt. I had heard that the video is made up of tiny chunks that are linked via quicktime container. Anyone know about this? Is audio and video in sync on these little chunks? Can you reconstruct them easily?

IB

[Stuart English]

The RED camera's Digital Magazine file system is based around FAT 32. That means you can read its contents from pretty much any Mac or Windows P.C.

Then as a result of FAT 32, no single file can be bigger than 2GB, which is a bit more than a minute of 4K RAW REDCODE at 24fps. Hence a clip can comprise of several independent video+audio+timecode+metadata files, each of which is capable of operating in isolation from each other. So if for any reason your data archive process corrupted one of these files, there would be no effect on the others.

The QuickTime files reference these video+audio+timecode+metadata files so that the clip appears as a single movie to Quicktime based applications.

So again, if for any reason the Quicktime file is corrupted there would be no effect on the original video+audio+timecode+metadata files.

[Adrian T.]

Stuart, will there be a possibility to recreate the QuickTime files from the video+audio+timecode+metadata files in REDCINE?

As far as I understand, you also need independent QuickTime files for all the proxy resolutions. It would be great if we could create files for missing resolutions in REDCINE.

[Stuart English]

That should't be a problem if its necessary. The 2GB data files are the most important elements...

[Adrian T.]

That should't be a problem if its necessary. The 2GB data files are the most important elements...

That's what I thought. Thanks, Stuart.

[Steve Freebairn]

Very nice on the data storage! Stuart, will the quicktime files be able to be named so that we add a tag and then the rest of the file name the camera creates? If not, how will the naming of the files be done?

[Michael Hastings]

Another surprising item in current research is how storing an archive drive, rather than using it, can actually increase the failure rate.

Up till now we've all known the biggest enemies of hard drives are heat, vibration/shock and, indirectly, dirty electricity. Apparently storage on a shelf has become a risk as well.

http://www.computerworld.com/action/article.do?command=viewArticleBasic&articleId=9012066

But isn't this mostly a risk that the overall drive will give a problem i.e. the drive spin up or head movement would be affected by sitting there with the lubricants gumming up or whatever. It seems to me the actual data on the platters would be in good shape, and could be accessed by removing them from the drive and putting it in another - not easy or ideal to recover but the point is the data would still be there. As opposed to other media - tape, optical, etc. that may degrade quicker at the actual data surface.

[RentCam]

In regards to the original poster's topic... Below is a direct link to Google Labs' recently published study on hard disk drive failure (Feb 2007). Interesting read for sure. It helps to put things in perspective.

Google Labs "Failure Trends in a Large Disk Drive Population (http://labs.google.com/papers/disk_failures.pdf)

Regards,

Jordan E. Miller
RentCam, Inc. (www.rent-cam.com)

[Stuart English]

The camera creates a 16 character base file name for all the files that relate to a clip - of course you are free to change these if you wish.

The name references -

Camera Letter

Digital Magazine number

Clip Number

Date in YYMMDD format

[Steven M. Bailey]

Most of the drive failures that I have worked on have been bearing related, and the disks were totally wrecked. I think the future lies in solid state solutions.:alien:

[MarcusX]

The majority of drive failures I encoutered were electronics related (about 70 drives, mostly cheap IDE models). I had about 10 drives in different computers which crashed in the same week due to a manufacturing error.

If you build a storage or backup system with harddrives, get different revisions or even different manufactures for drives and controllers.

[istvanttt]

The RED camera's Digital Magazine file system is based around FAT 32. That means you can read its contents from pretty much any Mac or Windows P.C.

Then as a result of FAT 32, no single file can be bigger than 2GB, which is a bit more than a minute of 4K RAW REDCODE at 24fps. Hence a clip can comprise of several independent video+audio+timecode+metadata files, each of which is capable of operating in isolation from each other. So if for any reason your data archive process corrupted one of these files, there would be no effect on the others.

The QuickTime files reference these video+audio+timecode+metadata files so that the clip appears as a single movie to Quicktime based applications.

So again, if for any reason the Quicktime file is corrupted there would be no effect on the original video+audio+timecode+metadata files.

Just a question, is this time of data recovery still possible on a RAID? I thought always that this is the biggest risk on RAIDs, that a recover center cannot just open the box and recreate the indexes?

[I Bloom]

Just a question, is this time of data recovery still possible on a RAID? I thought always that this is the biggest risk on RAIDs, that a recover center cannot just open the box and recreate the indexes?

In the case of RAID0 things could get a little hairy but I think it depends on the type of damage. If the recover center is able to pull a complete disc image off the damaged drive than it should be able to rebuild the whole RAID image. Not a simple task, but possible. If one drive is simply unrecoverable, or has an extremely corrupt filesystem than it may be impossible to recreate the RAID0. With RAID0 you have double the risk, since the failure of one drive in the pair, means the failure of both.

RAID0 has one advantage, speed. I think most of us view RAID0 as a temporary place to store footage before backing it up to a more robust solution. Because a RED DRIVE is a RAID0 many people have commented on this site that they wouldn't wait until their RED DRIVE was full to offload data.

IB

[I Bloom]

So again, if for any reason the Quicktime file is corrupted there would be no effect on the original video+audio+timecode+metadata files.

Stuart thanks, and more questions:

One of the big weaknesses I found in P2 early on was not really a Panasonic problem but an Apple FCP problem, the FCP P2 Importer couldn't deal with many (or really any) errors and unlike Quicktime Export the P2 Import function fails if you try to bring a large clip to a FAT32 scratch disk. I also found that occasionally the importer will drop one out of four audio tracks for no apparent reason.

Are the 2GB data chunks, that RED creates, one file that includes video+audio+timecode+metadata or does the RED seperate this data into seperate files ala P2?

If we loose a quicktime container file for some reason, what will the reconstruction process entail?

IB

[Stuart English]

Our 2GB data files contains all the video +audio+tiemcode+metadata for that section of the clip.

So there are no subsidiary files to loose track of.

[Steven M. Bailey]

The majority of drive failures I encoutered were electronics related (about 70 drives, mostly cheap IDE models). I had about 10 drives in different computers which crashed in the same week due to a manufacturing error.

If you build a storage or backup system with harddrives, get different revisions or even different manufactures for drives and controllers.

I have also experianced crazy anomalies from blitzing power supplys causing funky write errors and drive mis-reporting. The worst sound in the world is squeek-----squeek---squeeek--squeek-squeeksqueek-squeeeeeeeeeeeek. Thats when you know you have a new paper weight, or a new set of pocket mirrors. Makes you want to cry.

Whatever you find as a storage solution remember this; Dont put all your eggs in one basket.:bye2: