Help - Offloading Red Drives - Speed vs. Safety

[Dave Weber]

I was on a shoot the other night shooting interviews. When I went to offload the first drive through R3D Data Manager to 2 drives it gave me a time of about 2 hr 30 minutes for the Sata drive and over 4 hours for the FW800 for 126 GB. I ended up just doing a drag and drop transfer which took less time but not as safe without sum check.

What are you finding as the Fastest/Safest way to offload RED Drives?

[jimhare]

Hi Dave,

Many will call me insane, but I always just do a drag copy to a SATA drive, and then clone the drive. Never had a single issue with corrupted data. Twice I have had Red Drives fail, but nothing to do with the copy process and was before Red Undead.

Having said that, I always check the footage before allowing the drives to be erased.

Since I work on multi cam productions I work with up to 10 Red Drives at a time, usually full.

As I said, never a problem so while the risk is always there (as with anything) my personal experience is that there isn't a measurable danger in copying.
More about insurance and peace of mind.

[Alexis Hanawalt]

Just do a get info on folder you copied over and the copy and make sure the bit sum (the long number) is identical - this is just as accurate as the fastest checksum that R3D data manager offers. If the bit sums are different, copy over again.

[Conrad Hunziker]

Just do a get info on folder you copied over and the copy and make sure the bit sum (the long number) is identical - this is just as accurate as the fastest checksum that R3D data manager offers. If the bit sums are different, copy over again.

Hate to say it, but this is EXACTLY wrong.

You can have two different files with the same exact size with different contents. Are those contents important to you?

The 'bit sum', which I think you are referring to the folder size, is just a addition of all the sizes of the files contained in that folder. That says nothing about the content of each folder being exactly the same. The folder size is NOT the same as a checksum. Anyone who says that is dead wrong.

The only way to ensure that the data is exactly the same is to read it back on all destinations, and ensure that it reads exactly as the source read. Ask anyone who has had errors how important that is.

So the question really is, how important is your footage? If its you and your buddies, then who cares. If its a network show, Id certainly make sure all my ones and zeros were correct before you erase the source.

[Conrad Hunziker]

When I went to offload the first drive through R3D Data Manager to 2 drives

The issue is how you had those drives hooked up, and the throughput on the source. I dont know how you had the drive hooked up, but Im assuming it was FW800. In that case, copying to multiple drives at once divides the total bandwidth available to copy, so it will take a little more than twice as long. So in this case if you copied to a single drive twice, you would see it go significantly faster.

The next version of R3D Data Manager, due out shortly, will allow you to copy to several destinations at once, with no speed penalty. So the above would not apply.

[Alexis Hanawalt]

My apologies, it's "byte sum" under get info. The odds two folder's having this same number yet having different content are REALLY low, right? What is the quickest option in R3D data manager doing differently? My guess was that is was checking the byte sum of each individual file, rather than the whole folder.

[Dave Weber]

The issue is how you had those drives hooked up, and the throughput on the source. I dont know how you had the drive hooked up, but Im assuming it was FW800. In that case, copying to multiple drives at once divides the total bandwidth available to copy, so it will take a little more than twice as long. So in this case if you copied to a single drive twice, you would see it go significantly faster.

The next version of R3D Data Manager, due out shortly, will allow you to copy to several destinations at once, with no speed penalty. So the above would not apply.


Conrad,

Yes that is the case. I had eSata to one and FW800 to the other. I wish you could daisy chain esata drives.

Looking forward to the next version. Free upgrade? =) I love your product it's a great tool.

thanks again,

[Brook Willard]

The things that I've seen go wrong with drag & drop transfers would make your skin crawl.

[Conrad Hunziker]

My apologies, it's "byte sum" under get info. The odds two folder's having this same number yet having different content are REALLY low, right? What is the quickest option in R3D data manager doing differently? My guess was that is was checking the byte sum of each individual file, rather than the whole folder.

I think you may be confusing what sums are.

Byte sum is the addition of the number of bytes in a file. This would give you a size for a folder, but is not unique to that folder. There may be multiple folders with different content but the same byte sum.

A checksum is the mathematical method to add the zeros and ones of a specific file to create a unique number for that file. Its basically reading a file, taking the zeros and ones, adding them up and dividing in a specific method, to come up with a 32 (or 128 or 256) bit number that represents that file. If a single zero or one is out of place, switched or otherwise mangled, the 32 bit representation is vastly different.

By using this method we can say that the content of two files are exactly the same, as opposed to saying that two files are the same size.

Ive used R3D Data Manager personally to transfer 6Pb of material to date (rough estimate). In that time Ive had 9 errors. Sometimes it was a stupid human error. The majority of the time it was a hardware failure - a reader error or a write error. Things that drag & drop simply dont have the ability to detect.

Computers are just machines. They fail. By using a checksum method, you will know when it fails when you can still do something about it. To me thats far better than the editor calling.....

[Dave Weber]

The things that I've seen go wrong with drag & drop transfers would make your skin crawl.

I'm sure. I hate doing it, but (knock on wood) I've had no trouble the times I have done it. :shocked:

[Steve Sherrick]

I'm sure. I hate doing it, but (knock on wood) I've had no trouble the times I have done it. :shocked:

I know that not all of us agree on this one. There are many of us working professionally on all levels of productions that have various perspectives on data transfer. To date, I do not know of a single frame of R3D files that has been lost with the method I choose to use. So, for me that's what allows me to sleep at night. That method happens to be R3D Data Manager and the MD5 Checksum process. Other great minds, such as Ian Bloom have argued for drag and drop as being a perfectly valid method. Ian is a very smart guy. He knows a lot of the under-the-hood stuff. He has his methods.

Whatever works for you, I suppose that's what you go with. If there are errors, regardless of the method used, you will have to answer for it. So that's why it's important to have confidence in the method you choose because if that day comes when you get a call from post saying that File XYZ has corrupted frames, they will not really care whether it was done with R3D Data Manager or drag and drop. They'll want to know why they don't have their frames, i.e. why didn't you make sure the frames were all intact on set.

Again, how you get there is up to you. Drag and drop with visual inspection, a 3rd party verification system such as R3D data, or by telepathy. Do lots of tests. There are those of us that have seen drag and drop fail. There may be others who have seen the 3rd party apps fail. I know which one works for me. That's why I've stuck with it.

[Dave Weber]

Again, how you get there is up to you. Drag and drop with visual inspection, a 3rd party verification system such as R3D data, or by telepathy. Do lots of tests. There are those of us that have seen drag and drop fail. There may be others who have seen the 3rd party apps fail. I know which one works for me. That's why I've stuck with it.

I agree Steve.

So what I'm getting from you all is that as of now, I am using the "quickest/safest" method through R3D Data Manager.

thanks for all of your input. I appreciate the work you do and the suggestions you give!

[Steve Sherrick]

I agree Steve.

So what I'm getting from you all is that as of now, I am using the "quickest/safest" method through R3D Data Manager.

thanks for all of your input. I appreciate the work you do and the suggestions you give!
In terms of speed, making sure your data pipelines are as fast is possible will get you faster transfers. If you are using a laptop and it has an express34 slot, then you can add an esata card. This is a much better solution than using the one FW800 bus on the motherboard. When you move over to a tower, there are even greater possibilities, as you have far more options for expansion. You can create quite powerful data transfer pipelines.

In terms of safety, again R3D Data is what I use and it is what I consider safe for myself, but I can't say this is the only method. There are others out there. And a lot of this can be done if you know how to get under the hood of your system, but Conrad has provided a GUI that makes it more practical for on set, pressure cooker scenarios. And there are several useful tools that come with the app as well. The price is also a no brainer.

Now, if you run into a situation where time just won't allow checksum verification - find out if there are ways to make the transfers more efficient throughout the day, i.e. swapping media more often. You still have to find a way to verify that the data is there and there are no errors. So if you use drag and drop, you'll have to at the very least use scrub playback verification and in the best case scenario you will watch through the clips in realtime. A sure way to find yourself on the phone with an upset producer is to just drag and drop, never inspect/verify the integrity of the data, and then hand it over. Not a good idea, but one that is done.

[Noah Kadner]

Yeah I'm with Steve 100%- one frame of footage lost can mean your entire reputation blown. Nobody wants that.

Noah

[Wil Klassen]

As I've come to find it really does depend on the eSATA card you use. Bought the Sonnet Tempo for at work, and its blazing fast, reliable and haven't had a problem since buying it last year no matter what i throw at it. I bought an el cheapo one for personal use, and it was/is garbage, after one 500gb transfer between drives its now topping out at a whopping 2mb/sec.

[Mike Prevette]

This is like a filmloader telling me they don't use a changing tent because doing it in the light is faster.

IT'S THE GODDAMNED MASTER FOOTAGE! DON'T TAKE CHANCES.

[Noah Richoz]

In my opinion whatever method you want to verify your data is fine...but just make sure you do so. The way I've sped up my workflow recently has been through clever scripts and hardware. I'm now using a LEMO to ESATA cable that I find to be about 30% faster then FW800 when connected to a newer Mac Pro. I also have a ESATA CF reader.

[Alex Carr]

If I cannotuse R3d DM, there is another potentially 'safe' option. I DO NOT Drag and Drop. I use cp -R in Terminal , as a Backup. Finder can crash, gui's are goofy sometimes. A system command is a tiny bit faster transfer than Finder and will continue transferring even if Finder Crashes. Plus it can be done without an operating system loaded to a MacPro... by booting single user. I do cp -R for any media that is not R3d, like EX3, P2, 5D, etc... And in the situation where Multiple Cameras exist I try to get enough drives that I can Checksum through Terminal after the transfer is complete to save a little bit of time and save the md5 output of Source and destination to a txt file and leave it in the root of the Mag.

The other check I have is doing an 1/8 res Debayer on an entire Mag after transferred. If a corrupt clip exists the frame count will not match. I create a snapshot of the finished Batch window of RedRushes to prove that footage is not corrupt. I never offer these files to Production, they are usually horrible.

[Dustin Cross]

I am interested to hear what problems people have had with Drag N Drop? I can't even imagine the ones that would make my skin crawl.


Dusty

[Philip Allister Anderson]

The last show I did I used R3D and then scrubbed through all the footage. If something hinky happened with the camera, transferring or when I scrubbed I would transcode a 1/8 debayer just to be sure. R3D takes 33% longer I believe but it's well worth the added security. If you catch a problem while you are still at the location you look golden. If the assistant editor catches it....death :)

[Stephen Lovett]

I am interested to hear what problems people have had with Drag N Drop? I can't even imagine the ones that would make my skin crawl.


Dusty

Hi Dustin,

I generally do a finder copy to a separate raid prior to using R3D data manager to do the "official" copy to the authoritative source directory. (I generally push pretty hard to be working of CF cards instead of Drives so the finder copy is about five to seven minutes) It's pretty fast, and it's a get out of jail free card in the very unlikely event that the raid six array that I offload to has an issue.

I won't go into exhaustive detail here, but I've seen the file size in byte count be different on occasion when doing a Finder copy. I'm not talking about the first size number listed in the get info window which is effected by block allocation size, but the actual byte count for the file.

I saw this three times last week. A second copy to the same location resulted in the expected identical file size.

I've also had a couple of experiences where the finder completed the copy, and terminated the copy dialog normally, but it skipped sub directories more than one level below the root level of the directory being copied.

That's super rare, but I've personally experienced it.

This is one of those things like a multiple hard drive failure. Chances are it's never going to bite you (so to speak) :-), but do if it does, you're completely hosed.

One of the reasons the guy comes around and hands me a check at the end of the shoot is that he doesn't have to worry about such things, it's my job to make sure it all just works.

Having said that I've had a couple of DP's insist that R3D data manager is too slow and insist that I do only finder copies.

After the obligatory "sin upon your head dread sovereign" conversation, if the director / producer agrees... I shut up and row.

Steve

[J. Eric Camp]

Plenty of people are going to have vastly different and and staunch opinions here. All I will say is that no matter what process you are using, you should be doing manual verification past it.

I have had a "copy, check versus check sum" program report a perfect transfer, only to find on manual verification that the clip had corrupt frames. Now these were not the fault of the program, they were in fact the fault of a dieing cf reader. However, the check sum program saw the data it it's corrupted state and copied it thusly. With out the manual verification I would have left the "during transfer induced" corruption in the clip.

So I say, no matter your transfer process, you should be doing manual verification. As Conrad puts it, how important is your footage. Take the time to make sure its right.

[Steve Sherrick]

Plenty of people are going to have vastly different and and staunch opinions here. All I will say is that no matter what process you are using, you should be doing manual verification past it.

I have had a "copy, check versus check sum" program report a perfect transfer, only to find on manual verification that the clip had corrupt frames. Now these were not the fault of the program, they were in fact the fault of a dieing cf reader. However, the check sum program saw the data it it's corrupted state and copied it thusly. With out the manual verification I would have left the "during transfer induced" corruption in the clip.

So I say, no matter your transfer process, you should be doing manual verification. As Conrad puts it, how important is your footage. Take the time to make sure its right.
Agree 100%. At the very least scrub through, and best case scenario watch the clips all the way through. Not only will you possibly catch a corruption in the file, but there are those times when you might catch a visual problem that you will bring to the camera team's attention (using the correct chain of command of course). This is why having multiple systems that can handle various tasks is often times a necessity on set. You can be backing up on one machine, while visually inspecting clips on another.

[IAN SUN]

Watch the footage in real time, there is no substitute.
Check sums are great, but they won't flag dropped frames or any other nastiness in your clips.

[Brook Willard]

I have had a "copy, check versus check sum" program report a perfect transfer, only to find on manual verification that the clip had corrupt frames.

I have had the same thing happen consistently with one application. It was also a hardware issue, but a particular [and popular] application out there lets a scary amount of corrupt footage slip by even its most thorough check.

Thankfully the errors were all caught before the original media was scrapped in all cases. I've since moved on to R3D Data Manager with great success.

[Rich Schaefer]

It was also a hardware issue, but a particular [and popular] application out there lets a scary amount of corrupt footage slip by even its most thorough check.


Ok Brook, now that you said that much, you know you have to name names here buddy. Who is it? or give us the clue. ;)

Prost,
Rich

[Mick van Rossum, NSC]

http://www.reduser.net/forum/showthread.php?t=44596

[Rick Burnett]

I code on OSX and I am going to reiterate that I wouldn't trust Finder either. I've copied thousands of large files during some application development I was doing and I have noticed a variety of issues:

(1) Copies stalling and locking up
(2) Incomplete copies, especially if other applications are touching the files (a backup started when something was copying)
(3) Lost Files
(4) Inconsistent file lists

The 4th one amazes me and I can recreate it easily. If I have a directory with > 1000 entries, I've had it not sort the contents properly based on name and sometimes those items not show up. It's ALWAYS on drives with thousands of items in the root directory.

For my own critical stuff, I use 'cp' as well from the command line. It has never failed me. Finder scares the CRAP out of me!

[Monty Bass]

R3D Data Manager is a must!

[Gunleik Groven]

I have had the same thing happen consistently with one application. It was also a hardware issue, but a particular [and popular] application out there lets a scary amount of corrupt footage slip by even its most thorough check.

Thankfully the errors were all caught before the original media was scrapped in all cases. I've since moved on to R3D Data Manager with great success.

Would that be Shotput pro?