How to copy and back up?

[glik]

Hello, we are shooting a feature with the red one.
what is the best way to copy the rushes from red hard drive to another hard disk? what should I take care of? is it a simple copy paste action? does fire wire400 is faster than usb? should I use back up program? why?
Thanks,
Glik.

[Dylan Reeve]

Copy and paste is not ideal. It lacks any sort of oversight and i highly prone to human error and potentially machine error too.

There is an application out that is designed to automate the process somewhat. I can't remember what it is just now. Otherwise many people use backup or file synchronisation tools. Even Microsoft's own SyncTool is an improvement. Otherwise, a the very least a command line (XCOPY in Windows, or cp in OS X) tool is probably a better choice.

[coopercam]

For what it's worth, I've been using Shotput Pro from imagine products, and it's worked out pretty well. Go to www.imagineproducts.com and check it out...the same program can also download P2 and Sony sxs cards. I think the app is less than $ 90 USD. FW400 or 800 seem to be running faster than USB 2.0.

Hope this helps, and best,

J

[Cüneyt Kaya]

aasync its free

[David Mutchler]

good to know

[Kaku Ito]

One point is to concern is to use multiple ports when you are copying.

Firewire 800 is fast but if you mix with 400 then the speed is going to be 400 worth.
Also, when you connect two 800s then that will use up the whole bandwidth.

If you are on Mac book Pro, the fastest way is to use the internal FW800 and eSATA or FW800 on the Express34 slot.

These things will be improved when USB3.0 or Firewire 3G compatible CPUs and devices come out.

[Virgil Kastrup]

I highly recommend R3D data manager.
I use it on all RED projecs.
http://www.r3ddata.com/

[Conrad Hunziker]

One of the problems many users have found with copies via the finder is that if there is an error, finder will just stop - and it may not alert the user its done so. For example if there is suddenly no more disk space, it will just stop and not alert the user.

R3D Data Manager was designed to copy everything, check its all there and verify that the copies match the original, bit for bit. And it provides a way to ensure that the copies are valid in the future.

[Dylan Reeve]

The same issue with copies just stopping applies to copies made in the Windows file explorer too. While it will usually alert you that it's failed, it's very difficult to pick up from where it left off. You have to do the whole thing again.

[Conrad Hunziker]

Thats one of the things about R3D Data Manager that I like: it will list exactly what files it was not able to transfer, and (if the checksum doesnt match) even copy those specific files again.

[C_Skaff]

hey guys
I've seen R3D Data Manager and it looks good for copying redmedias on set...

But does anyone know a good app for copying a hole HDD to multiple destinations incl LTO3a? that also does a checksum before giving a report.

kind of the same stuff that R3D Data Manager does. But instead of copying just a CF-card/Red-drive over to HDD's. It should copy the HDD from Set onto a raid / LTO for archiving.

/carl

[conrad gaunt]

I have an early prototype of a program I had previously written, and now rewritten, called Safecopy, for windows.

It will copy an entire drive to another. All non-system, non-hidden files are copied to a folder on the destination drive, named source_volume_label.bak.

The copy is verified on the fly (ie, it reads back what its just written and verifies it against data its just read, before transfering next chunk of data)

All successful copies are logged into a text file in the root of source_volume_label.bak, as are skipped filenames and any file copy errors/verification failures.

This verified copy method is quicker than a cut and paste also.

At the moment i think its working, but would like anybody interested in helping me test it out to PM me, and I'll send a copy.

Testing should be performed whilst NOT on a shoot since the program is alpha release.

Also, please resist asking for a copy if you're not likely to give some feedback, feedback is payment, but also essential for finding any bugs with such a potentially critical application.

It will be freeware once tested, and will copy to multiple drives. thanks

[C_Skaff]

the ".bak"... is that kind of like an image-file of the coped drive?
Meaning after its done, I can't brows the new folder to see whats in it...

/carl

[Conrad Hunziker]

hey guys
I've seen R3D Data Manager and it looks good for copying redmedias on set...

But does anyone know a good app for copying a hole HDD to multiple destinations incl LTO3a? that also does a checksum before giving a report.

kind of the same stuff that R3D Data Manager does. But instead of copying just a CF-card/Red-drive over to HDD's. It should copy the HDD from Set onto a raid / LTO for archiving.

/carl

Guys-

This has got to be one of the BIGGEST misconceptions about R3D Data Manager. It reads from ANY MEDIA. A Red Raid, a red CF card, or your hard drive, or even a network drive - it DOESNT MATTER.

As long as your media appears as a mounted drive, it can be the SOURCE or DESTINATION for any copy. Infact, if you wanted to and were so inclined you could copy FROM a HDD to a RED CF CARD!

R3D Data Manager provides a END-TO-END copy and verification process, and you should use R3D Data Manager EACH and EVERY time you copy R3D files, to ensure they are complete and precise.

[Brian Master]

You might want to use StorageDNA software to manage your data.
Basically with a StorageDNA enabled workflow you could have automated local LAN or remote WAN synchronizations of your footage. If you want to try it out you can get a trial app from the company web site. http://www.storagedna.com Please be sure to mention that you are a RED user in the request form.

[Robert Trim]

Anyone have a typical data move time from RED drive to backup drive(s), FW800 usingR3D Data Manager? Maybe the time it takes to move 100 gigs.
thanks

[Conrad Hunziker]

The time it takes to transfer is highly defendant on your system - the speed of the drives, how they are hooked up and the type of checksums used in addition to other factors.

For instance, having 2 or 3 drives on the same bus will slow down transfer times by 30% to 70%. So I would recommend not using firewire 800 to write to if it is on the same bus.

In addition, copying to multiple destinations at once will slow it down when copying from the Red Raid - as the seek time trying to find the data for multiple sets of copies will drive your total transfer time down. Finder suffers from this also.

One last point - you have to consider that the only way to verify your data is by reading it 3 times - once to create the checksums, once to copy the data, and once to verify the copied files. Finder and Explorer dont to that, so it seems to be 'faster', at the expense of 'safer'

[JanneJansson]

Just use rsync as root. Is already on all macs. The down side is that it's a command line tool, so if you'r not used to that is abit ackward.

I think RED is saying you should use the DiskTool and make a image of all drived and medias.

[Ted Chu]

Anyone use ChronoSync for the backups? Might be useful?

http://www.econtechnologies.com/site/pages/cs/chrono_overview.html

[Mark L. Pederson]

I have not personally used R3D Data Manager in a while - so, someone correct me if this my comments here are outdated and my issue/concern has already been addressed.

My biggest issue with R3D Data Manager is that IT WRITES (CHECKSUMS) onto your digital negative.

I don't want ANYTHING writing to my digital negative except the camera. Period. All copies should be clones of the volume as the camera created it.

I never understood why R3D Data Manager doesn't allow you to SET A DESTINATION FOLDER for the checksums - so you can write them on a volume which is not your camera media.

Did this feature already get implemented?

[Thomas Mathai]

Has anyone tried using rsync in OS X Terminal?

[Chris Parker]

so what do you use now mark?

[Jonas Rejman]

We started to develop our own application called Cinezoid, because of R3D Managers limitations.

As Mark, I was horrified, when I found out, that it writes into the magazines. Also, its not able to copy to a backup storage from a main storage, but reads the camera media again, which takes awful long and blocks the CF card.

We took another approach in our program. It basically starts with the checksum on the first copied storage. I do not see any benefit in reading checksums from the CF cards, only to find out, that they do not add up on the first media. You have a huge problem in that case anyway - it can be the cables, readers, firewire port, camera CF reader... - just too much hurdles to take, where a checksum will not help you anyway.

As you simply HAVE to visually check the data on your first copy one way or another, Cinezoid generates the checksums on the first copy, re-reads the data and makes a backup copy, with another checksum. This all happens in the background, and gets interrupted, when you insert another card to offload.

With 2 Intel X-25M SSD's it takes about 15 minutes for 16GB to copy, read to generate checksums, read to verify checksums, copy to backup, read to verify backup checksums. With normal drives, it takes about 30 minutes.

Cinezoid has an itunes-like interface, its written in java 1.6 under netbeans and its a 64bit application, so it will work well with snow leopard and it will be open-source. Currently its an alpha version.

I have started a wiki about it here (http://kenai.com/projects/cinezoid/pages/Home), on the Kenai Forge.

We are looking for hardcore beta-testers, to iron out some bugs, people that would not mind to put the program through its paces and report issues at kenai Forge (http://kenai.com/).

PM or email me at jonas@rejman.com, but only if you are serious about alpha-testing. Its nothing pretty... :-D

[Conrad Hunziker]

I have not personally used R3D Data Manager in a while - so, someone correct me if this my comments here are outdated and my issue/concern has already been addressed.

Yes, you are a bit late on the information. The latest release already has code to mitigate this issue, due out shortly.



I never understood why R3D Data Manager doesn't allow you to SET A DESTINATION FOLDER for the checksums - so you can write them on a volume which is not your camera media. Did this feature already get implemented?

The next release will allow you to write checksums to a separate folder.

However, there are a couple of more issues to look at:

- Not everyone follows the same workflow. Many users offload their drives in increments, keeping data on the Red Media until it is truly full. For those users, it will be possible to have RDC folders that already had checksums created but will need to be re-created due to other factors. So that will cost them additional time in creating those checksums.

- It is entirely possible to have duplicate RDC folders from DIFFERENT cameras. By writing checksums information to the original RDC folders you can mitigate which folder is which, and use the correct checksum. However, you will not be able to do that when writing checksms to an external folder, as you will not be able to determine which folders are using the correct checksums. This was a huge issue for camera builds before build 15, but is still an issue to this day even after Red changed the folder naming format from containing the year to random letters. I personally have seen on 2 occasions RDC folders that were exactly the same name, the only difference being the camera letter (ie, camera A, B, C...)

[Mark L. Pederson]

Yes, you are a bit late on the information. The latest release already has code to mitigate this issue, due out shortly.



The next release will allow you to write checksums to a separate folder.


Good news Photocon - I expect we'll start using again then! I do like the app - except for this issue.





However, there are a couple of more issues to look at:

- Not everyone follows the same workflow. Many users offload their drives in increments, keeping data on the Red Media until it is truly full. For those users, it will be possible to have RDC folders that already had checksums created but will need to be re-created due to other factors. So that will cost them additional time in creating those checksums.


Understood - and I respect people work in different ways - IMHO it is NOT SMART to offload in increments for several reasons such as an increased chance of human error - more opportunity to corrupt a directory, etc. - it feels a lot like the old days of "remove 35mm mag from camera, breaking the neg in the mag, putting mag back on the camera" - sure it works - but you are increasing you margin for error. IMO this entire "data management game" is all about reducing the opportunity for human error and redcing the chance for a file table/directory corruption.

Again, horses for courses - but I think folks who want to offload in increments can pay the price of slower checksums.




- It is entirely possible to have duplicate RDC folders from DIFFERENT cameras.

I personally have seen on 2 occasions RDC folders that were exactly the same name, the only difference being the camera letter (ie, camera A, B, C...)

Yes - this CAN happen - EXTREMELY RARE - and if your tech does not have different camera ID letters on each camera - he/she deserves it. I DO NOT buy this as a good arguement for writing checksums to my digital negative.

Again, if you give me the option to write those checksums to another folder - outside the RED MEDIA - I'm very happy and will have my team start using the app again.

[JanneJansson]

Has anyone tried using rsync in OS X Terminal?

Yepp I'm using that method. Works very good, very fast and realiable.

[Conrad Hunziker]

Also, its not able to copy to a backup storage from a main storage, but reads the camera media again, which takes awful long and blocks the CF card.

This is a popular misconception with R3D Data Manager. It can copy from any media - the Red Media or non-Red Media. The original copy or copies of copies. There is no reason why you cannot copy from any mountable filesystem to any mountable filesystem.


We took another approach in our program. It basically starts with the checksum on the first copied storage.


I strongly urge you to re-consider your approach, and hope you do not deceive the public into thinking this flawed logic is correct.

At no time should anyone ever consider the checksum of a COPY to be the standard to which all other copies are to be made. I cannot emphasize this point strongly enough. If you consider the checksum of a COPY to be the standard, then all you are doing is ensuring that all of your COPIES will forever have any defects that are in that copy but NOT in the original. This includes slipped bits, backward bits, misreads and miswrites.

Again, this is such an important point, I need to be very very clear. Your checksums must always be done from the original source, not ever from a copy. The copy's checksums must always be verified to the original, not a copy.

Every modern hard drive, regardless of manufacturer, has a feature called ECC - error correcting code. They do this because the data read off the hard drive will have errors in it. Drives are packed so tightly these days that it is literally impossible to get a 100% reliable read at any sort of sustained speed. The drive manufacturer knows this and has code to help mitigate errors on the fly. The read errors from the hard drive can be in the 10s of thousands every minute (during a sustained read), and the code will hopefully catch each one of those errors. But how do you know it catches every one of them, or that the data it replaced was valid?

So by the time you are creating checksums from those copies, you may have 2 levels of read errors already incorporated into those checksums - one from the Red Media, and the other from the first copy drive. Now you are proposing to use those checksums as the source for further copying?

I respectfully submit that you seriously re-think your entire approach. And I hope that the people who use your software do not rely on its checksum features as currently implemented as a cornerstone for any loss mitigation process.

By the way, it is not possible to see the ECC errors that your drive is correcting for by looking at the SMART information for your drive. SMART drives do collect this info, but do not publish it. You'll need a third party program to divulge this info from the drive. And yes, you will find that your drive, under normal sustained read operations, will have thousands of errors per minute - even 10s of thousands are "normal" - which the drive hopes to correct before passing it on to your computer. This is the standard operating procedure for every modern hard drive made this century.


I do not see any benefit in reading checksums from the CF cards, only to find out, that they do not add up on the first media. You have a huge problem in that case anyway - it can be the cables, readers, firewire port, camera CF reader... - just too much hurdles to take, where a checksum will not help you anyway.

Honestly, if you cannot count on your Red Media via the firewire port, cables and CF reader to make accurate checksums, why should you count on those items to deliver that media to your copy destination without error? If those things are not generating reliable data that you can count on, you should be using different hardware, plain and simple. And in reality, if any of that hardware goes bad (as can be the case), you should know as soon as possible - not after several copies have been made and assumed to be good, and even worse, after the original data has been overwritten.

One more point about checksums: It really needs to be a three-read process to create an accurate checksum and copy. You need to read the data from the source once to create the checksum and once to copy. Then you read the data once again on the destination to verify the checksum. Only by doing that can you verify that the original checksum from the source is valid, and not full of random read errors itself. Every checksum generation process is subject to random read errors. The only way to combat that is to read the data multiple times. If you only read once, then you only know the checksum of that particular read session, not that its a valid read.



As you simply HAVE to visually check the data on your first copy one way or another, Cinezoid generates the checksums on the first copy, re-reads the data and makes a backup copy, with another checksum. This all happens in the background, and gets interrupted, when you insert another card to offload.

I will not disagree that viewing the material after a copy should be part of one's normal data management duties. However, I will submit that your order of operations will require that you erase a Red Media of its data before you view the first copy. Because of this, there may be perfectly checksumed and copied data that doesnt add up to anything visually. You really need to ensure that data is always 3-way verified: all files are copied, all files match, and all data is valid. Your proposed workflow on working sets will require that only one will be accomplished before the Red Media is erased. This, in my view, should not be the standard for any production.

[Conrad Hunziker]

Good news Photocon - I expect we'll start using again then! I do like the app - except for this issue.

Watch for the upcoming 5.0 release. BTW, it will only work in Mac 10.5



Understood - and I respect people work in different ways - IMHO it is NOT SMART to offload in increments for several reasons such as an increased chance of human error...

I wholeheartedly agree - but people like their workflows, and R3D Data Manager tries to accommodate all types. I personally feel that this is one of the ramifications of the way the release of the camera occurred. Thats not Reds fault - but there was so much mis-information out there that the opportunity to instill correct workflows was filled instead with bad information and people's fears. To this day I still correct people with information from camera build 8.

[Conrad Hunziker]

Has anyone tried using rsync in OS X Terminal?

I have a couple of comments about using rsync.

First, it doesnt provide a way to re-verify your data at a later date. It only provides a way of making sure the destination matches the source. Once you erase the source, there is no chance of re-verification.

Second, it doesnt provide a way of validating the source. It just assumes that the read data is correct.

Third, there is a whole world of opportunity for human errors in the process. If you are computer savy (not just point and click, but understand what the commands ls, cd, su, awk, sed, and vi do) and can handle the stress, rsync may work for you. But there will be this one time after 16 hours on set you accidentally copy to the wrong area, potentially overwriting data.

I use rsync on a daily basis for backup of the 3 dozen or so servers I maintain (+/- 4.6Tb daily, non-red related). It works well. But for on-set data management, Id prefer a one-click to take care of it all, so I can return to set duties. Choose what works best for you.

[Mark L. Pederson]

Watch for the upcoming 5.0 release. BTW, it will only work in Mac 10.5
Please make sure I have the option to COPY the media BEFORE performing any checksum operations. Get the copy done stat. Then check the data. This is VERY, VERY important. Especially when folks are using DRIVES.

[Conrad Hunziker]

Please make sure I have the option to COPY the media BEFORE performing any checksum operations. Get the copy done stat. Then check the data. This is VERY, VERY important. Especially when folks are using DRIVES.

Mark, perhaps you didnt read my diatribe above. Creating the initial checksum will always be first, not copying. You can, right now with the current software release, remove the Red Media while the software calculates the destination checksums. There is no issue doing that. However, you must create the initial checksums from the source - that is not ever going to change, for the reasons I outlined above.

You can use R3D Data Manager to just copy, and get the benefits over finder of ensuring that every file is copied, buy disabling the checksums altogether.

And one more point - if you are overwriting (reformatting or recording) the original Red Media before you do all 3 stages of verification (verify files, verify contents, verify valid data), then you are taking a risk. As you pointed out before data management is really risk mitigation. So if you are not verifying that all files copied over AND verifying the data in the files is correctly copied AND verified the data is valid image data, then you are taking a risk. R3D Data Manager - and really, any tool out there - can only do 2 of those. We are working with Red to provide a way to do #3 autonomously.

[Nick Shaw]

Please make sure I have the option to COPY the media BEFORE performing any checksum operations. Get the copy done stat. Then check the data. This is VERY, VERY important. Especially when folks are using DRIVES.

I'll second that request. It is the reason I do not currently use R3D data manager. I understand the importance of check-sums, but agree with Mark that the number one priority should be making a copy of a mag as soon as it comes of the camera.

EDIT (after reading Conrad's reply above):

Is it not possible to prioritise copying media, THEN make checksums FROM THE ORIGINAL, then copy those to the destination, then compare checksums. Surely copying first does not remove the option to make checksums from the original, it just delays it.

[Conrad Hunziker]

Yes - this CAN happen - EXTREMELY RARE - and if your tech does not have different camera ID letters on each camera - he/she deserves it. I DO NOT buy this as a good arguement for writing checksums to my digital negative.

One small point with this statement. Yes, the tech does deserve what they get for not having different camera letters and the cameras generate random letters that happen to be exactly the same for the RDC folder name. But as the author for software that has transferred literally pentabytes of data to date, I cannot, honestly or morally, release software that I know does not account for every issue that I know of. Since I have seen this occur to me personally on sets twice, I know it happens on other sets in a predictable fashion. Therefore software used day in/day out must account for it.

We have a solution that will be the best of both worlds, I think.

[Conrad Hunziker]

I'll second that request. It is the reason I do not currently use R3D data manager. I understand the importance of check-sums, but agree with Mark that the number one priority should be making a copy of a mag as soon as it comes of the camera.

Guys- Ill beat against this stone for as long as it takes. You cannot EVER create the checksums from anything else but the source. It must only be done from the source. You cannot release a Red Media from the computer until after the checksums are made. There is no getting around that.

If your workflow requires that, then you seriously need to re-consider your workflow. I will not write software that is setup to fail. By releasing a Red Media before the checksum is made, that is failure - end of story.

[Conrad Hunziker]

EDIT (after reading Conrad's reply above):

Is it not possible to prioritise copying media, THEN make checksums FROM THE ORIGINAL, then copy those to the destination, then compare checksums. Surely copying first does not remove the option to make checksums from the original, it just delays it.

Ill reply to this edit-

Thats not a bad idea. I need to do some simulations first before I say yes or no to it. In the 3 minutes I thought about it, it could work, but let me think for a bit to figure out all of the ramifications. There may be a small issue with the reads, but let me think it through.....

EDIT: (though about it some more)
There will be a performance hit. Right now R3D Data Manager creates the checksums while you enter data or add more sources. For example, while you enter data about the footage (script scenes) or setup destinations or add another source, it is already calculating the initial checksums in the background. So it would be waiting for you to start the copy (after doing all the above) before it actually calculates any checksums. The difference can be 5 to 10 minutes, depending on how fast you are.

[Nick Shaw]

Following from my comments above, while I agree that in an ideal world this should never happen because you have enough Red Drives on a shoot, I envisage a situation when a drive may need to be returned to the camera before the full download and compare checksums is complete. If the download is done first, then you may only be aborting the checksum operation, which while far from ideal does mean a mag can be freed up to return to the camera.

One example where this sort of thing might happen is a shoot I worked on recently where we had two Red Drives with the camera and a backup body which also (by chance) had one Red Drive with it. It was then decided (as often happens) that the backup camera should be used as a B camera, so we found ourselves cycling three Red Drives between two cameras. We wanted to swap mags at points convenient to the shooting, not dictated by the download process, so if I had used R3D data manager I would have worried that when I got a Red Drive with over 50GB on it, it might have been needed back at the camera, and the checksum would be complete but the copy would not. Making a checksum of 50GB of data over FW800 takes a long time!

[Mark L. Pederson]

THEN make checksums FROM THE ORIGINAL, then copy those to the destination, then compare checksums. Surely copying first does not remove the option to make checksums from the original, it just delays it.

This is how it should be done.

[Conrad Hunziker]

One example where this sort of thing might happen is a shoot I worked on recently where we had two Red Drives with the camera and a backup body which also (by chance) had one Red Drive with it. It was then decided (as often happens) that the backup camera should be used as a B camera, so we found ourselves cycling three Red Drives between two cameras. We wanted to swap mags at points convenient to the shooting, not dictated by the download process, so if I had used R3D data manager I would have worried that when I got a Red Drive with over 50GB on it, it might have been needed back at the camera, and the checksum would be complete but the copy would not. Making a checksum of 50GB of data over FW800 takes a long time!

Well, I respect your situation. That particular situation and other similar ones have happened on sets Ive worked. But I respectfully submit that it is the workflow, not the tool, that is the issue here.

Regardless of if the copying is done first or second during download, returning the drive back to the camera with footage on it that has not been 3-way verified is just asking for trouble. On one shoot I was on in particular (for clarification, I was the DP, not data manager) that exact situation happened where the data manager decided to return the drive to the camera before the data had been verified. And the AC promptly formatted it - and we lost the footage. The copied data had not yet been 3-way verified yet, but the data manager thought it would be fine. Turns out he was wrong, the the footage was never recovered, and we re-shot several days later.

So, in your situation above, I agree that the best time to reload is always when there is a convenient time to reload. But that carries the caveat that there is the ABILITY to reload. In your situation above, I would not have reloaded until the data was 3-way verified. Doing so increases the risk, and data management is really risk mitigation.

Im still thinking about the technical issues with doing the copy first, but I submit that has nothing to do with the issues above.

The next version of R3D Data Manager will come with a ETA, so you can better guess when the download will be done. In addition, if your download station has internet access and you have an email account setup in Mail, it will send a SMS text message or email to you to let you know its finished and the finished state (good/warn/bad).

By the way, I still hire that data manager, because he will never make that mistake again.

As an aside, the amount of time it takes to create a checksum on modern hardware should be the exact same amount of time it takes to copy the data via the same medium. The processor is usually waiting on data to come in, not the other way around.

[Nick Shaw]

Well, I respect your situation. That particular situation and other similar ones have happened on sets Ive worked. But I respectfully submit that it is the workflow, not the tool, that is the issue here.

I agree that it's the workflow not the tool which is "wrong" in the situation I describe. However I would like the software I use to give me the option to take a risk if the requirements of the shoot dictate, and all concerned understand the nature of the risk they are taking.

As an analogy, I prefer traction control on a car to have a switch to turn it off. That way I can choose to take a risk in return for a performance benefit, rather than have a computer tell me "You can't do that. It's not the approved approach"!

[Conrad Hunziker]

Let me think about it more and do some more simulations. Beyond the first performance decrease I mentioned above, I havent thought of another issue (but it is bedtime so Im not preforming well ATM), but let me run some test first to make sure.

Edit: spoke too soon. It will need to recalculate the checksums of footage retained on the drive, regardless of if it has been copied already. So if in the middle of creating checksums you decide to pull it, it will need to start that process over. This is because the checksums for the data can no longer be trusted to be accurate after they have (perhaps) been modified by another file system. So in reality, in the situation where you decide to pull it before the initial checksum completes, then it should really re-copy the entire drive in addition to re-checksum the drive, as perhaps the footage was screwed up by the other file system.

So you'll need to have 2 copies of the footage and 3-way verify the last version before you manually(!) delete the first copy. I dont like it, but thats what Im thinking would be necessary to completely mitigate all the risk when pulling the drive before the initial checksums are complete.

[Nick Shaw]

So you'll need to have 2 copies of the footage and 3-way verify the last version before you manually(!) delete the first copy. I dont like it, but thats what Im thinking would be necessary to completely mitigate all the risk when pulling the drive before the initial checksums are complete.

I understand what you're saying about "mitigating all the risk". However I am talking about having the option to take some unmitigated risk if necessary. It would not be the norm, but I want the option to return the drive to the camera for formatting, knowing that at least one copy has been made (but not verified) rather than hold up shooting.

When working on a bonded picture, I know there may be conditions imposed, and processes that HAVE to be followed. But on other kinds of shoot, there are always compromises to be made, and associated risks to be taken. I just want options.

My ideal would be for the software to copy everything from the Red media first, then checksum and compare one .RDC folder at a time so that in an emergency abort situation you knew which clips were verified. Ideally the ability to flag circle takes to be verified (and maybe even copied) first would be useful.

[Jonas Rejman]

Photocon, you told me to use other hardware. I can't really, because RED is build that way. I have to use firewire cables, I have to use CF readers, I have to trust the firewire ports of the readers and those of the notebooks.

I may be totally wrong here, but I think, that the biggest problem on the data's path to the post's RAID's is between the data written onto the CF inside the camera, and till the point, where it is first copied on the first offload drive.

This is the most dangerous part, this is where most of the mistakes and bitfilps might happen. The set conditions are rough: cold, weather, dust, people are tired, etc. We bought R3D data manager and while studying how it works, we found that if something is supposed to go wrong, it might just during the process when the data-manager is reading the data for the first time, creating the checksums.
But it might have copied the data by that time already. Because it just takes too long to deal with the data, before it actually makes the first copy.

That means, that for example, when the data-manager reads its first checksums from the media, and at this very moment, the CF card blows, or the CF readers decides to flip the data on the card, without the data-manager I would have copied the data by then, and have at least some parts of the data on another drive.

I am not saying my approach is as pragmatic and save as yours, I am saying, that practically, it might be faster, to make the first copy without any checks. VISUALLY check the data on the copy. Make that first copy as fast as possible, not reading the original media three times, THEN do the copy.

IF the data is ok, then make checksums, verify the checksums, and go for the next copies. If the data is NOT OK, if an error occurred during this first, most dangerous steps, then you have to make a thorough investigation anyway, to find out what went wrong.

There were users, turning off the checksum function of the R3D data manager because your approach just takes way too long. while being pragmatic, it just does not seem to me practical on the set. With the fastest CF reader at 50MB/s it still takes ages to read 16BG three times.

With my approach I can read from SSD's to create the checksums AND I have visually checked the data, while the data has been copied as fast and as soon as possible. Of course, I have the responsibility stating, that the first copy is a "new" MASTER, but I tend to think, that that forces the DIT to actually check what is inside the takes, rather to rely on a copy procedure.

Say the camera CF writer is bad and it writes something bad onto the CF-card in the camera. Say, there is one quarter of the picture in a moiree pattern, images we have sometimes seen. With the R3D data manager, the DIT might be in false security, that the image is ok ALWAYS, because you do the thorough check-summing.

With my approach, he HAS TO check the data on the first copied drive visually. Open them, check them, state that they are ok. There is no way for him to get out if this responsibility. In my eyes, this is where the money lies on the DIT.


We had a lot of clients using the R3D data manager while shooting on reddrives. They tend to shoot half a day, and then offload, trusting blindly a harddrive in raid 0. It took then about 5 hours to make a checksum checked copy of the data on another drive with R3D data manager. But they have never checked the data, because they were busy with the copies. When I did check the data and found, that one take was corrupted, because of a dying reddrive, they could not believe how this might have happened - they used the r3d data manager after all.
Of course they used older firewire HW, of course they should have offloaded more often. But unless they loose data, they won't do that. And if they do, they will blame the camera and/or the R3D data manager for that, who else?

One of the reasons I love the 16GB cards, is that they FORCE you to offload, but you still have almost as much material as on a 600m film roll.

Maybe some sort of a protocol should be established, and I wonder if this might be a compromise of speed and security:

0. OFFLOAD as often as possible
1. Make first copy of media to master storage as fast as possible, without checksums
2. FORCE DIT to visually check all data, while
3.1 Read media again, create checksum, store off original media
3.2 Simultaneously read master storage again, create checksum
4.1 Read media again, verify checksum
4.2 Simultaneously read master storage again, create checksum
5. IF all checksum match AND IF data visually ok, original media can be released


Does that make sense, and is practical?

Would it suffice, to NOT create checksums from the originals, but visually check first copy and create and verify checksum, to speed up the process?

Phocon, what is your take on bitfilps of SSD's, worse or better than HDDs?

How many shooters are really making thorough check-summing, and do not just rely on finder (that should have some sort of checks as well, no?), to speed things up?

[Ted Chu]

Can't agree more, the DIT needs to do a backup and check the footage quickly while you still have the ability to reshoot the scene if the file is corrupted. Then you can do your checksum backups. I have also been checking the "good" takes on the camera playback. In my experience, it seems like it won't playback if the file is corrupted.

[Raffi Kryszek]

I know the ability to offload footage over eSATA has been on the wish list of RED users since the camera’s inception. The R2E LEMO to eSATA cable lifts the burden off of a computer’s FireWire 800, FireWire 400 or USB buss. By simply powering the RED-Drive or RED-RAM using the original power adapter, plugging the LEMO end of the R2E cable into the drive and the other end into a standard eSATA port, the user gains instant access to RED footage with eSATA performance.

www.CoolCameraGear.com

http://twitter.com/coolcameragear

[gbalaji]

I've couple of things about R3D Data Manager. I use for for my 3 feature films and couple of ADs and it works great.

What needs to be done when the results shows errors, Do I need to start copy again. I don't find any answer for this?

When copying from Red Media, erase, leave option needs to be removed since it may cause panic for couple of users when the copy is done and they have not selected leave option the media will be deleted which no one wants at the set.

I'm using RAID-5 for couple of users and when I start copy to RAID from my source disk some days transferring takes lot of time in comparison with External Harddrives used as another backup. Any reason for this?

The above issues are pointed out to have great product and I believe the product is evolving over a period of time.