Copy & md5 Shell Script

[Dustin Cross]

Anyone who tries out the new percent script, please let me know how it works for you. It seems to be working fine for me.


Dusty

[Dustin Cross]

I did some testing today on the new version that tells you percent complete. Everything seems to be working fine. I uploaded a new version with a couple speed improvements.

I did some speed testing today comparing copy an 8GB Red CF card. Here are my results.

3:00 min - Finder 1 copy
11:30 min - Finder 4 copies
7:00 min - Finder 1 copy, then 3 copies from first

5:20 min - ShotPutPro 1 copy (byte checks)
37:00 min - ShotPutPro 4 copies (byte checks)

3:45 min - R3D Data Manager 1 copy (md5 checksums)
7:45 min - R3D Data Manager 4 copies (md5 checksums)

3:45 min - CopyScript 1 copy (md5 checksums)
8:00 min - CopyScript 4 copies (md5 checksums)

All copies were from the same CF card to the same destinations. Computer was shutdown between each copy. This wasn't too scientific, just a quick test to give me an idea how this script compares.

I was pretty happy with these results for my little CopyScript. Not much slower than a straight finder copy with no checksums.

For R3Ds RDM is still the best option, but my script works on almost any footage.

ShotPutPro surprised me. Having never compared it to something else I thought it would be much faster. When making four copies, I thought it was gonna kill the CF card and reader. I need to run that test again to make sure something odd didn't happen, but I didn't want to wait another 40 minutes.


Dusty

[Jonathan Carbonaro]

Dusty,

Thanks for a good script! This is great because you're not limited to R3D's!
I used it to backup Audio as well, and it ran great.
The only thing I noticed was that when it wrote the Checksum, the percent complete stayed at 0% until it got to 99% and then it updated and finished.
Other than that, it worked perfectly!
I tested on my 15" macbookPro 2.53 dual core using OSX 10.5.8
Going to test on the Tower tomorrow..

Jonathan

[Dustin Cross]

Jonathan,

Were you copying a single file? That would give you 0% and then 100% with the way I have it set up.

The way I am calculating percent complete is to add the size of each file completed and compare that to the size of all files being copied. So if there was only one file, it would read 0% until it finished that file and then it would be 100%.

Or if your files were so small that the checksum happened faster than the checksum caught up. I have a slight delay built in so it doesn't error out because the previous step hasn't finished.

If that is not your situation, I will have to figure out why.


Dusty

[Jonathan Carbonaro]

Nope, that was it!
The file was a 5gig file, but yes, it was only one. I had actually just realized that as I did further testing with multiple files within a folder.
Thanks, that makes sense

Jonathan

[Richard Goodwin]

You can do a binary compare of directories with the diff command (as a secondary check to doing md5 checksum)

There is also ditto, which is OSX specific but preserves metadata and whatnot.

[Dustin Cross]

Richard,

Never heard of ditto. I'll have to look into it.

Not sure what binary compare of directories would give over md5 of each file. How are you thinking binary compare would improve? I could do a diff -r and compare each destination directory to the source directory, excluding the md5 files the script creates.

Would this add anything that makes it worth the extra time?

I just did a quick test and this was very fast. Much faster than a full read or making md5s, so that makes me think it is not comparing files as completely as md5. I'll do more research into what diff is actually doing on directories.


Thanks,
Dusty

[Dustin Cross]

Has anyone tried the CopyScript_percent script? What did you think? Anything you would like to see changed?

I am thinking about adding stuff in the log file about what where files were copied from and all destinations, what time each step starts and finish, and a name for the person doing the work. Basically a bunch of stuff that can be used as a paper trail. What do you guys think? Anything you would like to see in there? Or do you think you wouldn't use something like that?



Dusty

[MasonJames]

+1 for the log file creation. Haven't been able to test this script just yet, but looks very exciting. Thanks for sharing!

[Dustin Cross]

Mason,

Thanks for the feedback. What would you like to see in the log file?

Let me know how things go when you test.



Dusty