Copy & md5 Shell Script

[Dustin Cross]

---------------------------------------------
Hi Dustin,

I hope you don't mind me contacting you privately but I really like your shell and wouldn't want to cast a stain on its reputation by publicly reporting a problem which is almost certainly related to deficiencies in my IT skills.

I was using it on an Alexa job a few days ago (after testing it and coming to the conclusion that it is fast and reliable as you said - at least in relation to the not fast and definitely not reliable shot put pro).

I tried to copy an Alexa folder to two destinations and it reached 96% then froze for several minutes even though the data was all copied. I checked the bits and they matched up and played through a couple of clips and everything seemed fine. I would have liked to transcode and ultimately retry your shell but it was in the heat of battle (I was data wrangling and ACing) so I just had to revert to the hated shot put pro and press on.

I'd love to have another crack of the whip with your shell but would feel more confident about using it if you had any idea why it had a little freak-out the other day. I'm quite busy and the lions share of jobs these days are on the Alexa so I will have the opportunity pretty soon.

On a related note - if you were thinking of developing it further, one thing I would dearly love to use is something that lets me copy within finder (drag and drop) then run a checksum retrospectively

i.e. First folder (Drag 1st folder here) 2nd folder (drag folder here) original folder (drag folder here) - perform checksum.

In my wildest dreams I'd do it myself but, as you have probably worked out from this message, I'm far too thick.

All the best from England

Have a good weekend

Alex Parkinson
---------------------------------------------

Alex,

I know exactly what happened. It is the problem Jamie and I were discussing on earlier in this thread about different file systems measuring directors differently. In the UDF file system used for Alexa SxS cards, directories actually have a different file size than in HFS+ used by OSX. So you run into a problem where the destination never gets large enough to end the loop I use to measure percent complete. The latest version of the script solves that problem is a really dumb way. I have the measure end at 90% instead of 98%. I really need to change the script so it only measures files and ignores directories.

Try the latest version of the script on some test footage and see how it works for you. It stops the loop at 90% now to prevent that problem. It could still happen if the source and destination are more than 10% different in size, so I really need to rewrite that whole section of the script.

A script that just does MD5 checksums would be pretty easy. I am about to start a new feature, so it might be after that before I could get to it. I need to make a few things about this script better first.



Dusty

[Dustin Cross]

Just added a new version with updated order of questions at beginning so the source drive is the last question asked.


Dusty

[ayarbro]

I applaud you Dustin. Just found this thread, cause I haven't had enough time to search for this until now. Something I've been wanting to do for sometime, but haven't had the time for. Way to go!

Hope you don't mind if I make some edits! I'll still keep your name in the comments of the script though.

[Chris Wayg]

Thanks Dustin for this very useful script! At first I considered buying Shotput Pro, but when I tested your script I found out it meets my needs fully and I can adapt it easily to various use cases. Hopefully someone here could help me with an issue I ran into:

Thankfully the script unsurfaced a problem with corrupted video files from my Canon EOS 7D when copied to my desktop Mac. (Strangely the Shotput Pro demo did not catch the corruption.) I copy from SanDisk Ultra 16 GB CF cards using a USB card reader to two internal hard drives. Every time about a dozen video files out of 50 to 100 clips are corrupted, meaning the MD5 do not match even though the files usually play fine in Quicktime. I suspect some crashes in FCP-X while editing may be caused by these corrupted files. I double checked with other MD5 software like checkSum+ and they confirm the corruption. Usually the files copy successfully after a second or third try of re-copying just the damaged ones.

I've checked this with three different card readers, various CF cards and cables and different hard drive targets. The problem only shows up on the desktop Mac and not on my 2010 MacBook Pro using the same card readers (- everything copies perfectly onto the MacBook). I've run Prime95 on the desktop, stress testing the CPU and memory for many hours and it did not show any errors. This leads me to believe, that probably memory or CPU errors are not the cause of the corruption.

What else could it be? I'm really out of ideas on what else I could check

Chris

[Dustin Cross]

Not sure why I have not posted this yet, but I added a little GUI front to make the script easier to store in your Apps folder and launch. It is still a command line script. The GUI just launches terminal and runs the script. The GUI also has a couple other scripts I have written.
http://www.sandust.net/SandustTools/SandustTools.zip

The first thing it does when you run the app is check the website to see if there is a newer version. If there is, it will ask if you want to download and install the new version. You can test this by downloading the old version and running it. The script is written to be install in Application folder so that is where it will install the new version.
http://www.sandust.net/SandustTools/...ls20110726.zip

right now it looks like nothing is happening when it is downloading the new version, but the new version will pop up once the process is complete. I should add some kind of counter to let you know it is doing something.

If you look inside the application package you can see all the scripts and tools that are in there, so you can still modify them to your needs.

Hope you find this useful.

PLEASE TEST and let me know if you find any problems.


Dusty

[Dustin Cross]

Chris,

That sounds really odd. Not sure what is causing that.


Dusty

[Nick Pasquariello]

The only advice I could offer, Chris, is that if the issue is only located on one computer, then you know it is either hardware or software on that computer that is causing it.

Maybe that sounds pretty obvious. But if you were to re-install the operating system and still get the issue, then you know it's entirely hardware related. Of course, since it's a somewhat inconsistent issue, it's kind of tricky to diagnose from there.

[Chris Wayg]

The only advice I could offer, Chris, is that if the issue is only located on one computer, then you know it is either hardware or software on that computer that is causing it.

Thanks for your replies and suggestions. Its hard to pin down the exact cause, as the issue cannot be reproduced consistently. I removed some RAM which I had added recently and I tried different USB ports. Now the issue seems to have been reduced, having copied two cards in a row without issues. I suspect that it is either the RAM or a faulty USB port.

It goes to show that we cannot rely on the operating system to protect us from data corruption during a copy operation, and that verification of checksums is essential to alert us to problems, before they become incorrigible.

[Chris Wayg]

I added a little GUI front to make the script easier to store in your Apps folder and launch. It is still a command line script. The GUI just launches terminal and runs the script.

Dusty, thanks for your updated script. I wouldn't have noticed the corruption of my video files in a timely manner without it.

I really like your idea of integrating the CopyScript with Pashua as a GUI frontend. I modified your script to use Pashua for all data input and added a Platypus wrapper for making it an app and for terminal-like output, thus eliminating the need to start a separate terminal session. I added more robust user error checking and modified it throughout to allow file paths with spaces. (I did not integrate the other scripts or your auto-update mechanism, as I'm mainly interested in the CopyScript.)

I wanted to ask under what license you consider your code? If its not public domain I would suggest an appropriate open source license such as the GPL or MIT. I uploaded my changes to a git repository at bitbucket. Please give it a try and tell me what you think. It hasn't been tested yet extensively, thus it may still contain bugs.

https://bitbucket.org/chriswayg/copy_verified
Here is my changelog:
# - added a Pashua GUI frontend for inputting the file paths and data
# - added Platypus wrapper for making it an app and for terminal -like output
# - added support for spaces in path and file names
# - added the -n option to cp to prevent from accidentally overwriting files
# - added sanity check for validity of paths
# - chanage MD5 files from .txt to .md5 to facilitate verfication with other tools
# - renamed to 'CopyNverify' to have a name that is less generic

[Dustin Cross]

Chris,

Nice.

I made a more complicated GUI version of the script using Pashua, but I got too complicated and it really slowed things down, so I scrapped it.

I am fine with GPL.

I made the MD5 files .txt so it was easy to review them. Simply select them and hit the space bar. Originally they were .md5, but .txt made life so much easier and I can easily change them to .md5 if I ever needed to.

In the GUI version I made, at the completion of the copy it went back to the main screen. That made it easy to start the next copy. And from there it is easy to quit the app if I don't want to make more copies.

I would like to see the output window not open until you start a copy and close once you acknowledge the completion.

Hopefully my lazy code didn't make it too annoying.



Dusty