Thread: Copy & md5 Shell Script

Reply to Thread
Page 1 of 15 1234511 ... LastLast
Results 1 to 10 of 147
  1. #1 Copy & md5 Shell Script 
    Senior Member
    Join Date
    Mar 2007
    Location
    Austin, Texas
    Posts
    1,240
    I have been playing around with a little shell script to make copies of footage and do md5 checksums on them. Mainly because things like R3D Data Manager only copy Red media and things like Shotput Pro don't do md5 checksums.

    Basically the script copies any folder to one to four destinations. Then makes md5 checksum of the source and each destination and compares them.

    Run it from the command line and then simply drag and drop folders from Finder to tell the script where things are. Pretty simple.

    I give no warranty and I wouldn't use this on important footage unless you have played with it and know what it is doing.

    I wrote this on OSX so some commands it uses might not exist on Linux or other systems. So test.

    Just thought I would share and let people play.


    Dusty


    edit -- I made a minor change to the script. I was trying to speed it up and compare the checksums in parallel and that was too fast for small files.
    edit -- changed the script so md5 deals with sub directories correctly (02/02/11)
    edit -- changed script so errors print to the screen instead of to /dev/null (02/02/11)
    edit -- cleaned up the code a little (02/10/2011)
    edit -- fixed a problem where the copy didn't work on files with "." in the initial directory name (02/17/11)
    edit -- made the md5 part ignore folders so it doesn't give errors on folders. md5 now only works on files and not directories or any special files (02/17/11)
    edit -- the script has problems with doing md5 on things like OSX applications which are actually some kind of hidden folder. The copy works, the md5 has issues. (02/17/11)

    edit -- added a new version (CopyScript_percent.sh) of the script for testing that has a config file, tells percent complete, and keeps timestamp of the source files. (03/10/11)
    edit -- Cleaned up things a little. Improved performance a little by having source checksum happen during second copy. Also made errors write to a log file. Will add more for this log file in the future. (03/11/11)
    edit -- added more info to the log file. /var/tmp/copyscript.log (04/19/11)
    edit -- modified the script so you can run multiple copies at once without conflict. So you can copy multiple different drives at the same time (04/19/11)
    edit -- fixed the source checksum 100% complete message. (04/22/11)
    edit -- changed the order of the questions at the beginning so the source drive is last.

    edit -- added link to GUI front version of the shell script. (01/18/12)
    http://www.sandust.net/SandustTools/SandustTools.zip

    edit -- link to the new GUI version of the script Chris created. (01/27/12)
    The latest version can be downloaded here: https://bitbucket.org/chriswayg/copy_verified/downloads
    If you find bugs please report them in the bug tracker here: https://bitbucket.org/chriswayg/copy...ied/issues/new
    Attached Files
    Reply With Quote  
     

  2. #2  
    Senior Member Tim Sutherland's Avatar
    Join Date
    Apr 2008
    Location
    Peyton, Colorado
    Posts
    652
    Nice. Thanks for sharing, this will be nice for copying audio files, etc, especially since we will still have the hard copy of that.

    I might try to make a little automator app that runs this with a little interface. If I do I'll post here for the same reasons.

    Along the same lines, I made a little app that opens a new instance of R3D Data Manager so you can put it it the dock and click it to open each instance you want to use. I'll get it off my desktop and upload it if anyone is interested.

    Tim
    Reply With Quote  
     

  3. #3  
    Senior Member
    Join Date
    Mar 2007
    Location
    Austin, Texas
    Posts
    1,240
    I was thinking about writing a Applescript GUI frontend for the script. I didn't think about trying something with automator.

    Right now I have the script set to copy to the first location and then copy from the first location to the second through fourth locations in parallel. This is because I always copy to my fast internal raid first and from the fast raid to all the shuttle drives and such.


    Dusty
    Reply With Quote  
     

  4. #4  
    Senior Member
    Join Date
    Mar 2007
    Location
    Austin, Texas
    Posts
    1,240
    Had to copy a bunch of folders with sub directories today and realized my md5 checks couldn't handle sub directories, so I fixed that.


    Dusty
    Reply With Quote  
     

  5. #5  
    REDuser Sponsor Gunleik Groven's Avatar
    Join Date
    Dec 2006
    Location
    Norway
    Posts
    13,406
    This is very nice. Thanks!

    G
    Reply With Quote  
     

  6. #6  
    Senior Member Keith_D's Avatar
    Join Date
    Nov 2008
    Location
    Los Angeles, New York
    Posts
    239
    Thanks!
    Reply With Quote  
     

  7. #7  
    You should be a little careful about redirecting standard errors to /dev/null; I ran this script on Linux (Ubuntu) out of curiosity, and it ran without errors - even if md5 does not exist on Linux (md5sum on the other hand). It copied the folders OK - but the checksum files were empty - false security.

    And a footnote on your development style - there is a thing called "for loop" that can get rid of lot of the copy-and-paste code.
    Reply With Quote  
     

  8. #8  
    Senior Member Tim Sutherland's Avatar
    Join Date
    Apr 2008
    Location
    Peyton, Colorado
    Posts
    652
    The last time I ran a script through automator, it worked a lot better than this time, so I guess I'll stick with terminal, unless you can get it to work better with a GUI.

    Thanks for sharing this.

    Tim
    Reply With Quote  
     

  9. #9  
    Senior Member
    Join Date
    Mar 2007
    Location
    Austin, Texas
    Posts
    1,240
    Quote Originally Posted by Erik Larsson View Post
    You should be a little careful about redirecting standard errors to /dev/null
    Doh. I did that because md5 was giving me lots of errors today on the sub folders. I should change it back for sharing. But that's why I said play with it and know what it is doing before using it for anything important. I changed it to print errors to the screen and reloaded it above.

    You are right. I should clean up the code, but it was easy to copy and paste. I guess it chould also check for things like md5, but that comes standard on OSX.


    Dusty
    Reply With Quote  
     

  10. #10  
    Quote Originally Posted by Dustin Cross View Post
    Doh. I did that because md5 was giving me lots of errors today on the sub folders. I should change it back for sharing. But that's why I said play with it and know what it is doing before using it for anything important. I changed it to print errors to the screen and reloaded it above.

    You are right. I should clean up the code, but it was easy to copy and paste. I guess it chould also check for things like md5, but that comes standard on OSX.


    Dusty
    Well, you could redirect errors to /dev/null - but make sure you exit with an error when things go to wrong - like not having md5 installed.

    As it is now, everything does look to work correctly - and it is kind of hard to know that it does something wrong when it does not tell you. So - if I did not know better, I would test this: No errors, copy OK, but the MD5 validation step failed without telling me ... Potential "deep shit situation".
    Reply With Quote  
     

Posting Permissions
  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts