Copy & md5 Shell Script

[Tim Sutherland]

Wanted to make this look nice in my doc, where I have it for launching, so I made a quick icon for it. This plus telling osx to always open with terminal makes it nice and clean.

Tim

[jamie parry]

That way find ignores anything that is not a file (-type f). the only problem with that is I have to find all files, du to get their size, then add all the sizes to get the total. It was so much easier to change 99% to 90% and be done.

find path/to/your/folder -iname \* -exec ls -l {} \;|awk '{print $5}'| awk '{ sum+=$1} END {print sum}'

will do all your quoted needs in one line of bash code if you want it!

and i've just tested a folder size across HFS+ and FAT32 and it comes back exactly the same.

HFS+ volume (folder on desktop of mac)
find /Users/jamie/Desktop/The_Best_Film_Ever/A003_R1P7_AMA -iname \*.mov -exec ls -l {} \;|awk '{print $5}'| awk '{ sum+=$1} END {print sum}'
74252205

fat 32 volume
find /Volumes/FATTY32/A003_R1P7_AMA -iname \*.mov -exec ls -l {} \;|awk '{print $5}'| awk '{ sum+=$1} END {print sum}'
74252205


Haven't tested UDF though. Would be good if someone here could do that...then i'd know as well!
cheers
j

[Dustin Cross]

Jamie,

Thanks. I was joking and being lazy. I know its pretty easy to do.

One thing I haven't found a good solution for is spaces in filenames or path. Being an old school unix guy, I never use spaces so I never even think about that, but they are common enough these days to be annoying. I have found a couple really ugly solutions for dealing with spaces. Do you have anything nice and neat for that problem or do you just ignore it, like I do?


Dusty

[Tim Sutherland]

Actually, after playing with the script a bit, it would be nice to have it notify you when the checksum of the source is complete. I don't think I'd want it to auto eject, though.

The Log looks great, can you put a line in at the end stating wether or not all checksums matched? It's possible that it does report bad checksums but I haven't had any in my testing yet to check. I'll force a bad checksum and see what happens. And it would be great if the log data was added above previous data, so that you don't have to scroll down forever to see current info after transferring hundreds of mags.

Also, it looks like the LOG file doesn't update until the operation is complete. If the computer crashes during a copy operation, no information is stored in the LOG. Is there a way to have it update the log file at the completion of each copy step? That way after a crash and reboot, you can quickly see what operations finished and what operations didn't. I know you can manually check the md5 values, but it would be handy to have in the LOG as well.

I'm also wondering if the script could check for already generated md5 value txt files, to save time on recalculating md5s in event of a crash or additional copy after the original copy operation.

And it would be cool if it were stored in a location like /User/Library/Logs/CopyScript.log or similar so that those without hidden files turned on can easily locate the log. Same for the config file.

Again, great work, this thing is getting better and better.

Tim

[Dustin Cross]

Tim,

All of that is doable.

The Source checksum complete was there, but I think the second copy is overwriting it. I will make it create a new line so it is visible along with the second copy percent complete.

I can have a line that confirms the diff had no errors or tell what the errors are. I have been meaning to add this, but I always manually verify the checksums, so it hasn't been a priority.

Writing the log in reverse order. I will have to look into the best way of doing that. Shouldn't be a problem.

The log makes a temp file while it is working and at the end puts the temp file in the main log. This is because of multiple instances, so each mag is in its own section and not all mixed up. So if the copy fails in the middle, look at those temp files and you should find what was completed.

Checking for existing md5 files should be easy.

Changing the location on the log files and config files is easy. I will put everything in /User/Library/Logs/CopyScript/

Anything else you can think of to put in the log?


Dusty

[Tim Sutherland]

Maybe in the confirmation line, have a nice little summary, telling how many folders and files were copied and what the results were, something like.

(A001_0807RX) -> Copied (1) folder containing (12) files to (2) destinations on (Dest Drive 1) and (Dest Drive 2), with (no errors).

where each () contains a variable

That way it's something to quickly show as proof of a good copy if necessary, since when it hits the fan, the person managing the data is often the first to be blamed.

Also some option to re copy bad checksummed files without copying the whole thing again would be nice, but that might be more complicated.

Tim

[Dustin Cross]

I can add a nice summary at the end.

Do you like how it is keeping each mag together, or would you be fine with everything mixed together with just time stamps?

I think I will hold off on the recopy stuff. I want it to notify when there is a problem, but I think fixing the problem should require manual intervention. I will look into it though.


Dusty

[Tim Sutherland]

I can see that about the re copy. Maybe sometime you could do a separate script to recopy non matching checksums for when you're in a pinch.

The way mags are grouped is fine, although maybe some tabs to make it easier to read.

Tim

[Dustin Cross]

Tim,

When you say tabs, do you mean indent?


Dusty

[Tim Sutherland]

Yeah, sorry, just a way for it to line up neatly when looking at the different lines of results.