Copy Entire Contents Of Mag To Hd

[dino g]

News Flash:

I know many of you just copy the folders containing the r3d files + the proxies which the camera makes onto a hard drive to extract/ingest your/clients footage.

DO NOT CONTINUE TO DO THIS

new procedure, and you will thank me for this later...

create a folder on your hard drive called MAG<#>, i.e. MAG1, MAG2, MAG3....or what ever you want to call it to designate a specific CF Card.

put the entire contents of each mag <CF CARD> into that folder as the copy of the footage.

those 3 other files the camera makes will come in handy one day real soon.

thank you for reading this.

[Mike Prevette]

I have the "option" key on my camera laptop taped green, and big letters P-touched onto the laptop that explain how to copy entire volumes. This really should be the ONLY WAY of doing this. I can't tell you how many calls I get from people that fucked up a P2 volume by not copying the whole thing. I would assume one the red it would be even more important.

[Mark L. Pederson]

News Flash:

I know many of you just copy the folders containing the r3d files + the proxies which the camera makes onto a hard drive to extract/ingest your/clients footage.

DO NOT CONTINUE TO DO THIS

new procedure, and you will thank me for this later...

create a folder on your hard drive called MAG<#>, i.e. MAG1, MAG2, MAG3....or what ever you want to call it to designate a specific CF Card.

put the entire contents of each mag <CF CARD> into that folder as the copy of the footage.

those 3 other files the camera makes will come in handy one day real soon.

thank you for reading this.

I (and others) have requested that the camera make a root folder on the mag at format (REEL_A001 or MAG_A001, etc.) for this, and a few other reasons. And PLEASE don't use finder to copy - use a sync app with a checksum verification.

[I Bloom]

I (and others) have requested that the camera make a root folder on the mag at format (REEL_A001 or MAG_A001, etc.) for this, and a few other reasons. And PLEASE don't use finder to copy - use a sync app with a checksum verification.

Which sync app are you using?

[donatello b]

i've been testing a Nexto & Hyperdrive (space)units with 160gig 5400 rpm laptop drives in each ... with both units powered by internal batt & using a extreme III 2 gig card .. 1 gig of data ( 252 files) .. the nexto transfers in 1:23 ... the hyperdrive in 2:02 ....
nexto takes 1:19 to full verify for a total time of 3:42 to copy & verify... hyperdrive 2:34 for a total time of 4:56 to copy & full verify ....
i have used them with 8gig RED cards but didn't time the transfers - the larger the files the faster the transfer ( so two 1 gig files will transfer a little faster then 70 files ( that equal 2 gig) ...

basically the difference :
Nexto is a 2 step process to copy & verify .. you 1st copy the card - then pull out the CF card .. you then put it back in and it will verify ....
the hyperdrive is 1 step - you set it up before hand to copy & verify .. you put the card in and it will copy a file & then verify it - copy a file -verify etc ..
on both units you have the choice to either full verify or only verify the 1st 128k of each each file ( the hyper also gives choice to random verfiy ) ...
the nexto only accepts CF cards .. hyperspace takes many different memory cards

both units had to be returned when i received them .. nexto because it wouldn't charge the internal batt .. hyper because the verify button didn't work .. no problems returning the units .. both replacements work ...

http://www.nextodi.co.uk/prod_nd2525.html
http://www.hypershop.com/shop/index.php

[Mark L. Pederson]

Which sync app are you using?

trade secret for the moment. what do you like? DOUBLE checksum pass is the move -
We just tracked an issue to a RAID 1 enclosure - we are abandoning that method for two individual copies with checksum to separate drives.

[I Bloom]

trade secret for the moment. what do you like? DOUBLE checksum pass is the move -
We just tracked an issue to a RAID 1 enclosure - we are abandoning that method for two individual copies with checksum to separate drives.

Interesting. Think its the specific enclosure or a general issue.

Meanwhile here's a little checksum wrapper, you might use for direct comparison of R3D files on a Mac.
http://www.macupdate.com/info.php/id/23168

If you duplicate this program in your applications folder you can run two instances at the same time and thus compare a CF card to its copy.

Also most of what you need is already built into Unix and part of OSX. So if you gain some familiarity with the Bourne Again Shell you can do transfers using the Terminal.
If you have that kind of mind I recommend doing this because it's an all keyboard task, and it lists files as they are being transferred. Dragging files from one finder window to another is subject to slippage.

Ian

[dino g]

trade secret?

come on, we are in this together, and you guys are 3000 miles away from us and we are sending you business, certainly this is not going to turn into an us and them marketplace right away.

we are all trying to push this forward and share info, i have been doing that for free and my payback is more understanding on the part of the client. and then a greater likelihood of getting the gig.

personally, i am loosing too many jobs to 35mm or the varicam because the executive producers catches wind that the director and dp want to use RED and the exec gets scared and pulls the plug...

so if you figured something out, please share; that is what this specific thread is about, i may know something that you may not or do not, but since i can not say anything specifically right now, i wanted to let you all know/ warn you/ that if you are not saving the digi mag files, you should start doing so.

[Mark L. Pederson]

trade secret?

come on, we are in this together, and you guys are 3000 miles away from us and we are sending you business, certainly this is not going to turn into an us and them marketplace right away.

we are all trying to push this forward and share info, i have been doing that for free and my payback is more understanding on the part of the client. and then a greater likelihood of getting the gig.

personally, i am loosing too many jobs to 35mm or the varicam because the executive producers catches wind that the director and dp want to use RED and the exec gets scared and pulls the plug...

so if you figured something out, please share; that is what this specific thread is about, i may know something that you may not or do not, but since i can not say anything specifically right now, i wanted to let you all know/ warn you/ that if you are not saving the digi mag files, you should start doing so.

LOL!!

I was breaking iBloom's chops - there are many copy utilities with checksums - many are free -

We are using AAsync right now on the feature but testing a few others at the shop -

[I Bloom]

trade secret?

come on, we are in this together, and you guys are 3000 miles away from us and we are sending you business, certainly this is not going to turn into an us and them marketplace right away.

we are all trying to push this forward and share info, i have been doing that for free and my payback is more understanding on the part of the client. and then a greater likelihood of getting the gig.

personally, i am loosing too many jobs to 35mm or the varicam because the executive producers catches wind that the director and dp want to use RED and the exec gets scared and pulls the plug...

so if you figured something out, please share; that is what this specific thread is about, i may know something that you may not or do not, but since i can not say anything specifically right now, i wanted to let you all know/ warn you/ that if you are not saving the digi mag files, you should start doing so.

That's how Mark answers questions when he thinks you might have a better answer than his. :shifty:

My current workflow is to use terminal to copy files between volumes with the cp command. Once the files have copied, I check them over manually, with two finder windows, keeping an eye on file sizes. I have two copies of SuperSFV in on my dock that I run as two different instances. If it all looks good I drag the contents of the CF card into one SuperSFV instance and the contents of my new folder into the other SuperSFV instance. I then visually compare checksums. This also works for P2, but it's obviously a little slow and meticulous.

Update: My currrent workflow is a program called AAsync that I discovered in the series of tubes. This new tool is really going to put me ahead of all my competition. On the west coast they haven't even heard of this shit.

[Mark L. Pederson]

Update: My currrent workflow is a program called AAsync that I discovered in the series of tubes. This new tool is really going to put me ahead of all my competition. On the west coast they haven't even heard of this shit.

terminal hackin' smart-ass!

[dino g]

wow, impressive, seems like i have got to put on my computer geek hat and try and figure out what they hell you just said.

thanks for that.

[I Bloom]

wow, impressive, seems like i have got to put on my computer geek hat and try and figure out what they hell you just said.

thanks for that.

AAsync sounds like a good solution, I had been looking into something like it because my method is difficult to train people on. But I will say that on the flip side anytime you are incorporating an automated tool you need a human sanity check as well.

One other tip is that Finder does a slightly different calculation of file sizes between FAT32 (your CF card or Red Drive) and HFS Plus (a Mac OSX formatted drive). If you do a spot check between two finder windows you will discover a difference in file size of about .1MB

I think I need to get one of these CF reader hard drive units for december though. Anyone know where you can pick one up in NYC? B&H?

IBloom

[Mark L. Pederson]

One other tip is that Finder does a slightly different calculation of file sizes between FAT32 (your CF card or Red Drive) and HFS Plus (a Mac OSX formatted drive). If you do a spot check between two finder windows you will discover a difference in file size of about .1MB


I don't trust finder for sensitive data movement - ESPECIALLY in Leopard at the moment - I could just be paranoid - but I have seen things that scare me in the world of OSX finder ....

[I Bloom]

I don't trust finder for sensitive data movement - ESPECIALLY in Leopard at the moment - I could just be paranoid - but I have seen things that scare me in the world of OSX finder ....

That's true. You might also try this move:

Open the terminal.
type: cd /Volumes
type: ls
there will be a list of mounted drives
type: cd RED*ONE
(use * for filenames with spaces)
type: ls -altR | less
("-altR is not alt+R its letters A L T R. That character in the middle is the one above the return key)

Open another terminal window next to that one:
type: cd /Volumes/My*Drive*Name/My*Copy*Folder*Name
(put in the names of your drive and the entire copied Red cf card folder)
type: ls -altR | less
(Pressing "q" gets you out of less.)

If you start seeing spooky sh*t on Finder, (which actually happened on a Red shoot) use the terminal instead. Thirty Five Years of Unix developement never lies.

How to copy in terminal is a trade secret though. I'm gonna be first to market with that.

IBloom

[Mike Prevette]

I think you forgot " rm *.* "



EDIT: THIS IS A JOKE DO NOT ATTEMPT

[I Bloom]

That's almost not funny Mike.

With -fR it's not.

[reality]

\rm -rf is cooler

[ChristopherKenworthy]

I don't trust finder for sensitive data movement - ESPECIALLY in Leopard at the moment - I could just be paranoid - but I have seen things that scare me in the world of OSX finder ....

Could you elaborate on the Finder issues? You've started to me me paranoid, even though I've used the Finder to copy countless TBs of data without a single problem. Is the Finder notoriously rubbish at this? I've tried AAsync, but found that it slows the workflow to an impossibly slow level (even with a top of the range Mac.)

If I offload the CF, and watch the rushes every night and they play back OK, do I really need to checksum? What's the worst that can happen if I don't?

[Mark L. Pederson]

Could you elaborate on the Finder issues? You've started to me me paranoid, even though I've used the Finder to copy countless TBs of data without a single problem. Is the Finder notoriously rubbish at this? I've tried AAsync, but found that it slows the workflow to an impossibly slow level (even with a top of the range Mac.)

If I offload the CF, and watch the rushes every night and they play back OK, do I really need to checksum? What's the worst that can happen if I don't?

I can't say for certain that mac osx finder will ever fail you - but I have personally seen this ... and it hurts -

http://tomkarpik.com/articles/massive-data-loss-bug-in-leopard/

Not sure why you say AAsync is "an impossibly slow level" - we are shooting a feature right now and making TWO copies of 130G per day off of 8G CF cards with AAsync - and it's going like butter -

Do you really need checksum?

Well ... do you REALLY need insurance?

Not unless you have problems. Then it's pretty handy.

[Mike Prevette]

That's almost not funny Mike.

With -fR it's not.


Ya I thought about that when I posted it. I've edited it now.

[MikeCurtis]

I was thinking about starting a thread entitled "Digital Magazine, Oh How I Hate Thee, Let Me Count The Ways" but I'll just put it in here.

Red has gone to the trouble of creating a file naming system that tries not to create duplicate file names. This is good.

BUT....you CAN do it (get repeat file names that match exactly, a venal sin in a file based workflow) by pre-formatting your cards, or by switching from 4K to 2K for high speed work (been bit by that TWICE in the heat of production, and Force Reel scares me since I don't necessarily know what exactly is going on or where we are, so I've been switching to B then C camera designation when switching formats mid-job, but that's another post/thread).

ANYWAY....they've attempted to make a system that won't create dupe file names and are human readable. GOOD. BUT....then they create a file that is Somehow Important (somebody PLEASE explain to me what's in those two digital magazine profile files, and WHY WE NEED THEM!)....and every F(*(*&^*&^ing last one of them has the EXACT same file name.

And it gives you no clue what files are associated with it. Therefore you have to keep'em in same folder with the .R3D folder stuff all through the process, and if you have an "orphan" you are fuxored, since there is no way to tell where it went.

If you go by numbered Mags in folders, makes it tougher to traverse, find, verify you have two copies, etc.

What I'd like to see:

those files should be renamed to have a unique name. If they are created when card is formatted, OK - give it a name based on camera, date/time, etc. that is UNIQUE. Digimag_071129_(serial#/PIN)_4K_2398 or somesuch.

Within the file, right at the top of the header, in plain Engrish, should be the shots that it is associated with.

I hate to sound so ignorant on here, but REALLY - I don't understand what these files are doing for me. Somebody clue me in, PLEASE, because they are making my on-set data management life significantly more difficult.

And kee-riste, if it is causing issues for propellerhead me, woe be unto any Joe who rents and "just wants to shoot, ya know?"

Rant off.

-mike

PS - EDIT - I really LIKE the idea of putting all shots in one Big Bit Bucket on a drive - I can see what shots I do and don't have, and when working as Red Tech and digital data backup on some jobs (anybody else get clients bitching about two people on jobs for this "low cost" camera?), the ability to queue up a double copy (from CF to two separate drives, no seeks to shred transfer performance) and walk away (insert new workflow with AAsync or metric equivalent) is NICE - so I'd like to maintain that if possible.

When there's 8 cards on set and some are hot 2K and hot 4K and has that one been backed up, wait help reconfigure this, did that copy finish, is it all there, etc., is made NOTABLY more difficult if we MUST do Mag#(x) setups - which mag# was that? Oh wait, let me copy/find/paste, etc....bag that - I should just be able to LOOK and see if A009_C012_071124 is there or not.

OK. NOW I'm done.

[ChristopherKenworthy]

Not sure why you say AAsync is "an impossibly slow level" - we are shooting a feature right now and making TWO copies of 130G per day off of 8G CF cards with AAsync - and it's going like butter -



Thanks for that. I may well be doing something wrong, which is making AAsync struggle. How long does it take you to transfer 130GB with AAsync?

[Mark L. Pederson]

Thanks for that. I may well be doing something wrong, which is making AAsync struggle. How long does it take you to transfer 130GB with AAsync?

5-6 minutes for 8G card.

[ChristopherKenworthy]

When there's 8 cards on set and some are hot 2K and hot 4K and has that one been backed up, wait help reconfigure this, did that copy finish, is it all there, etc., is made NOTABLY more difficult if we MUST do Mag#(x) setups - which mag# was that? Oh wait, let me copy/find/paste, etc....bag that - I should just be able to LOOK and see if A009_C012_071124 is there or not.
.

That is so funny. I laughed for far too long, because I wake at night thinking about this.

The best solution I've come up with for this is a physical one. Have way more CF cards than we need (about 30), and have a storage case for blank cards. We take cards from there, and only from there, to the camera. As soon as the card is full it's copied to the MacBook Pro, and then goes into the Halfway House case. At night, the Halfway House case is completely copied to a couple of RAID 1s. After all shots are checked, the CF cards are wiped and returned to the first case.

Not simple, nor foolproof, but if there's no wiping on any given day, and we just shoot-copy-store one card at a time, there's hope.

Better ideas are always welcome.

[ChristopherKenworthy]

5-6 minutes for 8G card.

Thanks. Great news. I'll keep testing and get it to work.

[Mike Prevette]

For me as an "on set" guy, I like the idea of things separated into "mags". Maybe it's having stuff beaten into me when I was a film loader, but my mind just works in the world of mags. And when I get into the edit, I have my head organized in the chronologies of Mags.

[MikeCurtis]

Christopher - that was originally my plan, when the client said we'd probably shoot maybe 20 minutes of footage. I had 8 CF cards, so wasn't too concerned. But we ended up shooting 83 minutes of footage (I pointed out we could shoot 72fps at 2K, and oh boy, does that zip out the frames!), so I ended up cycling through all the cards more than twice. So it wasn't an option on this shoot.

[I Bloom]

Could you elaborate on the Finder issues? You've started to me me paranoid, even though I've used the Finder to copy countless TBs of data without a single problem. Is the Finder notoriously rubbish at this? I've tried AAsync, but found that it slows the workflow to an impossibly slow level (even with a top of the range Mac.)

If I offload the CF, and watch the rushes every night and they play back OK, do I really need to checksum? What's the worst that can happen if I don't?

I think the biggest danger with Finder is just slipping with the touchpad. But in addition, I have seen ghostly folders that appear and disappear on external drives. After that I plugged one of those little lights into my USB port, so that I could sleep.

All that said, I have never lost a file because of Finder.

I like terminal because its a little more deliberate and cp -v gives you a report as it's copying, when I'm copying in finder, I don't want to walk away while the blue bar is moving.

I don't think you need checksums really, after all, file size is a kind of checksum, just much less accurate. But if you are on a set spending possibly thousands on each setup, how can you not use every means at your disposal to insure that the data is copied securely.

IBloom

[Mitch Deoudes]

Thanks for that. I may well be doing something wrong, which is making AAsync struggle. How long does it take you to transfer 130GB with AAsync?

It depends on what mode you're running it in, disk performance, etc. Read through the documentation - AASync goes through a number of "phases", which include reading the source (CF card) in order to generate "database information" (which I'm assuming is basically a checksum), copying the files, then reading both the source and destination *again* to see if they match.

I ran some tests this morning, and saw times ranging from about 45s/Gb to 90s/Gb, depending on type of drive, which FW800 port, daisy-chain, etc. I've also seen times in excess of 100s/Gb with a drive that's normally fast - no idea what caused it. But in general, I find that AASync is just as fast as doing the reads/writes/compares from the command line.

It can be a bit challenging to keep up with the camera dept., when they're shooting a million takes ("Hey - it's digital!"), 'cause I'm essentially reading the card four times. But it's worth it, to not have to tell production that we need to re-set the lights/camera for a lost scene.

As for the Finder issues: me, I don't trust it for heavy file-slinging. And I have a vague memory of getting wildly faster copy times from the command line - but that may have been a couple OS versions ago. One trick: if the file size/count that Finder shows is stale, you can kick-start it by creating and deleting a folder underneath the tree you're looking at, which will force Finder to recalculate its totals.

mitch

[MikeCurtis]

UPDATE - talked to Stuart English - there is NO NEED to keep the digital magazine files. On P2, you need the other stuff, but NOT for Red.

Throwing it all the .RDC folders into a common folder is JUST FINE.

[Rob Lohman]

Please do NOT follow that advice. There's future software that might use the two .profile files in the root of the magazine. Just copy the complete magazine.

[Stuart English]

Oops. I stand corrected! (One of the great things at RED is we constantly have new ideas bubbling up.. )

At this time there is no NEED to carry those files over, but its sounds like it could be good practice to do so....for the sake of future post software.

[Steve Freebairn]

Are there any plans to create a new file naming system so that no filename is every repeated?

[indivfx]

I'm a Unix Programmer i do this for living. Let me know if you guys need some help in automating this process.

[BASSAM MSSALATIE]

I'm a Unix Programmer i do this for living. Let me know if you guys need some help in automating this process.

sounds great..:biggrin:

[J.D. Frey]

I can't say for certain that mac osx finder will ever fail you - but I have personally seen this ... and it hurts -

http://tomkarpik.com/articles/massive-data-loss-bug-in-leopard/

Not sure why you say AAsync is "an impossibly slow level" - we are shooting a feature right now and making TWO copies of 130G per day off of 8G CF cards with AAsync - and it's going like butter -

Do you really need checksum?

Well ... do you REALLY need insurance?

Not unless you have problems. Then it's pretty handy.

I've had finder bomb on bulk copying of large files- not a fun time.

[Jarred Land]

Not simple, nor foolproof, but if there's no wiping on any given day, and we just shoot-copy-store one card at a time, there's hope.

Better ideas are always welcome.

Nothing is foolproof to a sufficiently talented fool... not deleting amongst the chaos in the middle of a shooting day, no matter how smart you are, is a great idea christopher.

[Lachlan Ward]

Update: My currrent workflow is a program called AAsync that I discovered in the series of tubes. This new tool is really going to put me ahead of all my competition. On the west coast they haven't even heard of this shit.

LoL. Ah yes the good old series of Tubes. "Its not a dump truck you can't just pile things in it. Its more like a series of tubes!"


Nothing is foolproof to a sufficiently talented fool... not deleting amongst the chaos in the middle of a shooting day, no matter how smart you are, is a great idea christopher.

Its what we all should look out for, talented fools truly are the bane of all existence.

I'm glad this thread started up, everything other than what I am saying now has been actually useful, but I will continue to rant uselessly anyway, enjoy.

It sounds like it might be a good idea to take 50 - 100 CF cards with you into battle, and fist full of red drives. Also I heard rumor that a Spanish film was recently shot on red, or at least they shot the film in Spain and they kept every CF card they used. It doesn't sound like such a bad idea really. The CF cards are 1/3 the cost of 35mm so you still come out ahead by a long way not to mention the tech is cheaper.
Mike you should demand all your clients do that from now on. Tells me and every one how it goes, you know, weather its a sound business stratagem.

Thought I should shift some paradigms... Don't you worry about work flow, let me worry about blank.

Lachlan

[Jance Allen]

Hey, the Build_14_Ops_Guide_v2.0.3.pdf (latest as far as I can tell) says this:

"Note: You don’t need to copy the PIN.log, digital.magazine.profile and digi-
tal.magdynamic.profile. files, but will also do no harm if you do."

There's no clue there that we should actually be copying those for future features. So I haven't been, which makes me feel, um, silly is one way to put it. Should the Ops_Guide be updated?

Are there any current workflows/software/hacks that are using those at this point?