Ballooning Data Rates and Crippling Workflow

[dev.doyle]

I was just on a job with two MX cameras shooting 4KHD RC42. The nature of the show required long takes, which naturally inflated file size. Running on a 2.8 GHZ MacBook Pro w/ a FW800 express card adapter I had three destination drives. Two were connected directly to the FW800 adapter, and the third was daisy chained behind one of those direct drives. The RED RAIDs were being connected directly to the MacBook's FW800 port.

Being a diligent data manager I began using R3D Data Manager to ingest the footage, but the combination of daisy chained drives and steep shooting ratio I began to get backed up (on a side note - R3D used to be so much faster, two upgrades ago it suddenly took 30% longer to dump - what happened?). Anyways, when pushed to the point of running out of media to hand the ACs I switched over to ShotPut Pro which allowed me to make up some time (I'm well aware of the pitfalls of ShotPut, and whenever using it I'm sure to watch all the footage to verify integrity).

All day it was a struggle to catch up, and at end of day required an additional 3 hours to transfer the rest of the footage. The question I pose is this: with data rates growing every build (which is great for image quality) what's a data manager to do? I'd use eSata for the destination drives but if you're limited to FW800 out of the RED RAID isn't it a moot point? Daisy chaining with eSata makes sense I suppose to boost speed between the two...or does it? Can anyone shed any light on that?

What are people doing to cope with these ballooning data rates? I don't know about you guys, but after a 15 hour day on location after 4 company moves the last thing I wanted to do was hang around for an additional 3 hours. Transferring the footage safely and verifying is priority one, but surely something can be done to make this faster. Any gear or suggestions anyone has would be greatly appreciated!

[Jannard]

I was just on a job with two MX cameras shooting 4KHD RC42. The nature of the show required long takes, which naturally inflated file size. Running on a 2.8 GHZ MacBook Pro w/ a FW800 express card adapter I had three destination drives. Two were connected directly to the FW800 adapter, and the third was daisy chained behind one of those direct drives. The RED RAIDs were being connected directly to the MacBook's FW800 port.

Being a diligent data manager I began using R3D Data Manager to ingest the footage, but the combination of daisy chained drives and steep shooting ratio I began to get backed up (on a side note - R3D used to be so much faster, two upgrades ago it suddenly took 30% longer to dump - what happened?). Anyways, when pushed to the point of running out of media to hand the ACs I switched over to ShotPut Pro which allowed me to make up some time (I'm well aware of the pitfalls of ShotPut, and whenever using it I'm sure to watch all the footage to verify integrity).

All day it was a struggle to catch up, and at end of day required an additional 3 hours to transfer the rest of the footage. The question I pose is this: with data rates growing every build (which is great for image quality) what's a data manager to do? I'd use eSata for the destination drives but if you're limited to FW800 out of the RED RAID isn't it a moot point? Daisy chaining with eSata makes sense I suppose to boost speed between the two...or does it? Can anyone shed any light on that?

What are people doing to cope with these ballooning data rates? I don't know about you guys, but after a 15 hour day on location after 4 company moves the last thing I wanted to do was hang around for an additional 3 hours. Transferring the footage safely and verifying is priority one, but surely something can be done to make this faster. Any gear or suggestions anyone has would be greatly appreciated!

Be happy you aren't shooting uncompressed?

Sorry... couldn't help myself. Someone qualified needs to help with the real answer.

Jim

[Deanan]

Epic/Scarlet will have built in checksums for each frame so it'll be easier to verify after a copy rather than during the copy.

We to often see productions go out with much less media than they really need. Sometimes it helps if you can get safety media as part of the rental that you open up only if you are in an emergency.

The other recommendation is to double up the backups. Get an older laptop and do the backups in parallel so you can turn around faster.

[Brian Wells]

One of the reasons I dislike this job - 19+ hour days on two camera shoots.

On several occasions we've shot in excess of 400GB per day on commercials. On a two day shoot, we once did 830GB in just two days! Ok, now, if this was common, the solution would obviously be higher end hardware, such as a Mac Pro / RAID setup. But, when clients normally only shoot 60-150GB per day, it's difficult / nigh impossible to expect them to pay for a "real" downloading station 100% of the time. It always ends up being a laptop for budget reasons. I wish I had the answer, the OT is sometimes nice. But, what would be even nicer is to have a normal work day, go to bed at a reasonable time, and earn just a little less money. Or, own your own drives, and get a second day to actually replicate them, instead of doing it all night long. Haven't yet decided to go all out on a downloading system yet, but I probably will to save my sanity on the next 2 camera job. Again, if you're doing episodic television or features, you gotta have that stuff. But for many of us, it's only an occasional need.

[Evin Grant]

For professional level 2+ camera shoots a tower with at least two available busses (1FW800 and 2+ esata) is imperative. You can handle a one camera shoot that is primarily CF card based with a laptop but beyond that you seriously need to build yourself a proper data management cart, ideally with some sort of fast RAID. It's amazing how fast FW800 can be when you free up the bus to just read. That said we're really looking forward to trying the new Red Station card reader.

[Greg M]

As Evin said you need a faster rig. MacBooks are not appropriate for that much data, nor are they safe. Build a professional rig or don't take on the larger jobs, it's as simple as that.

[Brook Willard]

Epic/Scarlet will have built in checksums for each frame so it'll be easier to verify after a copy rather than during the copy.

That's fantastic news.

As for dev, the MacBook Pro is not a tool that will have an extended future for data handling on higher-end production. It just sounds like you had the wrong tools for the job.

When we're shooting 3-7+ cameras on a movie or TV show, it requires a certain amount of equipment that most people do not own. There are carts out there that you can rent [generally $750-1,250 per day] that have the necessary data throughput to enable data transfers at the speed necessary for larger productions. You can build one yourself, but the cost of entry is very high.

Just as an example, here's a picture of the old cart I built two years ago for Southland:

http://homepage.mac.com/brookwillard/oldcart.jpg

My current cart is roughly twice as fast with many times more storage. Keep in mind that this sort of tool is designed as a DIT cart - not as a loader's cart - so its primary function is accurate color correction with data handling falling into the background. Needless to say, one has to have some very powerful equipment to turn a task that would take hours on a laptop into a task that takes minutes in the background of a tower.

Get the right tools for the job! If it's taking more than 15 minutes after wrap [or so, depending on certain situations], somebody messed up.

[Wendy Woods]

what cart is that digitalfx?

[Greg M]

They are both carts we built for our DIT's

[joe hedge]

Just got off a 14 hour data-tech day here in beautiful downtown Burbank...

Basically I try to never let the RedRaid drives get over 20% full, so the readout on back of the camera never goes under 80%. A lot of times I end up taking the drives home with me and either taking them back to the set the next day, or if it's the last day of the shoot, taking them back to the rental house the next day myself. Couldn't do that tonight though, as the owners are on dawn patrol for Vegas first light tomorrow...

[joe hedge]

For professional level 2+ camera shoots a tower with at least two available busses (1FW800 and 2+ esata) is imperative. You can handle a one camera shoot that is primarily CF card based with a laptop but beyond that you seriously need to build yourself a proper data management cart, ideally with some sort of fast RAID. It's amazing how fast FW800 can be when you free up the bus to just read.

Aren't the FW and eSata busses separate on a MacBook Pro with the express34 slot doing eSata? So you do have 2 available busses w/ a MBP, right?

[Blair S. Paulsen]

IF all you have is a MBP laptop, and it has the Express/34 slot so you can add the 2 port eSATA card, the fastest set up I have tried is reading off the RedDrive via the R2E cable and back out via eSATA to a G-Tech 2 drive RAID. Second best is reading off the RedDrive via FW800 and writing out via eSATA. There are many issues with this set up that make it less than ideal.

As Evin & Brook have noted a serious integrated solution with a MacPro tower on a cart is a sensible way to handle bigger demands. The Red Station should look pretty good on a DIT cart if you're shooting CF.

A wish list item from RED would be a Red Station module that I could slide a Red Drive into, that has the Lemo receiver. This is basically what the R2E cable does, but if RED actually made a module to do this that could compensate for the high signal levels out of the drive, send a proper signal to the computer's eSATA tap and fit the form factor...

My personal solution is another step beyond that as I have both a MacPro and a Globalstor PC in my van so I can divide up the copy operations (plus a lot of other cool stuff beyond this thread).

Cheers - #19

[joe hedge]

IF all you have is a MBP laptop, and it has the Express/34 slot so you can add the 2 port eSATA card, the fastest set up I have tried is reading off the RedDrive via the R2E cable and back out via eSATA to a G-Tech 2 drive RAID. Second best is reading off the RedDrive via FW800 and writing out via eSATA. There are many issues with this set up that make it less than ideal.

Yeah but w/o an eSata port multiplier your R2E cable leaves you 1 port short because you are backing up to 2 separate drives, but you only have 2 ports and you need 3.

And yes the G-Raids are faster, but Raid 0 for backing up R3D's makes me nervous...

[I Bloom]

Aren't the FW and eSata busses separate on a MacBook Pro with the express34 slot doing eSata? So you do have 2 available busses w/ a MBP, right?

For us lowlifes with our Macbook Pro's the answer is: using firewire and E-Sata together makes sense because they are on separate buses. So you will get the full throughput of the FW-800 coming from the Red-raid to your E-Sata drives.

When you daisy chain you must also understand that the information still passes into the computer and then out again. Thus with overhead you get less than half of the throughput time when you source and destination are on the same line. The same will be true if you have 2 Firewire ports on the side of a laptop, they are still on the same bus, so exactly the same as daisy chaining.

If you copy to two destination drives simultaneously from a single source drive you are forcing the source drive to search back and forth for different files. Imagine the needle on a record player trying to play two songs at once. The needle would spend most of its time traveling between the two grooves instead of in the grooves playing the music. That's what happens when you copy to two places at once on a mechanical hard drive, the transfer time is much more than double the time of one copy operation because the source drive is spending all of it's time searching between places. Instead do one copy and then the next.

On more little secret, most people don't do checksums. They're in the closet though and this is not a safe space so don't bother asking. Be your own checksum, by testing and inspecting the data on your destination drives. (The frame checksums do sound amazing though... bravo Deanan)

I carry on two 17-inch MBPs. Not much fun pushing a DIT cart when a glacier just calved and you have to run for your life ;)

Cheers,
IBloom

[dev.doyle]

Evin and Brook - thanks for the thoughtful insight. Brook we met before my nightmare job the other day at Keslow. I come from the camera assisting side - worked at Panavision for two years, have been camera assisting for a couple years, and have gotten in to data management as it's become the new loader position. Obviously there's a very clear line between digital loading and doing DIT work. On one end you're dumping and verifying, as well as lending a hand to the camera crew when you can - fulfilling the second/loader role. On the other end of the spectrum (IE: Brook and Evin) it's a committed DIT position with full post-production support brought right on set.

I suppose it took that job for me to realize where that line is. Full on RED teching w/ a $20k cart isn't where I'm heading, nor what I'm interested in. That being said, I'd like to operate at maximum efficiency in the capacity of a digital loader, and there have been several helpful tips above.

The R2E cable - it doesn't work with drives that have the SATA_TRANS_FAULT fix does it? Which nowadays is most of the drives? Say you were able to snag some old 320GBs, now that you're losing an eSata port strictly for ingestion, daisy chaining one of your destination drives via eSata would be a zero sum gain wouldn't it? Or would the gain in transfer speed nullify the additional lag on the daisy chain? Wonder when/if the R2E boys are going to fix that.

I've tryed to stay away from ingesting the RED RAID via fw800 and writing to destination drives thru the eSata adapter as it always gave me checksum errors with R3D Data Manager. That's great news about EPIC though...

[joe hedge]

If you copy to two destination drives simultaneously from a single source drive you are forcing the source drive to search back and forth for different files.

Is this true if you are using dedicated software, or does this just apply for dragging and dropping? Because ShotPut Pro, at least, copies faster to 2 drives simultaneously than if you did one at a time, and it's only a little slower than it would copy to 1 without it. 60 gigs takes about 40 mins, for instance...

[Conrad Hunziker]

If you copy to two destination drives simultaneously from a single source drive you are forcing the source drive to search back and forth for different files.

The current version of R3D Data Manager for windows does not have this issue. One read from the source to dozens of destinations. We are just about finished porting the Mac version of this. With this, your copy is as fast as your slowest destination.

As for the original post, in addition to Brooks' insight, I suggest getting rid of firewire wherever possible. When producers come to me with firewire-only drives, I send them back. Minimum connection, eSATA.

In addition, you wouldnt approach a multi-camera shoot with one AC. So you shouldnt approach it with one computer. Adding parallel processes can only help.

One last suggestion: you can do the third copy later. You said the third is daisy-chained, which mean both of those drives are getting less than half the original bandwidth. Because you are using checksums, you know the second copies are exactly like the originals. So take advantage of it, and do the third drive later (still asap, just not in order).

[Dave Weber]

The current version of R3D Data Manager for windows does not have this issue. One read from the source to dozens of destinations. We are just about finished porting the Mac version of this.

Conrad,

When do you think the Mac version will be done?

[Tom.Wong]

also keep in mind that's y the red team is work on RED STATION, for all these grow needs for more bandwidth for safe redundancy. I actually had the idea of trying to build one myself with maybe 70% of the specs they put together in their mock up. Just never really had the money to "experiement" not knowing how well it would work.

plus faster ports are emerging, usb 3.0, esata is becoming more common on a lot of drives.

my quickest solution though for current MB Pro, red drive with esata cable connection to express card (or cf to express card reader). FW800 to a raid 1 drive, and create redundants later, or offloading to a usb drive at the same time too. the key is to keep all ur buses independant and at full use to single drives only. daisy chaining is only gonna cause problems.

[Conrad Hunziker]

Conrad,

When do you think the Mac version will be done?

ASAP.

Unfortunately unlike our favorite camera manufacturer, we cant release beta products. We are directly dealing with peoples data, and people trust us and the software.

That said, I was showing a beta version at REDucation in Vegas last week.

[I Bloom]

Is this true if you are using dedicated software, or does this just apply for dragging and dropping? Because ShotPut Pro, at least, copies faster to 2 drives simultaneously than if you did one at a time, and it's only a little slower than it would copy to 1 without it. 60 gigs takes about 40 mins, for instance...

Basically any program will do this if it is accessing the same drive at the same time for two different reasons, even if you are doing the same exact copy at the same exact time the operating system treats them as two distinct actions. It's not really a software issue, it's a hardware issue. However software can overcome the problem easily just by waiting to make the second copy after the first is done.

This is only really and issue with mechanical hard-drives such as Red drives. Solid state drives might add some overhead by switching but even if they do it's muuuch less.

IBloom

[Blair S. Paulsen]

FWIW my best benchmark times when working on a MBP with the eSATA Express/34 adapter was to hook up the RedDrive via R2E and the catch drive via the eSATA as well, make one copy, swap the catch drives and make another copy. Faster than any daisy chain. Faster than FW800 off the RedDrive and eSATA to 2 catch drives. Obviously if you have an upgraded drive the R2E cable is out. The option to have software that forces the copies to be sequential sounds like a winner to me. Anyone know of a current methodology for that, or are we waiting for the Mac port of R3D Manager?

Cheers - #19

[Conrad Hunziker]

The option to have software that forces the copies to be sequential sounds like a winner to me. Anyone know of a current methodology for that, or are we waiting for the Mac port of R3D Manager?

Cheers - #19

You can do this today with the current version of R3D Data Manager. Open multiple copies of R3D Data Manager:

Copy 1: Red Media to destination 1
Copy 2: Destination 1 to Destination 2
Copy 3: Destination 2 to Destination 3
...

You still cant do that simultaneously. However, by just making Destination 1 an eSATA RAID setup, and destination 2 a eSATA device, you can do this round robin with zero downtime.

The next version of R3D Data Manager is days away.

[Conrad Hunziker]

However software can overcome the problem easily just by waiting to make the second copy after the first is done.

Theres many ways to overcome this issue, the suggestion above is the slowest, and not used by R3D Data Manager.


This is only really and issue with mechanical hard-drives such as Red drives. Solid state drives might add some overhead by switching but even if they do it's muuuch less.

SSDs and CFs still run into the issue of data throughput on the bus. If your destination is faster than your source (especially as you add multiple destinations), then this will always be an issue no matter what the source technology. The current windows version and next mac version of R3D Data Manager eliminates this issue - the single slowest device is the fastest you will copy at. Of course, you will still need to avoid doubling up on a single firewire bus.

[Noah Yuan-Vogel]

I didn't realize R3D Data manager for Mac couldn't do one read for many writes, but then I haven't used it. I was always impressed that the Win version supported this but then I don't use it because I found the software to be too buggy and provide two few details of copies in progress.

I recently built a portable workstation with pci-e2.0, Sata6G, and USB3 and an internal RAID for just this reason, esata or sata is mandatory in my mind, as are backup destinations that can handle writing 100MBps+ sustained, FW800 is a backup if necessary. I did a couple low budget jobs last year that only had 4-5 8GB cards and required copying RC36 footage on a laptop to FW400/USB hard drives (thats what was provided by production) so I know the ridiculousness of dealing with this kind of data without proper equipment and media. What's worst is doing that in the front of a van or on top of a car. One bumped cable and the whole production is held up waiting for a new card to shoot to. With the Red Drive, I am often forced to do many partial copies throughout the day and/or forgo one backup drive by not formatting the 640GB Red Drive until after the post house has received and backed up a transport drive to keep up with data and avoid coping for hours at the end of the day. From now on I will insist on the portable workstation for RED shoots. It's so new I haven't brought it on a RED shoot yet, but I did a few SxS shoots and copying a 16GB card in under 3 minutes (80+MBps) is a beautiful thing, that's twice as fast as my old laptop did to its internal drive and about 4 times as fast as it did on average with external drives.

I think software will really help this, I know there are a few solutions that do it but it should be standard to be able to write to all destinations from a common buffer and a single read and have stable well-laid-out software that allows canceling without freezing the software or leaving partial files, allows resuming transfers, and gives full real-time details on file copy speed, progress, and verification.

[Conrad Hunziker]

I didn't realize R3D Data manager for Mac couldn't do one read for many writes, but then I haven't used it. I was always impressed that the Win version supported this but then I don't use it because I found the software to be too buggy and provide two few details of copies in progress.

Too buggy? To date we have not received a single bug report of the windows version, from anyone. If you have some, please submit them.

We will bring the windows version up to the same reporting features as the Mac version, as soon as this mac port is finished.

[Noah Yuan-Vogel]

I will admit my experience with R3D Data Manager for windows is rather limited. I really only tried a trial version once and when I saw it gave me so little information about the copy (I expect to see speed, progress, current file, etc) that was in progress I canceled the copy and the program froze up and I had to eventually end the process after it didn't unfreeze for a few minutes. I apologize as calling the program buggy could be a significant mischaracterization since my experience was very limited and anecdotal. It didnt give me a great first impression of the software, though. I was also off-put by the fact that the versioning and feature set was not unified between the mac and windows version as I use Windows far more often than Mac due to the greater quantity and stability of software tools and drivers in windows in my experience (Macdrive, Teracopy, expresscard-pcie bridge hotswap support, bad sector remap in chkdsk, trim function for SSDs, etc).

[Rob Ruffo]

Here's a really sleazy solution I used one time, being my own DT on an 18 hour shoot: Go to an anonymous big box store, the kind that put mom and pop out of business, buy another laptop, then, after that one job where more speed was needed, return it on the 20 day no questions. I know, it's ghetto, but it might save your sanity, and it only costs gas and/or shoe leather.

[Jay Gannon]

I think what it comes down to is if your paying for a DIT he should have a MacPro based station.
I wouldnt DIT anything short of a free gig on a Macbook Pro their just not up to the job.

Saying that theirs a 15" and a new 13" in the Jeep at all times as emergency devices.