PDA

View Full Version : Who wrangles, and how?



Robert Atkins
01-14-2009, 01:44 AM
[I'm new 'round here, if this is the wrong forum feel free to move this post]

I've done a weekend helping some friends out data wrangling on a Red One shoot. It was fun and I'm curious to see if there's enough work around to make a business out of it.

I'm a "computer guy" rather than a "film guy"; so I know all about RAID levels and storage equipment and how to do off-site backups properly and such.

Aside from getting the data off the CF cards/Red drive and safely into the hands of the editor, what other duties are typically performed by a data wrangler? Is "data wrangler" even usually a separate job to "camera tech" on a professional shoot?

Finally, who's doing this today for all you people shooting on Red? Someone who's just fallen into the job and learning as they go or are there already lots of experienced professional data wranglers with well-tested workflows and bullet-proof gear out there? What rates do people charge? Is there much work about?

I'm in Sydney, Australia.

Cheers, Robert.

Zakaree Sandberg
01-14-2009, 02:25 AM
its not a real position.
DIT is the position.
you must know all the camera functions, menus, settings, be able to troubleshoot codec errors, camera failure,
exposures, histograms so on..
on set CC (depending on delivery form)

backing up data is 10% of the real job

Christopher Grant Harvey
01-14-2009, 02:36 AM
its not a real position.
DIT is the position.
you must know all the camera functions, menus, settings, be able to troubleshoot codec errors, camera failure,
exposures, histograms so on..
on set CC (depending on delivery form)

backing up data is 10% of the real job

True.

I just finished DIT'ing (a real word? :huh:) on a feature and the tasks listed above pretty much cover the duties I had to perform. Luckily we never had a major problem. I also updated the firmware of the camera to B18 and did a black-shading twice. I mostly formatted and handled the drives and cards coming from & to the camera.

As for data backups, I had 3 backups at any given time. I also kept a copy of the sound files from the sound recordist, who used a Fostex FR2 recorder. Just copied the files from the CF card.

I also provided dailies for the DOP and Director to view from the R3D's which was blazingly fast. During lunch I could have a few clips from the mornings shoot ready to view.

I used a very fast dual core PC laptop armed with Build 17 & 18, the users manual, 12 hard drives, some external housings, a few card readers, and some software.

Zakaree Sandberg
01-14-2009, 02:54 AM
Robert,
since you said your more of a computer guy..
I encourage you to get onto as many red sets as possible,
or rent a camera and learn it.
Spend a few months reading this entire site, learning all the possibilities.
It is a Fun and rewarding job:)
so keep up the learning:) you came to the right place

Robert Atkins
01-14-2009, 06:20 AM
I didn't think it'd be that simple :-).

A quick google comes up with http://www.4klondon.com/dit.html which looks like more of a complete job description, yeah?

Cheers, Robert.

Nicholas Shields
01-14-2009, 08:05 AM
The most effective and sought-after DIT's are the ones that understand the filmmaking process. It's the ability to anticipate problems that arise in the normal course of production and prevent loss of time/data that make a DIT valuable. The only way to be this responsive is to know as much about the filmmaking process as possible. For example, as the crew is setting up an action sequence and you overhear someone mentioning "120fps", you need to anticipate the need for 4 times the amount of data per second being recorded and coming your way. It's the little things that will make you a wanted man. I don't believe that either background (IT or production) is in a better position than the other to succeed- but either way, be prepared to take on a new expertise. Best of luck with your endeavors.

Nick.

Conrad Hunziker
01-14-2009, 03:12 PM
You asked about software, and Id like to point out that R3D Data Manager helps to take care of the actual copying process, in addition to being able to do quicktime renders. For more info, see the link in my signature.

That helps to take care of the 10% of your job that data management comprises. You'll have to figure out the rest on set.

Zakaree Sandberg
01-14-2009, 03:19 PM
You asked about software, and Id like to point out that R3D Data Manager helps to take care of the actual copying process, in addition to being able to do quicktime renders.

havent used this particular software yet, so I cant comment on the quicktime rendering.. but I will never trust sofware to do all the copying and backup.
it may be a good starting point.. but always double check.

Christopher Grant Harvey
01-14-2009, 03:21 PM
Oh yes for PC guys check this out: http://www.codesector.com/teracopy.php

I used it to handle all my copying and it works amazing.

Copy files faster - TeraCopy uses dynamically adjusted buffers to reduce seek times. Asynchronous copy speeds up file transfer between two physical hard drives.

Pause and resume file transfers - Pause copy process at any time to free up system resources and continue with a single click.

Error recovery - In case of copy error, TeraCopy will try several times and in the worse case just skips the file, not terminating the entire transfer.
Interactive file list. TeraCopy shows failed file transfers and lets you fix the problem and recopy only problem files.

Shell integration - TeraCopy can completely replace Explorer copy and move functions, allowing you work with files as usual.
Full Unicode support.

Conrad Hunziker
01-14-2009, 03:51 PM
havent used this particular software yet, so I cant comment on the quicktime rendering.. but I will never trust sofware to do all the copying and backup.
it may be a good starting point.. but always double check.

Well, Id invite you to use the software.

This software picks up where finder/explorer leaves off. Neither finder nor explorer verify that files actually made it across, nor do they do any sort of file verification, and many times they ignore critical errors. R3D Data Manager checks for all of this. And its one-click copying, which helps to remove the human element.

You make a point about never trusting software, but this software will show you whats gone wrong when finder doesnt, and provides ways to ensure the copied data is an exact duplicate of the original, even when the original is long gone years from now.

Christopher Grant Harvey
01-14-2009, 03:58 PM
Neither finder nor explorer verify that files actually made it across, nor do they do any sort of file verification, and many times they ignore critical errors. R3D Data Manager checks for all of this.

Teracopy does all this for Free for PC guys who would be interested...

Conrad Hunziker
01-14-2009, 06:09 PM
Teracopy does all this for Free for PC guys who would be interested...

I dont see where teracopy does checksums, verifies file data integrity or provides for copying redundancy. R3D Data Manager also allows you to check to ensure that files you copied years ago are still valid, even if the source doesnt exist. Even the pro version, which does cost money, doesnt do any of that.

Christopher Grant Harvey
01-14-2009, 11:26 PM
It comes pretty close considering it is free. Perhaps my choice of words should have been, "it comes quite close and offers similar functions that work in the exact same way to ensure you never lose or miscopy data."

It has it's own system of checking that the files are copied correctly without problems which works perfectly fine. It verifies after each file has been copied and then after the whole process has finished it verfies again letting you know of any problems.

Can't see why you would need more than that...

If the files you copied don't open from years ago then they aren't gonna open so how would R3D data manager solve that? Just wandering...

Uli Plank
01-15-2009, 12:04 AM
I can only second this– R3D Data Manager is a very helpful piece of software.

Von Thomas
01-15-2009, 12:22 AM
I have 8 years experience doing digital tech work for the photo industry, and willing to work for free to learn Data Wrangling on RED systems.

I owned a digital capture service in NYC which handled digital still photo shoots (advertising, fashion, editorial). I've recently moved to L.A., and I am transitioning over to digital video capture.

The still cameras I've been working with are the latest 30 and 39 mega pixel rigs. The computers I use are Mac Pro's and MacBook Pro's. I typically had these computers outfitted with the fastest everything (video cards, drives), and minimum 8GB ram to 16GB, to handle the large amount of data.

As I understand from reading this thread, there are many similarities to the still world. I have to know all digital cameras and backs I work with, the operation, the settings, set-up, trouble shooting both cameras and computers if necessary, the software, back-up procedures. This was a learning process that took years. Well I'm ready for a new challenge.

I'm looking to getting hands-on experience with RED cameras to learn the workflow. To give you an idea of what I do, here is a link to my site. http://www.digitaltechnyc.com.

Von
347.463.3731

Conrad Hunziker
01-15-2009, 01:21 AM
It comes pretty close considering it is free. Perhaps my choice of words should have been, "it comes quite close and offers similar functions that work in the exact same way to ensure you never lose or miscopy data."


But it doesnt do it the exact same way, thats the point.



It has it's own system of checking that the files are copied correctly without problems which works perfectly fine. It verifies after each file has been copied and then after the whole process has finished it verfies again letting you know of any problems.


It does? I dont see that mentioned anywhere in the software. If it does, they dont list it as a feature, which would be pretty big to miss. In fact, unless they are doing checksums there is no way to future-proof that the data you have is correct.



Can't see why you would need more than that...

Ok - heres a real-world instance, that has happened more than once in the past year.

You have made a copy of your data, and made an exact verified backup. Now you have your 2 perfect copies. You put them on the shelf. You come back a month later and look at both copies. Something happened to the media and the file sizes differ! Which one is right?

Your program wouldnt have the foggiest idea. In fact your program would be useless. R3D Data Manager would be able to tell you.



If the files you copied don't open from years ago then they aren't gonna open so how would R3D data manager solve that? Just wandering...

Though the checksum is one-way, if its just a couple of bits in the wrong place you can figure it out by figuring out what you would need to alter to get your checksum to match. Its not that hard really, but would take time.

The bigger point is that R3D Data Manager provides a verified way to make up to 4 copies at once, and while doing so will satisfy studio and bond completion requirements. If you are going to shoot anything with money behind it, the money people want an audit trail - which R3D Data Manager provides.

I dont mean to steal this thread on the merits of one software vs another. I do wish to point out that as a DIT, the data that you are moving around costs tens to hundreds of thousands of dollars to produce. Its worth more than the sum of the data. You really need to consider your workflows and options you provide for the productions you work for. The more reliable you make the stuff around you, the more it will pay off, both in terms of jobs, but also in terms of data security.

Having worked as a DIT and/or Operator and/or DP on nearly 2 dozen RED shows at this point, including a feature currently playing at sundance, theres a level of reliability and safety that producers will require - especially since for a lot of them this is their first venture into anything Red. There simply is no software on the market designed to make multiple copies with a single click that will verify your data independently of the source being present. Thats why I wrote the program within the first week of the first Red being released and had a 1.0 version by January 2008.

I have yet to loose one byte of data for any Red show. Thats not only because of the software, but also because of the management practices on and off set. Even if Im the DP or Operator Ill pay close attention to what the DIT is doing and help them accomplish their mission - not to be overbearing, but because thats gold in them-there drives!

Christopher Grant Harvey
01-15-2009, 01:28 AM
I have yet to loose one byte of data for any Red show.

Same here, just using Teracopy.

I see where this discusion can go... I don't want it to go there... :sick:

Robert Atkins
01-15-2009, 05:22 AM
Photocon: nice thread hijack, well executed ;-)

I did actually find R3D Data Manager when doing some research before the shoot I was on, but even $70 wasn't in the budget so I did md5 sums from the command line to make sure the data transferred onto the backup drives ok. I know R3DM does more than this and I would definitely purchase it if I was doing a paid gig.

Nicholas: Yes, of course, to be in demand you would need to, in the eyes of the director/2nd AD, "skate to where the puck is going to be".

I think my approach will be to help out on a few more low/no-budget projects and get more familiar with the equipment and environment and if I still enjoy it and have something to offer, try and scare up some paid gigs.

So if anyone in Sydney needs a hand with DIT/wrangler-type duties, send me a PM.

Nelson Goforth
01-23-2009, 04:58 PM
I just did a job as "Data Wrangler" using the R3D Manager software.

I'd looked into running checksums myself, and thought perhaps I could put together a perl script to do the repetitive work, but when I read about R3D Mgr it seemed to do the job with a level of security that I could accept. The md5 checksums and the reports are critical, and the R3D Mgr makes it pretty effortless. I was copying from a Red Drive onto two 500GB backups and it worked like a charm.

My only comments would be (1) make a demo version available - I was hesitant to purchase without really knowing the software. Only recommendations by others settled me. And (2) we need better documentation - and not online (didn't have connectivity on the set). For some reason the initial checksum wasn't running automatically for me and there was no documentation to help me discover why. And while I understand what md5 is, I'm not sure most people would. There are a number of obscure choices in the software and the 'tooltip' instructional method is okay as quick reminder, but it doesn't replace a manual - even an electronic one.

But on the whole I was very well pleased with the software and it quietly sat there and did it's job.

Nelson

Nelson Goforth
01-23-2009, 05:13 PM
.. but I will never trust sofware to do all the copying and backup.
it may be a good starting point.. but always double check.

How do you double check?

I'd thought early on about running md5 checksums from the command line, but for a lot of files that could be tedious (and therefore prone to operator error), and literally rewatching each shot (twice for two backups) seems counter-productive.

I have used R3D Manager and it does what I'd planned on (md5 checksum) and quite a bit more (reports with any errors, etc), but without the headache.

Of course I don't completely trust software either, so I watched samples of a number of shots from each backup drive. What's more, the md5 checksums (and R3D Mgr can do even more secure checksums) WILL tell you if you have digitally identical files... but they won't tell you if the original file was any good to begin with :biggrin:

Thanks,
Nelson

Cam Crowley
01-23-2009, 06:48 PM
If you are using a mac and serious about your DIT duties then R3D Data Manager is money well spent IMO.

But nothing is as thorough when it comes to checking footage as actually watching the clips play through from start to finish and checking them against the camera reports. And this includes all the copies of the footage.

Cam

Chris Parker
01-23-2009, 06:53 PM
zakaree, i hate to break it to you, but the term DIT is a misnomer when dealing with RED. A TRUE DIT works on set and actually paints the digital picture with other HD and digital cameras, that don't shoot RAW. The fact that RED shoots RAW makes the RED tech job more a hybrid of DIT and data monkey. By virtue of REDs system, a DIT is not needed on RED Shoots. Sure you need to know the menu, and how to troubleshoot, and how to read exposure meters, but it is a far cry from what TRUE DITs do. Red/Data tech is a better description....

Conrad Hunziker
01-23-2009, 07:14 PM
How do you double check?

...
Of course I don't completely trust software either, so I watched samples of a number of shots from each backup drive. What's more, the md5 checksums (and R3D Mgr can do even more secure checksums) WILL tell you if you have digitally identical files... but they won't tell you if the original file was any good to begin with :biggrin:


Precisely - However, by default if you know the MD5 sums on 2 copies are correct from the original, and watch the copy on backup 1, then backup 2 must also be correct.

I think the original poster was talking about if the file was copied over - i.e. a filesystem check. But again that only means that the files are there, not that they are correct. By doing a filesystem check, md5 check and viewing the proxy file (so you can see each frame) you have checked all possible methods for corruptions.

R3D Data Manager checks for two of the three. We are looking into ways of doing the third (with Red's help)

Dane Brehm
01-26-2009, 12:58 AM
Though Checksums and Verification are incredibly important to maintaining integrity from drive to drive, I feel that paired with a calibrated monitor and a UPS (uninteruptable power supply) are a mission critical part of the Data Tech's job.

On commercials and episodics, I am typically working the camera from the body back working closely with the 1st&2nd to assure the DP is taken care of and always ready to shoot.

As a 600 DIT I've been shoe-horned into traditional film heirarchy but a traditional DIT's job is to match multiple camera's pictures and allow a camera's such as the F900's 10-bit 4:2:2 capture fit into a 8-bit 3:1:1 HDCAM tape. They also manage exposure, gamma, knee, etc via a paintbox.

a Red Tech's short list of duties (I am also a Phantom, S.Two, and Codex Tech)

- all Sound, HD Video and Timecode cables are attached and functional.
- TC is sync'd via Lock-it or Deneke box
- All drives/cards are formatted and labeled
- Verify Camera Data and Transfer to Redundancies, IT
- DP/Directors monitors are Calibrated.
- Assure Modulus or Wevi HD wave is operational.
- Manage and Define LUT with DP
- Assure Editorial and Post's specific needs are met
- Assist DP with Exposure
- Organize and Anticipate all Red specific issues
- Anwser Questions about Workflow even recommend best workflow for job

I'm sure there are a few thing's I'm missing. Just Understand that each time you do a Red Job your representing The Red Camera and the Red Tech's alike:)

Troy Smith
01-26-2009, 01:05 PM
Sounds like Code Sector is a great tool to use for the pc side at least until r3d manager releases for windows.

Thanks for posting the link to Code sector.

hoylecd
04-21-2009, 01:35 PM
I find it a misnomer to call a data wrangler a DIT. As others have pointed out, this is already a very well defined position, and just because the RED doesn't necessarily require one, does not mean the job has suddenly vanished. I think Digital Loader is by far the best term used to describe the individual who is; backing up media, creating dailies, working with post. My reasons for this are simple, a Loader already exists in the film world, and while a Film Loader and Digital Loader don't match up perfectly as far as responsibilities are concerned, they are very similar. This also allows for people working in the film world to easily understand the position. Calling yourself a DIT when in fact you are a Digital Loader can mislead what a crew may expect from you. A Digital Loader doesn't have to be working on a RED shoot. They could be working with SxS media, or P2, in which case labeling yourself as a DIT could really cause problems. Imagine showing up on set and the D.P. is expecting you to paint two HVX200s. If your a Digital Loader, you'll have neither the knowledge or tools to pull this of, and end up with a very unhappy camera department. Even terms like Data Wrangler, or Digital Asset Manger are better suited for the position of backing up media. Hopefully local 600 will push a term (digital loader, data wrangler, etc...) that will clear up this mess and stop the confusion this position seems to carry with it.

Zakaree Sandberg
04-21-2009, 02:48 PM
zakaree, i hate to break it to you, but the term DIT is a misnomer when dealing with RED. A TRUE DIT works on set and actually paints the digital picture with other HD and digital cameras, that don't shoot RAW. The fact that RED shoots RAW makes the RED tech job more a hybrid of DIT and data monkey. By virtue of REDs system, a DIT is not needed on RED Shoots. Sure you need to know the menu, and how to troubleshoot, and how to read exposure meters, but it is a far cry from what TRUE DITs do. Red/Data tech is a better description....

yes i know... however the 600 doesnt...

there is no "data monkey" position in the union.

DIT is the closest

Von Thomas
04-21-2009, 07:58 PM
Wow. Well what should I call myself? This past weekend I teched a three camera job, now I specialize on the RED camera, and I not only managed all the data and backed it up, I supervised the exposure settings for every take, made sure a color card was inserted in key shots for color reference in post, oversaw playback, and made sure the infomation was ok before moving on.

With the RED there are more things to consider, it is a different way of capture, to think of it in terms of a film or video camera, is to undermine your project. Now there are data loaders, but to understand the ways of RED, you have to take it up a notch, especially for camera dept not familiar with how RED works.

hoylecd
04-22-2009, 10:38 AM
Wow. Well what should I call myself? This past weekend I teched a three camera job, now I specialize on the RED camera, and I not only managed all the data and backed it up, I supervised the exposure settings for every take, made sure a color card was inserted in key shots for color reference in post, oversaw playback, and made sure the infomation was ok before moving on.

With the RED there are more things to consider, it is a different way of capture, to think of it in terms of a film or video camera, is to undermine your project. Now there are data loaders, but to understand the ways of RED, you have to take it up a notch, especially for camera dept not familiar with how RED works.

I think RED Tech would better describe your position on these jobs. What you're doing is a cross between what a traditional DIT and Digital Loader does. I realize that smaller productions may not want to pay for a digital loader who's focused on data security and a smooth post process. But on larger productions, the need for a dedicated Digital Loader becomes essential. Merely dragging and dropping is one thing, but providing data integrity with RAIDs, LTO decks, creating dailies, rendering on set, and working intensively with post is a full time position, and is not what a DIT does. I believe true DIT's are probably unhappy with the fact that their position is being misrepresented by so many people. The same goes for Digital Loaders. This is a growing field, and one that is constantly evolving. And even though this is REDUSER.NET, lets not forget that there are plenty of other digital formats out there, with plenty more to come. So the need for a clear distinction between a DIT and Digital Loader is crucial as the transition from film to digital really starts to speed up. Should a Digital Loader have a working knowledge of the camera the production is using, absolutely. But this does not make them a DIT, just like being able to copy a hard drive doesn't make you a Digital Loader. We need to start thinking beyond RED, and look at these two positions on a larger scale.

Nick Gardner
04-22-2009, 12:06 PM
yes i know... however the 600 doesnt...

there is no "data monkey" position in the union.


tru, but there is a fully functional loader position, which is exactly what a card/drive monkey does. There is already a neat little room on the camera truck where you can put the loader, loaders understand the chain of custody and responsability of thier position, AND they know how to fill out everyones time cards. And again, they can cover the 2nd when he's 10-1, and are a functional part of the camera dept.

I see the Dit going away and getting back to loaders. DIT was really a transitory position to hold film DPs hands when shooting HD.

Nick

Mike Prevette
04-22-2009, 01:20 PM
Gotta say I totally agree with Nick. Now that more cameras are shooting "raw" we are goig to return to the already well ironed out hierarchy.

_mike



tru, but there is a fully functional loader position, which is exactly what a card/drive monkey does. There is already a neat little room on the camera truck where you can put the loader, loaders understand the chain of custody and responsability of thier position, AND they know how to fill out everyones time cards. And again, they can cover the 2nd when he's 10-1, and are a functional part of the camera dept.

I see the Dit going away and getting back to loaders. DIT was really a transitory position to hold film DPs hands when shooting HD.

Nick

Zakaree Sandberg
04-22-2009, 02:06 PM
tru, but there is a fully functional loader position, which is exactly what a card/drive monkey does. There is already a neat little room on the camera truck where you can put the loader, loaders understand the chain of custody and responsability of thier position, AND they know how to fill out everyones time cards. And again, they can cover the 2nd when he's 10-1, and are a functional part of the camera dept.

I see the Dit going away and getting back to loaders. DIT was really a transitory position to hold film DPs hands when shooting HD.

Nick

true.. however.. if im getting paid loader rates.. i will NOT touch the camera, settings or consult on exposure..

Cüneyt Kaya
04-22-2009, 02:12 PM
the more post possibilities come the more an onset guy is needed/wished by directors and DOP´s (my experience)

-databackup
-Footage checking (focus, exposure etc.)
-dailies
-onset editing operator---test if the scene works or not with the director for example
-onset grading
-3rd AC
- playback operators
-onset syncing
- connection between Posthouse,VFX, Editor and Set (cam and sound department)

with red rocket and other nice tools it will be much more comfortable for directors and Dops.
i see the classical videooperator fading away.

doing all this i saw sets with more than one "DIT/Datawrangler"
i even saw DIT/Wrangler Assistants

and a DIT earns twice as much as a Clapper Loader here

Nick Gardner
04-22-2009, 02:13 PM
true.. however.. if im getting paid loader rates.. i will NOT touch the camera, settings or consult on exposure..

I'm sure the Operator, the DP and the 1st will be thrilled by that ;-)

Seriously, why would you consult on exposure ? totally the DPs gig. Telling the DP how to expose a shot is a great way to piss him/her off. And really, there is no reason for a dit dam monkey whatever you want to call him to touch the camera, there are 3 fully qualified people doing that already - Op, 1st AC, 2nd AC. And by the way, loaders help out with whatever they can.

Cüneyt Kaya
04-22-2009, 02:22 PM
last fall for example

1st unit 2 cams, 2nd unit one cam.

crew:
1st unit

1xdop
2xoperator
2x1st ac
2x2nd ac
1xprotocoll guy
1xDIT
1xvideooperator

2nd unit
1xoperator
1x1st ac
1x2nd ac
1xprotocoll guy
1xDIT
1xvideooperator

and one fat Bus. headquarter of the DIT/wranglers equipped with 2 macpro`s, several mb pros and all the things you need.
Everybody called the Bus "the Lab"

hoylecd
04-22-2009, 08:22 PM
true.. however.. if im getting paid loader rates.. i will NOT touch the camera, settings or consult on exposure..

This is an interesting point. Not to knock on traditional film loaders, they are highly trained and have a large responsibility on any film production. But a Digital Loader has much more to keep track of and works with many more departments than a traditional Film Loader. As a Digital Loader, you are in constant communication with the Post House / Editor to make sure the workflow is consistent. You are reviewing footage for for quality control, rendering dailies, possibly syncing sound, all while making sure every piece of media is safe and secure. A film lab would never call a Film Loader on how to develop 35mm, but a Digital Loader could easily get a call from the post house or editor needing advice on how to best handle the media (this has happened to me personally). Again, I'm not trying to belittle the Film Loader, this is an essential and respected position. But I believe a Digital Loader should have a higher pay rate because the nature of the job requires more from the individual. And while I'm sure there are many producers who cringe at that thought, lets not forget the money that's being saved by not having to buy film, develop, and scan digitally. That adds up to a lot even on smaller shoots, and I think easily makes up for paying a Digital Loader what they deserve. I'm glad to see these topics being discussed, as I believe the position of Digital Loader is going to become much more prevalent over the next couple years, and these issues need to be figured out.