View Full Version : Digital Lab Software and Camera Reporting... how would YOU like it to develop?

Jody Neckles
06-07-2011, 01:24 PM
This thread is to discuss what DITs, 2nd ACs and DOPs would like to see in future Digital Lab software.

There are few tools (Colorfront, Storm, Truelight, 3CP, Clipfinder and perhaps RCX for example) that are designed with onset use in mind. As such tools have not been around very long, and are still quite limited, we have to be resourceful by utilising other software packages to deliver the lab service our clients require. The area between acquisition and post is growing and this is confirmed by the number of software vendors like Assimilate and SGO moving into the digital lab market by utilizing existing (DI centric) technology to create new products for onset use (Scratch Lab and Mistika Live). I hope and expect other companies to follow this trend.

Today's Digital Lab software is still in it's infancy and there are a number of areas I would like to see improved: in particular the areas of Backup, Asset Management and Reporting:

Here are my thoughts and I would love to hear yours... I write hoping the software developers are listening :-)
For the record I use Storm and Scratch on my onset labs.


- Fully functional backup with checksum options to multiple drives and LTO built into the lab software.


- Metadata merge and import from systems like Qtake, Live Play and 3D systems
- Databases based on metadata tagging with the option of organizing material into folder structures
- Integration with other databases like CatDV, Avid Interplay, Final Cut Server via API or export functions.

Improvements in reporting is something I think is long over due I would like to see the following features implemented on a per clip basis
- Backup Log including: Checksum result, multiple destination drive backup confirmation, total backup time, LTO backup and verify conformation, Visual Verification conformation
- Add Board info: Name, Slate, Scene, Circle take,
- Clip info: Project FPS, Resolution, Aspect Ratio, Shot FPS, ISO, Color Temp, Shutter, Color space, Gamma, Date, Time, Timecode, Edgecode, Frames, GB, TOTAL GB
- Camera info: Serial Number, Mag Serial Number, Build, Format (RED, Alexa etc), Lens, Stop, Focal Distance, Filter, Sound onboard, Sound Ext, Viewing LUT, Sync Ext/Int
- Other notes: Production, Production Company, Director, Producer, DOP, Focus Puller, 2nd AC, DIT (with contact details), Location.
- Issues: Exposure, Focus, Lens, Dropped Frames, Sensor, Timecode, Sync
- Look Communication: The ability to easily track when a clip has been graded. Perhaps a before and after them nail with a table of changed perimeters (I like what Gamma & Density do). Reports should include thumbnails of clip, histogram, waveform, vectorscope and false color map. This can be used to communicate DOPs intent to the post facility.
- Dailies log: Sound/Video Synced, L/R Eye Synced, a log is kept of all dailies generated (noting the preset used to create them).
- Export Options: PDF, ALE, XML, .CSF, TIFF snapshot, HTML contact sheet with embedded video proxies :-)

As a DIT the lack of streamlined reporting tools has bugged me the most on recent projects. I would like to hear your thoughts on the above and in particular how you would like to see the Report functionality develop?

Dustin Cross
06-08-2011, 12:39 PM
Here is a rehash of some e-mails I recently sent to a company developing this type of software.

1. LIVE GRADING with HDlink (or similar)
Being able to see live HD-SDI feed from camera and make ACS-CDL compliant color corrections to the image on the monitor with something like the HDlink. Right now we do this with tools like Panavision GDP and Truelight do this with special hardware, or we can use 3cP with the HDlink, or LinkColor is really nice and simple, but lift, gamma, gain wheels would be better than the nine sliders. Linkcolor can't receive an SDI signal, so it is limited. This is something that a Professional DITs needs in a tool to use on higher end shows.

the ability to save the live ASC-CDL grade data and be able to associate it back to the footage later via timecode or something. So if shooting the Alexa the DIT can grade the live HD-SDI signal coming to the monitors, later when the data manager downloads that card the grade is automatically applied to the footage. Maybe by keeping track of what timecode was being sent when a certain grade was done. Then being able to export an FCP XML of the footage with the grade applied. I think this could easily be done by applying something like Colorista Free to the footage with the ASC-CDL values of that grade. Or maybe your own plug-in. On a Red shoot it would be best if the grade was applied as Red Metadata. A way to take the live grade being done on set and automatically apply it, non-destructively, to the footage when downloaded is what all these software developers need to be working on.

3. LUTs
the ability to import LUTs and apply them to footage is required. Something like the LUTs created by the Arri website for Alexa. Also the ability to export LUTs of the grade we do in your software is required. That is where keeping simple with ASC-CDL compliant controls becomes helpful. Plus we don't need more control on set than that.

The ability to copy ANY kind of footage fast and safe to multiple locations and make checksums to verify the copies. These apps that only copy Red or only copy Alexa are a pain and the ones that don't do checksums are pointless. We have to deal with all kinds of cameras and sound and transcodes and LUTs and so much stuff that a super limited copy app is a waste. I had to write my own script to do this.

Most of us at the higher level of doing this work have fast systems with very large fast raids. I have a 20TB raid6 that is capable of over 900MB/s on my system I take on set. The fastest way for me to backup media on that system is to copy to my raid and then make other copies from my raid. Having the option to tell the data management software to make the first copy to my fast raid and then make other copies from my fast raid really speeds things up. Not everyone has this, so the option to tell the software the sequence of the copies is very useful. All the tools I have used that do one read and four copies from that one read all write to the speed of the slowest drive. So if I have a USB drive in the copy along with my fast raid, everything writes at USB speed.

I prefer the checksums outside of the folder that was backed up. I also like two checksum files with each copy. One with the ckecksums of the the source footage and one with the checksums of this copy. When I hand off the drive those files go with it, so everyone knows what the checksums are and they can verify the footage six months later if they have a problem. It is a little pointless have a copy of the source checksums and the destination checksum since they are identical, but Productions really loves that. It is something a producer can see and understand. The copy apps already do the work, so just write the results.

I prefer checksum files to be simple text files with one line for each checksum. Something simple like this:

c0b4c87e3f158ed166fbd5587ccde6a4 B043C001_110531_R1XH.mov
9982fa2845feafa717889d0d0056a4ec B043C002_110531_R1XH.mov
35e0a3d20ab91b215ce07ee559b5ce91 B043C003_110531_R1XH.mov

I just need the checksum and file name.

I also like to have one checksum from the read of the source footage and one checksum from the read of the copy. Something like this:


This way I can select those two files and hit my space bar to review those two files. Then all I need to do is quickly switch between them and I can easily see if there are any differences. I know the software does this, but I like to be able to see it.

Plus I like to be able to give production, this is from the source and this is from the copy. I know they should be the same, but most of the production people we have to deal with do not know anything about technology. They like seeing a file that says source and one that says copy.

I use .txt because it is easier to open and review files with .txt extension and not .md5

Also I don't like having the checksums in the folder with the footage. I put those files outside the folder and not in the folder with the footage. So if later someone uses a different tool to copy the files, it isn't copying and verifying my checksums.

I have recently heard of a company that is working on a XML standard for checksum files. has lots of useful stuff in there. One thing I found really interesting was using public key encryption to sign the checksum file. This way if someone modifies the checksum file you sent, you can prove that it is not the same file you sent. Sounds like overkill, but CYA is always a good thing.

Another thing that people really like about my copyscript is the log files it makes of every operation. Every time it copies a file or makes a checksum or compares a file or anything it logs the time and what it did. It even writes the results of every checksum in the log. That way I can always go back to my log to trouble shoot problems or to verify something if production calls me six months later. That data management log is very useful and I recommend it.

3cP has some great reports. More tools need reports that say how much footage was downloaded, if things were successful or not, where copies where made, info about grading, all the stuff that would be useful to someone not doing the work on set. I also really like the header slates in tons of useful info that 3cP can render into footage. More tool need to add this, so we don't need burn in on teh footage, but still get all the needed info and a lot more. 3cP has visual timecode in there info about the grade, before and after images, all kinds of good stuff.

Dustin Cross
06-08-2011, 12:40 PM
You have the ability to review footage. One thing the might be nice to add the some kind of color coding so that downloaded footage that has been played once from start to finish has a green box around it or something. Footage that hasn't been played has a Red box around it. Maybe a dashed green border for footage that has been scrubbed though. That kind of stuff would also be great to have in the log. The only way to know for certain the footage is okay is to watch it. Their are a couple proprietary software packages that do something similar to this. Basically an easy way to know if footage has been played or scrubbed or if know one has look at it yet.

it would be great to have the ability to auto sync sound based on timecode. Right now I use Sync-N-Link and it is great. Being able to do that inside the tool is pretty much required. Also the sound metadata is very important.

if you don't have auto sound syncing, I think rendering is almost not required. We need to sync sound most of the time. Sometimes we can render without syncing, but not all the time.

Red rocket support is great. It would also be great if you supported Matrox MAX for speeding up rendering h.264 files. Most of us use h.264 for our dailies and MAX greatly improves speed of that.

The ability to render DHxHD MXF files with ALE is desperately needed. That is the BIGGEST problem with Alexa right now it there are NO tools for making MXF files. There is Alexicc and I wrote my own shell script that is similar to Alexaicc using opensource tools. A proper GUI based tool people can use to transcode graded Alexa footage to DHxHD MXF files with ALE for Avid is desperately needed. We can make MXF files with Red footage, because Red wrote the software to do it. No one has done that for Alexa. The ability to make graded and synced MXF files for editorial from all cameras (Red, Alexa, 5D, 7D, Panasonic, Sony, GoPro, and even iPhone video) is desperately needed.

The ability to do multiple different renders for offline, for dailies, to marketing, for DVD, for iPad, for on-line. I think ColorFront has this working the best right now. BUt we need to be able to render lots of different formats. Also need to be able to name some of the renders based on the scene and take info, but retain timecode and the actual filenaem somewhere. Also renders must be in the background and need to me able to use all my CPUs. I don't want to have to stop working because I have a render going and it locks up teh app. Also I hate apps that batch render and only use on CPU. Most of us have 8 or 2 CPUs, max those suckers out. Use the GPU to render. Do what is needed to render as fast as possible and sill leave me a little processing so I can continue to work.

The ability to export FCP XML files is great. The ability to import and FCP XML could be useful. Add a way to add color corrections to the FCP XML and that becomes even more useful. We really want to be able to be on set grading the live feed on the monitor. Then send some kind of project files or XML or something to the data manager with my color data, so when he get download the media, the grade work is automatically applied to the downloaded footage. Also the ability to export for Avid is needed. Everyone makes tools for FCP, but professional editors don't use FCP, they use Avid, so a proper professional tool must support Avid.


Dustin Cross
06-08-2011, 02:57 PM
Thought I should add a few things:

being able to write to LTO would be great, but I don't see this happening unless LTFS starts being usable. There are just too many standards. LTFS seems great until you try to use it and then you realize it needs more work, but it seems to be going in the right direction.

3D - All of these applications need to be able to do the basic stuff of 3D.

All the metadata that Jody lists in his reporting would be great to have. The stuff Storm is doing seems nice, but all the technical data as meta data needs to be done. It would be great if Red made an iPad app (or someone else) for ACs that was a camera report that saved as metadata in the R3D. Scene take lens, height, stop, all that stuff as metadata in the file from an iPad app. I am really like the app Red showed at NAB so we can color the camera from the iPad. That is going to be AWESOME! I can be looking at my monitor grading the image to taste and the grade is saved as metadata in the camera WIRELESSly. PERFECT!!!! Make a new tab of that app for the ACs to input all their data.