Click here to go to the first RED TEAM post in this thread.   Thread: Komodo REDCODE RAW Explained

Reply to Thread
Page 1 of 8 12345 ... LastLast
Results 1 to 10 of 74
  1. #1 Komodo REDCODE RAW Explained 
    Moderator Phil Holland's Avatar
    Join Date
    Apr 2007
    Location
    Los Angeles
    Posts
    11,830
    Hello there humans and occasion spam bots,

    Lots of chatter surrounding Komodo and how REDCODE RAW works, wanted to make a dedicated thread rapping about it. Today I'll put on my RED Whisperer hat and drop some light nerdy stuff in here, but will keep it wrangled to simple concepts, and I'll spice things up with my personal perspective.




    What is REDCODE RAW?

    REDCODE RAW is a compressed RAW codec that provides the flexibility of 16-bit digital cinema imaging while keeping file sizes down with compression. Over the years REDCODE RAW has advanced a great deal as the RED cameras have evolved and there are layers to how this technology works. Over the years RED deployed a wavelet based (Discrete Wavelet Transform/DWT) scheme for REDCODE RAW and with the release of the less power hungry and compact camera, RED is using a Discrete Cosine Transform. This change is likely for a multitude of reasons and I'll attempt to provide some perspective, fun history, and what this all means when it comes to image quality and workflow.


    Storage Costs and the Film Industry!
    When I began in the industry in the 90s we were indeed scanning film in at 4K and typically targeting a 2K finish in full uncompressed 16-bit Cineon Log. This evolved to 6K and beyond scanning throughout the 2000s as the emergence of digital cinema began. Early days of uncompressed digital cinema image acquisition in RAW were somewhere between a comedy, a tragedy, and horror film. Carts with mountains of drives, cables that could support bridges, and more issues than any comic book.

    One thing was apparent as digital workflows evolved. Storage costs were going to be "a concern". New drive types, connections, data rates; all sorts of things to take into consideration and you had to pay to play.

    Price per 1TB in the last 30 years:
    1990s = $12 million to $15,000 USD
    2000s = $15,000 to $7,000 USD
    2010s = $7,000 to $70 USD
    2020s = ?! Cheapest I can find currently is in the realm of $15-$36 bucks per TB as we step into the unknown.


    So the big storage drop in the 90s and technology advancing opened up some possibilities. For the film industry where we purchased film in 400', 1000', and 2000' foot cans plus all the courier, development, and print run costs as well as the workflow ecosystems to support it; digital storage coming down in price played a huge role both for film and digital workflows.

    Also, handling digital footage, specifically uncompressed 16-bit stuff and expecting to work efficiently with it back then was rough. This is before GPU acceleration and a lot of the thread optimization and the proxy workflow was truly born of this world before the job title of DIT even existed.

    Anyways, the questions were there. Good digital cinema cameras ranged in costs of up to over $300K at the time. RED found a way to make a digital film alternative with a 4K camera that recorded to a high quality digital RAW format. RAW is key for professionals and flexibility with it's lossless workflow which we all know so well now, specifically at this time there was a lot of crap MPEG-2 lossy codecs. Good for filming your vacation, not good for a paying audience in theaters. REDCODE RAW was a key innovation and truly slapped a fresh coat disruption on the walls of the industry that was going in a few different directions at the time.


    Wavelet Versus Cosine

    Looking back at interviews and some of the conversation on REDuser over the 2000s some of the reasons that RED went with a wavelet based compression scheme were apparent. At very high compression ratios you could maintain a very high quality image at the cost of encoding and decoding. This is why the early days of working with REDCODE RAW were a bit frustrating on the post side. However, major, and I can't stress this enough, MAJOR computational advancements came with faster CPUs, more processing cores, RAM, storage technology, and GPU technology.

    I'll underline GPU tech here as the two major improvements in my mind have been the implementation of GPU Acceleration, which enhanced playback a bit, but truly changed the world of what the time commitments were in rendering your footage to a deliverable codec or image sequence. When i got my first RED footage at my old studio I built a dedicated machine to render nonstop as footage came in. And it would render all week to uncompressed image sequences and was actually what we referred to as our digital scanning workflow back then.

    RED explored dedicated GPU acceleration for a while via RED ROCKET and RED ROCKET-X cards which also helped on the render and decode. But the most recent ground breaking advancement in GPU tech has been the implementation of GPU Decode Acceleration. Which is why i can playback REDCODE RAW 8K footage even on mobile workstation.

    Back to Wavelet encoding and decoding. Pluses, small file (REDCODE RAW allows for up to 22:1 compression ratio!) sizes great image quality. Minuses, needs a super computer shoved into a tiny camera box and all the supporting tech surrounding workflow I mentioned above. Bare in mind much of the conversation on REDuser early on was best practices and compression ratios for film production and I was heavily involved with some of the tests for recommended practices. Generally we worked at 5:1 through 8:1 REDCODE RAW with DSMC and DSMC2 and as the image quality and resolution increased that opened up new worlds for things like 10:1 and 14:1 to be downsampled to a 4K or 2K finish. During the era of DSMC2 I pretty much from day one said 12:1 should likely be your highest compression for cinema production. And most higher end through mid-range projects truly just have been riding the lowest compression ratio possible for your frame rates and resolution choices during the last several years.

    It also should be mentioned that variable compression ratios have allowed for high frame rate filming while still keeping file sizes down as there is a data rate ceiling for whatever goes on inside a camera brain.

    Cosine encoding, chunkier though easier to encode and much easier to decode. Cons, limited compression ratio range and certainly not doing 22:1 to get cinema grade images. But you will have some reasonable range to expand recording times. And the benefit, and I mentioned it early on when I was looking at Komodo 6K footage, I was able to turn my GPU acceleration off and get realtime 6K REDCODE HQ playback. That will allow more modest hardware to be used, speed up encoding and decoding, and on the camera side opens up the world to smaller and less power hungry professional tools.

    I've mentioned many times that all of what we do is a balancing act between our wants, desires, and needs. Often there are pros and cons. I imagine for RED they have a bunch of costumers chatting with them on the daily, which is why they pushed so hard for the GPU advancements, which have in effect changed our entire industry in how we work with this material.

    And not to get too spicy, but yes, Apple has chosen a DCT scheme for their ProRes RAW


    Personal Perspective

    The Douglas Adams in me will always go with the "Don't Panic" perspective first despite a fairly large audience of panic prone peeps abound. This is still REDCODE RAW and the maturely developed and globally adopted REDCODE RAW workflow. In the case of Komodo this is part of the equation that has allowed this camera to be so small, achieve this level of image quality, in a camera system that uses far less power than it's bigger brothers.

    Just to highlight that specific point, a 150Wh V-Mount brick gets me this much up time currently:
    - RED Monstro 8K VV = 2 hours and 20 minutes
    - RED Komodo 6K S35 = 6 hours and 8 minutes

    This is also in a camera that is typically rigged at 1/3rd of the weight of my typical DSMC2 shooting package. That's sort of what Komodo is all about. Small, built like a tank, runs forever. Doesn't have the motor or octane to do high frame rates or Dual Record, but the big cameras have big fans and lots of little robot brains for that reason.

    I've said it a bunch in the last several years as things became pretty clear to how this industry is moving forward. Don't expect data rates to go down on professional level image acquisition systems. At least, not too much. This is mainly because we are always looking for high quality image capture. More Dynamic Range, More Color, More Resolution, More Frame Rates, More features (Dual Record y'all), etc. Where you will see a high level of optimization and shrinking sizes is in delivery codecs like H.264, H.265, VP9, and now AM1 and H.266 which are the new cool. Optimized code, hardware, and more coming for those new bad boys on the block which will lead to audiences getting better quality footage in home or wherever/however they watch.

    Remember, one of the selling points on DSMC2 was/is it's higher data rates compared to DSMC (same moving up from the RED One as well) and yes it does have an impact on image quality. Yes, budgets aren't going on, and thankfully storage prices are going down. But you are investing in a 16-bit digital cinema camera and if that level of color and dynamic range performance is important to you and what you create, that's where the fun is here. And I'd say that's going to be pretty important moving forward as HDR achieves more adoption and deployment.

    Also, certainly a growing trend on more efficient workflows. Not everybody is running a movie studio even if they want cameras that can shoot movies.
    Last edited by Phil Holland; 07-23-2020 at 02:34 AM.
    Phil Holland - Cinematographer - Los Angeles
    ________________________________
    phfx.com IMDB
    PHFX | tools

    2X RED Monstro 8K VV Bodies, 1X RED Komodo, and a lot of things to use with them.

    Data Sheets and Notes:
    Red Weapon/DSMC2
    Red Dragon
    Reply With Quote  
     

  2. #2  
    Moderator Phil Holland's Avatar
    Join Date
    Apr 2007
    Location
    Los Angeles
    Posts
    11,830
    What's this mean for Komodo?

    At the moment, RED Komodo 6K S35 has REDCODE RAW in HQ, MQ, and soon an LQ. Data rates are not fully locked for LQ as RED is tinkering. Early orange birds like me have been shooting HQ and MQ. How this effects data rates is sort of yet to be seen, but you do get access to higher frame rates and longer roll times as you go from HQ to MQ to LQ.

    For the numbers folks, and again, things are still being tweaked via firmware updates even now as Stormtroopers are going out into the wild and this is purely based on my testing as of today.

    REDCODE RAW HQ is about the same data footprint of 2.65:1 - 970GB/h
    REDCODE RAW MQ is about the same data footprint of 3.08:1 - 834.5GB/h

    And I guess just for fun, 6K 24p 16-bit uncompressed is about 9.38TB/h.

    To wrap up, DCT and DWT are encoded differently and you do actually generally speaking want a higher data rate with DCT. It's not so cut and dry thinking about things in simple ratios and what we've grown used to over the years. And I'm certain there's lots of developments occurring as Komodo's ecosystem matures. I mean mine is still in toddler phase, though it seems to have discovered how to fly recently before even walking.

    So far, I've seen people just rock MQ mostly. At the moment that's how Komodo even defaults. Not really seeing any visual hit there so it's sort of the I guess 3:1 versus 5:1 mentality I have now perhaps, though different actual numbers. I'm expecting LQ to actually be a bigger data rate gap once it's put into the cameras. It would be interesting if RED gets it to 5:1-6:1-ish rates to get about double roll time on media.

    And yep, at the moment Komodo uses inexpensive and readily available CFast 2.0 media, which at the moment tops out at 1TB capacity. So thickest data rate scenario 1TB an hour basically. But without spoiling anything, I have indeed recorded to something larger in capacity, you know, for fun.


    Alrighty, hope that helps answering all the guessing and speculation. I'm certain there's other logical reasons RED did this on a technical hardware and image quality levels as well, but this is what is allowing Komodo to do what it does the way it does without being bigger, heavier breathing, and hotter running.

    It's my birthday today. Going to enjoy my pandemic isolation cake and shoot some footage.
    Phil Holland - Cinematographer - Los Angeles
    ________________________________
    phfx.com IMDB
    PHFX | tools

    2X RED Monstro 8K VV Bodies, 1X RED Komodo, and a lot of things to use with them.

    Data Sheets and Notes:
    Red Weapon/DSMC2
    Red Dragon
    Reply With Quote  
     

  3. #3  
    Senior Member Bastian Meier's Avatar
    Join Date
    Apr 2013
    Location
    Graz, Austria
    Posts
    331
    Thanks a lot for the nice write up.
    Enno your birthday and I hope we see some footage of the isolation cake soon :)
    Reply With Quote  
     

  4. #4  
    Senior Member Robert Hofmeyr's Avatar
    Join Date
    Sep 2009
    Location
    South Africa
    Posts
    763
    Happy birthday Phil! Thanks for this useful explanation.

    Do you know how the quality of Red's DCT compares to their wavelet compression? ie what DCT compression ratio would i need to use to get similar quality to wavelet 8:1.

    Interesting that you get higher frame rates with more compression. I have not seen that mentioned before. Can you get higher than 40p in 6K FF?
    Reply With Quote  
     

  5. #5  
    Senior Member Patrick Tresch's Avatar
    Join Date
    Aug 2007
    Location
    Switzerland, Lausanne
    Posts
    5,273
    Thanks for your thoughts and post. Happy birthday Phil.

    Patrick
    Reply With Quote  
     

  6. #6  
    Senior Member
    Join Date
    Apr 2008
    Location
    Costa Rica
    Posts
    874
    Happy Birthday, Phil! All the best.
    Reply With Quote  
     

  7. #7  
    Senior Member
    Join Date
    Oct 2009
    Location
    Los Angeles
    Posts
    251
    Happy Birthday!
    Ranger Gemini
    Reply With Quote  
     

  8. #8  
    Senior Member
    Join Date
    Mar 2008
    Location
    Gotland, Sweden
    Posts
    112
    Happy Birthday Phil!

    I listened to Jarred on a recent video, using the low storage cost of today as an explanation of how the thoughts have been during development of Komodo, just like Phil here above. There is one thing that is bothering me though and that is the maximum length of a take. I know Komodo is thought about as a special kind of film camera but everyone knows that it will be used as an A-camera by many, thereby being used for documentations and such. I think that a maximum recording time of a few hours on one storage disc would be desirable.
    Reply With Quote  
     

  9. #9  
    Senior Member Alvise Tedesco's Avatar
    Join Date
    Dec 2006
    Location
    Milan Italy
    Posts
    588
    Happy birthday! And enjoy your pandemic isolation cake. We had 3 (kids and wife) in April, here in Milan
    Reply With Quote  
     

  10. #10  
    Senior Member
    Join Date
    Jul 2018
    Location
    Boiling Springs, PA
    Posts
    195
    This is awesome Phil, thank you and Happy B-day! I am very happy to know that HQ/MQ is indeed very low compression (2.65:1 - 3.08:1) and helps to explain the data rates / recording times. With my Gemini at 5k (2:1) @ 24fps I can get 26min on a 480gb mini-mag and at (4:1) I can get 52min so Komodo is looking to be on par even at 6k res!
    Last edited by Andrew Reese; 07-23-2020 at 06:57 AM.
    DSMC2 GEMINI + STORMTROOPER KOMODO
    Reply With Quote  
     

Posting Permissions
  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts