Data rates and storage options

[Phil Becque]

Hi Folks, I'm so excited! And English men don't normally get excited - Phew!

Anyway, I was wondering what kind of disc space we'll be using for one minute of recording at the various output rates we can get from our RED's e.g. RAW, 4K, 2K etc, etc

I'm used to DVCAM and I know it's going to be ALOT more!

Any thoughts/numbers?

Thanks, Phil

[jbeale]

It is definitely not DVCAM (25 Mbit/sec). Last I heard, 24 fps 4k REDCODE is about 27.5 MBytes/s or 220 megabits per second, so a 320 GB Raid array (2x160 GB) attached to the RED could hold just over 3 hours of 24 fps, 4k material. See also: http://en.wikipedia.org/wiki/Red_Digital_Cinema_Camera_Company#Compression_.26_ Workflow

Comments from the Red team on this forum have mentioned that the Redcode compression scheme looks good even at lower data rates, but I don't know if they plan to make the datarate user-selectable.

[Phil Becque]

Brilliant! Thanks mate! I'm really new to all this high end stuff but it all sounds amazing!

What kind of storage option will you go for?

[Brook Willard]

Uncompressed RAW 4K = 323MB/s
REDCODE RAW 4K = ~27.5MB/s
REDCODE RAW 2K = [guessing] ~10MB/s
REDCODE RGB 2K/4K = currently unknown

RED FLASH magazine = unknown capacity
RED MINI RAID = 160GB
RED [fridge] RAID = unknown capacity

The smart money [<$1,000] is on the MINI RAID shooting REDCODE RAW 4K.

[Phil Becque]

Thanks Brook -

Why would anyone want Uncompressed RAW 4K = 323MB/s ?

How would you use that?

It doesn't sound possible to edit that kind of data rate?

I guess I'm missing something ??

[Brook Willard]

You never edit something like that online [read: at full quality]. You downconvert to a lesser format, edit that format and then conform back to your finish quality at the end.

Most humans won't want to shoot uncompressed RAW. Frankly, 323MB/s is chump change for 4K. It's the studios that would go uncompressed 4.5K [the maximum resolution and sensor area that the RED ONE is capable of] all the time.

Furthermore, the camera cannot compress 60p 4K onboard - it's just too much data. If somebody wanted 60p 4K, they'd need to go uncompressed. For the humans out there it'll probably come down to renting a fridge RAID for those days.

[Phil Becque]

Got it - thanks

Do you think it would be feasible to use a FireWire or USB 2 drive at one of the lower rates e.g. 1080p & 25fps ?

[Brook Willard]

No. As far as we know, you will need to purchase some form of media from RED in order to record [that is, until a 3rd party releases an option]. Jim has been adamant in stating that the storage pricing will be very reasonable. You will be able to offload footage onto those hard drives assuming you have a computer to negotiate the swap.

You could plug the SDI or HDMI output on the camera to a recording device and capture 1080p if you were so inclined.

[MikeCurtis]

I posted some speculation the other month at:

http://www.hdforindies.com/2006/12/mikes-conjecture-on-redcode-data-rates.html

...but it has been stated to be incorrect in some areas where I've been guessing, so only use as the vaguest of possible guidelines.

Read the article & updates for disclaimers, do NOT just skim the charts and think those are valid.

[Jeff Kilgroe]

Thanks Brook -

Why would anyone want Uncompressed RAW 4K = 323MB/s ?

How would you use that?

Woohoo!! - Up to 60fps in uncompressed RAW @ 4K (120fps max @ 2K), absolutely zero compression artifacts/issues (sounds like we won't have much of any with the wavelet based REDCODE RAW/RGB anyway).


It doesn't sound possible to edit that kind of data rate?

I guess I'm missing something ??

You need a massive RAID with massive bandwidth. Other than that, you're not missing much... Oh, it takes a pretty powerful computer too and anything less than a 2.6GHz dual-core with 6GB RAM need not apply. And even that's a stretch... Plan to do a lot of off-line editing by proxy or small sequence compositing/keying at full res.

We'll cope with it the same way people currently cope with digitized film res. Or the same way we dealt with HD resolutions a few years ago, which are mostly real-time now.

[Jannard]

is on the MINI RAID shooting REDCODE RAW 4K.

My personal money will be spent shooting 4K REDCODE RAW onto the RED DRIVE (mini raid) or REDFLASH onboard. At some point I can say the name of the person (and his team) that viewed 4K RAW side by side with 4K REDCODE RAW on a Sony 4K projector and guessed wrong which was which. I can see NO reason to shoot 4K RAW. REDCODE will be shown to be as important to the program as the Mysterium sensor.

Jim

[Brook Willard]

That's what I love to hear... I can't wait to see the current generation REDCODE... I'm sick of DPX filling up my hard drive.

[Thom Steinhoff]

...At some point I can say the name of the person (and his team) that viewed 4K RAW side by side with 4K REDCODE RAW on a Sony 4K projector and guessed wrong which was which.

This is fantastic news and can't wait to hear who.

Does his name rhyme with "Eter Ackson?"... or "Orge Ukas?" :)

[Blair S. Paulsen]

I heartily agree that a visually lossless wavelet codec that makes it possible to work with hi-res moving images using affordable and widely available NLEs running on commodity hardware is an awesome achievement. The more I talk to people about the RedOne camera system the more I am starting to think the next big hurdle after the creation of the sensor and the codec (and getting it working :) ) is educating people in our business what the heck it is and why they should care. Strangely enough, that may be the toughest challenge.

[Shawn Bannon]

I have to admit that the REDCODE RAW is just as exciting as the camera itself. It will allow all editing and processing (not finishing & output) to be done on standard hardrives if you wish. It really is the icing on the cake.

I'm not interested in archiving 33 terabytes (around 30 hours of footage) of 4k uncompressed for a feature. With Redcode RAW we could be down around the 3 terabyte area.

Now if there were an editing program to read the REDCODE RAW CODEC to save a little rendering time, is that even possible?

I'd probably still down-rez for editing because not much of advantage for my crew to edit in 4k anyway.

[Mike Devlin]

It is excellent that REDCODE RAW is visually indistinguishable from Uncompressed RAW. However, my understanding is that the real test (or "a" real test) for a compression scheme is how well it holds up through multi-generational processing in post production. Has there been any testing of REDCODE RAW in such situations to see if artifacts arise after multiple effects are applied? Of course, that would require comparison to the same effects applied to Uncompressed RAW.

I would be interested in seeing footage that had motion tracking and/or stabilization, wire removal, a time effect (I know, better done in camera, but...), and then color correction -- comparing REDCODE RAW to Uncompressed RAW.

I guess this is something we can all try ourselves when we get our cameras to see if Uncompressed RAW or REDCODE RAW is most appropriate for our application. Practically speaking I imagine that initially we will do most of our capture using REDCODE RAW.

[Stephen Gentle]

My personal money will be spent shooting 4K REDCODE RAW onto the RED DRIVE (mini raid) or REDFLASH onboard. At some point I can say the name of the person (and his team) that viewed 4K RAW side by side with 4K REDCODE RAW on a Sony 4K projector and guessed wrong which was which. I can see NO reason to shoot 4K RAW. REDCODE will be shown to be as important to the program as the Mysterium sensor.

Jim

Sounds great, but isn't REDCODE limited to 30fps in 4K and 60fps in 2K? High speed shooting might be one reason to output raw.

-Stephen

[Jason Francois]

High-speed would be nice, but my wife said if I had to spend as much on harddrives as the camera to do a feature that I couldn't have the camera. Come on REDCODE, do your thing. :)

[Gregory Karydis]

I have placed an order with my supplier for the Brand Sparkling New Seagate Savvio 15kRPM 2.5" SAS (http://www.seagate.com/www/en-us/products/servers/savvio/savvio_15k/) drives.
My idea is to make a portable RAID 5 configuration for when I'm travelling.
The deciding factor for going this way was that I could test-drive the technology and maybe find a way to use something simillar with my Red when I get it.
Heck, maybe a RAID 0 with a couple of these babies could work for 4k RAW.

I have ordered bulk so I will be getting a box of 10 drives to experiment on.

[Damien Molineaux]

It is excellent that REDCODE RAW is visually indistinguishable from Uncompressed RAW. However, my understanding is that the real test (or "a" real test) for a compression scheme is how well it holds up through multi-generational processing in post production. Has there been any testing of REDCODE RAW in such situations to see if artifacts arise after multiple effects are applied? Of course, that would require comparison to the same effects applied to Uncompressed RAW.

I would be interested in seeing footage that had motion tracking and/or stabilization, wire removal, a time effect (I know, better done in camera, but...), and then color correction -- comparing REDCODE RAW to Uncompressed RAW.

I guess this is something we can all try ourselves when we get our cameras to see if Uncompressed RAW or REDCODE RAW is most appropriate for our application. Practically speaking I imagine that initially we will do most of our capture using REDCODE RAW.

The idea with RAW, and REDCODE RAW, is to convert to your format of choice for editing offline (or online if you want a DVCproHD master for example), then conform your REDCODE RAW (like you would a negative) to make a master in the format of your choice, ie uncompressed RAW to then make a 35mm print for example.

As of today, REDCODE RAW is not designed to be an editing format, but who knows what the future holds?

Cheers,
Damien

[Brook Willard]

REDCODE RAW is RAW. Uncompressed RAW is RAW. We must be careful when using the term "RAW."

And one cannot edit RAW data without first debayering it. A RAW file has no image data.

[david farland]

Gee am I confused......

Redcine can take an array of RAW/RGB outputs from the camera, demosaic(if raw), color correct, resize/crop and then output to codec of choice? Is this all?
If you offlined in some low rez format, did all you edits/effects etc, can you then 'conform' to a Redcine format?
And to show how completely confused I am....Do you output this 'edited' Redcine file to some online format...dpx, tiff???? for final color correction or can you color correct on a native Redcine format.
Help!!
DF

[Rob Lohman]

Now if there were an editing program to read the REDCODE RAW CODEC to save a little rendering time, is that even possible?

Not really, you still need to do a high-quality debayer first etc. Unless you are talking about a lower resolution preview.

[digitalinnovations]

My understanding is that you output to whatever format you wish to edit in from REDCINE (i.e. DVCPRO HD for example) Then import that into your NLE for edit, color correction, etc. I am correct here?


Gee am I confused......

Do you output this 'edited' Redcine file to some online format...dpx, tiff???? for final color correction or can you color correct on a native Redcine format.
Help!!
DF

[Rob Lohman]

It is excellent that REDCODE RAW is visually indistinguishable from Uncompressed RAW. However, my understanding is that the real test (or "a" real test) for a compression scheme is how well it holds up through multi-generational processing in post production. Has there been any testing of REDCODE RAW in such situations to see if artifacts arise after multiple effects are applied? Of course, that would require comparison to the same effects applied to Uncompressed RAW.

You cannot do multi-generational processing with REDCODE RAW. That codec is input only. However, you could encode it out to REDCODE RGB which will allow you to do multi-generational tests. I think Graeme has done some tests and he was very pleased. The nature of Wavelet compression greatly helps with this.

Obviously you can output uncompressed (like TIFF / DPX) or lossless compressed (which is an option of REDCODE as well) files to avoid any generational loss.

[Rob Lohman]

Sounds great, but isn't REDCODE limited to 30fps in 4K and 60fps in 2K? High speed shooting might be one reason to output raw.

That is correct. With the RAW port you can go to 4(.5)K at 60 fps & 2K @ 120

[Rob Lohman]

As of today, REDCODE RAW is not designed to be an editing format, but who knows what the future holds?

You can't output to REDCODE RAW indeed. However, you should be able to load it into any QuickTime accepting application on Windows / Mac OS X.

[Rob Lohman]

Redcine can take an array of RAW/RGB outputs from the camera, demosaic(if raw), color correct, resize/crop and then output to codec of choice? Is this all?

In broad strokes, yes.


If you offlined in some low rez format, did all you edits/effects etc, can you then 'conform' to a Redcine format?

No, REDCINE is for RAW processing only. You will be able to export your original footage in a higher resolution and then use the normal conform tools to conform that to your offline. Or replace the lower resolution files with high resolution ones (in your NLE etc.) for the final render.


And to show how completely confused I am....Do you output this 'edited' Redcine file to some online format...dpx, tiff???? for final color correction or can you color correct on a native Redcine format.

You can do some early color work in REDCINE to get it in the ballpark. Final color correction in your NLE or post grading application. What format you use for a professional grading application depends on that application. Usually they accept Cineon / DPX / TIFF or other file sequences. If they accept QuickTime you could also export to our REDCODE RGB codec. However that might be slower than going to a file sequence if you have a (very) fast RAID array (depending on resolution of your files).

[Mike Devlin]

You cannot do multi-generational processing with REDCODE RAW. That codec is output only. However, you could encode it out to REDCODE RGB which will allow you to do multi-generational tests. I think Graeme has done some tests and he was very pleased. The nature of Wavelet compression greatly helps with this.

Obviously you can output uncompressed (like TIFF / DPX) or lossless compressed (which is an option of REDCODE as well) files to avoid any generational loss.

I understand that you don't process the REDCODE RAW itself (or Uncompressed RAW). Ideally we will export Avid DNxHD (or some other HD rez format) from REDCINE for off line in Media Composer and 4k DPX files for the on line. The question here is for the online conform how much difference is there between DPX coming from REDCODE RAW versus DPX from 4k Uncompressed RAW. Sounds like Graeme's initial tests indicate that we do not give up much just capturing REDCODE RAW, depending on how many layers he was dealing with. I imagine for effects-heavy shots it might occasionaly still be worth capturing 4k Uncompressed RAW, but maybe not!

[Phil Becque]

Hello again and thanks to all the contributors to this thread, particularly the RED team - it's been a real eye opener. I should say that my film making editing smarts are severly limited so if this is a really dumb question forgive me I often miss some crucial point:

What are the plans for RED-CINE in terms of existing HD Codec support for editing. For example the Edius Pro editing system I use: has a HQ codec for High Def - but as far as I know - that's only 1080p . I know that RED-CINE can downsample to 1080p but I'm still not sure - even if I wanted to do that what I'll have to edit with?

And then if I want to downsample to a higher res e.g. 2K or 4K what Codec support will there be for other editing systems?

I guess what I'm really asking is: will I have to get Avid or FCP if I wanted to edit something - say 15 months from now?

Many thanks for bearing with me on this one - Phil

[Nick Shaw]

REDCINE will encode to any codec installed on your system, as I understand it, so you should be able to encode 1080p in the Edius codec. As for 2k or 4k, if the Edius supports those, REDCINE can convert to any codec Edius can use.

Nick

[Phil Becque]

Great thanks!

[Gregory Karydis]

In other words RecCine is what TeleCine is for normal film cameras...
You feed the RAW to redcine, select the type of output and codec and redcine does the rest.
Did I get it right or confuse it even more?

[Chris Kenny]

You can't output to REDCODE RAW indeed. However, you should be able to load it into any QuickTime accepting application on Windows / Mac OS X.

So, REDCODE RAW can be played back, but not exported, from QuickTime? Is the codec smart enough that you can play back at a lower resolution in real-time from any QuickTime app, or is this really just more of a convenience for getting the data into QuickTime apps like Shake for processing, where real-time playback isn't a factor?

[Rob Lohman]

REDCINE will encode to any codec installed on your system, as I understand it, so you should be able to encode 1080p in the Edius codec. As for 2k or 4k, if the Edius supports those, REDCINE can convert to any codec Edius can use.

Keep in mind that at least for the moment it needs to be a QuickTime codec. If not you need to export to something else that Edius or the application in question can load.

[Rob Lohman]

So, REDCODE RAW can be played back, but not exported, from QuickTime? Is the codec smart enough that you can play back at a lower resolution in real-time from any QuickTime app, or is this really just more of a convenience for getting the data into QuickTime apps like Shake for processing, where real-time playback isn't a factor?

Realtime is a bit of a problematic word. That will highly depend on your hardware (no, we don't know yet) and certain other factors.

Yes, there will be options for lower resolution playback and such.

[david farland]

Say I import 4K Redcode RAW into Redcine, demosiac, then do some Red colour correction.....What is the highest resolution I can save to using the Redcine codec.........None, 1080p???

Or to keep in 4K, must I output to a non-Red codec i.e..dpx,tiff etc?

DF

[Rob Lohman]

If you are talking outputting it to our REDCODE RGB then 4K is definitely possible. I don't know if there are any other QuickTime codecs out there that support such a high resolution.

Obviously the still image file formats have no problems either.

[Lucas Wilson]

If you are talking outputting it to our REDCODE RGB then 4K is definitely possible. I don't know if there are any other QuickTime codecs out there that support such a high resolution.

Obviously the still image file formats have no problems either.

Hi Rob...

I have worked with several QT codecs at 2K, but have never tried at 4K. Realtime with QT at high resolutions is a big issue.

Cheers,

Lucas
------
ASSIMILATE, Inc.
Los Angeles (but right now... jet-lagged in Singapore)

[david farland]

I guess (meaning I know shit about this stuff..), the missing piece is getting these 3rd party NLE's to conform Red's REDCODE RGB (or RAW??) files.

DF

[Storyline]

How does the workflow compare with Cineform's? I experimented with their trial version running on PremierePro. It de-Bayers the footage to preview quality in realtime when playing and decompresses to full quality when playing stops - revealing the full-quality image for that frame. Corrections and FX processing (similar capabilites to what is being proposed by Red) can be viewed at full resolution on the parked frame. To see everything in full-quality, one hits the render key and comes back afterward. Is this the Red workflow for "off-line" to "on-line" or does it differ?

[MikeCurtis]

Some thoughts -

1.) All this worry and concern about 4K output/workflow/grading...very, VERY few projects are mastered at 4K - I think a lot of folks here may be worrying too much about it.

2.) There are only two ways to output a 4K file so that folks can watch it - either filmout or DCI Spec digital cinema masters (forget the exact term, but 4K digital projection masters to work with the DCI spec servers and projectors...and only then if you have a 4K projector, a very expensive piece of gear)

3.) 4K is really, REALLY hard to work with in realtime - most 4K grades are done at 1K or 2K proxy, and final rendering done non-realtime. And again - how are you going to monitor that signal at that res, let alone in realtime?

4.) If you wanted to, you could in theory do compressed HD or 2K using something like Color Finesse or Colorista in Final Cut, then use Automatic Duck to get it into After Effects for a high quality 4K render pipeline and maintain all the quality of the source 10 bit log or lin or 12 bit lin source

5.) As I've stated before, I'm betting the vast majority of stuff shot on Reds will get mastered to HD - and it'll look GREAT. 4K RAW (even compressed) downsampled to 2K or HD is going to look very clean and smooth. And even a feature, if shot 4K Redcode RAW, downconverted to whatever for offline mastering, then color corrected in 10 bit 4:2:2 or 4:4:4 1920x1080 or 2048x1080 will look really, really good (if well done, of course). And the nice thing about it? There are already good tools out there to get good results for mastering 10 bit 4:2:2 out there with off the shelf hardware/software that is relatively easy to use and doesn't cost an arm and a leg. But with extra work and software and time, you can even do 10 bit 4:4:4 pipelines (or even 2K/4K 10 bit log or 12 bit lin pipelines) on desktop class computers, lots of storage and a decent amount of render time.

The focus should be on good results, at the budget level you can afford, in the time you have. For INDIES, the quality difference to be gained between a 10 bit 4:2:2 HD master and a 12 bit 4:4:4 2K or 4K master is, in the end, up on the theatrical screen or high def Blu-ray/HD-DVD master, not so huge as compared to the effort/resources/skills/time required. Time/effort perhaps better spent on the cut, the FX, the marketing/promotion?

All that said, I'm personally dedicated to finding the best way to master all this stuff. Actually, finding the best solutions for mastering this stuff at various quality and price points, starting with SD and ending at 4K mastering off of Red originated footage, all at various points on the price/speed continuum.

...and (pardon the brief shill) I'll be making that info available to my clients. I want to be the Go To Guy on all this stuff.

-mike

[Steve Freebairn]

Great Post Mike.

My question is much more reasonable. (I'm not planning on editing 4k for a while :)

If I am targeting a 1080P master, and I output from Redcine to 480P or something else really simple to edit, What is the process of getting it onlined back at 1080P. Can I import an EDL into RedCine or do I need to manually get the clips that I used the first time and export them?

It would be really nice to be able to load up an EDL from any of the major applications Final Cut, Avid, Premier Pro and then just select the new output format of choice.

[Chris Kenny]

Great Post Mike.

My question is much more reasonable. (I'm not planning on editing 4k for a while :)

If I am targeting a 1080P master, and I output from Redcine to 480P or something else really simple to edit, What is the process of getting it onlined back at 1080P. Can I import an EDL into RedCine or do I need to manually get the clips that I used the first time and export them?

It would be really nice to be able to load up an EDL from any of the major applications Final Cut, Avid, Premier Pro and then just select the new output format of choice.

REDCINE won't actually do your conform, but it will, apparently, be able to load an EDL and automatically export just the clips you need for your final conform in whatever format you want.

[Michael Schrengohst]

(edited) I want to be the Go To Guy on all this stuff.

-mike

I notice on your website, available for consulting. So will you be available for consulting re: RED - Capture, edit,delivery? And are you getting a RED yourself?
Thanks

[Bmoreshaun]

I just put my reservations down for 2 red ones and the 300mm primes(more lenses to be rented)...I'm planning on shooting a feature(keeping but not being able to view my 4k master) by doing a(I've never done before) 2k(or lesser) offline edit in final cut(can you do 2k in final cut?)...Thus being able to have the ability to have my project blown up on film(or projected on the Sony) and still be able to send my usual trailers,etc quickly...My major questions are..."Keeping Cheap" in mind....

1: How much (terrabytes) storage will I need for say 15hrs using the redcode raw format(small sequences doing 60fps in RAW)(will have to have storage for both 2k and 4k files?,I'm sure...Could I go lesser format for an offline edit and have no problems going back to my 4k master... I was thinking of useing the red raid (on site) then download (on site) onto a raid system I can have built(would my mac book pro cut it for (on site) transfer, or would I lug the mac pro?)

2: What is the best monitor for editing(and feild)(keeping in mind wanting to blow back up to film)

3:I need to get an editing system(useing g-5 now:(...I'm looking at the mac pro(dual 3.0), say 8gigs of ramm ) and a Kona card....Again can final cut make the cut for 2k edit,& 4k master?...if not,how should I go?...

4: besides file storage(on site and editing), editing system, proper monitors, etc... is there anything else I'm forgeting?(certian softwear for getting any of these into final cut/shake/logic,etc....?), anything else I might not have thought of...Co$t wise...(on site and off)...besides normal production costs

[Rob Lohman]

It's all a bit premature to talk about this. Storage rates for RED-RAID have not yet been given and monitoring & editing hardware increase in features and drop in prices almost monthly.

As I have advised others: wait with purchase of computer gear, monitoring solutions etc. until at least after NAB. There will be 1000+ people getting their camera's before you. Use that time to see and learn which workflows and equipment work best for them. Then buy what you need and can afford.

To walk buy your questions:

1) unknown yet, however, onboard REDCODE RAW (4K at 30 fps should be around 30 MB/s). RED-RAID will have much larger data rate than REDCODE RAW though!

2) see my points above

3) see my points above, also new products / software releases tend to be announced at NAB (April in Las Vegas), so wait what gets released there

4) backup for archival? And if you do the full processing I guess you want to start looking into Blu-ray & HD DVD mastering for HD delivery? Unless you solely do TV or film out.

[MikeCurtis]

I notice on your website, available for consulting. So will you be available for consulting re: RED - Capture, edit,delivery? And are you getting a RED yourself?
Thanks

Red Guy - YES I specifically will be available for consulting on all these Red related issues, and YES I've got a Red reserved for myself (and a zoom) for rental.

-mike

[Gregory Karydis]

At some point I can say the name of the person (and his team) that viewed 4K RAW side by side with 4K REDCODE RAW on a Sony 4K projector and guessed wrong which was which. ... Jim
Awesome!
Would this (http://www.reduser.net/forum/showthread.php?p=10401#post10401) be the person in question?