Red and Blue-Ray

[Brandon Freeman]

So, I'm thinking about archiving issues, and I know there's a ton of different opinions, but I'm just wanting to know, if I have a 50GB Blue-Ray disc, how much RAW 4K footage would fit on that 50GB? As in, hours and minutes (most likely minutes)...

[Chris Forbes]

The REDDRIVE is 320 gb and will hold about 3 hrs of footage (not exact) we haven't gotten final confirmation on data rates yet. So your 50 GB Disk will hold about half an hour of raw footage. OUCH

[Craig Schober]

So, I'm thinking about archiving issues, and I know there's a ton of different opinions, but I'm just wanting to know, if I have a 50GB Blue-Ray disc, how much RAW 4K footage would fit on that 50GB? As in, hours and minutes (most likely minutes)...

figure about 25 minutes of 4k at 24fps per 50gb blu-ray disc. but the real issue is the cost of each dual layer disc and transfer speed. right now i'm seeing just under $1/gig and transfer speeds of just under 10mb/sec. that's just under 90 minutes to transfer 25 minutes of actual footage. not great but both will get better. hopefully soon.

[Brandon Freeman]

Okay. So, if I decided to shoot 2k instead, does the footage get recorded as 2k to the hard drive or is it still RAW 4k, and downconverted when transfered?

And if it does record in 2k, am I getting a smaller file size, or the same size, but a picture less compressed?

Sorry for all these questions -- I've studied the chart for a while now, but can't seem to get these questions answered.

[Brandon Freeman]

figure about 25 minutes of 4k at 24fps per 50gb blu-ray disc. but the real issue is the cost of each dual layer disc and transfer speed. right now i'm seeing just under $1/gig and transfer speeds of just under 10mb/sec. that's just under 90 minutes to transfer 25 minutes of actual footage. not great but both will get better. hopefully soon.

Wow. Looks like I'm still waiting to make a decision for what I'm shooting my next film with.

[Brandon Freeman]

Sorry for the triple post, but...

Isn't redcode a bit more compressed than RAW? Would it not be advisable to shoot 4k 24fps w/ REDCODE?

Confused noob, sorry.

ADDED SO AS NOT TO QUADRUPLE POST :) :

So, it appears that there are three quality formats. I don't know how I missed this. Am I correct, then, in assuming that REDCODE RAW (which is different from RAW, correct?) will be at about the same bit-rate as XDCAM HQ (35mbps)? (I understand that it's a different encoding style entirely, so I'm not worried about encoding 4k with only 30 mbps.) Then, would a 50 GB blue-ray hold about the same amount of REDCODE RAW as XDCAM HQ data?

I hope I'm not bugging anyone with all these questions...I'm just learnin'. Thanks for your time in advance.

[vanguy]

Brandon; I'll try to answer...

Here goes. Feel free to correct me, anyone.

Redcode is a way of compressing RAW files. RAW is not a compression scheme, but describes the unprocessed data coming off the sensor (as opposed to YUV or RGB)

It is not the same bitrate as HDCAM HD. Bitrates are quoted using upper and lower case letters, mainly to confuse noobs, i think.

b= bit
B= byte or 8 bits.

XDCAM HD =35 Mb/sec
Redcode = approximately 30 MB/sec or almost 8 times as much data.
If my math is correct, that works out to 240 Mb/sec.

So a 50 GB Blu-Ray wouldn't hold as much.

Incidentally, XDCAM HD 35 uses a high-rate compression scheme similar to HDV. Although it looks pretty good, RedCode will be much better, just going by the numbers. And about 5x as many pixels.

[Brandon Freeman]

Yeah, I just figured out the b and the B on wikipedia. Doh.

Wow. So I am really going to need to figure out the whole storage space thing if I do go with Red.



So, I guess that makes it about 1.5 GB per minute. Hm.

[vanguy]

If it makes you feel any better, it's about the same data rate as uncompressed SD video.

And check out the other threads. There's lots of ideas on storage.

[Brandon Freeman]

And the thing I have to keep remembering too, is this is a camera that would easily sell for 3x its price, yet is only $17.5k -- so after all the lenses and stuff, you spend around $50k or maybe a bit more for an entire set-up that in any other circumstance would be $500k or more.

Ho...ly...crap. Red is awesome.

[vanguy]

is that $17.5k or $17.5K?

[Joe Aurili]

Yea, I heard a rumor that they could have priced it at $5000, but it would have caused too many nervous breakdowns, so they up the price for the good of mankind.


And the thing I have to keep remembering too, is this is a camera that would easily sell for 3x its price, yet is only $17.5k -- so after all the lenses and stuff, you spend around $50k or maybe a bit more for an entire set-up that in any other circumstance would be $500k or more.

Ho...ly...crap. Red is awesome.

[Greg Voevodsky]

I'd wait for the holographic hard drives to become more affordable.

http://en.wikipedia.org/wiki/Holographic_Versatile_Disc

Until then, I'm backing up on 2 or 3 hard drives. :-)

[Häakon]

I'd wait for the holographic hard drives to become more affordable.

http://en.wikipedia.org/wiki/Holographic_Versatile_Disc

Until then, I'm backing up on 2 or 3 hard drives. :-)
We've been hearing (and waiting) for these things for years. Not that they won't ever see the light of day, but I'm not holding my breath...

[Vincent Rice]

There is no sensible archiving strategy at the moment that I can see apart from multiple hard drives. Tapes and opticals just don't cut it. Something the whole world is having to come to terms with.

[vanguy]

You can get a 2 TB drive from LaCie for $769 US. I'd guess you'd want about 10 TB to store a feature (100 hours or a 50:1 shooting ratio, to be generous), so double it for safety, and your storage costs are $7690. Shelf space is about the size of a medium beer cooler.

HDCam tape is about $4000 at the cheapest, DCVProHD is about $4900. Double that if you want a backup. HDV is much cheaper, but you don't want to shoot on HDV, do you? And film? About a quarter million. Without processing.

Now if you really want to save, you can buy raw drives for under .25 / GB, which would make your feature storage costs about $5K, with backup.

To save even more money, you could double-backup just the media in the final cut of the show (using the media manager), and single-backup the raw footage, and reuse the drives on the next show. That'd save you almost half. Three years later, it'd be a good idea to rotate your media, copying it to new drives, then rewriting it back. Good work for an intern, copying 20 TB of files.

But it all seems fairly doable to me.

[Joe Aurili]

Have you been looking in my plan file??? ;) Because that is pretty much my back-up plan for now.


You can get a 2 TB drive from LaCie for $769 US. I'd guess you'd want about 10 TB to store a feature (100 hours or a 50:1 shooting ratio, to be generous), so double it for safety, and your storage costs are $7690. Shelf space is about the size of a medium beer cooler.

[vanguy]

Someday there'll be this 73 cent plastic sheet you feed into your printer and it records a hundred terabytes on an 8 1/2 x 11 " sheet, and it does it in 49 seconds, and it lasts for eternity, surviving supernovae and three-year-old drool.

I long for that day.

[Joe Aurili]

And when that comes out I will back-up the HDs to it :)


Someday there'll be this 73 cent plastic sheet you feed into your printer and it records a hundred terabytes on an 8 1/2 x 11 " sheet, and it does it in 49 seconds, and it lasts for eternity, surviving supernovae and three-year-old drool.

I long for that day.

[Rob Lohman]

There is no sensible archiving strategy at the moment that I can see apart from multiple hard drives. Tapes ... just don't cut it. Something the whole world is having to come to terms with.

How do you figure? The IT world has been managing GB's & TB's of data on a daily basis through various systems like tape backups. Complete with automatic tape libraries & robots.

This stuff exists and is reliable. Of course you can backup to either an offline or online hard drive system, but that doesn't have nearly the same track record.

It might be cheaper to store it on drives, but I don't see why tapes "just don't cut it".

How did you think the IT world do their backups?

[Craig Schober]

How do you figure? The IT world has been managing GB's & TB's of data on a daily basis through various systems like tape backups. Complete with automatic tape libraries & robots.

This stuff exists and is reliable. Of course you can backup to either an offline or online hard drive system, but that doesn't have nearly the same track record.

It might be cheaper to store it on drives, but I don't see why tapes "just don't cut it".

How did you think the IT world do their backups?

i think we're spoiled. we've been so accustomed to fast random access at such cheap prices that many of us overlook some very real solutions like tape backup. my first experiences with tape backup was about 10 years ago when it was much cheaper than the hard drive equivalent storage. it was a hassle but i saved money using tape. now, especially to users like me, it's not so clear. why should i invest in $5000 tape machine at $75 a tape. why not just keep buying and building sata raids? they will only get cheaper, bigger and you still have that convenient and fast random access. i'm not up with the latest tape backup innovations but they don't seem to keep pace with more consumer-like solutions like hard drive. i think i'll just keep buying drives, occasionally archive to even bigger drives and hope the holy grail of hologram storage will finally surface one day. until then, at least i know i'm spoiled.

[feb31films]

You can get a 2 TB drive from LaCie for $769 US. I'd guess you'd want about 10 TB to store a feature (100 hours or a 50:1 shooting ratio, to be generous), so double it for safety, and your storage costs are $7690. Shelf space is about the size of a medium beer cooler.


Not to diminish the discussion of storage because it is a very important issue, but a 50:1 shooting ratio? OMG! This is what (affordable) digital acquisition has done to us. Back when I was in film school (and is was not THAT long ago) a 15:1 shooting ratio was considered excessive. That's because they taught us to use things like Storyboards, Rehearsals, and pre-viz. Granted we didn't have Hollywood budgets, but just because we are replacing the film equipment doesn't mean we have to replace all the film techniques. Many of them are still valid and beneficial.

[vanguy]

I picked 50:1 to be extra generous. Doc work can easily go that high. But I'm allowing for the Rodriguez style of "let the camera roll; tape is cheap" dramatic filmmaking, which is sure to happen when you're using disk media.

Speaking as an editor, I'd sure like it if my clients (most of whom use digital acquisition of one kind or another) adhered to those planning techniques. I usually get boxes and boxes of tape, all of which has to be viewed and sorted. The project I'm working on now has over 200 hours of source media on the drives. Fortunately, most of it is logged.

That said, Red allows you to do things you would've looked twice at before, like shooting the full scene from all the angles, and not aborting to save stock. Or trying a new camera technique knowing you can delete the take if it's bogus. And many image control options.

But it would be nice not to have to sort through all that media.

[Keith Nealy]

Large amounts of source material is where camera liketh XDCAM-HD shine because of the disk based file system. You can immediately see icons of your clips and delete unwanted ones. There is even software for your laptop to allow clip sorting, logging, and meta tagging on location.

If RED can eventually have this capability it will make life so much easier and and also cut into Sony's advantages with XDCAM.

Sorting though source material is drudgery. Now that video is file-based, technology can help ease the pain so you can put your time and money on the creative.

Keith

[oldphart]

... why should i invest in $5000 tape machine at $75 a tape. why not just keep buying and building sata raids?

It depends on how you rate reliability. Tape is the safest archival medium we have. It is also fast and compact, and for really big storage volumes it is quite cost effective.

Example: The LTO-4 (which costs rather less than 5000 USD) will store 800GB uncompressed per cartridge, and transfers 120MB/s. A 20-pack will cost around 2700USD, for a storage cost of 17 cents(US) per gigabyte.

The 16 TB storage (drive+1 pack tapes) is a bit over 7000 dollars. Compare this to the cost of 80 400GB SATA-drives you need for RAID-1 with lower reliability, and the cost of convenience is not negligible after all.

You can reduce the number of drives by using RAID-5, but with lower reliability and the risk that you will be locked in to a specific controller or software.

I use SATA disks for my home videos (and other personal stuff), but then I copy important stuff to other media. Even so, I am tempted to get a used tape drive even for home use. For work there is no alternative to tape backup.

http://www.tandbergdata.com/index.php?option=com_content&task=view&id=173&sid=1&Itemid=245&pdbl=1&a=8

[Brandon Freeman]

I picked 50:1 to be extra generous. Doc work can easily go that high. But I'm allowing for the Rodriguez style of "let the camera roll; tape is cheap" dramatic filmmaking, which is sure to happen when you're using disk media.


Well, I can tell you that I most certainly won't be thinking "disk is cheap" when shooting with a RED. :)

[vanguy]

I thought I'd break it down (because that's the kind of mind I have):

Tape: total backup time 30 hours for about 10 GB. You must be there to change tapes every hour or so. Cost $4K (one-time) for the drive. $1600 for media. Remount: 30 hours, attended. So you gotta add the cost of a tape monkey for every show. Or buy a robot ($$$$$).

Disk: Total backup time 37 hours. Can be set up to run completely unattended. Cost $3245 for firewire 800 drives, or $2400 for raw drives plus maybe $600 (one-time) for enclosures. Remount: instant for firewire, maybe a half hour for raw drives.

Disk may require double backup or RAID 5 parity storage which increase the cost by 100% or 50% respectively. But the price of HDD storage is dropping according to Moore's law, and will drop below tape in a year or so.

Both methods seem to have a shelf life of 3-5 years according to people's experience (including mine; I've had more DLTs go squirrely than HDDs). So both would require rearchiving every 3 years or so.

Seems like about six of one and half a dozen of the other, both cost-wise and reliability-wise, but HDDs are a lot easier.

[vanguy]

Well, I can tell you that I most certainly won't be thinking "disk is cheap" when shooting with a RED. :)

It's still as cheap as HDCam tape (almost) and VERY cheap compared to film. It ain't as cheap as HDV. But try to find a 4K HDV tape...:detective2:

[Vincent Rice]

How do you figure? The IT world has been managing GB's & TB's of data on a daily basis through various systems like tape backups. Complete with automatic tape libraries & robots.

This stuff exists and is reliable. Of course you can backup to either an offline or online hard drive system, but that doesn't have nearly the same track record.

It might be cheaper to store it on drives, but I don't see why tapes "just don't cut it".

How did you think the IT world do their backups?

I am completely aware of how the IT world do their backups. I am also aware of how much it costs, the time required, the personnel required, the storage conditions required and the complete pain in the arse that it is to restore something from say a year ago. Tapes and tape systems are not getting cheaper, discs are ridiculously cheap. I used to use tapes extensively and in my experience it is necessary to upgrade your tape system on about the same time scale as you would your disc back-ups making the supposed reliability advantage moot.

I think making tape archives make sense for an organisation where the information is rarely restored. These days I seem to be creating mountains of video and graphics data and I need to have instant access to that data for at least a couple of years, or more. Now in three years time do I archive that on tape or simply quadruple my online storage at minimal cost?

I think disc/file systems will be getting a lot smarter (ZFS?) allowing for relatively risk-free 'on-line all-the-time'. Tape's days are numbered.

[Rob Lohman]

I think we have three tiers of data access:

1) projects currently in the pipeline, probably want those on fast raid's or nas/san solutions with a high-speed network

2) completed or on hold projects you might want to keep online for a while. These could be moved to a larger but slower access system, probably still disk based though

3) you should also have a backup system in place for long-term storage and backup of the previous two tiers

When talking tapes I was a) responding to your comment that they don't cut it and b) I was thinking about tier 3.

I never said they were in-expensive or did not require maintenance (which any long-term archival solution needs!!!). The only thing I said is that it works and has been proven to be reliable (through various means).

Depending on what amount of data you need to store and how reliable you want that to be you'll pay a certain amount for a tape system. This can be as low as couple of hundred bucks for a DAT based system in a DELL PowerEdge server or go all the way up to hundred of thousands of dollars for a tape robot.

My main point is that you should not discard what has been proven to work just like that. Managing stacks of hard disks has its own problems.

However, personally I would probably be inclined to store it on both drives & tape to reduce the risk of either one failing. Of course the tape will be restored (like a verified copy to a hard drive) to verify it has been properly written.

In both cases you will need to check / maintain your archive to make sure everything still works and is sufficiently backed up.

You can create some pretty neat backup systems that automatically copy data from one tier to another (access getting slower and slower) to facilitate multiple backups.

[vanguy]

The one thing about disk-based shooting is that it starts to follow an IT model rather than a media model. Which is new for us media-only types, and sometimes just plain too much work. I can't really afford all the infrastructure for a three tier storage system and the time to manage it properly. I'm a one-man show.

I need something that approximates videotape or film; shoot, bring it online for editing, put it in the vault. Forget it.

My experience has been corporate/educational/documentary with some drama. The drama usually gets archived forever, almost never being brought back online after completion.

But the other stuff gets reused for years, sometimes decades. And you need to bring it online relatively quickly. I've had some experience with DLT backup in the past, and it was always faster to redigitize from the camera tapes. And if you weren't sure which videotape something was on, DLT would have taken silly amounts of time compared to scanning videotapes.

So I kind of wince at an IT Solution which involves thousands of dollars of investment in the kind of technology that normally goes obsolete in a very few years, takes many hours of management on every show, and is rather difficult to bring back online. And if I put "data management" as a line item on an invoice, my clients won't complain, they'll go elsewhere.

It has to be stupid easy. Putting a show on firewire drives fits that model. I'm not dissing data tape as a reliable, cheap long-term storage medium. Go ahead and use it. But it won't work for me.

[Ace]

Disk is still a linear media when you think about it. It just has a much lower seektime to tape. You want real random access, flash is it. The great thing about tape is that unlike disk, the data is scattered accross hundreds if not thousands of feet of media. Your eggs are not all in one basket. If You archive everything on one platter, and that platter goes, then your screwed (If storing on drives, you would have 2 mirror drives at the least anyway). On the other hand, if part of your tape physically gets damaged, you can cut and splice it like you would film and attempt data retrieval. It would be good if REDCODE could be stored as seperate frames so that any corruption in backup wont effect the entire file..

[vanguy]

I think someone's gonna have to come up with a way to cheaply, easily, and safely store the petabytes of data we're all gonna have in our vaults. Holograms? 1.8" 1 TB mini-drives? Ginormous flash RAM? Self-backing, error checking storage arrays?

[Paul Leeming]

My archival storage plan will center around the modern, ubiquitous 3.5" SATA hard drive, sitting on a shelf, wrapped only in one of those soft silicone protective bumpers and labelled with the relevant info. They will only ever be used twice - once to record the camera's footage to them and verify that it is correct, and a second time to transfer that footage to the editing system drive (again verifying its integrity as you go) to do with what you want (and make further backups if you are that paranoid).

I come from the IT side of the fence and having seen plenty of drive failures in my time, it basically boils down to this - unless it has a head crash while operating, the data on the physical platters will be fine even if the electronics board dies or some other mechanical part of the drive dies. This means worst case scenario (remember, we're talking IF you need to restore from the original archive for some reason) you take your drive to a recovery centre, pay your money and they recover the info to a new drive.

For my own footage (my movies and projects) I will have two archival copies of all the original data in separate locations. For my clients, I will retain one copy of the data and I will give them the second copy as their master. How they back up from there is their concern, but they can always come back to me for the other copy if needed. Everything will be on 3.5" drives, with the client paying for the backup copy I retain (which at roughly JPY 8,000 per 320GB SATA drive is hardly a huge cost in the scheme of things).

Neat, simple and for all intents and purposes providing a secure backup system.

[Harva Raj]

Guys, holographic storage system is HERE!
http://technology.guardian.co.uk/weekly/story/0,,2080840,00.html

The world's first commercial holographic storage system is launched this autumn, with the product able to store the equivalent of 64 DVD movies on a disc about the size of a CD.

Holographic storage offers extremely fast data transfer rates - currently up to 160Mbit/sec, though there are plans to increase this.

The first holographic products are certainly not mass-market - a 600GB disc will cost around $180 (£90), and the drive costs about $18,000. Potential users include banks, libraries, government agencies and corporations. (Filmmakers..anyone?)

The source material (such as a photographic image or video footage) is encoded with error correction and channel data, but instead of the encoding being carried out on a bit-by-bit basis (as with say, CD or DVD), around 1m bits are generated at one time, which are recorded as a single page. The process uses light from a single laser split into two - a signal beam, which carries the data, and a reference beam. Where the two beams intersect, the interference pattern (hologram) is recorded within a photosensitive storage medium.

[Jeff Kilgroe]

Er... InPhase is already selling their holographic drive with 300GB media. 600GB is due by the end of the year and 1.2TB media sometime middle of next. I stopped by their HQ (about a 25 minute drive for me) and saw the demo. I have to buy through one of their dealers, but I contacted two of them to get quotes. Both quoted full MSRP and wouldn't budge from it (those slimeballs... maybe I should become a dealer). But MSRP is $16,999 plus any cables and interface controllers that may be needed. That price gets the drive unit and one piece of 300GB media. InPhase is also selling the drive under other brand labels like AGFA.

160Mbps is not extremely fast... That's actually extremely slow. Only about double the speed of BluRay, which is not fast. Many tape systems are much faster than this and hard drive storage races past this quite easily. 160Mbits is only 20MB/s.

The InPhase Tapestry system as well as the Maxell holographic product, the media is very durable. However, it doesn't have a very long shelf life at this point -- roughly 15 years estimated as a maximum. The holographic film currently being used by these manufacturers degrades over time as badly as many of the original organic dye based CD-R media. InPhase has been very open about this and they are working on correcting this as a top priority.

[Curran Giddens]

I'm currently using a Blu-ray burner for my G5. I needed another burner since my Superdrive stopped working. To burn a 50GB BD-R DL at 2x speed takes like 90 min.! :sad:

[Jeff Kilgroe]

It will get better, obviously... Just think back to burning CD-R media with the first generation burners... 80 to 90 minutes a disc. But everyone thought it was the coolest thing ever -- we were writing to a CD! Wooohooo! I paid almost $1800 for my first CD writer and blank discs were $14 each. So, BluRay isn't much different. Actually cheaper to start out with. Prices will drop and performance will improve. I just hope Sony doesn't use their format licensing to control BD media prices with the lame excuse of controlling piracy. Which they have already started to do in some sense. Not cool...