Volume Recover on an 8GB Red Flash Card [Archive]

Morning Glory

11-14-2007, 12:14 PM

Sorry, this thread should be titled, Unerase on an 8GB Red Flash Card

On day 5 of 5 we have one camera roll missing, and nothing on the card except the redone formating files, and nothing on the harddrives we are copying to. Seems like it could be user error that erased the card before copying. Or something else entirely, but either way it's not the question I need answered.

The empty card was not used again, so, I'm assuming the same theory of hard drives applies where when you trash something the data is not lost, only the reference to it, until new data is put on the drive. If this is true, I'd love to hear peoples favourite volume recover software for the mac. Or if I'm totally off on my theory. Some unique takes were on this roll, so any help is greatly appreciated.

jbeale

11-14-2007, 12:54 PM

I know there is a lot of software made for situations of this kind (usually it's been formatting a CF card that held DSLR stills). googling "CF undelete mac" gets some hits, I'm not a mac user so I can't comment on those.

Babu Kantamneni

11-14-2007, 12:54 PM

Greg M

11-14-2007, 01:05 PM

we did some tests with most of the commercial software out there and couldnt find one that worked on the Red cards....seems most are looking for jpg or tiff files.

The one app that seemed to work was File Salvage, although it crashed more than it ran.

Damien Molineaux

11-14-2007, 01:24 PM

You may want to give "Data Rescue" a try. I've used it before to recover data off a hard drive (never a CF though) and it worked rather well, more info here :
http://www.prosofteng.com/products/data_rescue.php

Best of luck,
Damien

spitfire

11-14-2007, 01:27 PM

can only offer advise on a product that has saved my butt more than once, easily recovering files off several different makes of cards.

It even picked up files from one or two formats back, provided a new file didn't overwrite the real estate they occupied.

http://www.lc-tech.co.uk/software/rescueprodetail.html

We've tried most of the obvious flavors of recovery and RescuePRO is by far the best we found.

Good luck.

Jeff Kilgroe

11-14-2007, 02:33 PM

I spent weeks trying to recover a P2 card that got quick-formatted in-camera with no luck. The problem is data recovery software doesn't know what it's supposed to be looking for. Most recovery applications are good at finding common files like Word, Jpeg, PDF, etc.. Most known file types. But recovering MXF files (or R3D files in this case) isn't going to happen without some help.

I don't recall the program's name, but I ran across one recovery software on the Mac that allowed the user to supply file header information so that the application could then use that as a template in its search. Unfortunately, it didn't have any provisions for files that keep header other pertinent info at the end of the file.

My advice in this situation is don't do anything with the card just yet. Don't even run data recovery software on it just yet. The first thing you should do is use a program that will create a bit for bit image of the entire 8GB card as it exists. Once that image is created you can then use recovery software and experiment on copies of that image and you can go back to using the original CF card.

I think in the not too distant future, a data recovery program that understands R3D files and how RED formats CF cards can be made. So if you accidently wipe a CF card, (quick format, not a full format) you could theoretically run the "RED OOPS" software and get your files back.

Some of your high-priced data recovery expert services could probably get your data back. They may need some technical info on R3D files though to isolate the files out of the jumble of bits. Could get expensive and there's no guaranty.

Brent J. Craig

11-14-2007, 05:52 PM

Garth, I have a bunch of methods I use the recover files of all types. If you are still in trouble tomorrow (Thurs) give me a call and I can give it a shot.

Lexicon

11-14-2007, 06:16 PM

If you have access to a Windows machine, try ZAR or some other program that specializes in FAT data recovery. ZAR has some great advanced functionality that might be useful in your case. It's possible you can recover the files by rebuilding the entire FAT structure of the card (most data recovery programs just go after files) but corruption is a possibility.

Morning Glory

11-14-2007, 06:50 PM

Thanks for all the suggestions, comrads, I have tried file salvage, and it found some of the files (all the circled takes anyway!!!) but it maxed out each file at 500MB. Which as you may conclude is too short for most takes. Is this an inherent limit in file salvage? is there a way to change it? The undelete didn't work as R3Ds are not a recognized format, but in expert mode i was able to drop in dozens of R3Ds for it to analyze as a file format it could support and found 5 files, but presented them as 500MB each and they all end before the take is over.

Thanks Jeff for the back up suggestion. i am now using carbon copy cloner to make a few test copies. Thanks to Lexicon for the PC suggestion, the FAT 32 format made me think PC machines might offer more robust solutions. Perhaps I can take a test to a PC machine.

Brent, Thanks for the offer. I may well need your help tomorrow. Please PM me with your phone #.

I will keep you all posted, as I can only assume this is a situation that may well happen to anyone at some point.

Mark B.

11-15-2007, 12:59 AM

I've had good luck with R-Studio, but then that's a Windows application.

Brent J. Craig

11-19-2007, 11:08 AM

Update:

I downloaded and tested 6 different Windows file recovery programs. Since I didn't have a Red CF card or R3D files, I used a normal CF card with JPGs from my digital camera and did a quick-format to erase them. Only 4 of the programs could even see that the card had contained JPGs, and only 3 actually recovered them.

Recovered JPGs:
-Data Doctor Recovery for FAT
-Data Doctory Recovery for Removable Media
-GetDataBack for FAT

Saw filenames but couldn't recover:
-R-Studio

Did absolutely nothing:
-Ontrack Easy Recovery Professional
-Runtime Software Data Recovery

I met up with Garth on Friday and tried the 4 best file recovery programs on both his original card and on a bit-for-bit copy.

The Data Doctor programs found several files from a previous format and wanted to recover those. They recovered what looked like the .R3D files and the Quicktime proxies with the proper filenames and the expected sizes, but none of them were openable in RedCine (PC and Mac), RedAlert or Quicktime.

On one run they found a filename of one of the deleted files we wanted but the recovered file couldn't be opened.

GetDataBack found nothing
Easy Recovery Pro found nothing

The Windows file recovery route seems to be a bust.

One important note: When a Mac formats a card, it seems to immediately create the folders Trashes and Miscellaneous. This means it starts writing to the card right away, possibly overwriting bits of the files you may want to recover. Would it be smarter to delete the files on the card rather than formatting it?

As Garth mentioned, File Salvage on a Mac was successfully recovering files but stopping at 500Megs. He was in contact with the program's makers and seemed to be getting prompt replies. We will see how he made out.

I tried. It was a good learning experience.

Mike Prevette

11-19-2007, 11:17 AM

I''ve always dissed reformatting in OS vs on device. I take it this scenario they formated on the mac and not in camera? I wounder if the results would be different if done the other way around.

_mike

Morning Glory

11-19-2007, 02:11 PM

We always formatted in Camera. It was one of our failsafe devices, the files would be deleted on the mac, then the camera would reformat the card. If there were files on the card camera would send it back to the DIT. In this case, the situation of the missing roll was discovered before the card was reformatted in the camera.

Thanks to Brent from giving it the old college try.

As of now I am still waiting to hear back from the developers.

Cam Crowley

11-19-2007, 04:13 PM

I recently came close to losing all the data of a 320GB FW drive when my powerbook's drive became corrupted. Was staring down the barrel of losing months of work.

After unsuccessfully trying other methods (eg: Tech Tools and Nortons) I found DiskWarrior to be a really good program for recovery of files that have not actually been lost but cannot be accessed. It has a really good 'scavanger' mode and was able to recover all my data - including 6GB+ QuickTime files. I've checked most of the recovered files (roughly 250GB of the FW and 70GB of my now junked PowerBook) and absolutely everything has been intact.

DiskWarrior saved my arse - it took a few days and a few goes to get everything off the drives but all was good in the end and I learned quite a few important things in the process. Along the way I stumbled across a quote which I would like to share with you all because it has become my 'words to live by' when dealing with data:

"If you do not have both a backup and an off-site backup of your files, you do not own them. You are simply borrowing them from Fate."

Best of Luck - hope you get the files back.

Cam

Deanan

11-20-2007, 12:42 AM

When prepping cards to be cycled back to the camera, I usually just
removing the two 'profile' files. That way, if I need to get files
back for whatever reason, they're still there. The camera will see a card
without the profiles as unformatted and will ask if you want to format.

Brent J. Craig

11-20-2007, 06:08 AM

When prepping cards to be cycled back to the camera, I usually just removing the two 'profile' files

That sounds like an excellent practice!

I Bloom

12-11-2007, 08:44 PM

Disk Warrior is no good for this problem. It does not support FAT32.

IBloom

Casey Green

01-31-2008, 03:29 AM

We've been doing a lot of tests with this very issue and are still tryingto find some answers.

Has anyone been able to recover a .r3d file from a CF Card?

The thought of "what if a card gets formatted accidentally" or "what if a card gets corrupted during mounting / unmounting" has been talked about, and even thought it is a rare possibility, I think it would be great for RED to released a utility for both PC and Mac to scan the card and recover the files.

The closest I have gotten so far on the Mac has been scanning with Virtual Lab and Boomerang Data Recovery software. Both seem to show the .r3D files, and Virtual Lab even lets you enter in the file type and size limitations, since most recovery programs only look for typical files such as .txt and .jpg, etc.

The only problem is that these programs then charge quite a bit. between $175-$200 for 8GB of data recovery. And that isn't even gauranteed to work.

So I would like to know what anyone else has tried.

It seems to make sense that a formatted card or one that lost the header information still should have the rest of the .r3d files intact and be recoverable.

thanks for any help, as we want to be prepared for anything for our clients. :)

Casey Green

02-18-2008, 01:51 AM

Has anyone had any other success at this?

With such important data only an accidental format away from being lost, we would like to offer our clients the peace of mind that RED offers a way to scan for .r3d files to possibly get them back.

Would a simple utility be possible to offer us?

Maybe someone on the RED TEAM can chime in here?
Is this possible to create a scan and recover utility for .r3d files?

Dylan Reeve

02-18-2008, 02:04 AM

I'll try and take a look at this issue when I can in the next week or so, but I don't have cards available to test with at the moment.

One point is that the Mac is messy. From a network management point of view it's a huge pain it the ass, it creates all sorts for metafiles wherever it goes. I have directories on my server full of Mac dot-files. With P2 it's often recommended to write-protect the cards before putting them in a Mac, as it immediately writes various folder information onto the cards as they are mounted I believe. I think the same will happen with the RED cards.

Only deleting the Profile files is a good approach, because it means that if the Mac does write more data on to the card it can only write over the sectors previously occupied by those files, rather than writing over sectors previously occupied by RAW data.

[blatant plug]
This is almost exactly the sort of situation we're creating the SolidStore to avoid... I know first hand how easy it can be to get a little out of whack when copying files over and over again all day. Everything starts to look the same.

I think human-error is by far the biggest risk to data on set. It's so easy to make a mistake, and once it's made it's really hard to fix it.
[/blatant plug]

Dylan Reeve

02-18-2008, 02:15 AM

A few things I've read (in the last few minutes) suggest this tool is very effective in many cases where others fail:
http://foremost.sourceforge.net/
It's a linux application.

I will try to build myself a test image to test with tomorrow. Can you confirm exactly the circumstances? You mounted card on Mac. Selected entire contents of card and dragged to trash? Then emptied?

Brent J. Craig

02-18-2008, 07:16 AM

With P2 it's often recommended to write-protect the cards before putting them in a Mac, as it immediately writes various folder information onto the cards as they are mounted ...

If someone were to make a Read-ONLY compact flash reader they would sell a million of them! (Every Red user, every still photographer)

Dylan Reeve

02-18-2008, 11:05 AM

It would probably be possible to implement at a driver level in the operating system... You can certainly write-protect a device once it's mounted, but it may be too late by then.

Alternatively I suspect it's possible to make the default mount option Read Only in OS X. I might look into that.

Casey Green

02-18-2008, 11:26 AM

In our case, we are doing a test with a formatted card. Some clients have been curious about this and we would like to offer them a solid answer.

In a scan with Virtual Lab (made by BinaryBiz.com), the .r3d files showed up and looked to be the correct sizes. But the way Virtual Lab works, they want a license fee + you purchase a quota of MB or GB for a certain $ amount.

For an 8GB card, were talking $175 just to try and recover with no guarantee that anything will be recovered. I emailed Virtual Lab, and they are not interested in doing a test or trying to work with us. I informed them that they could have another great addition to their business with RED business, since nothing else we have found seems to show the .r3d files at the correct size, but they were not interested.

For people recovering jpgs, their business model may work, but for 2GB .r3d files, I was not going to spend the $ on just a test. Even if it were $40-$50, maybe, but paying $175 with no guarantee plus having to pay each time per GB is not the way to go.

I suppose if it worked and was the only solution, it's a small price to pay and very much worth it.

So far, the trial version scans the files (after you put in the .r3d filetype and maximum size) and reports back what looks to be the correct files and sizes.

I think the real solution here if for RED to release a small utility that does this. I really think it would be a simple thing for them to crank out. Maybe after build 15 is released?

It would be a great way to offer even more insurance and a feeling of reliability. Eventually, there will be a large production where someone accidentally formats the card in Camera and then this will all of a sudden be an emergency. Why not put out a utility that does this now? Last NLE company I worked for released a utility that re-built the clip index and it was very much appreciated by customers.

Stuart, or perhaps Deanan, can you comment on this as a possibility? REDRECOVER?

Zach Gray

02-18-2008, 11:47 AM

I spent significant time trying to recover p2 MXF files.

This software was by far the best I tried.
http://www.runtime.org/ (http://http://www.runtime.org/)

I wasn't able to get the p2 data back because of fragmentation in the file system. The software did recover the files, but they were garbled.

Read this FAQ on their site... you may get lucky, but fragmentation on very large files may create a problem that no one can solve.

http://www.runtime.org/recoverability.htm

clip:
* Fragmentation
A very common situation, caused by file deletion, format or partition deletion is a missing FAT entry. As long as the file size is smaller than the cluster size (e.g. 32 KB, depending on the drive size) you will get a perfectly recovered file because you actually do not need the FAT entry.

When the file is larger, it is usually allocated in consecutive clusters. This is why the most promising strategy for a data recovery software is to assume consecutive clusters when it rebuilds the file without FAT entry. This works for most of the files but runs into problems for files that increase over time. These files will necessarily be fragmented if they cannot be allocated consecutively because other files meanwhile use these clusters. Sadly, many important files fall into this category: Email files, databases, large documents and directories.

GetDataBack employs several techniques in order to recover even fragmented files correctly. These techniques include taking the allocation of other files into consideration. GetDataBack also is capable of reassembling fragmented directories. But make no mistake; these efforts are doomed to fail for large and heavily fragmented files.

As annoying as it is, although their content is still somewhere on the drive, these files are unrecoverable.

There is no automated data recovery software available that can solve fragmentation satisfactory. If you want to recombine a file consisting of 10 clusters on a 20 GB drive you must analyze, given a cluster size of 32 KB, all possible combinations of one known cluster with 9 other clusters out of possible 625000. There are 6250009 possible combinations, a number with 52 digits.

The only possible and more intelligent approach is a "manual" data recovery for a particular file. With Runtime's DiskExplorer you would begin at the known cluster and search downward looking for data you know belongs to the missing part of the file. Finally, you put all your findings together into a new file. The limitations of this approach are obvious. This can only be done for a couple of files with a known content.

Even data recovery service companies will most likely not produce better results. While they might have a couple of tools, e.g. for extracting readable text, they do not have your knowledge about the content of the file.

Dylan Reeve

02-18-2008, 12:21 PM

I wouldn't think fragmentation would be a huge problem on RED CF cards, we're talking about relatively few large sequentially written files...

Convince the NSA that the card had terrorist secrets on it, they'll get the data back! :)

BigLu

02-18-2008, 01:19 PM

I have seen this happen on set once already.
Card shot and oops reformated wrong card, but we have not rolled yet.
We didnt have a fix, stressful hour yelled at by the director.

I would PAY nice money for a program that would save me on set when this happens.
[programmers]

Dylan Reeve

02-18-2008, 01:22 PM

I've played a little bit with a card here, and I can't get any good results really.

I think it's a problem that is going to be largely unrecoverable, certainly not in an automated 'click to fix' kind of way.

Casey Green

02-18-2008, 01:44 PM

Anyone at RED, we'd love to hear from you. :)

Dylan Reeve

02-18-2008, 01:50 PM

Unless there's some way RED can prepare the file system to cope with this I don't think it's something they're going to have any more luck with. These are issue that are inherent with the filesystem, and are made more complex by the large file sizes involved.

The best bet is products like our SolidStore and the software tools I believe Offhollywood is working on, that will help manage that flow and reduce the likelihood of human error of this sort. Even then it's still going to be important that the card get handed off to the right person and put through the process correctly.

Little things may help. Perhaps different coloured CF holders for 'exposed' cards. All cards coming out of the camera are put in a red holder for example and not used again until the DIT puts them in a green holder to signify a processed card.

Casey Green

02-18-2008, 03:00 PM

Yes, of course.

But if there is anything else that can be done in the form of a RED utility, it is well worth looking IMO.

I have tried tests with at least 12 different recovery programs, both on MAC and PC, but they don't really know what to look for and how the file structure is built.
I believe there are options out there to recover P2 data. Has anyone experience with this?

If RED could make a custom app, there might be a decent chance to recover .r3d files.
I wonder why we haven't heard from them on this. I can't find anywhere else on the forum where this issue has been addressed. :(

Casey Green

02-19-2008, 11:56 PM

I realize that the .R3D file system is still evolving, and that that makes such a utility difficult to make, but perhaps a utility app could be released and updated along with the firmware.

This is a very important part of the equation and I hope we see this released soon.

Anyone else have anymore luck with other File Recover programs.

Perhaps there is something still out there... low level, that can restore .r3d.

Dylan Reeve

02-20-2008, 12:24 AM

The problem is that it's all about the FAT32 file system which is an existing standard.

mdo

02-20-2008, 11:17 AM

This definitely shouldn't be difficult to solve as long as the designers of the file format are on board with helping to make the files easily recoverable. Dylan is right that fragmentation shouldn't be an issue if files are written consecutively and no gaps are created by deleting earlier files while later ones exist. Even the creation of new files after format by the Mac shouldn't be a problem if the media was formated on Mac to begin with, since it will only write files that already existed there before, in exactly the same location, which would not overwrite any data.

FAT32 is a fairly simple file system in that it doesn't store cluster chain information with information on the file. It's all stored in the File Allocation Table (FAT). When a file is deleted, the first character of the directory entry is set to ASCII 229 and the cluster chain in the FAT is zeroed out. This means that full file start information will remain as long as no new file has overwritten the directory slot or the same clusters on the disk. Since this directory entry tells you where where the file starts and how long it is, if the file is not fragmented, recovery is about as simple as it comes.

Formating completely zeros out the root directory and FAT, so file start information will be lost for files that are located in the root directory. I don't know how files are stored on the Red, but if they're all in the root directory, then no file system metadata will exist after a format. Since these data recovery programs are having no luck, I would guess that files are being stored in the root directory, which means that a signature search will be the only way to find the files.

I have done file recovery based on signature searches for a variety of files and I'm usually able to construct an algorithm that enables me to find the start and end of files. Unfortunately, guessing at the file structure wouldn't be sufficiently reliable for putting together a tool. This is where the designer of the file system comes in. Only the author of the file type knows for sure what set of rules can be used reliably to find the beginning and ending of files of that type, particularly since those rules are subject to change. If the designer of the file type can take care to ensure the rules remain valid, or are updated, when changes are made to the file structure, then the tool would remain a reliable tool. Also, if the process of building the set of rules that define the beginning and end of the file shows that there is no way to do it -- no reliable rules exist -- the designer of the file system can add a highly unique beginning and ending signature to the file, thereby completely resolving the issue. As long as those beginning and ending signatures remain as the first and last bytes in the file, it wouldn't matter what changes occur to the rest of the file.

So here's the question: Is this a sufficiently important issue that the file system designer should become involved in defining a set of rules for beginning and ending signatures, and possibly for creating highly reliable beginning and ending signatures, to maximize the ease of recovering mistakenly formated media?

How important is it?

I'll post a poll on it.

Casey Green

02-20-2008, 03:49 PM

OK so, an update:

I've tested more with BinaryBiz's "Virtual Lab" software, and it seems to have potential. (www.binarybiz.com)

Here are the steps taken:

When you run a scan, you must select the option for "I don't know why my files / folders are missing".

Next, select "Start a new Virtual Lab recovery session". This connects you to their server where they create you an account which you will eventually use to pay for a MB or GB "quota" of data recovery "credits". I don't think this is right, where you have to pay by the data recovered plus the software license, but I think they do this to prevent piracy of their software.

Anyway, you then will be prompted to select your media (Lexar ATA flash card), and once you do that, choose the option for "Recover Initialized Drive" and then "Start New Scan".

Next, you will be asked, "Would you like to include a filetype Search?"
This is very important. Select YES and you will now be able to add the .R3D file type to the filetype list to select from. Select Add. When you type in the file extention,Make sure it is in ALL CAPS.

The way this works is that you must at least 5 known good .R3D files to create a "signature" for the filetype. So you will next click Add again and browse for 5 or more good .R3D files to add to the signature list. Once you have done this and have at least 5, click on Start, and Virtual Lab will analyse the file format to build the signature and then allow you to name it: .R3D, of course. (ALL CAPS)

Next, select the .R3D filetype from your list (it should show up in the unsorted folder) and select it's check box. Make sure no other filetypes are selected, unless you perhaps are trying to also recover the .mov quicktime proxy files. But I wouldn't worry about those right now. Also, change the maximum size of filetypes you are trying to recover to <8 GB, as the default is <10MB. Then Click on OK to Start Scanning.

Results should show within a few minutes (as tested on my MacBook Pro with Express Media 34 reader and full 8GB REDCard).

Here are my results from the scan. Virtual Lab wants about $110 for the software plus another $65 for about 8GB. I have not paid yet, since I am not sure it's going to work, and I am only doing it as a test.

Anyone want to go in on it with my and we'll set up a PayPal account or Fundable Account and pitch in for the test? I will be happy to do the work and post all the results.

Let me know and I'll set it up. Maybe we do $1 a person or $5 to get it off the ground quicker.

Here is a screen grab from my test results. As compared to what I know should be on the card, this looks accurate. Now, I just have to pay to recover and see if they are intact. (The files on this card are not from a commercial shoot, but just shot for testing purposes.)

What do you guys say? Shall we set up the test?

http://www.reduser.net/forum/uploaded/971_1203551337.png

Dylan Reeve

02-20-2008, 04:30 PM

Here are my results from the scan. Virtual Lab wants about $110 for the software plus another $65 for about 8GB. I have not paid yet, since I am not sure it's going to work, and I am only doing it as a test.

Anyone want to go in on it with my and we'll set up a PayPal account or Fundable Account and pitch in for the test? I will be happy to do the work and post all the results.

Let me know and I'll set it up. Maybe we do $1 a person or $5 to get it off the ground quicker.

Casey, let's not get carried away.. Contact the vendors, point them at this thread, and let them know what you're trying to do and see if they'll let you do the testing free. It should be worth it to them if they can be seen to be a good option for fixing this problem for people.

For this test, do you have copies of the original 'lost' files? If so, it would be good to do an MD5 checksum of the resulting recovered files against the originals. Will let us know if they've made it through okay. If not, then a diff might be in order to see how damaged the recovered files are.

At that point is when it would be really handy to have an app from RED to split or fix broken R3D files, so you can at least recovered the no damaged footage.

Ken K

02-20-2008, 05:47 PM

Have you tried dcfldd to image the card and Foremost to extract the .r3d files? Both are open source and I don't see why they wouldn't work. That combo has worked like a charm for me in the past.

dcfldd:
http://sourceforge.net/projects/dcfldd/

Foremost:
http://sourceforge.net/projects/foremost/

Casey Green

02-20-2008, 07:32 PM

Casey, let's not get carried away.. Contact the vendors, point them at this thread, and let them know what you're trying to do and see if they'll let you do the testing free. It should be worth it to them if they can be seen to be a good option for fixing this problem for people..

Thanks for the thoughts, but I have already gone this route. It's unfortunate, but the company is not interested in doing any tests with .R3D files. They are focused on their main business of .jpg and .mp3 files that their average customers have. Sad, but they are not interested.

The great thing about their software is that it analyses file formats by asking for 5 or more similar files, and builds a signature. It found the files and looks promising.

I'm not sure what you mean by "let's not get carried away". This could be a very good chance for us to get a solution. RED has replied to me that they are interested in working on this but they won't start on it until the file type solidifies more, and that is still months away. Plus they have said they have a lot of other issues to work on that are higher priority, so I don't see a utility from them any time soon.

If you are not interested in participating, that is fine, but I am going to set up an account to collect contributions and do a test.

We need some hope here until RED addresses this issue.

Those in favor of contributing to a test of Virtual Lab, please let me know, or PM me. I know we will all benefit from the results.

Dylan Reeve

02-20-2008, 07:44 PM

Let's not get carried away was more about paying this company money to essentially evaluate their product in this way...

mdo

02-20-2008, 07:57 PM

You could open the five files with a binary editor like WinHex (free version at http://www.winhex.com/winhex/index-m.html will do the trick) and see if you can find sequences of bytes common to the five files near the top and bottom of the files. Most signatures will be close to the top and bottom. There can be signatures significantly offset from the beginning and end, but the likelihood is much lower. If you find the signatures, you should be able to proceed with a good deal of confidence. Still might work otherwise, but less likely.

I'll kick in $10 to the pool. I'm curious what you'll find.

Dylan Reeve

02-20-2008, 07:58 PM

Casey, can you make an image of the 'deleted' CF card with dd and upload it somewhere? It should compress up pretty small. I don't have any cards here at the moment.

mdo

02-20-2008, 08:02 PM

Casey, can you make an image of the 'deleted' CF card with dd and upload it somewhere? It should compress up pretty small. I don't have any cards here at the moment.

It's not likely to compress small if it's a dd sector-by-sector image, since the R3D compression algorithm is presumably fairly efficient. Doubt it will squeeze up much smaller than the original file that was stored on it.

Dylan Reeve

02-20-2008, 08:26 PM

Yeah, good point :)

I'll get a card again soon.

Casey Green

02-20-2008, 09:07 PM

I'll kick in $10 to the pool. I'm curious what you'll find.

Cool. I've set up an account and put in $20 to start it at Fundable.com

https://www.fundable.com/groupactions/groupaction.2008-02-21.0551838552

As soon as we raise the goal, I'll go ahead and purchase the license and data quota to do the test.

Remember, this software continually needs to be purchased for the use of the data quota, so I will not be able to use it more than our test without paying for the next amount of data "credit". This keeps it fair, since I won't be benefiting more than anyone and the test will remain on test footage only.

I very much hope that this works as it will be a nice way to help clients feel more comfortable.

thanks everyone for contributing to the cause.

https://www.fundable.com/groupactions/groupaction.2008-02-21.0551838552

Ken K

02-20-2008, 09:29 PM

Dylan Reeve

02-20-2008, 10:42 PM

Before you guys shell out $175 for something that can most likely be done for free, let me try it with dcfldd and Foremost. I don't have a Red CF card, but I can download some .r3d files and copy them over to a CF card, image it and see if I can extract them.

I'll try the same thing sometime soon as well with a genuine RED card.

There are few scenarios I can imagine that should be tested:
1) Card is placed in computer and all files are deleted (there are a few ways to delete, so that's more too).
2) Card is placed in computer and formatted (not recommended by RED, but it happens).
3) Card is placed in camera and formatted.

In Scenario 1, there will be differences depending on the OS and technique as well. On a Mac dragging a file to the trash creates a .Trashes directory on the device, with a directory for the user and a .Trash directory in there. This is creating quite a few new file pointers, but as the R3D files haven't actually be removed at this stage, it should no overwrite them. When the trash is emptied files in those directories may be altered or rewritten, which could overwrite sectors from the source files. In windows I believe there is a similar process, although I'm not clear on quite what changes are made.

In Scenario 2, it will depend on format tool and technique. Also, whether any further files are created after the format (again the Mac adds some of it's own meta files).

In Scenario 3 there are immediately some files created as part of the format process. These are small, but stand a good chance of clobbering sectors of previous files.

What works in one case may not work in others, or even all that often in that specific case.

Casey Green

02-21-2008, 02:34 AM

I welcome multiple tests, however I think we should also try this software, since the other attempts I have made with over 12 different methods, haven't been successful.

I will continue with the other test when we make our goal of contributions.

Thanks everyone for helping make this happen.

http://tinyurl.com/36u9qh

Garrett M. Smith

05-28-2008, 10:28 AM

Did anyone ever find solutions to these problems? I downloaded Foremost, but I don't know header/footer info for R3D files

I find myself facing a card issue.

Garrett

Nikolai Pigarev

08-04-2008, 06:34 AM

I was planning on writing it for a while now but there you go:

A way of recovering footage from formatted CFs or RED RAIDs.

First a bit of history, we have managed to format a RED RAID with about 30GB of data. How? Easily nobody marked the disk, and the backup person just automatically backed up one, and it was the wrong one. Cause all that information was passed from one person to the other.

I highly recommend never to keep a single copy of your material, never to postpone the backup to the next day, and never assume everyone understood what you said :)

I have used FileSalvage by SubRosaSoft.com it is fairly cheap for what it does.

1) Go into Explore A Drive - EXPERT

2) Press "Supported File Formats..."

3) Drag & Drop 10 or more R3D files. To generate a signature.

4) Press "Continue" and then "Start a new scan"

5) wait...wait...wait...

6) After its done you ll notice it found 3 types of files:
R3D
QuickTime Media (Proxies)
MP3 (don't have a clue where that comes from, may be the embedded sound

Now because the RED writes files form the beginning of the media, and the media is usually formatted i found it safe to assume that the Index pretty much shows the sequence of files. First come the 4 QT Proxies then 1 R3D

But not so quick, unfortunately you can not just recover it all and be a happy camper, you need to recreate the file structure and the file names. How?

7) Recover the files and place them in folders. 4 QT Proxies and 1 R3D in each folder, according to their Index.

8) If you preview any QT file in FileSalvage it gives an error saying it can't play without some R3D file, so any QT nows the name of the R3D. Open QT in any text editor and in the first lines you ll see the name of its R3D file. Reconstruct the structure using that name.

Thats it, worked for me, i have managed to recover footage dating months back...

BUT in our case not only did we format the media we managed to write over it and then all hope is lost!!! You can recover something using statistical analysis i have been told but nobody knows how... talk to the special services they say :)

I hope it helps somebody. But DO BACKUP YOUR DARM MEDIA

Caleb Heymann

08-06-2008, 04:11 AM

Thank you Nikolai! This is great news to hear that a successful recovery of R3D files is possible. I'll try and verify your process as soon as I get the chance.

ericyoung

09-06-2008, 08:05 AM

If someone were to make a Read-ONLY compact flash reader they would sell a million of them! (Every Red user, every still photographer)

You mean like this?
http://www.digitalintelligence.com/products/forensic_card_reader/

There are others. Just google "forensic compact flash reader". When it's forensic, it's supposed to be completely read only/write blocked to avoid changing computer storage evidence in law/security enforcement.

However, whether they're as fast or reliable as Lexar/Sandisk readers or handle very large non-typical files is another question.

Casey Green

09-06-2008, 08:15 AM

By the way, here is the thread that has the results to the Virtual Lab software recover tests from a while back:

http://www.reduser.net/forum/showthread.php?t=9205&highlight=Recover

thanks for everyone involved. With the SDK now available, it would be great if someone were to take on writing a .R3D recover application.

kevinmaistros

01-05-2009, 09:30 AM

I probably have the most gut-wrenching story involving lost .R3D files on a music video I did for Universal Music Group's band, Puddle of Mudd. Accidently lost 3 weeks worth of footage shot in IRAQ for the video.

This was in JUNE of last year, and I'm just now getting a resolution. I'll make a new thread about the entire process, and the steps that were taken to recover the files. Many lessons learned.

Spike Baumann

01-05-2009, 10:08 AM

Yikes! I bet you have ulcers now.

Ivan Cortazar

03-26-2009, 07:30 PM

Rob van Gelder

04-01-2009, 05:31 AM

OK, guys I think here is a solution to the recovery of R3D files.

My assistant stumbled upon a recovery program that he needed to un-delete/ un-format his girlfriends hard drive......

This program is named Filerecoveryangel and it does exactly what we need.
I just tested this with an 8Gb card, recorded about 40% full in 5 clips, downloaded that first on my MBP with Download manager, then went back to the camera, did a format and next I go to my WINDOWS machine to recover with Filercoveryangel!

It scans the card and find so called "lost files", which have still the original name and with their proxies and checksums, etc.

In this case I only selected the r3d files to recover, which worked flawless!
Copy to an external HD and import in your MBP, or you could use Bootcamp to switch between XP and Mac.

This is really amazing and very very re-assuring. We now have a way out of format-mistakes .
It is also a more secure feeling for and towards our clients.
I did not test a repeated format or a partial re-recording, but it might also work.
Please check it out.

Rob

Ivan Cortazar

04-02-2009, 12:32 PM

A way of recovering footage from formatted CFs or RED RAIDs.

I have used FileSalvage by SubRosaSoft.com it is fairly cheap for what it does.

OK, guys I think here is a solution to the recovery of R3D files.

My assistant stumbled upon a recovery program that he needed to un-delete/ un-format his girlfriends hard drive......

Rob

Thanks for posting your discoveries. Because of a very unfortunate situation , I have also been in the process of trying to recover some lost footage. :crying:

Thanks Nikolai for your guide, it was really helpful. I have used filesalvage with some success. It found a lot of R3ds, even really oldones. I didn't worry so much about the quicktimes and the namings. I can create new proxies in Red alert, so I just opened the R3d in red alert and regenerate the proxies. These are my findings. R3D files that are larger than 2GB, filesalvage combines them into a longer 3rd (ie 6GB) and that causes Redalert to crash. So unless someone tell me how to split them back that leaves this program only to be able to recover short clips.

Regarding the sound. The R3d files get the sound when you regenerate the proxi in Redalert. I don't know what those mp3 files are. They won't open or play.

Rob, I am going to test that program and see if I can get some better results, specially with the over 2GB r3ds. Thanks