Cataloguing LTO-3A tapes?

[Mel Matsuoka]

(x-posted from FCP-L and fxphd....apologies to anyone who already saw this)

So we broke down and got a Quantum LTO3A unit (our punishment for trusting external hard-drives for backups and archive). So far, other than the horrible Java FTP client, it works great.

One question I have for fellow LTO3A owners, though. How are you cataloguing the contents of your backup tapes? Since the unit is FTP only, I can't use standard OSX media cataloguing apps like CDFinder or MediaCatalog to catalogue the tape contents into a database. Well, actually, I take that back...I *can* mount the unit on the Desktop using the built-in FTP capabilities of Finder, but as many of us know, the Finder FTP client is flakier than than a dude in a Head & Shoulders commercial. It works to catalog stuff this way some times, and other times, it just continually disconnects at random times.

I assumed that I could just log in to the unit via command-line FTP, and issue a recursive listing ("ls -laR") that I could pipe out to a text-file, but the damned built-in FTP server does not support the fricken -R flag!!! GRRRR!!!!!

I've tried Cyberduck, Transmit, Finder, MacFUSE ftpfs and none of them can hold a connection to the drive, even when uploading to it.

Any ideas how we can simply catalogue our tapes?

Thanks in advance,
mel

[Scott Mason]

Aloha Mel

We make folders called LTO_001 etc and put 400gbs in it for archive. Then drag the whole folder to the lto tape through the "lovely" java ap that came with the drive. I imagine you could catalog the folder and it would be labeled the same as the tape. We have for this and it works well. I have heard of people having a 500gb drive in the computer and fotmating it with the name of the tape. You could catalog this way as well. We are on the process of making final cut server do this and the folder nameing system seems to work but I have not tested completely.

Also hdlog is designed to work with this lto drive but it does not have the same functionality as final cut server so we have moved towards that instead. I have at least a month before I move final cut server into the post workflow as I am testing it on a temporary network until I am satisfied. So far it works very well. Call me up or come by the office if you want to see it in action.

Hope this helps
Aloha
Scott

[Dan Graetz]

We've struggled to a find a good cataloguing solution as well. Scott's folder method sounds feasible.

As far as FTP clients go - we've had great success with Fetch. Not sure if this would provide any cataloguing (I'm not much of a command-line user unfortunately)

Could you use something like Media Catalog (Mac OSX app - halfduplex.net) to catalog the LTO_001 folder (using Scott's method) and then export that catalog to CSV or similar?

[Jeremy Newmark]

Aloha Mel

We make folders called LTO_001 etc and put 400gbs in it for archive. Then drag the whole folder to the lto tape through the "lovely" java ap that came with the drive.

Just an FYI. Quantum does not suggest putting more then 360gigs on a tape at the moment with the LTO-3A. Apparently there is a bug with the current software that can cause some kind of indexing error and make some files in-accessible. The media can be recovered, but the tape has to be sent to Quantum to do so. Supposedly the new software will fix it and should be available in a few weeks.

disclaimer: I do not work for Quantum, just an LTO-3A owner.

best regards,
jeremy

[Greg M]

where did you buy the LTO-3A?
How much are they?

[Tom Goldberg]

Quantum consultant here - We do have customers who've successfully used freeware applications "Disk Librarian" and "CD Finder" to catalog their tapes. Note however you do have to mount them in Finder (Go > Connect to server > ftp://root@youripaddress) to use these.

There are also good asset management tools available with A-Series interfaces including Pictron's "Network Tape Gateway" and Imagine Product's "HD Log."

BTW, we do not recommend Fetch or other third party FTP Clients because they do not correctly handle Apple resource forks and may time out with typical tape drive latency.

The 360GB recommendation is true for v2.6.0 and earlier firmware. It is fixed in v3.0 which should be out this week - I'll post when it is.

[Edited 10-11-08: v3.0 is now available for stand-alone systems on the Quantum web site (not yet ready for SuperLoaders)]

[Mel Matsuoka]

Quantum here - We do have customers who've successfully used freeware applications "Disk Librarian" and "CD Finder" to catalog their tapes. Note however you do have to mount them in Finder (Go > Connect to server > ftp://root@youripaddress) to use these.

Thanks for responding. As I mentioned in my post, I use CDFinder as my primary media cataloging app, and have already tried using it to index LTO-3A tapes that I've mounted via FTP in the Finder. However, I've found this method to be 75-90% unreliable. Even with the options to index stuff within zip/rar/etc. archives turned off, CDFinder cannot hold a connection to the drive long enough to complete the index.

For now, Scott's method of creating virtual tape folders is the only method that works reliably. It's a bit of a pain in the ass though, since I already have an established file management system/hierarchy in place, and shuffling files around just to do this is rather kludgey...but this works fine for now.

Could I make a feature request? I think it would be great if the drive wrote out a text, xml or csv file listing of the tape contents to the root of the tape every time it gets ejected. That way, you can just download one small file in order to quickly get a listing of the tape's contents. An XML file would be even better, so people can write their own indexing tools that use that data.

The 360GB recommendation is true for v2.6.0 and earlier firmware. It is fixed in v3.0 which should be out this week - I'll post when it is.

Thanks for keeping us posted. Is there any way that drive owners can be kept updated on driver/FW updates via email (since most of them aren't REDuser readers, I would hazard to guess)?

[Tom Goldberg]

I believe the unreliability you experience with CD Finder is a function of OSX timeouts on mounted volumes and I've been unable to discover how to make that longer.

There is a way to do an ls -laR on our tapes, although it isn't pretty. Mount the drive as an FTP volume as you have been for CD Finder and then use Terminal to navigate to /Volumes and do the ls on the tape from there. You can redirect the output to a file and use any Unix tools, or copy the listing and paste into the application of your choice using Mac OS utilities.

We have noted your feature request and will keep it in mind for the future - it sounds like a good idea to me.

All Drive owners can check for the availability of updates on the Quantum site under Service and Support > Software and Documentation > Downloads. Any drive owner is also welcome to send me an email to get on my own mailing list.

[Martin Jaeger]

Here's how we're going about it... With a new tape we create a directory on the tape (as has already been suggested) called 0001_tapename_date. We then use the terrific FTP app Transmit on OS X and copy project files/media etc to this directory. Could do this through the Java applet (ostensibly more reliably) but it is indeed a little clunky. Have not had issues so far using Transmit. Anyway, after the tape is full we mount the drive on the desktop using "connect to server". We then use CDFinder to catalog the "tapename" directory.

We have also experienced the "connection to server interrupted" issues during cataloging. The trick here is when the finder displays the interrupt dialog, rather than click "disconnect" and try the whole thing over, just ignore it... After a few seconds to a minute it will reconnect and the cataloging will resume. It may even disconnect a couple of times during a session. Again, just let it alone and it will reconnect and eventually finish the session.

Would love to have "A-Series" elaborate on the problems with 3rd party FTP clients with regard to improper handling of OS X resource forks and what problems that could create for archiving or retrieval of data.

Best,

M

[Tom Goldberg]

Would love to have "A-Series" elaborate on the problems with 3rd party FTP clients with regard to improper handling of OS X resource forks and what problems that could create for archiving or retrieval of data.

Every File on a Mac can have additional metadata beyond what is tracked by the file system in a separate but invisible file called a "resource fork." This can contain a variety of information but most importantly it tracks the Creator and Type of the file so it knows what application it is associated to even if there is no file extension.

When Apple went to OSX, they supported but deprecated the use of these forks in favor of the more traditional three letter extension, but many applications continue to use it extensively (i.e. Photoshop). When we first started archiving FCP 5 files we found that the resource fork was mandatory for restored managed projects to work properly. Apple has supposedly fixed this in FCP 6 but too late, we now always store the resource forks with our FTP applet. The only other FTP application which I am aware of that can do this is Fetch - but it implements AppleSingle which modifies every file by adding the resource fork. We implemented AppleDouble to keep this file separate and hidden for better performance and reliability.

This resource fork is undoubtedly not required for original red files but we do not know if all the files you might be backing up require theirs and thus for security sake always keep them and always recommend you use our applet.