DIY Raid along with Fibre Channel network

[Steve Freebairn]

I'm looking at building an Intel based PC server with an Areca 24 port Sata II raid 6 card. I know the card will handle the formats I'd like to do, my real question is how I can allow others (3 other computers) access to the speed of the drive. I know that iScsi is an option.

Does anyone have any experience using multiple NICs with iScsi?

Another option is to use a quad 4gb fibre channel card and then put fibre channel cards in each system.

My main question with this approach is how much speed will we lose? The Raid card is capable of 1600+ megabytes/sec, will 3 users be able to get about 300 mb/sec each using 4gb fibre chanel?

I know that 4gb fibre is capable of 400 or 500 megabytes/sec depending on whether it is 10bits/byte or 8 (see Sata2 if you have questions about this, 3gb/sec means 300 megabytes in that case). So I assume that a quad
4gb/channel card would be capable of feeding up to 4 users 300-400 mb/sec.

So, does anyone know if my assumption is right? If I'm only using my server as a "raid box" how fast can the data be passed from the raid card to the fibre channel network (or iScsi if that would work)? Any help or thoughts would be appreciated!

Ideally we'd all have enough for the camera and several hundred thousand for accessories, but in this case, we don't. The server will be used for working storage and we'll be using a tape backup drive to archive projects.

[Joe Carney]

Interesting question. Have you looked into 10gEthernet? I am wondering how I'm going to handle all this RED video and vfx stuff without breaking the bank.

Joe C.

[Jeff Kilgroe]

kinobairn,

What you're looking to build isn't that elaborate as far as the actual drive aray and/or system itself to handle the data at those speeds. 15Krpm hard drives -- about 14 to 20 of them in a proper RAID-5, 6 or 7 configuration should handle the 1.6GB/s rates in most situations.

The biggest challenge will be connecting your workstations to this RAID system and maintaining the performance you desire. And that isn't too much of a stretch as far as the technology is concerned, but it doesn't come cheap.

I would steer well away from iSCSI. The whole SCSI over IP model isn't all that great and doesn't have the performance you might think. Fiber Channel or Infiniband are your two best choices. 10G Ethernet is also a possibility, but there isn't much on the market yet for interface and switching hardware and what is out there is very expensive. ...Actually the same can be said for Infiniband. FC is good, but there are several variations between manufacturers and you'll find that connectors and compatibility isn't guaranteed across the market. With FC, at least it's possible to get 4Gbit interfaces into your workstations at < $600 each. There are 4x4Gbit host adapters for your server, but depending on the host adapter and implementation you may or may not be able to connect a single workstation to each 4Gbit channel without going through a FC hub or switch.

You also brought up the 10bits/byte issue. Some FC implementations do have the 2bit overhead, as does Infiniband, but there are some FC implementations that do not. It depends on the interface model and the chosen protocols operating over it. While I'm talking Infiniband, also be aware that it's 2.5Gbit/channel. It can be configured as single, quad or 12-channel. There is an 8-channel Infiniband movement out there, but it's not an official part of the spec (which seems dumb, but that's the way it is). So if you're thinking Infiniband, you would probably need dual quad-channel interfaces on your server or a 12-channel interface. Then a proper Infiniband switch to connect each workstation.

You may also need SAN software to provide more functionality to accomplish everything you're thinking of. Given the price and that most of this tech just isn't quite "here" yet. You may do better with a regular Gbit Ethernet network and a moderate server and give each workstation their own local high-speed RAID. In the long run it would probably be less problematic and will save you a bunch of money that you can invest into a better system in another year or so as the tech becomes more mainstream.

But if you're serious about this, it's entirely do-able. I know a fair bit about some of the fiber channel options out there and might be able to recommend something. I could also pick the brain of a friend who does this sort of thing for a living. He knows FC and Infiniband options inside and out and where to get what and what you should use with what systems, etc..

[Karbon Arc]

Hey kinobairn, i'm also trying to do a similar thing. i've read up a bit on ethernet linking or link aggregation (same thing as far as i know). This is when you link multiple ethernet connections in parallel to get more bandwith. I currently have 8 drives on a rocket raid on my dual G5 with a black magic card. I'm getting just over 400 true megabytes a second, direct. Now, i want to have 3 other people get the same rate over a network (and obviously a much bigger raid with more drives).. I also looked into 10gig ethernet but didn't find many affordable options.. i think Jeff's suggestion is probably the safest with the FC. I'm going to do some research on that and see if i can afford it. If anybody knows if and how link aggregation works, i'd be happy to hear.

[Jeff Kilgroe]

When you start building aggregate ethernet links or bonded ethernet channels, it can get really messy. There is a sharp jump in latency and a huge falloff in performance as the number of links increase, or at least that's the case with 100Mbps ethernet in most situations. You see this a lot in datacomm applications DSL, TI/DS1 style circuits. Even retro-fitting 100Mbps performance out of older 10Mbps networks since it's cheaper to bond several ethernet channels together on a network backbone in a large facility than it is to upgrade all their old wiring throughout the building. But I don't know where you would get the hardware to start bonding gigabit ethernet together to accomplish something like this. It seems it would be far more efficient to just go to 10G ethernet. 10G over copper needs CAT-6e grade wire or better. It will also work over CAT-6 over runs of less than 75ft or longer if shielded wire is used. But just use 6e, it's cheaper now than 5e was 2 years ago. 10G interface cards are not cheap though, about $2K each for the Myri 10G PCI-E, which is an 8-lane PCI Express card. Not many systems out there can handle or fully support one of these as most systems dedicate their 16-lane slot to video use and leave the remaining slots all locked at 4-lanes each. There's several options out there for dual quad-channel Infiniband on a single card (two connectors, two cables), this provideds 10Gbps. But you lose out on overall bandwidth due to the 10bits/byte format.

Take a look at www.aspsys.com -- these guys are in my area, but there's some info on their site about potential products. Also look at www.Myri.com Myri probably has the best 10G Ethernet products out there right now...

For kickers, their 10G Fiber Ethernet card is Here (http://www.myri.com/Myri-10G/NIC/10G-PCIE-8A-R.html). This could very well be a potential interface card for a DIY RAID system for interfacing to RED's RAW port. It supports 10G E over Fiber and can fun in both single mode and multi-mode. I think the cost of this card runs about $2400, but I would have to double check that.

[Joe Carney]

Being a complete newbe on this..I thought with Redcode being wavelet based with visually lossless compression, throughput rates were considerable reduced compared to uncompressed rgb type options. So maybe for small shops, off the shelf equipment using GigE or dual GigE (nVidia chipsets) might work?

[Jeff Kilgroe]

Being a complete newbe on this..I thought with Redcode being wavelet based with visually lossless compression, throughput rates were considerable reduced compared to uncompressed rgb type options. So maybe for small shops, off the shelf equipment using GigE or dual GigE (nVidia chipsets) might work?

I suppose it depends on your application. If you're not going to be fitting your RED(s) with a RAW port, that eliminates a lot of the concern for a very fast RAID and a fast interface right there.

But even for a 1-man operation, 10G ethernet or other very high speed connection could have its advantages... Like if you're using Mac and PC workstations, but want the bulk of all your data on a central RAID or SAN device, accessible from any of your workstations.

OTOH, this level of network infrastructure is very expensive. By the time you factor in Fiber Channel interface cards or Infiniband or 10G interfaces for each system, you have already far surpassed the cost of equipping each workstation with 1TB of fast RAID-0 storage and simply having a central backup and archival tape system.

For most small to mid-sized operations, local high speed disks for each workstation and then a conventional 1Gbps network is probably more than just fine. After all, 1Gbps / Gigabit Ethernet can sustain over 110MB/s on a proper installation with a theoretical max of 125MB/s. That's plenty fast enough to handle three streams of REDCODE RAW 4K @ 24fps. Not that you would be streaming it anyway, but just an example for comparison of data rates. Gigabit ethernet is also 1.25X the speed of Firewire 800.

[Trevor Meier]

How about a system like this:

* Local Mac Pro storage x2: 2TB of RAID-0
* Windows Server box: 7-10TB of RAID-5 for nearline backup, recent projects, media library etc.
--> (running ExtremeZ-IP? Or some other system that allows mac filenames)
* Single LTO-3 (or 600A) tape drive on server for backups & archives (or a tape library if you have the $$)


The reason for the Windows box is it's the cheapest way to get a 12 or 16-way RAID-5. Buy a large case, big power supply, and a 3Ware or other multiport SATA raid card and drives. But windows as a file server for the mac has issues... anyone with experience with ExtremeZ-IP (or any alternatives)?

[Joe Carney]

I suppose it depends on your application. If you're not going to be fitting your RED(s) with a RAW port, that eliminates a lot of the concern for a very fast RAID and a fast interface right there.

But even for a 1-man operation, 10G ethernet or other very high speed connection could have its advantages... Like if you're using Mac and PC workstations, but want the bulk of all your data on a central RAID or SAN device, accessible from any of your workstations.

OTOH, this level of network infrastructure is very expensive. By the time you factor in Fiber Channel interface cards or Infiniband or 10G interfaces for each system, you have already far surpassed the cost of equipping each workstation with 1TB of fast RAID-0 storage and simply having a central backup and archival tape system.

For most small to mid-sized operations, local high speed disks for each workstation and then a conventional 1Gbps network is probably more than just fine. After all, 1Gbps / Gigabit Ethernet can sustain over 110MB/s on a proper installation with a theoretical max of 125MB/s. That's plenty fast enough to handle three streams of REDCODE RAW 4K @ 24fps. Not that you would be streaming it anyway, but just an example for comparison of data rates. Gigabit ethernet is also 1.25X the speed of Firewire 800.

Thats what I was thinking, powerful Islands in a workflow stream. Still might want an affordable NAS for asset storage, and workflow purposes, but not for realtime streaming.

[Joe Carney]

How about a system like this:

* Local Mac Pro storage x2: 2TB of RAID-0
* Windows Server box: 7-10TB of RAID-5 for nearline backup, recent projects, media library etc.
--> (running ExtremeZ-IP? Or some other system that allows mac filenames)
* Single LTO-3 (or 600A) tape drive on server for backups & archives (or a tape library if you have the $$)


The reason for the Windows box is it's the cheapest way to get a 12 or 16-way RAID-5. Buy a large case, big power supply, and a 3Ware or other multiport SATA raid card and drives. But windows as a file server for the mac has issues... anyone with experience with ExtremeZ-IP (or any alternatives)?

Do Macs work with Linux and Samba?

Joe C.

[Jeff Kilgroe]

The reason for the Windows box is it's the cheapest way to get a 12 or 16-way RAID-5. Buy a large case, big power supply, and a 3Ware or other multiport SATA raid card and drives. But windows as a file server for the mac has issues... anyone with experience with ExtremeZ-IP (or any alternatives)?

Windows as a file server just has issues. Unless you go to one of the Windows Server versions, but then it still has issues with non-windows clients.

I would recommend going Linux on the server box, but that isn't perfect either. If you plan to keep your actual workflow Mac based, and you're only entertaining a Windows or Linux system as a storage server or SAN node, then you may want to seriously consider an Apple option there as well. There are several SATA RAID options for Mac as well -- unfortunately 3Ware isn't one of them though. For an Apple option don't discount the XServe either. It's not priced bad at all compared to similar PC options. And the build quality is second to none -- good support too on the business/corporate end. the XServe is a great server for mixed environments and the storage can all be in a companion rack-mount RAID enclosure connected via fiber channel or SCSI or whatever, just install the proper interface card into the Xserve.

[Steve Freebairn]

Applied Visual,

Thank you for all the info. My main thinking right now is similar to what you have suggested, just using gigabit ethernet to connect each computer to the central bank of files. Then have each station use it's own 2 tb+- raid system for working with the data. I'm looking at the Areca 24 port card for a few obvious reasons. It takes up to 2gb of DDR2 ram, it has 24 ports, and it has the newest intel IOP processor for raid 6. I think we'll hold off on the fibre channel until we see what direction we'd like our workflow to go.

[MikeCurtis]

skipping through all this, short version - it doesn't scale as well as you'd think, it isn't as fast as you'd think, it won't provide data as consistently as you need - average rate is one thing, minimum guaranteed delivered datarate is quite another. The seek times from multiple simultaneous hits is a HUGE hit on throughput as well. The DIY solution won't cut it for high throughput, mission critical (don't drop frames) applications - gotta go with a SAN, and a good one that that.

-mike

[Steve Freebairn]

Mike, great article on Uncompressed workstations.

It might not be super clear in what is written, but are you saying that the 24 port raid solution won't work for uncompressed HD? We want a lot of storage, and we aren't planning on onlining at 4k until we beef up the system. The main point of this raid is for double redundancy (Raid 6) and for a decent amount of speed for uncompressed 1080P. Do you think it will work? What are your suggestions for fibre channel cards?

[hatton]

Do Macs work with Linux and Samba?

Joe C.

Absolutely! Macs can work with Linux native NFS or Samba. Samba is just a Windows server implemented in Linux/BSD/OSX/...., just better.

Connect like you would to any windows server.

There are also implementations of AFP server for Linux, but I've never tried them.


Michael

[Gavin Greenwalt]

I woud question the definition of the word "Better" based on the samba implementations at my studios (windows domain networks).

[Dustin Cross]

So long as you won't be using the Raw port of your Red, then GigE is all you need. I have been using GigE to share footage from my 3TB Mac file server to four edit stations for a few years now with no problems. 10bit uncompressed SD, DVCProHD, HDV, DV, pretty much everything except uncompressed HD.

On my file server I use a Rocket Raid 2322 raid 5 card (upgraded from a 2220) with 8 400GB SATA-II drives. I also have a Small Tree 6 port GigE card that I use to directly connect each edit system to the fileserver. This way each station has its own bandwidth. There is no cpu load on the file server. The Rocket Raid card with 8 drives in Raid 5 has more than enough throughput for a few stations and I can add a second Rocket Raid and larger drives if needed.

I use a Mac for the fileserver, because I am a Mac shop, it is so simple to set-up, but also because AFP is faster than SMB. I haven't done much performance testing with NFS, because AFP was more than I needed and easy. I would think with Linux properly configured and a properly tuned NFS setup you could get descent speed and have access to more than 8 drives in an array.

If you really need a lot of storage consider using something like a Lacie Biggest S1S instead of single drives. It is four 500GB drives raided together and then presented as a single SATA device. You can make it raid 1 or 5. It is not the fastest drive at around 55Mbytes/sec, but 8 of these will give you close to 16TB of storage and depending on your config, over 400Mbytes/sec of throughput. And its a pretty cheap set-up. If you do raid 5 on the Lacie and Raid 5 on your controller you get extra redundancy and still 12TB of fast storage.

The guys at Small Tree created (or licensed) a network file system for just this sort of thing that was supposed to have awesome performance (BlaseFS), but I never tested it and I don't see it on their website any more. When I was talking to them about it they said AFP maxed out in throughput below 100Mbytes/sec in their tests, if I remember correctly.

What I am considering upgrading to is a 10G card in my Mac server and one edit station and a D-Link DXS-3227 for my switch. This switch gives me 24 GigE ports and two 10G ports for around $2000. Add two DEM-411T 10G Copper CX4 modules for about $500 each. 10G copper CX4 cards for the server and edit station from Small Tree are $1300. So that's around $5600 to bump my file server and one workstation to 10G and have enough bandwidth to do uncompressed anything that exists right now.

My other stations are still limited to GigE, but that is enough for everything I do anyway and I plan on shooting compressed Redcode 4k, so GigE is all I really need.

The problem right now with upgrading to 10G is finding a network file system that can take advantage of the available bandwidth. A cheap 8 SATA-II drive raid 5 setup can out perform GigE (maintain over 300Mbytes sec read and write even 90% full).

The problem FC, ATA-over-Ethernet, and I think iSCSI (though I don't know much about iSCSI) is that only one system at a time can access the device. Meaining you still need a file server in front to share the storage if you want multiple system to be able to access the same data at the same time. With FC you can use things like Apples xSAN or SGIs CXFS, but they add a lot of expense and complexity.

You need to look at what your requirements and technical capabilities are. For me I try to keep costs down and I have over 15 years experience as a network engineer and Unix administrator, so the roll my own option doesn't bother me. I am my tech support.

Hopefully that was useful.


Mahalo
Dusty
www.sandust.com

[Joe Carney]

I woud question the definition of the word "Better" based on the samba implementations at my studios (windows domain networks).

Do you mean not as good as Windows file sharing, or problems using it in a windows domain system?

[david farland]

Very excellent post, Dustin.

Have you considered software raid across multiple SAS controller cards , say 2 drives per controller?

Cheers,

[Dustin Cross]

David,

I have used software raid on Linux, Solaris, and HPUX with no problems, but hardware raid cards are not that expensive.

I don't know of any SAS cards for Mac and there is no software raid 5 for Mac. Raid 5 or better is really a requirement for dealing with tapeless acquisition. For me sticking with Mac is just so much less work right now and I can get plenty of disk space and plenty of bandwidth with what is available without breaking the bank.

With one server I can put two RocketRaid 2322 raid 5 cards and one 10G ethernet card. I can then connect 16 Lacie Biggest S1S drives configured as raid 0 to those two cards. That would give me 32TB of storage (2x 16TB arrays), with each array capable of 400Mbytes/sec throughput and all of it protected at raid 5 for approx $36k. With both arrays using full bandwidth of 800Mbytes/sec I am still not maxing out my 1000Mbytes/sec capability of the 10G ethernet.

There is enough room for growth on the simple Mac side of things, that I haven't considered all the other options out there. Plus keeping back-ups of my 3TB is enough of a pain. Having to deal with 32TB would kill my 600GB DLT tape drive.

If I wanted to get crazy I think I could build a Linux system with two 24 port SATA raid cards and two 10G ethernet cards and connect 48 Wiebetech 5 drive raid disks (similar to the Lacie's) with new 1TB drives and get around 230TB of usable high speed storage. I think that would come in at less than $1000/TB of storage.

I think a more expensive purpose built systems would be more responsible though.


Mahalo,
Dustin Cross
www.sandust.com

[Gavin Greenwalt]

Do you mean not as good as Windows file sharing, or problems using it in a windows domain system?

Problems using it with windows in general. Samba shares would cut out, randomly change permissions and this was with us never touching it, the thing just ran 24/7 but it had a life of its own, it was a nightmare. We finally put it out if its misery and installed windows server 2003 and never had a problem after that. And it wasn't a question of expertise I had never even been on a windows 2003 box let alone configured one but in less than 2 hours over a weekend I had the whole shop switched over to a domain, it was foolproof and worth the $$$ for the reliability.

Later a few editors wanted to add 3 macs onto the network and they had millions of problems... so it's not perfect. But if you run windows and only windows it works perfectly.

[gtfx]

Folks,

I would assume that a constraint is also to be portable, and reliable while capturing from RedCamera, one would need a field usable device! How about a quarter rack, with self contained UPS, to allow for switching power source from mains to gen-set or solar etc.

A server (taking on from David's post) say SuperMicro or Chenbro with SATA-II or SAS drives, (3 RU 16 disks), triple 760 Watt power supply (2+1 redundancy), 4x4 port SATA or SAS raid cards, each Raid card with 2xRAID 1 pairs or 1x RAID10 set and a software Raid0 (MD) (OS could be Gentoo or similar booted through USB key, vm.Swappiness=0) across all the HW RAID sets. One would need PCI-E or PCI-X capable cards for all devices, and a server grade motherboard with 1024mhz bus, 2 64bit processors, about 6 GB RAM (to provide caching).

I estimate (based on my limited testing) this will provide non blocking write speed of ~1+GBPS. Which matches the 4k estimated data flow via a 10GBE connection.

Assuming one is using 750 GB SATA-II or 300 GB SAS disks, one would get approx 3 to 2 TB usable redundant, protected usable space in a field device.

One can stack about 4 such hosts in a quarter rack along with a UPS too! Could be house n a station wagon, etc. Will require air conditioning support, however, thats easy with some of the available rack options.

what say thee wise folks? again, above is purely from the perspective of capture (in field) only. an above will not break the bank.

Your comments, much appreciated.
best,
S.