Storage

[Stacey Spears]

This might belong in the main workflow thread, but I am going to post it here for now.

What is everyone using or planning to use for storage in their Adobe workflow? Medea, San, Fiber, SCSI, SATA II, etc...

I am looking at a Burly rack mount myself: http://www.burlystorage.com/ccp0-prodshow/MGBurly9RPM.html

I am looking at a 3ware RAID controller: http://www.3ware.com/products/serial_ata2-9650.asp

As far as drives, I will hold out until I get closer to receiving the camera and hope that the cost of the 1 TB drives comes down. If not, I will get eight 750GB drives.

[Kevin Halverson]

Are you asking about archival or working storage? For archival I use DLT. For working, a combination of SATA, SCSI and NAS. I haven't purchased any 1 TB drives yet, but I have been using 750's for about 18 months now without any complaints.

[Edgar Pitts]

We are going with an all Dell solution. For us the reliability and support are worth the slight premium in price.

3 x MD1000 with 500GB Sata II drives in Raid 5 yields 18TB usable:

http://www.dell.com/content/products/productdetails.aspx/pvaul_md1000?c=us&cs=04&l=en&s=bsd

Powerconnect 6248 Switch:

http://www.dell.com/content/products/productdetails.aspx/pwcnt_6248?c=us&cs=04&l=en&s=bsd

PowerEdge 2970 Server:

http://www.dell.com/content/products/productdetails.aspx/pedge_2970?c=us&cs=555&l=en&s=biz

PowerEdge 24U Rack:

http://www.dell.com/content/products/productdetails.aspx/pedge_2410?c=us&cs=04&l=en&s=bsd

Total Cost: $35,000

[Kevin Halverson]

That's an impressive storage solution. I am wondering how many users you are supporting and what types of applications they will be using. Are you a post house?

[Ben Feuer]

I'm probably gonna rock some combination of these;
http://www.g-technology.com/Products/G-SPEED-es.cfm

You get a 15 percent discount if you're a Digital Cinema Society member, which is extra nice.

[RivaiC]

For GRaid - But it's OS X only ? no XP support ?

[Mike McCarthy]

Wow I am dissapointed that the rest of this thread is gone. It was a really interesting. Luckily I have a txt backup of my really long post.

[Edgar Pitts]

Wow I am dissapointed that the rest of this thread is gone. It was a really interesting. Luckily I have a txt backup of my really long post.

Please repost!!! I too was disappointed. I found some of it in the Google cache, but not all. Thanks.

Edgar

[Kevin Halverson]

I was told that there was a HD failure on the reduser server. The backup is apparently not as up to date as one would have hoped for. If you do have a .txt backup, it might be worth posting, there was some great information in this thread.

[Edgar Pitts]

This is what I could pull from the Google cache. It is missing some from the 1st and 2nd pages.

-----------------------------------------------------------

Quote:
Originally Posted by McCarthyTech
I guess it would be good to answer the original question of the thread as well.

I use a number of solutions on 6 different stations.
4TB Huge Raid from Ciprico (10 400GB IDE drives in RAID50 connected with Dual SCSI 320 via ATTO card)
5TB ProAVIO Array (14 400GB SATA Drives in RAID 50 connected with Dual SCSI 320 via LSI card)
2x 2TB Internal SATA Arrays (8 250GB SATA Drives in Raid 5 connected to Broadcom RaidCore PCI-X card)
2x 1.2TB Internal Arrays (3 400GB SATA Drives in Raid 0 to integrated NForce Raid)
1TB Internal Array (4 250GB SATA Drives in Raid 5 to Promise 4200 PCI-66 Card)
Next purchase: 8TB Fibre array, with 4Gb SAN infrastructure (16 500GB SATA Drives, ATTO and QLogic HBAs)

McCarthyTech, you have a very interesting case there with different configurations under the same roof. Could you provide a read/write benchmark for each of these configurations? I would be interested to know both the burst and sustained abilities of each.

khmuse

--------------------------------------------------------------------------------

Quote:
Originally Posted by khmuse
McCarthyTech, you have a very interesting case there with different configurations under the same roof. Could you provide a read/write benchmark for each of these configurations? I would be interested to know both the burst and sustained abilities of each.

I have found that Blackmagic has a great benchmarking tool, but requires their hardware in system. Other benchmarking tools require you to set a block size. I usually get the best performance withabout 4MB blocks, but I have no idea if that emulates real world performance. Anyone know the average I/O block size when editing AVI files? Or DPX? If I can get an answer to that, I can benchmark them all for you guys. I assume you don't just want the optimal transfer rate if that is at an unrealitstic setting.

As a rule of thumb, my dual SCSI solutions provide 400MB/s, and my 1TB SATA arrays around 200MB/s. I am anticipating my dual channel 4Gb fibre SAN to be around 500 MB/s, but that will be shared to at least 3 systems.


McCarthyTech

--------------------------------------------------------------------------------

Quote:
Originally Posted by Edgar Pitts
Could you please provide companies that you would recomend?


Rorke, 1Beyond, Apple (heaven forbit) and a few others, will provide workstation SANs, while Dell, HP, or IBM only provides SANs for servers and datacenters. Some of those products have requirements like "Unix only" or whatever, making them useless to us. There are probably 20 companies that make shared SAN solutions that are optimized for media production. I can't think of many more at the moment.


Quote:
Originally Posted by Edgar Pitts
So if the FC switch has 12 ports, a quad link to the array would leave 8 available links to the workstations (2 dual links and 4 single links). What do mean by connected with faster interface?


I meant more channels for the array than the workstations. Ideally as many channels for the array as the workstations total, butthat rarely is feasible. Your situation seems best served by initially using single links to stations, and dual linkto the array. The first upgrade past that would be to double the array performance by adding another whole array, which would be 2 more channels on the array side, balancing the 6 workstation channels. Only after that would I worry about dual channel to any of your workstations.


Quote:
Originally Posted by Edgar Pitts
If you are smart about the dual channel array you purchase now, it the future you should be able to purchase another identical one, stripe them together, and boom, you have quad channel solution, with twice the capacity, making use of all of your original investment, but now supporting uncompressed 2K to 4 or 5 stations at once. Could you please elaborate on this?


You can use Raid 0 to aggregate the performance of multiple arrays. Usually large arrays are split that way already. I am buying an 8TB array, which will be divided into to 4TB halves, one on each channel of fibre. My workstations will stripe data between the two halves for double the performance of a single half. If I buy another physical array, I will have four 4TB parts to stripe together, doubling the performance again. That would be 16TB equally devided over four channels of Fibre, returning about 1600MB/s if I am lucky. A single workstation with a quad channel HBA could recieve this, or 4 workstations with single channel HBAs will never be effected by each other. If I have 4 workstations each with dual channel HBAs, they each have a peak rate above 800MB/s, but not all four can achieve that at once. See how that works?

Basically buy extra ports on your switch, and make sure your specific model of Fibre array is going to be available for a while. You may want to buy dual channel HBAs even if you are only going to initially use single channel. Otherwise single channel HBAs will be obselete to you once you ungrade.

McCarthyTech

--------------------------------------------------------------------------------

McCarthyTech,

Do you have an opinion on RAID 5 vs. 6 in this scenario?

Any opinion on SATA brands? Maxtor, Seagate or WD?

Stacey Spears

--------------------------------------------------------------------------------

I've heard good things about Terrablock...any thoughts?
http://www.facilis2.com/

zeke

--------------------------------------------------------------------------------

Quote:
Originally Posted by McCarthyTech
I have found that Blackmagic has a great benchmarking tool, but requires their hardware in system. Other benchmarking tools require you to set a block size. I usually get the best performance withabout 4MB blocks, but I have no idea if that emulates real world performance. Anyone know the average I/O block size when editing AVI files? Or DPX? If I can get an answer to that, I can benchmark them all for you guys. I assume you don't just want the optimal transfer rate if that is at an unrealitstic setting.

As a rule of thumb, my dual SCSI solutions provide 400MB/s, and my 1TB SATA arrays around 200MB/s. I am anticipating my dual channel 4Gb fibre SAN to be around 500 MB/s, but that will be shared to at least 3 systems.


Thanks for the offer. I would be interested in three measurements. Say a 1 MB, 10 MB and 50 MB block sizes. Both burst (say 10 - 50 blocks) and sustained (at least 30 seconds or more) would give a great sense of how much real world performance can be expected. I have also had great success with SCSI configurations on both PCs and SGI workstations.

khmuse

[Edgar Pitts]

--------------------------------------------------------------------------------

Here are my benchmarks burst/sustained in MB/s at these block sizes:

4TB Huge Raid from Ciprico (10 400GB IDE drives in RAID50 connected with Dual SCSI 320 via ATTO card)
512KB 220/215
1MB 232/230
4MB 455/370
16MB 460/380

5TB ProAVIO Array (14 400GB SATA Drives in RAID 50 connected with Dual SCSI 320 via LSI card)
512KB 400/365
1MB 425/395
4MB 425/395
16MB 420/420

2x 2TB Internal SATA Arrays (8 250GB SATA Drives in Raid 5 connected to Broadcom RaidCore PCI-X card) (Currently a degraded Raid 5)
512KB 1380/97
1MB 1020/140
4MB 1010/140
16MB 155/140

2x 1.2TB Internal Arrays (3 400GB SATA Drives in Raid 0 to integrated NForce Raid)
512KB 305/155
1MB 400/150
4MB 335/155
16MB 128/155

1TB Internal Array (4 250GB SATA Drives in Raid 5 to Promise 4200 PCI-66 Card)
512KB 190/158
1MB 193/160
4MB 190/170
16MB 180/172

I used a utility called HD_Speed which another editor recommended to me awhile back. I can't find the source link right now. These are all read tests. The write tests are destructive, and I don't have time to deal with them. I usually find write speeds to be 20% lower as a rule of thumb.

McCarthyTech

--------------------------------------------------------------------------------

Excellent work McCarthyTech, I really appreciate the effort!

I am more than a bit surprised by the massive difference between the burst and sustained read of the 8 drive SATA Array. The 16MB test seem very reasonable, but the three lower block sizes are wildly different.

I am not too surprised that the Dual SCSI array is the winner in terms of both consistency and sustained throughput. With the large number of spindles, availability is rarely ever an issue.

Again, thanks so much for doing this test!

khmuse

--------------------------------------------------------------------------------

Quote:
Originally Posted by khmuse
Excellent work McCarthyTech, I really appreciate the effort!

I am more than a bit surprised by the massive difference between the burst and sustained read of the 8 drive SATA Array. The 16MB test seem very reasonable, but the three lower block sizes are wildly different.

I am not too surprised that the Dual SCSI array is the winner in terms of both consistency and sustained throughput. With the large number of spindles, availability is rarely ever an issue.

Again, thanks so much for doing this test!

Kevin


Look closely, that one is currently a degraded Raid 5, I will post new numbers when I have replaced the failed disk, and the controller is not regenerating data on the fly. As far as the high burst number, I believe that is due to the direct interface, or just an anomaly due to the bad drive in some way.

McCarthyTech

--------------------------------------------------------------------------------

After doing some research, it looks like the Ciprico MediaVault 4210 offers the most bang for the buck Dual Channel 4Gbit Fibre Channel array:

http://dv411.com/mv4210.html
http://www.ciprico.com/Products/Medi...ediaVault_4210

Anyone care to comment on this system? We would likely start with one (probably 7.5TB) and add another to get up to quad channel.

What kind of performance hit is there going from Raid 0 to Raid 3? Is the limiting factor the dual 4Gbit ports?

Thanks.

Edgar Pitts

--------------------------------------------------------------------------------

Quote:
Originally Posted by Edgar Pitts
After doing some research, it looks like the Ciprico MediaVault 4210 offers the most bang for the buck Dual Channel 4Gbit Fibre Channel array:

http://dv411.com/mv4210.html
http://www.ciprico.com/Products/Medi...ediaVault_4210

Anyone care to comment on this system? We would likely start with one (probably 7.5TB) and add another to get up to quad channel.

What kind of performance hit is there going from Raid 0 to Raid 3? Is the limiting factor the dual 4Gbit ports?

Thanks.

Edgar


I have the SCSI version of the product you are speaking of. With a conservative estimate of 50MB/s per drive, 10 drives will deliver 500MB/s in Raid 0. In Raid 3, you lose the banwidth of two drives, leaving you 400MB/s. The other limiting factor when moving to RAID 3 is that the controller can cause a bottleneck. Making the pressumption that is it similar in capability to the older SCSI version, I have never experienced any bottleneck or performance loss since switching over from Raid 0 to Raid 3 a few months ago. The same can not be said of cheap internal PCI Raid cards.

What workflow are you planning to use to edit (AJA, Decklink, Desktop, etc.)? A 7.5TB array introduces some interesting issues into the workflow. At Raid 3, that solution will provide two 3TB volumes that must be striped together. Windows XP 32 is limited to 2TB volumes unless you increase the block size of the drives. This is not supported by some software. Adobe does support it, Matrox recently added support, can't confirm Decklink or AJA, but I believe they do. Just one more thing to check on, to allow your workstations to access more than the first 2TB of each half without a complex solution. I know, not what you wanted to hear, but better to find out now. The same feature that makes the gigabit solution really slow, masks this problem so Windows is not limited in the part it can see over the network. This is the same reason that Macs can write to NTSC drives over a network, because the host PC masks the nature of the drive, and translates the data.

This is why all of the storage solutions I have are under 2TB to a side. My 14drive array is divided in half, and each side of seven disks includes a hot spare, and a drive worth of parity data. The remaining five 400GB disks gets me right to 2TB without needing to increase the block size. Striping the two halves in Windows gives me a 4TB volume. These were all setup before our Matrox systems supported larger block sizes. My new SAN will have the same limitation for other reasons. My 8TB will be divided into four 2TB chunks, and combined in pairs into two 4TB drive letters.

McCarthyTech

[Edgar Pitts]

--------------------------------------------------------------------------------

Quote:
Windows XP 32 is limited to 2TB volumes unless you increase the block size of the drives.

Ignoring the cost of the OS for a second, what about using Server 2003 SP1 with GPT enabled?

Stacey Spears

--------------------------------------------------------------------------------

I wonder Ciprico has its own format to be shareable between Win and Mac natively. It says it supports it, but i don't know what it really means by support it.

Rivai

--------------------------------------------------------------------------------

Quote:
Originally Posted by Stacey Spears
Ignoring the cost of the OS for a second, what about using Server 2003 SP1 with GPT enabled?

That only works if you are sharing over an ethernet network. Data passes through server, server translates, allowing saving to GPT disks not otherwise supported by XP32. In a SAN, the data never passes through the server, although certain solutions require a server to monitor metadata to prevent data corruptions and overwrites. Metadata servers cannot translate to mask drive formating incompatibilities. Only exception I know of is that I believe MetaSAN has a way of allowing Macs to write to NTFS on a SAN. They still can't solve the 2TB issue by having PCs write to HPFS, although that is supported, the 2TB limit remains, unless you increade the block size.

McCarthyTech

--------------------------------------------------------------------------------

Quote:
Originally Posted by Rivai
I wonder Ciprico has its own format to be shareable between Win and Mac natively. It says it supports it, but i don't know what it really means by support it.

Ciprico advertizes that their products support Mac and PC, but not at the SAME TIME. Their arrays can only be connected to ONE system at a time, so you use NTFS for PC or HPFS for Mac. The exception is if you are using their fibre array on a SAN, at which point you will need SAN software to share files in any given volume, and that software will establish the Mac/PC sharing limits.

NO hardware can support direct connection and sharing of a volume between multiple systems, without dedicated sharing software, either from the OS in the case of ethernet sharing, or from SAN software in the case of fibre channel. That is what has me confused about your CalDigit claims. Their PCIe solution is very innovative, and should work very well as direct attached storage to an edit workstation. Unless they have some MAJOR tricks up their sleeves, there is NO way to connect and share that unit on a fibre SAN. The best you could do is 10Gb ethernet or trunked Gigabit possibly, but I doubt that will be the most economically feasible solution to achieve reliable uncompressed data rates on a shared solution.

McCarthyTech

--------------------------------------------------------------------------------

Quote:
Originally Posted by McCarthyTech
Windows XP 32 is limited to 2TB volumes unless you increase the block size of the drives. This is not supported by some software. Adobe does support it, Matrox recently added support, can't confirm Decklink or AJA, but I believe they do. Just one more thing to check on, to allow your workstations to access more than the first 2TB of each half without a complex solution.

Can you briefly describe the advantages and disavantages of increasing the block size?


Quote:
Originally Posted by McCarthyTech
I know, not what you wanted to hear, but better to find out now.

You are providing extremely valuable insight. I truly thank you for your time!!!

Edgar Pitts

--------------------------------------------------------------------------------

Quote:
Originally Posted by Edgar Pitts
Can you briefly describe the advantages and disavantages of increasing the block size?

Increasing the formating block size from the default of 512 bytes to 4096 bytes (8x increase from .5K to 4K) increases the maximum drive size windows can recognize from 2TB to 16TB.

The long answer is: as a 32bit OS, XP can address 2^32 blocks (just over 4 billion) so if each block is .5K or 2^9, then 2^9 x 2^32 = 2^41 (add the exponents). 2^41 is just over 2 trillion, therefore 2 terabytes. If we increase the block size to 2^12 (add 3 to the exponent, or multiply by 8 with is 2^3) our total addressable size increases from 2^41 to 2^44, which is 16 trillion bytes, or 16TB. (Quick reference for exponents, 2^10 is approximately 1000, so for every digit in the tens place, append ",000" to the result first digit result. 2^4=16, 2^24= about 16,000,000)

The disadvantage is that certain software doesn't support this very well, probably because it makes certain assumptions about your drives, in order to attempt to deliver better performance. Older hardware (controllers and arrays) don't support it either, but most recent products should.

McCarthyTech

[Mike McCarthy]

Thanks for reposting all of that. I have one more long post that I authored that I happen to have saved. I also have a lot of related info on my tech website www.hd4pc.com (http://www.hd4pc.com) so feel free to check that out. The following post was intended to explain why the traditional model of client/server data sharing over ethernet is not ideal for high-end post-production workstations:



The easy one first. The speed of the server in this specific application will have little impact on the throughput.

The problem with you proposed setup is that ethernet is a very inefficent interface protocol. Gigabit should provide 125MB/s (1000mb/8 bits in a byte), but in practice I have never seen more than 50MB/s, due to the limitation of network adaptors, switches, cable line noise, the mood of server gods, or whatever. The TCP/IP protocol is designed for reliability over long distances, not throughput. I believe it cuts transfer speeds in half any time it experiences an error, and in half again if the problem reoccurs. Other interfaces perform much closer to their specifications, like fibre channel, SCSI, or SATA. This is the same reason the Firewire400 is preferable USB even though USB is 20% faster on paper. In reality the overhead of certain protocols prevents anyone from actually experiencing maximum performance.

As far ar utilizing the equipment you had speced, a shared SAN does not pass the data through a server before it reaches the workstation. Certain SAN software solutions require a server to monitor array and direct traffic at the file level, but the data never passes through that server, so performance is not as much of in issue for a machine in that role. The MD1000 arrays you speced would work well as direct attached storage for your workstations since they are eSATA based, but Dell only supports them for server attachment. Even if they did, there is no way to easily share the data on them besides the slow option of ethernet. Similar arrays that interface with fibre channel would work better for you. Dell makes these fibre arrays as well, but once again limits their connection to PowerEdge Servers. I usually like Dell products, and am writing this on my Dell XPS 1210 laptop, next to my Dell Xeon workstation, but unfortuneately Dell does not offer any form of SAN product suitable for media creation or post-production. I am not too impressed with HP or IBM's offerings either, although I know IBM is working to improve that. The problem is that they are all used to the server mindset that doesn't work for post-production. Most of the good shared SAN solutions are provided by slightly smaller companies that are implementing the same basic technology in slightly different ways, to provide the workstation performance that post-prodution requires. What other industry needs 300MB/s to the average worker's desktop? Usually all of the high performance processing is done on servers that have the fast storage attached, and only the results are sent to the client system.
Your network engineer seems to have fallen into the same mindset, since he provided you a solution with extremely fast storage access from the server (2.4GB/s which is 24Gb/s) but only Gigabit connectivity to the clients (100MB/s which is 1Gb/s). So the server connection is 24 times faster, but you are only trying to serve 6 clients, so not matter what, at least 3/4 of your disk throughput (24-6= 18GB/s) is wasted.

As far as your other questions, yes that is my Qlogic switch, yes other vendor's solutions should be comparable. I agree $1K per seat license fee is a racket, but I believe the resulting functionality far outways the cost IF you have multiple users editing the same content.

As far as channels go, 4Gb Fibre theoretically delivers 500MB/s, Dual channel is 1GB/s, and Quad Channel is 2GB/s. 2Gb Fibre gives half the max performance at those settings. I would put the reality factor at 20-30% loss, so you should expect 350-400MB/s per channel PROVIDED that your disks support that rate. I estimate about 50MB/s per SATA drive. Also remember that you have two links to factor in. Each workstation has a dedicated link to the switch, but the connection between the array and the switch is shared by all the systems. Therefore you array should be connected with faster interface. In your case Quad channels to your array would be ideal for sharing to six single channel stations, but in reality, a dual channel link should be sufficient if you are doing compressed work.

If you are smart about the dual channel array you purchase now, it the future you should be able to purchase another identical one, stripe them together, and boom, you have quad channel solution, with twice the capacity, making use of all of your original investment, but now supporting uncompressed 2K to 4 or 5 stations at once.

[Peter Karlsson]

Im just building a new storageserver and after months of investigation I finally received all building blocks. Im not a big spender, and I like to put together stuff on my own, so if I can save a dollar here and there its all for the good.

The situation. We´re a bunch of freelancers working under the same roof. Our projects usually ends up at eating more than 100GB each. I usually do about 2-3 projects/week (3D renderings etc.). So storage space has always been an issue. Now I got about 10GB of freespace on our last 4TB server. The network load is also huge since everyone is accessing files on a single 1Gbe connection.

So, the criteria for a new server. Be able to grow over time but don´t cost a fortune initially. Have enough throughput for all eight people to use videofiles/famestacks etc + having 20+ computers at the renderfarm crunching data. I´ve decided against fibre because it simply cost too much for us right now.

The first thing I did was looking at auctions/2nd sales for cheap old SCSI Chassis. I found two for about $100 each, with PSU and can hold 12 drives each. I threw everything out except the PSUs, and plugged the drives into the HDD containers.

Then I waited until 3ware released their latest 9690SA cards. I received one a couple of days ago. Now Im looking at buying a 36-port SAS expander from Rancho Systech (they are really hard to get in europe, so Im investigating how to get one straight from the US). I think the price is about $2400, but can connect up to 32 SAS/Sata drives with SAS->4xSATA expanders. Another option is to buy 16-bay rack chassis from Xtore. They seem quite solid, and doesn´t cost a fortune either (you can stack them on top on eachother, with a maximum of eight = 128 Sata drives).

Then I bougth a Intel PT Quad server adapter card. The nice thing is that you can do something called port trunking/bonding or as Intel calls it "Static Link Aggregation". This can be used for up to eight adapters, sharing the throughput on the switch. So theoretically I won´t see less performance compared to a fibre channel network solution.

Well.. Soon I will test this setup, and see how it works. Its damn cheap compared to a pre-built setup from a vendor. But I expect it to work quite well. The nice thing about SAS is that I can build another server with the same components, and use the same disk array. This will also help balance the network throughput, and have fault tolerance on a server level.

I hope someone founds this interesting! :)

/Regards Peter K

[Kevin Halverson]

Thanks for the post Peter, I found your ideas to be very interesting and applicable. The concept of "trunking/bounding" is clever, I would love to hear more about your experiences once you finish your testing. Please do post some bench marks for this configuration if you find the time.

Thanks again for the post,

Kevin Halverson

[Stacey Spears]

Then I waited until 3ware released their latest 9690SA cards. I received one a couple of days ago.

I just purchased one of these as well. I got the 8 external version. I am using eight 750GB Seagate .10 drives in the Burly rack mount. I am still waiting on the drives and rack mount to arrive. Since I am direct connecting the storage to my workstation, I am using GPT to have one volume to edit on.

[Mike McCarthy]

I just purchased one of these as well. I got the 8 external version. I am using eight 750GB Seagate .10 drives in the Burly rack mount. I am still waiting on the drives and rack mount to arrive. Since I am direct connecting the storage to my workstation, I am using GPT to have one volume to edit on.

You need to have a 64bit OS access GPT disks, correct? I know Adobe supports this, but few of the usual I/O cards do. Blackmagic is the only one I am aware of with 64bit drivers, is that what you are using?

[Stacey Spears]

You need to have a 64bit OS access GPT disks, correct?

No, 32-bit Vista and 32-bit Server 2003 SP1 both support GPT. If you want to run XP, then you need 64-bit.

[Mike McCarthy]

No, 32-bit Vista and 32-bit Server 2003 SP1 both support GPT. If you want to run XP, then you need 64-bit.

Good to confirm that. I had heard conlicting rumors about GPT support in the 32 bit versions of Vista. One of the few benefits of Vista-32 I guess. I hope to be able to run XP64 before I make the jump to Vista. Hopefully Vista SP1 will improve things significantly on that front.

[Joe Carney]

Since my post got lost in the crash..what are peoples opinions on
TerraBlock SANs?. They seem to be an affordable option for small boutique shops.

[Stacey Spears]

I am torn on which OS to run.

1. Vista 32-bit - This is what I am currently using. Had trouble installing CS3. Now I can't get CS3 updates to install. GPT appears to be working.
2. XP 32-bit - Would like to switch back to XP, but no GPT support.
3. Server 2003 SP1 32-bit - More or less XP with GPT support. Problem is CS3 blocks install on Server so they don't have to support. Have to do manual work to get CS3 to install.

4. 64-bit anything - I would love to run 64-bit, but there are the issues of driver application support. Unclear how well CS3, Redcine and SCRATCH will work on a 64-bit OS, not to mention other applications I will want to use.

[Kevin Halverson]

Seems like the most stable option is 2, but yeah no GPT support. Can you repartition your storage so you don't exceed 2TB segments?

Would be interested to hear from anyone using XP Pro x64.

[Stacey Spears]

Can you repartition your storage so you don't exceed 2TB segments?

Yes. How do applications deal with this? Lets say I create a feature length edit on the R3D in [insert favorite app here] and then export as [favorite uncompressed file sequence]. Does the application automatically move to the next volume when low on disc space?

[Mike McCarthy]

Yes. How do applications deal with this? Lets say I create a feature length edit on the R3D in [insert favorite app here] and then export as [favorite uncompressed file sequence]. Does the application automatically move to the next volume when low on disc space?

If your favorite App is After Effects, and you are going to a frame sequence, then yes, it does support that, and has for many versions. It is called storage overflow, and you can set it to kick in when your drive is say 95% full, and you can que up at least 4 volumes. And that would be one of the LEAST efficient solutions to the problem. (Right up there next to installing Vista purely for GPT support)

There are many ways to access a volume over 2TB. One is to use Vista or XP64 to access GPT disks. Downsides of these are limited driver support, so it depends on your hardware. Do you have an HD-SDI card or any other unique hardware that needs special drivers? XP64 isthe preferable option, and will offer the best performance, provided all of your HARDWARE supports it. CS2/3 and most other SOFTWARE support it automatically.

The next option is to use larger block sizes. I posted info about this last week:


Increasing the formating block size from the default of 512 bytes to 4096 bytes (8x increase from .5K to 4K) increases the maximum drive size windows can recognize from 2TB to 16TB.

The long answer is: as a 32bit OS, XP can address 2^32 blocks (just over 4 billion) so if each block is .5K or 2^9, then 2^9 x 2^32 = 2^41 (add the exponents). 2^41 is just over 2 trillion, therefore 2 terabytes. If we increase the block size to 2^12 (add 3 to the exponent, or multiply by 8 which is 2^3) our total addressable size increases from 2^41 to 2^44, which is 16 trillion bytes, or 16TB. (Quick reference for exponents, 2^10 is approximately 1000, so for every digit in the tens place, append ",000" to the result first digit result. 2^4=16, 2^24= about 16,000,000)

Are you planning to use RAID 5/6 for your 8x 750GB drives? If not, stripe them in XP, no further problem. For RAID 5/6, you will have a 4.5 or 5.25 TB volume, so a 2K block size will allow XP32 to address the whole volume. In order to increase the block size, both the RAID controller and the apps must support it. I can't confirm the 3ware card, but most do, you may have to call them to check. Most software supports increased block sizes, Adobe does for sure, but be aware it could be an issue for some "accelerated " apps.

The last option is to stripe or span 2TB segments together in Windows disk manager. 3Ware has an Auto-Carving feature that I assume works on this principle. I have been told that striping between to volumes on the same physical disks can hurt performance, but I can't confirm that, and spanning should be fine.

I was previously limited by Matrox's lack of block size support, so all of my storage is designed around 2TB segments (8x250GB disks, or 5x400GB disks, or 4x500GBdisks, usually plus one for RAID 5 redundancy) They now support larger block sizes, but I haven't purchased storage in a while. XP64 with GPT might not be a bad way to go if you have full hardware support. Otherwise increase the block size for no hit on performance, but the minimum file size increases to 2K, no big deal for post production.

[Stacey Spears]

(Right up there next to installing Vista purely for GPT support)

The original reason for Vista was GDI+ 1.1. I wanted to take advantage of the 8x8 antialiasing (vs. 8x4 in 1.0) in an application that I have been developing. It turns out that if I copy the GDIplus.dll from Office and drop it in the same folder as my app, all is well.

I am planning to run some RAID0, 5 and 6 tests. RAID5 is probably what I will use. I will also try increasing the block size as you have suggested. I will get all of this sorted before I settle on the final OS.

[Mike McCarthy]

By all indications, Vista will introduce a significant performance hit, so I would make sure it is really necessary before using it on an edit system. It seems that you don't have a dedicated system, so there are obviously other factors you have to take into account. It seems pretty clear that XP64 will offer the best post-production performance in most situations, but my hardware config currently prevents it.

[Edgar Pitts]

Does XP64 allow for FC SAN systems? (Software & Hardware)

[Mike McCarthy]

Fibre Channel SANs were originally designed for servers, so hardware support for 64bit OS is widespread. (Some HBAs don't even have XP32 drivers, 64bit OS only) The software is a different issue, for metadata management, but I am sure certain solutions work. MetaSAN does, and StorNex should as well.

As with most technology advancements, as long as all the little individual pieces support it, you will see major benefits. And there are FC-SAN pieces that support 64bit, but not every single option will, so you will have to check carefully.

[Sven Seynaeve]

I've been doing some searching but could not find a descent answer on this case. Could we technically not built a storage device as good as the dulce or even higher end by ourselves using a supermicro 24 bay chassis in combination with 3ware 9690 card (maby 3of them??) , eventually working as 3x8 with the new hitachi 1TB disks?? Just the case with the SAS expander and you you should wire the cabling if this is possible I couldn't find out by myself. IN case you use a supermicro chassis to build the pc as well (as I did in january for my new workstation) you could put in 8 drives as well., this would be 32TB of storage for a very reasonable price,, but still..... if this works?????

[SalaTar]

Quote:
Windows XP 32 is limited to 2TB volumes unless you increase the block size of the drives.

Ignoring the cost of the OS for a second, what about using Server 2003 SP1 with GPT enabled?

Stacey Spears
nope
explain the 3.18 TB I have on xp 32 x3

[Peter Karlsson]

I haven´t done all my different tests yet BUT. You don´t need three 9690 cards. You can already hook up 128 Sata devices (Only 64 SAS drives) on one card. Then you can from what I understood from 3ware, build three Raid-5 Units (say 10 drives each) and then wrap those three units together as a Raid-0 unit.. Im investigating this right now, how it works in practice. But sofar everything about the 9690 card has been great..

If you build an Internal storage. My money would be on a AIC 40-drive case.. Those are reasonable priced, and fit all the storage you need for a couple of years ;)

I'll be back with benchmarks for the ones interested in the 9690SA card!

/Regards Peter K

[Stacey Spears]

Peter,

How do you extend the 9690? I have the 8E, but it is not clear how to add more to it.

[Kevin Halverson]

I'll be back with benchmarks for the ones interested in the 9690SA card!

I would be interested to learn the out come of your benchmark testing for both burst and sustained reads and writes of various file sizes.

[Peter Karlsson]

Peter,

How do you extend the 9690? I have the 8E, but it is not clear how to add more to it.

You'll have to use some kind of "extenders". There are alot of different kinds of extenders. What you want for your 8E card is something with mini SAS (SFF-8088) connectors.

I just received my XJ-SA24-316R case from Xtore. It has dual IO modules in the back. So from the 3Ware card, there is two SFF-8088 cables going out from the card and straight into the Xtore case. From that case there are two mini SAS connectors. So I´ve put two SFF8088->4xSATA cables (these cables are also "extenders") so I can hook up eight more drives. Giving you a total of 24 drives!

Now.. Those Xtore cases are pretty expensive (not crazy expensive, just expensive if you consider other "not approved/tested by 3ware" options). Another cheaper way to go, if you can find cheap cases just to host the drives are the 36 port expander from Rancho Systech (I think these are the only one in the world doing these kind of fanout extenders). This expander has 9 x miniSAS IO. So you can connect one of your 3ware cables to it. And plug 8 x SFF8088->4xSATA cables giving you 32 drives!. Now this is cheaper/drive if you already have a old cage.. I looked at ebay, and you can get cheap old SCSI cases for nothing. I will buy a Rancho expander later on. I had a deadline, so I needed something fast, so I went with Xtore, since I could get them in 2-3 days. Since it serial, you can build pretty much what you want :)

[Michele Gavazzeni]

IF YOU DON'T NEED TO SHARE YOUR STORAGE WITH MULTIPLE WORKSTATIONS

CIPRICO HAS THE BEST PRODUCT YOU CAN BUY
MediaVault 5108/5116
SATA II At a raw speed of 20Gbps

[Mike McCarthy]

IF YOU DON'T NEED TO SHARE YOUR STORAGE WITH MULTIPLE WORKSTATIONS

CIPRICO HAS THE BEST PRODUCT YOU CAN BUY
MediaVault 5108/5116
SATA II At a raw speed of 20Gbps

In reality, don't expect more that half of that, but 1.2GB/s should be more than enough for almost any need. I agree that the new PCIe concept to the most efficient design for a high speed storage interface, IF you want direct attached storage insteadof Shared SAN capability.

[Peter Karlsson]

Here´s some initial benchmarks on the 3ware 9690SA card:

I´ve used HD_Speed for benchmarks. Unfortunately one of my 1TB drives was broken, so the larger array isnt up and running yet. I will do a test of 10 drives as soon as I get a new drive back from the reseller. I expect them to do better than my old Western Digital drives..

8 WD drives 300GB Raid6 Read Benchmarks 1.7TB 3ware 9690SA External (direct attached)

Block Size 512KB 387.5MB/s / burst 419.8MB/s
Block Size 1MB 438.0MB/s / burst 473.2MB/s
Block Size 4MB 438.4MB/s / burst 240.0MB/s
Block Size 16MB 438.4MB/s / burst 352.0MB/s

/Regards Peter K

[Stacey Spears]

Peter,

What settings are you using on the 9690 when you create a unit?

Write cache?
Auto Verify?
Queuing?
SaveStor (Protection, balance or performance)?

Have you had any problems with a drive suddenly going inoperable?

Frys had 500GB WD SE16 drives for $99 this weekend, so I picked up 8. Having trouble during verify with at least one and possibly two others.

My drives are mounted in ICY DOCK removable enclosures. (MB123IK-1B)

[Stacey Spears]

Both Seagate and WD offer RAID versions of their drives for a premium. Anyone have an opinion, one way or another, on them?

[Peter Karlsson]

I´ve turned on Write cache and queuing, and went with savestor performance. I´ve tested the seagates drives going from raid-6 with 6 drives, then migrated to raid-5 with eight, and now went to raid-6 again with 9 drives. Worked flawless, but the rebuild sure takes a long time :/ I did the verify after the rebuilds without any problem. It will become very interesting once I start adding more drives to the raid. I hope the performance will follow along nicely as I keep adding drives.

I had one drive (the broken one) going inoperable. It behaved very oddly. I read that you can get time out with some drives if you set them to sata-II instead of sata-I. But all other drives seems to go well.

Do you direct attach them with SAS->4xSata cables?

[Stacey Spears]

Yes, SAS->4xSata cables. Each output is connected to 4 drives. I tried w/ and w/o the ICY DOCK enclosure.

I am hitting the inoperable error a lot. When I reboot, the drive returns and the verify continues.

I just sent mail to support to see if they have any ideas. My machine is still Vista at this point. Until I am done experimenting, I don't want to reinstall the OS. They don't have Vista specific drivers, which might be part of the problem.

I was thinking it might also be a heat issue. My MB has two PCIe slots with a PCI in between. The graphics card only works in slot 1. It is a Quadro 4600, which runs hot. It ends up right next to the 9690. I just mounted a fan to keep them both cool and started a new verify.

If I can get this working, I think I will go the Rancho route and have 24 drives in RAID50. (3x RAID5)

I am testing as JBOD and RAID0 at this point. Don't want to sit through the initialize time on RAID5/6 until I work the kinks out. How long does the migration or initial initialize take?

[Stacey Spears]

I made some measurements this evening. I tested two RAIDs.

MB Tyan 2895. Dual 290 Opteron w/ 4 GB RAM. Vista Ultimate 32-bit.

RAID # 1. nVidia RAID, built into Tyan 2895 MB. 4x WD Raptor 10k drives in RAID0.
RAID # 2. 9690SA-8E. 7x WD SE16 500GB drives in RAID0.

I used two tools, HD Speed and HD Tune 2.54.

RAID # 1 - HD Tune 2.54
Transfer Rate:
Minimum - 108.5 MB/sec
Maximum - 183.3 MB/sec
Average: 142.4 MB/sex
Access Time - 7.8 ms
Burst Rate - 109.2 MB/sec
CPU Usage - 10.0%

RAID # 2 - HD Tune 2.54
Transfer Rate:
Minimum - 82.0 MB/sec
Maximum - 139.9 MB/sec
Average: 120.6 MB/sex
Access Time - 10.2 ms
Burst Rate - 81.3 MB/sec
CPU Usage - 2.5%

RAID # 1 - HD Speed
Size Read Burst
512 331.6 434.8
1024 331.7 441.0
2048 331.8 432.3
4096 331.6 166.6
16 MB 332.2 296.0

RAID # 2 - HD Speed
Size Read Burst
512 415.1 436.6
1024 491.8 515.7
2048 510.6 560.4
4096 511.6 579.2
16 MB 513.6 596.8

It is interesting that HD Speed and HD Tune numbers are so different. I only ran read tests on HD Speed.

[Mike McCarthy]

Good demonstration as to why it is unnecessary to invest in 10k rpm disks if you need a large video array. The new Seagate 1TB drive beats the Raptor in every benchmark except rotational latency. We need a larger scale for capacity, so why not depend on scale for speed as well?

[Peter Karlsson]

For me the rebuilds takes 3-4 days actually.. I know its crazy! But I have about 30 rendernodes + eight workstation working against the 3ware controller, while Im doing the rebuild. So I think the disk access doesn´t leave much time for the controller to do the actual rebuild.. But it works! Thats the important part :)

I would recommend raid-6 though. I doesnt seem to be any speed penalty against raid-5.

As soon as I got this new rebuild wrapped up, I'll do tests in HD Speed, HD tach and HD tune.. HD Speed seems.. Uhm a little unreliable for raid setups. I get very strange result for burst tests.

3ware are known for flaky drivers for their first releases.. So I guess your troubles with disconnecting drives will go away in next release. Btw.. Do you have the BBU module?

[Luke Goddard]

Another thing when comparing the 10k drives to 7.2k:

The outmost edge of the 10k is around the 150GB mark (maximum capacity), which is only at the 20% full mark of the 750GB 7.2k drives... if you compare the speed at these points you can see why 10k isn't so much of a bonus! Particularly when you need all the fast storage you can get...

[Stacey Spears]

No, I did not get the BBU, I am am depending on a UPS.

[Stacey Spears]

Now Im looking at buying a 36-port SAS expander from Rancho Systech...I think the price is about $2400

I contacted them today about the RSI-SAS36EXP and they quoted me $8200. That puts a slight crink in my plans. :)

[Peter Karlsson]

Ouch!! $8200 was pretty far from the numbers I found hidden in a re-sellers list. But I guess $8200 makes sense since I got £5000 from the official UK dealer :(

I´ve now running on the Xtore 3U 16-drive + 8 directly attached drives, and sofar everything has been running smooth. I got decent figures from ATTO benchmark (it can simulate a better "real world performance" since you can tell it to do a Queue depth of 10 files (like having 10 users accessing different files at the same time)... I get at an average of 200mb/s for read/writes from ATTO.. I hoped for a little higher, but I only got a 4xPCIe slot on my machine, and the 3Ware is a 8xPCIe card..

Anyway, Im not looking at running out of diskspace any time soon :)