Thread: Storage

Reply to Thread
Page 2 of 5 FirstFirst 12345 LastLast
Results 11 to 20 of 48
  1. #11  
    Senior Member
    Join Date
    Aug 2007
    Location
    Dallas, TX
    Posts
    166
    --------------------------------------------------------------------------------

    Here are my benchmarks burst/sustained in MB/s at these block sizes:

    4TB Huge Raid from Ciprico (10 400GB IDE drives in RAID50 connected with Dual SCSI 320 via ATTO card)
    512KB 220/215
    1MB 232/230
    4MB 455/370
    16MB 460/380

    5TB ProAVIO Array (14 400GB SATA Drives in RAID 50 connected with Dual SCSI 320 via LSI card)
    512KB 400/365
    1MB 425/395
    4MB 425/395
    16MB 420/420

    2x 2TB Internal SATA Arrays (8 250GB SATA Drives in Raid 5 connected to Broadcom RaidCore PCI-X card) (Currently a degraded Raid 5)
    512KB 1380/97
    1MB 1020/140
    4MB 1010/140
    16MB 155/140

    2x 1.2TB Internal Arrays (3 400GB SATA Drives in Raid 0 to integrated NForce Raid)
    512KB 305/155
    1MB 400/150
    4MB 335/155
    16MB 128/155

    1TB Internal Array (4 250GB SATA Drives in Raid 5 to Promise 4200 PCI-66 Card)
    512KB 190/158
    1MB 193/160
    4MB 190/170
    16MB 180/172

    I used a utility called HD_Speed which another editor recommended to me awhile back. I can't find the source link right now. These are all read tests. The write tests are destructive, and I don't have time to deal with them. I usually find write speeds to be 20% lower as a rule of thumb.

    McCarthyTech

    --------------------------------------------------------------------------------

    Excellent work McCarthyTech, I really appreciate the effort!

    I am more than a bit surprised by the massive difference between the burst and sustained read of the 8 drive SATA Array. The 16MB test seem very reasonable, but the three lower block sizes are wildly different.

    I am not too surprised that the Dual SCSI array is the winner in terms of both consistency and sustained throughput. With the large number of spindles, availability is rarely ever an issue.

    Again, thanks so much for doing this test!

    khmuse

    --------------------------------------------------------------------------------

    Quote:
    Originally Posted by khmuse
    Excellent work McCarthyTech, I really appreciate the effort!

    I am more than a bit surprised by the massive difference between the burst and sustained read of the 8 drive SATA Array. The 16MB test seem very reasonable, but the three lower block sizes are wildly different.

    I am not too surprised that the Dual SCSI array is the winner in terms of both consistency and sustained throughput. With the large number of spindles, availability is rarely ever an issue.

    Again, thanks so much for doing this test!

    Kevin


    Look closely, that one is currently a degraded Raid 5, I will post new numbers when I have replaced the failed disk, and the controller is not regenerating data on the fly. As far as the high burst number, I believe that is due to the direct interface, or just an anomaly due to the bad drive in some way.

    McCarthyTech

    --------------------------------------------------------------------------------

    After doing some research, it looks like the Ciprico MediaVault 4210 offers the most bang for the buck Dual Channel 4Gbit Fibre Channel array:

    http://dv411.com/mv4210.html
    http://www.ciprico.com/Products/Medi...ediaVault_4210

    Anyone care to comment on this system? We would likely start with one (probably 7.5TB) and add another to get up to quad channel.

    What kind of performance hit is there going from Raid 0 to Raid 3? Is the limiting factor the dual 4Gbit ports?

    Thanks.

    Edgar Pitts

    --------------------------------------------------------------------------------

    Quote:
    Originally Posted by Edgar Pitts
    After doing some research, it looks like the Ciprico MediaVault 4210 offers the most bang for the buck Dual Channel 4Gbit Fibre Channel array:

    http://dv411.com/mv4210.html
    http://www.ciprico.com/Products/Medi...ediaVault_4210

    Anyone care to comment on this system? We would likely start with one (probably 7.5TB) and add another to get up to quad channel.

    What kind of performance hit is there going from Raid 0 to Raid 3? Is the limiting factor the dual 4Gbit ports?

    Thanks.

    Edgar


    I have the SCSI version of the product you are speaking of. With a conservative estimate of 50MB/s per drive, 10 drives will deliver 500MB/s in Raid 0. In Raid 3, you lose the banwidth of two drives, leaving you 400MB/s. The other limiting factor when moving to RAID 3 is that the controller can cause a bottleneck. Making the pressumption that is it similar in capability to the older SCSI version, I have never experienced any bottleneck or performance loss since switching over from Raid 0 to Raid 3 a few months ago. The same can not be said of cheap internal PCI Raid cards.

    What workflow are you planning to use to edit (AJA, Decklink, Desktop, etc.)? A 7.5TB array introduces some interesting issues into the workflow. At Raid 3, that solution will provide two 3TB volumes that must be striped together. Windows XP 32 is limited to 2TB volumes unless you increase the block size of the drives. This is not supported by some software. Adobe does support it, Matrox recently added support, can't confirm Decklink or AJA, but I believe they do. Just one more thing to check on, to allow your workstations to access more than the first 2TB of each half without a complex solution. I know, not what you wanted to hear, but better to find out now. The same feature that makes the gigabit solution really slow, masks this problem so Windows is not limited in the part it can see over the network. This is the same reason that Macs can write to NTSC drives over a network, because the host PC masks the nature of the drive, and translates the data.

    This is why all of the storage solutions I have are under 2TB to a side. My 14drive array is divided in half, and each side of seven disks includes a hot spare, and a drive worth of parity data. The remaining five 400GB disks gets me right to 2TB without needing to increase the block size. Striping the two halves in Windows gives me a 4TB volume. These were all setup before our Matrox systems supported larger block sizes. My new SAN will have the same limitation for other reasons. My 8TB will be divided into four 2TB chunks, and combined in pairs into two 4TB drive letters.

    McCarthyTech
    Reply With Quote  
     

  2. #12  
    Senior Member
    Join Date
    Aug 2007
    Location
    Dallas, TX
    Posts
    166
    --------------------------------------------------------------------------------

    Quote:
    Windows XP 32 is limited to 2TB volumes unless you increase the block size of the drives.

    Ignoring the cost of the OS for a second, what about using Server 2003 SP1 with GPT enabled?

    Stacey Spears

    --------------------------------------------------------------------------------

    I wonder Ciprico has its own format to be shareable between Win and Mac natively. It says it supports it, but i don't know what it really means by support it.

    Rivai

    --------------------------------------------------------------------------------

    Quote:
    Originally Posted by Stacey Spears
    Ignoring the cost of the OS for a second, what about using Server 2003 SP1 with GPT enabled?

    That only works if you are sharing over an ethernet network. Data passes through server, server translates, allowing saving to GPT disks not otherwise supported by XP32. In a SAN, the data never passes through the server, although certain solutions require a server to monitor metadata to prevent data corruptions and overwrites. Metadata servers cannot translate to mask drive formating incompatibilities. Only exception I know of is that I believe MetaSAN has a way of allowing Macs to write to NTFS on a SAN. They still can't solve the 2TB issue by having PCs write to HPFS, although that is supported, the 2TB limit remains, unless you increade the block size.

    McCarthyTech

    --------------------------------------------------------------------------------

    Quote:
    Originally Posted by Rivai
    I wonder Ciprico has its own format to be shareable between Win and Mac natively. It says it supports it, but i don't know what it really means by support it.

    Ciprico advertizes that their products support Mac and PC, but not at the SAME TIME. Their arrays can only be connected to ONE system at a time, so you use NTFS for PC or HPFS for Mac. The exception is if you are using their fibre array on a SAN, at which point you will need SAN software to share files in any given volume, and that software will establish the Mac/PC sharing limits.

    NO hardware can support direct connection and sharing of a volume between multiple systems, without dedicated sharing software, either from the OS in the case of ethernet sharing, or from SAN software in the case of fibre channel. That is what has me confused about your CalDigit claims. Their PCIe solution is very innovative, and should work very well as direct attached storage to an edit workstation. Unless they have some MAJOR tricks up their sleeves, there is NO way to connect and share that unit on a fibre SAN. The best you could do is 10Gb ethernet or trunked Gigabit possibly, but I doubt that will be the most economically feasible solution to achieve reliable uncompressed data rates on a shared solution.

    McCarthyTech

    --------------------------------------------------------------------------------

    Quote:
    Originally Posted by McCarthyTech
    Windows XP 32 is limited to 2TB volumes unless you increase the block size of the drives. This is not supported by some software. Adobe does support it, Matrox recently added support, can't confirm Decklink or AJA, but I believe they do. Just one more thing to check on, to allow your workstations to access more than the first 2TB of each half without a complex solution.

    Can you briefly describe the advantages and disavantages of increasing the block size?


    Quote:
    Originally Posted by McCarthyTech
    I know, not what you wanted to hear, but better to find out now.

    You are providing extremely valuable insight. I truly thank you for your time!!!

    Edgar Pitts

    --------------------------------------------------------------------------------

    Quote:
    Originally Posted by Edgar Pitts
    Can you briefly describe the advantages and disavantages of increasing the block size?

    Increasing the formating block size from the default of 512 bytes to 4096 bytes (8x increase from .5K to 4K) increases the maximum drive size windows can recognize from 2TB to 16TB.

    The long answer is: as a 32bit OS, XP can address 2^32 blocks (just over 4 billion) so if each block is .5K or 2^9, then 2^9 x 2^32 = 2^41 (add the exponents). 2^41 is just over 2 trillion, therefore 2 terabytes. If we increase the block size to 2^12 (add 3 to the exponent, or multiply by 8 with is 2^3) our total addressable size increases from 2^41 to 2^44, which is 16 trillion bytes, or 16TB. (Quick reference for exponents, 2^10 is approximately 1000, so for every digit in the tens place, append ",000" to the result first digit result. 2^4=16, 2^24= about 16,000,000)

    The disadvantage is that certain software doesn't support this very well, probably because it makes certain assumptions about your drives, in order to attempt to deliver better performance. Older hardware (controllers and arrays) don't support it either, but most recent products should.

    McCarthyTech
    Reply With Quote  
     

  3. #13  
    Thanks for reposting all of that. I have one more long post that I authored that I happen to have saved. I also have a lot of related info on my tech website www.hd4pc.com so feel free to check that out. The following post was intended to explain why the traditional model of client/server data sharing over ethernet is not ideal for high-end post-production workstations:

    The easy one first. The speed of the server in this specific application will have little impact on the throughput.

    The problem with you proposed setup is that ethernet is a very inefficent interface protocol. Gigabit should provide 125MB/s (1000mb/8 bits in a byte), but in practice I have never seen more than 50MB/s, due to the limitation of network adaptors, switches, cable line noise, the mood of server gods, or whatever. The TCP/IP protocol is designed for reliability over long distances, not throughput. I believe it cuts transfer speeds in half any time it experiences an error, and in half again if the problem reoccurs. Other interfaces perform much closer to their specifications, like fibre channel, SCSI, or SATA. This is the same reason the Firewire400 is preferable USB even though USB is 20% faster on paper. In reality the overhead of certain protocols prevents anyone from actually experiencing maximum performance.

    As far ar utilizing the equipment you had speced, a shared SAN does not pass the data through a server before it reaches the workstation. Certain SAN software solutions require a server to monitor array and direct traffic at the file level, but the data never passes through that server, so performance is not as much of in issue for a machine in that role. The MD1000 arrays you speced would work well as direct attached storage for your workstations since they are eSATA based, but Dell only supports them for server attachment. Even if they did, there is no way to easily share the data on them besides the slow option of ethernet. Similar arrays that interface with fibre channel would work better for you. Dell makes these fibre arrays as well, but once again limits their connection to PowerEdge Servers. I usually like Dell products, and am writing this on my Dell XPS 1210 laptop, next to my Dell Xeon workstation, but unfortuneately Dell does not offer any form of SAN product suitable for media creation or post-production. I am not too impressed with HP or IBM's offerings either, although I know IBM is working to improve that. The problem is that they are all used to the server mindset that doesn't work for post-production. Most of the good shared SAN solutions are provided by slightly smaller companies that are implementing the same basic technology in slightly different ways, to provide the workstation performance that post-prodution requires. What other industry needs 300MB/s to the average worker's desktop? Usually all of the high performance processing is done on servers that have the fast storage attached, and only the results are sent to the client system.
    Your network engineer seems to have fallen into the same mindset, since he provided you a solution with extremely fast storage access from the server (2.4GB/s which is 24Gb/s) but only Gigabit connectivity to the clients (100MB/s which is 1Gb/s). So the server connection is 24 times faster, but you are only trying to serve 6 clients, so not matter what, at least 3/4 of your disk throughput (24-6= 18GB/s) is wasted.

    As far as your other questions, yes that is my Qlogic switch, yes other vendor's solutions should be comparable. I agree $1K per seat license fee is a racket, but I believe the resulting functionality far outways the cost IF you have multiple users editing the same content.

    As far as channels go, 4Gb Fibre theoretically delivers 500MB/s, Dual channel is 1GB/s, and Quad Channel is 2GB/s. 2Gb Fibre gives half the max performance at those settings. I would put the reality factor at 20-30% loss, so you should expect 350-400MB/s per channel PROVIDED that your disks support that rate. I estimate about 50MB/s per SATA drive. Also remember that you have two links to factor in. Each workstation has a dedicated link to the switch, but the connection between the array and the switch is shared by all the systems. Therefore you array should be connected with faster interface. In your case Quad channels to your array would be ideal for sharing to six single channel stations, but in reality, a dual channel link should be sufficient if you are doing compressed work.

    If you are smart about the dual channel array you purchase now, it the future you should be able to purchase another identical one, stripe them together, and boom, you have quad channel solution, with twice the capacity, making use of all of your original investment, but now supporting uncompressed 2K to 4 or 5 stations at once.
    For more information on PC based Post-Production, check out my tech website at www.hd4pc.com
    Reply With Quote  
     

  4. #14  
    Senior Member
    Join Date
    Jan 2007
    Location
    Stockholm
    Posts
    377
    Im just building a new storageserver and after months of investigation I finally received all building blocks. Im not a big spender, and I like to put together stuff on my own, so if I can save a dollar here and there its all for the good.

    The situation. We´re a bunch of freelancers working under the same roof. Our projects usually ends up at eating more than 100GB each. I usually do about 2-3 projects/week (3D renderings etc.). So storage space has always been an issue. Now I got about 10GB of freespace on our last 4TB server. The network load is also huge since everyone is accessing files on a single 1Gbe connection.

    So, the criteria for a new server. Be able to grow over time but don´t cost a fortune initially. Have enough throughput for all eight people to use videofiles/famestacks etc + having 20+ computers at the renderfarm crunching data. I´ve decided against fibre because it simply cost too much for us right now.

    The first thing I did was looking at auctions/2nd sales for cheap old SCSI Chassis. I found two for about $100 each, with PSU and can hold 12 drives each. I threw everything out except the PSUs, and plugged the drives into the HDD containers.

    Then I waited until 3ware released their latest 9690SA cards. I received one a couple of days ago. Now Im looking at buying a 36-port SAS expander from Rancho Systech (they are really hard to get in europe, so Im investigating how to get one straight from the US). I think the price is about $2400, but can connect up to 32 SAS/Sata drives with SAS->4xSATA expanders. Another option is to buy 16-bay rack chassis from Xtore. They seem quite solid, and doesn´t cost a fortune either (you can stack them on top on eachother, with a maximum of eight = 128 Sata drives).

    Then I bougth a Intel PT Quad server adapter card. The nice thing is that you can do something called port trunking/bonding or as Intel calls it "Static Link Aggregation". This can be used for up to eight adapters, sharing the throughput on the switch. So theoretically I won´t see less performance compared to a fibre channel network solution.

    Well.. Soon I will test this setup, and see how it works. Its damn cheap compared to a pre-built setup from a vendor. But I expect it to work quite well. The nice thing about SAS is that I can build another server with the same components, and use the same disk array. This will also help balance the network throughput, and have fault tolerance on a server level.

    I hope someone founds this interesting! :)

    /Regards Peter K
    http://www.cgsweden.com Vfx, Motion Capture, Motiongraphics and Character Animation.
    Reply With Quote  
     

  5. #15  
    Thanks for the post Peter, I found your ideas to be very interesting and applicable. The concept of "trunking/bounding" is clever, I would love to hear more about your experiences once you finish your testing. Please do post some bench marks for this configuration if you find the time.

    Thanks again for the post,

    Kevin Halverson
    Reply With Quote  
     

  6. #16  
    Senior Member Stacey Spears's Avatar
    Join Date
    Sep 2007
    Location
    Sammamish, WA
    Posts
    1,098
    Then I waited until 3ware released their latest 9690SA cards. I received one a couple of days ago.
    I just purchased one of these as well. I got the 8 external version. I am using eight 750GB Seagate .10 drives in the Burly rack mount. I am still waiting on the drives and rack mount to arrive. Since I am direct connecting the storage to my workstation, I am using GPT to have one volume to edit on.
    Reply With Quote  
     

  7. #17  
    Quote Originally Posted by Stacey Spears View Post
    I just purchased one of these as well. I got the 8 external version. I am using eight 750GB Seagate .10 drives in the Burly rack mount. I am still waiting on the drives and rack mount to arrive. Since I am direct connecting the storage to my workstation, I am using GPT to have one volume to edit on.
    You need to have a 64bit OS access GPT disks, correct? I know Adobe supports this, but few of the usual I/O cards do. Blackmagic is the only one I am aware of with 64bit drivers, is that what you are using?
    For more information on PC based Post-Production, check out my tech website at www.hd4pc.com
    Reply With Quote  
     

  8. #18  
    Senior Member Stacey Spears's Avatar
    Join Date
    Sep 2007
    Location
    Sammamish, WA
    Posts
    1,098
    You need to have a 64bit OS access GPT disks, correct?
    No, 32-bit Vista and 32-bit Server 2003 SP1 both support GPT. If you want to run XP, then you need 64-bit.
    Reply With Quote  
     

  9. #19  
    Quote Originally Posted by Stacey Spears View Post
    No, 32-bit Vista and 32-bit Server 2003 SP1 both support GPT. If you want to run XP, then you need 64-bit.
    Good to confirm that. I had heard conlicting rumors about GPT support in the 32 bit versions of Vista. One of the few benefits of Vista-32 I guess. I hope to be able to run XP64 before I make the jump to Vista. Hopefully Vista SP1 will improve things significantly on that front.
    For more information on PC based Post-Production, check out my tech website at www.hd4pc.com
    Reply With Quote  
     

  10. #20  
    Senior Member Joe Carney's Avatar
    Join Date
    Dec 2006
    Location
    Northern VA
    Posts
    1,022
    Since my post got lost in the crash..what are peoples opinions on
    TerraBlock SANs?. They seem to be an affordable option for small boutique shops.
    Reply With Quote  
     

Posting Permissions
  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts