16 core/32 thread/64GB RAM Mac setup, but not fully working yet.

[Joseph S.]

Well, I got the new system built today, but the drivers aren't fully working yet.

Still, check this out. 16 real cores, 32 threads, 64GB RAM. (EVENTUALLY coming soon to Apple, or not).

http://cinespeed.com/3264.jpg

[Joseph S.]

Edit Update: CineBench CPU test is 24.5 in Win7.

Problem is, the latest Lion kernel is not made for this chipset yet. But everything is extremely fast in Windows 7.

This is the most recent set up with 128GB RAM and RAID test.

Get a Red Rocket if you need it.

http://cinespeed.com/128e5.jpg

[Jeff Kilgroe]

Which motherboard did you decide on?

I've got a couple SuperMicro X9DAi boards on order. waiting....

[Joseph S.]

Hey Jeff. Yup, same board. X9DAi. This is running dual E5-2690's. Lion loaded like a charm, but driver issues. If I had the patience to keep trying drivers, I'm sure it would work.

I'm only using a Quadro 1800 card, but I'll try running some benchmarks. Let me know if you have any favorites. (windows 7 for now).

[AndrewThomas]

My word.

[Josh Beadle]

I found out tonight that my 2010 Mac Pro 12 core only supports 64GB of RAM. Tried 96GB and the fan would kick on full speed after running low stress work. Would only shut down if I put the thing to Sleep. Did the SMC reset and even took it to the Mac Store (worthless).

Also surprised that 3 SSD 6G OWC drives in RAID 0 off the main docks run at 540mbs while the external G-Speed 24TB with Atto board runs at 800mbs in RAID 6. Seems that the SSD's have some kind of compression going on that slows them down.

Waiting for a BT Red Rocket this week. Thing plays back at 1/4 now. Could be happy with that but looking at transcoding speed as well.

BTW: Jeff Kilgroe is THE MAN!!

Would pay admission to see his workspace

[Jeff Kilgroe]

I found out tonight that my 2010 Mac Pro 12 core only supports 64GB of RAM. Tried 96GB and the fan would kick on full speed after running low stress work. Would only shut down if I put the thing to Sleep. Did the SMC reset and even took it to the Mac Store (worthless).

Interesting... Did you get the RAM from OWC? If so, talk to them about it. Have you rotated modules around to make sure you don't have a bad one?


Also surprised that 3 SSD 6G OWC drives in RAID 0 off the main docks run at 540mbs while the external G-Speed 24TB with Atto board runs at 800mbs in RAID 6. Seems that the SSD's have some kind of compression going on that slows them down.

They do, as all SandForce chipset based SSDs do, use on the fly compression. It's great for the right kinds of files. But for incompressible files, such as H264, ZIP, R3D, JPG, etc.. it doesn't help. And those advertised speeds that seem so amazing on the SSDs will fall to half or even a third of what they claim when you transfer those files. Especially large ones like R3D clips where you can easily have 100's of GB worth!


Waiting for a BT Red Rocket this week. Thing plays back at 1/4 now. Could be happy with that but looking at transcoding speed as well.

Rocket will help the transcodes for sure. Playback at higher resolutions also helps, but you'll find that anything over half is kinda unnecessary unless you're spot-checking for noise/grain or other artifacts. On most systems, RC-X still struggles even with the Rocket at half res playback or better because the GUI interface can bottleneck with scaling and LUT application. They also don't use a whole lot of RAM for cacheing of what the Rocket processes ahead on playback and that can get real annoying at times.


BTW: Jeff Kilgroe is THE MAN!!

Would pay admission to see his workspace

Hehehe.... You would want your money back. ;)

[Josh Beadle]

@Jeff
Yup, OWC RAM. I'll try rotating it tomorrow - worked after removing 2 x 16GB chips so left it at that. Really, 64GB is plenty since, as you say, RCX doesn't seem to utilize all that much.

What is the speed % diff between having 2 chips in each dock vs 3? Heard (probably from you) that best results had from installing RAM in 3 out of 4 slots. Got 2 x 16GB in each dock now. Seems speedy.

[Will Keir]

+1

Good way to waste a few hours.


even took it to the Mac Store (worthless).

[Sven Seynaeve]

really want to build similar system , however I'd like very much to use Smoke for Mac on it. Would this be possible at all???

[Joseph S.]

So far so good. Got it completely stable with all the drivers except for the Intel i350 ethernet. I do however use the Apple USB ethernet and it works. Audio is next. Lion updated fully as well.

So far zero crashes.

Once I connect the raid, I'll do an RCX test.


http://cinespeed.com/e51.jpg

[Joseph S.]

Even with the fastest CPU's, and 1,300MB/s raid reads, it still can't do full res playback at full speed (does full res at 12fps), but it does 1/2 res at 20-22 fps speed (sometimes 24).

Conclusion is, if you need a Red Rocket, you should buy one :)

[Jose Lomeña]

Try davinci resolve

[Eric Z]

Joseph,
Thanks for the info. Very en-lighting.
You need something that can decode JPEG2000 on-the-fly in hardware, which the RED Rocket does.
Even the best GPUs can't seem to handle this heavy chore. Intel CPUs also seem to choke with 5K R3Ds.
I'm not sure if there's an alternative JPEG2000 card, so the Rocket is probably your best bet for accelerating R3Ds and getting them to play/edit at 5K Full-res with good frame-rates.

[Joseph S.]

Jose: I have to download and try Resolve. Haven't used it before. I just mess around, so I don't have any real footage to grade.

Eric: I actually got a solid 15-16fps in full res. I changed it from a quadro 1700 to quadro 1800 and it went up a few frames. Maybe a better card and I'll get close to full res. Maybe not.

[Brian Merlen]

while your at it can you try full rez playback in a few NLE programs? thanks

[luigivaltulini]

So far so good. Got it completely stable with all the drivers except for the Intel i350 ethernet. I do however use the Apple USB ethernet and it works. Audio is next. Lion updated fully as well.

So far zero crashes.

Once I connect the raid, I'll do an RCX test.


http://cinespeed.com/e51.jpg


Joseph...
WOWO ;) :scared:
Very nice setup
For the network there are solutions ora.Non there are drivers. You can recompile the kernel using the kernel http://www.opensource.apple.com/ Apple's open source, but would become complicated.
I am very curious to see how it handles this pretty pci Supermicro's motherboard, can you save me a dump with IOREGistryexplorer?
http://www.osx86.net/view/93-ioregistryexplorer.html

I thank you, and congratulations again for the Beast;)

One more question you set 5.1 in the SMBIOS? try changing this parameter is important to enable the speedstep. Maybe changing it gives you a little more power.

Luigi

[Joseph S.]

Luigi, I know you're the master here of the hackintosh machines. I really appreciate your kind words.

I'll try the bios change. Supermicro actually sent me a new bios that isn't released yet, which fixes a few speedstep issues. Haven't upgraded the bios yet as it's installing Final Cut.

As far as kernel compiling, I've done it in Linux, but I don't think I have the skills to learn it on this. It's probably similar. I'm using the Apple USB network adapter for now.

I'll keep you guys posted on my next tests.

Joe

[luigivaltulini]

Luigi, I know you're the master here of the hackintosh machines. I really appreciate your kind words.

I'll try the bios change. Supermicro actually sent me a new bios that isn't released yet, which fixes a few speedstep issues. Haven't upgraded the bios yet as it's installing Final Cut.

As far as kernel compiling, I've done it in Linux, but I don't think I have the skills to learn it on this. It's probably similar. I'm using the Apple USB network adapter for now.

I'll keep you guys posted on my next tests.

Joe

thanks, but it's just a hobby.
Now the KING of all is her.
We think they're crazy, you have the best MAC PRO "Hackintosh" available on the market.
A round of applause even from the InsanelyMac team.
The nice thing is that the machines are more powerful forum of REDUSER, as always I like to repeat that here there are people who want to change and move on and experiment. And there are super competent people. Jeff Kilgroe I consider the best, when one needs some information he is ready for them, a show, and has much experience, I follow a lot on this forum.
Jose Lomena knows many tricks on Hackintosh, I like it very fast and precise.
Sen Subhadip a bible of information, always surprises me.
Many many others, in any situational in this forum all give a hand.

Do you follow 'in this forum on your advance, maybe do some tests to see where so Davinci is the limit of this machine;)
GOOOOd luck friend;)

thanks
Luigi

[Joseph S.]

I was finally able to get full res playback at 24 FPS. I have to wait for it to cache. But it loads a 60 second 5K R3D in RAM, and plays back and scrubs in realtime. I guess that's normal for anyone though, so nothing special.

The Cinebench score in win7 was 24.5 for the CPU test.

[Michael Vera]

Sweet fancy Moses!

[Gabriele Turchi]

I was finally able to get full res playback at 24 FPS. I have to wait for it to cache. But it loads a 60 second 5K R3D in RAM, and plays back and scrubs in realtime. I guess that's normal for anyone though, so nothing special.


how long it take to cache it in RAM ?

thanks
g

[Jose Lomeña]

24.5 is a really good score, but in my opinion is not enough for realtime full debayer with 4k redone footage... but you have a beast for after effects!.

[Joseph S.]

how long it take to cache it in RAM ?

thanks
g

5K, 4:1, 30 second clip took around 5 minutes to fully cache with a Samsung 830 SSD. RAM usage is 1GB per 1 second of footage (30 second clip is 30GB for RCX app). So I'm sure a 120 second clip would take up all the system's 128GB of RAM, and max out in full res.

Same footage, 1/2 res took around 2 minutes to load, but much less RAM of around 7GB. 1/4 res took about a minute.

It plays back that footage at 23.98 fps smooth as silk, and scrubs clean, as fast as you can move the mouse.

Since I've never used a Red Rocket, I have no clue what the norm is. Maybe someone can tell me.

[Jeff Kilgroe]

RED Rocket can cache your frames at real-time, or even a bit faster depending on which R3D format. Where most people run into issues on playback and scrubbing is with their available RAM. RC-X seems to do some funny things when it comes to how it manages that cache. On playback it tends to run to the end of the cache and then begin cacheing more, so you can run into stuttering, even with the Rocket. But I think it's this way more so that you can continue to scrub back and forth and tweak your look rather than just playing back endless minutes of footage.

While I'm thinking of it, what RAM did you install for your 128GB Joseph? I'm still waiting on my motherboards, and therefore waiting until they arrive to order my RAM. Selection of proper RAM out there sucks. 1600MHz modules are hard to come by, near impossible for ECC memory. As of right now, I'm probably going to install 128GB via 16 x 8GB of G.Skill Ripjaw X modules. They're available as PC3-12800 (1600MHz) with 9-9-9-24 timings. Mushkin also has the same product in their Blackline series, but it's more expensive and harder to get ahold of. I've been having good luck with G.Skill lately in other systems. Mushkin too, I'm partial to them, they're located near me and I'm kinda biased. :)

There's also the Kingston HyperX 1600MHz ECC modules, but they're at CAS-11 and seem to perform rather poorly overall.

[Joseph S.]

Since supermicro is really picky with RAM, especially with that much RAM, I bought the certified memory from the X9DAI drop down menu.

Samsung M393B1K70DH0-CK0 is the 8GB module. Works perfectly with 16 sticks. I think it's CAS 11 at 1600MHz.

[Jeff Kilgroe]

Since supermicro is really picky with RAM, especially with that much RAM, I bought the certified memory from the X9DAI drop down menu.

Samsung M393B1K70DH0-CK0 is the 8GB module. Works perfectly with 16 sticks. I think it's CAS 11 at 1600MHz.

Cool. I'll take another look.... Ah they have more RAM listed there than the last time I checked.

Pulling the specs, both the Samsung and Micron models look nearly identical for the ECC 8GB modules. Samsung are CAS 11, Micron are CAS 10. I'm going to call my guy at Micron and see what he says. According to when I talked to him last week, they didn't have any modules with these specs...? But their site says it's in production and Supermicro has them on their approved list, I'll see what he says. I can't seem to find them for sale anywhere.

[Paul Schneider]

Very interested in setting something like this up. Can you post a list of components or direct to a site. Thanks.

[Joseph S.]

@Paul, here are some parts below. Obviously the cpu, mem, drives, video cards are all your choice.

I'm still waiting for the EVGA SR-X as it might work better with MacOS, but it's still unknown.

Chassis/case:
http://www.supermicro.com/products/system/4U/7047/SYS-7047A-T.cfm

Board (installed/wired with case above):
http://www.supermicro.com/products/motherboard/Xeon/C600/X9DAi.cfm

CPU:
http://www.newegg.com/Product/Product.aspx?Item=N82E16819117261&Tpk=e5-2690

256GB SSD drive:
http://www.newegg.com/Product/Product.aspx?Item=N82E16820147135&Tpk=MZ-7PC256D%2fAM

The EVGA SR-X has 12 memory slots, and not 16. So the max RAM using 8GB is 96GB.
http://www.evga.com/products/moreInfo.asp?pn=270-SE-W888-KR&family=Motherboard Family

And the Asus board has 8 sticks, but it's not the C606 chipset.
http://www.asus.com/Motherboards/Intel_Socket_2011/Z9PED8_WS/

[Jeff Kilgroe]

If you populate all the slots on the SR-X board, I believe it will drop the RAM to running in dual-channel mode. I'm still waiting for more info on the board... Can't say I'm too wild about the PCIe configuration either.


ASUS board looks alright for smaller configurations, but I haven't really looked at it.


I just ordered a bunch of RAM for my two incoming SuperMicro boards using the 8GB DDR-3 Registered ECC 1600MHz modules from Micron. They're kinda pricey and in short supply. Their part number is: MT36JSF1G72PZ-1G6M1 and is listed as being tested/approved on the Supermicro X9DAi product page. I ordered direct from Micron.

[Paul Schneider]

@Paul, here are some parts below. Obviously the cpu, mem, drives, video cards are all your choice.

I'm still waiting for the EVGA SR-X as it might work better with MacOS, but it's still unknown.

Chassis/case:
http://www.supermicro.com/products/system/4U/7047/SYS-7047A-T.cfm

Board (installed/wired with case above):
http://www.supermicro.com/products/motherboard/Xeon/C600/X9DAi.cfm

CPU:
http://www.newegg.com/Product/Product.aspx?Item=N82E16819117261&Tpk=e5-2690

256GB SSD drive:
http://www.newegg.com/Product/Product.aspx?Item=N82E16820147135&Tpk=MZ-7PC256D%2fAM

The EVGA SR-X has 12 memory slots, and not 16. So the max RAM using 8GB is 96GB.
http://www.evga.com/products/moreInfo.asp?pn=270-SE-W888-KR&family=Motherboard Family

And the Asus board has 8 sticks, but it's not the C606 chipset.
http://www.asus.com/Motherboards/Intel_Socket_2011/Z9PED8_WS/


Thanks sir.

[Jeff Kilgroe]

Waiting for all my components is painful...

I could have my motherboards, etc.. here already if I had bought the 7047A-T barebones that Joseph listed above. Newegg has them in stock at a reasonable price, too! But I didn't... I'm going the liquid-cooled route and therefore ordered my motherboards on their own. ...waiting.

Several places, Newegg included, have the Supermicro X9DRF motherboard in stock and some people may be interested in it. They also have it bundled and pre-wired in a tower if that fits your needs too and it's in stock that way as well. The X9DRF is nearly identical to the X9DAI, but has a few differences with eSATA ports and does not have USB3 ports on-board. I went the X9DAi route because I can't afford to burn a slot to add USB3 ports.

[stu aull]

Cool. I'll take another look.... Ah they have more RAM listed there than the last time I checked.

Pulling the specs, both the Samsung and Micron models look nearly identical for the ECC 8GB modules. Samsung are CAS 11, Micron are CAS 10. I'm going to call my guy at Micron and see what he says. According to when I talked to him last week, they didn't have any modules with these specs...? But their site says it's in production and Supermicro has them on their approved list, I'll see what he says. I can't seem to find them for sale anywhere.


Jeff, Joe
Is this exotic (?) RAM you guys are talking about useable by us regular 2010 MacPro Tower folks?? And if not, any suggestions? Fascinated/confused by this thread but would love some insight into what would make my Mac a bit punchier in RXC...?

thanks
Stu Aull
Alaska

[Jeff Kilgroe]

The RAM is only exotic in the sense that 1600MHZ LRDIMM modules are just hitting the market now and are a bit scarce, especially in 8GB and larger module sizes. But 1333MHz modules are common components that can be bought everywhere. An 8GB module sells for about $80 to $100 in 1333MHz form, depending on brand, reseller, market volatility, etc..

As for the 2010 Mac Pro, it uses 1333MHz RAM. The ECC registered variant isn't necessary at capacities of 128GB and less. ECC is an Error Checking and Correction protocol using a parity arrangement. I guess it helps to make sure your bits don't get scrambled by errant cosmic rays or something. It makes for a more reliable system when operating with huge RAM capacities. But not something to concern yourself with here anyway and Apple actually has OSX locked to not use more than 96GB for some odd reason. But the Mac Pro does ship standard with ECC RAM and that's what the better vendors sell for use with it -- like OWC. The modules we're talking about here are just newer and presumably what the next version of the Mac Pro will use if/when it finally arrives.

As for making your system punchier, there may be some ways to help out, but we would need more information on what you're starting with. More RAM always helps, within reason anyway. And you need to make sure you have the proper RAM type and arrangement to optimize its performance. Faster storage in the form of a performance RAID helps too. RED Rocket, of course... Other than that, there's not much else you can do. The system can only run so fast and RC-X is just what it is... And it's not terribly effective at multithreading or using all your CPU cores, nor is it really capitalizing on any serious GPU acceleration -- RC-X is heavily optimized for the Rocket card at this time. There's also a lot going on under the hood of the software, more than a lot of people realize. Doubly so if using effects like HDRx blending and Alchemy.

[Aaron Kondziela]

Ok you X9DAi guys, do you have these up and running nicely? I'm planning out nearly the same machine right now, and I'm curious how it performs (it will be win7 for me).

And where are you finding the ECC 1600 for that board? I'm coming up dry for the most part. Would like the fast stuff in there, as one of my main goals with the build is to eliminate bottlenecks everywhere I can.

[Joseph S.]

Aaron, it's been rock solid so far with Windows 7 64-bit.

[Aaron Kondziela]

Thank you, Joseph! I'll share details of my build with everyone once it's done.

[Jason Enzer]

joe,

that evga sr-x board is HUGE. i only found a few cases that fit it. one was a huge supermicro case. its factor is called HPTX which is 2 to 3 inches larger than e-atx. it really is a monster. good luck fitting it in a case :)

[Jeff Kilgroe]

The RAM is hard to come by, especially now that people are buying the motherboards and trying to get RAM too. I ordered direct from Micron and still don't have it. It seems the only commonly available modules that are almost suitable are the Kingston HyperX PC3-12800 (1600MHz) ECC ones, but they're CAS-11 and still out of stock most places. In some ways, if you need the system up and running it almost makes sense to buy a smaller amount of RAM like 32GB and use commonly available 1333MHz modules until the faster ones are more commonplace.

You can run with non-ECC RAM up to 64GB I think and there may also be a limit to the number of non-ECC modules you can install. I don't know if Supermicro has certified anything there. One intriguing option for that is the G.Skill Ripjaws X modules at 1600MHz. They're reasonably priced and should work OK as long as you stay within limits. OTOH, Supermicro boards can be more temperamental about RAM choices than other boards from ASUS, Gigabyte, etc.. All depends on if you want to experiment or not and if non-ECC RAM limitations will ultimately still serve your needs.

[Jeff Kilgroe]

Things are falling together a bit more here. So, I thought I'd list some numbers for comparison.

HP Z820 Inbound Configuration (adjusted)
90% Efficient 1125W PSU
Dual E5-2687W 3.1GHz 8-Core 20MB CPUs
HP Water Cooling
64GB RAM (8 x 8GB)
No Video Card
2TB 7200rpm HDD 1
HP Blu-Ray Writer
Standard Keyboard & Mouse
Windows 7 64bit recovery CD
Free shipping
Total after coupons / less tax: $8,549.60



Build for SuperMicro X9DAi workstation
Cooler Master HAF X RC-942 system case
Thermaltake 88% efficient 1275W Peak PSU
SuperMicro X9DAi motherboard
Dual E5-2687W 3.1GHz 8-Core 20MB CPUs
Cooler Master Water Cooling system for dual LGA 2011 & special tweak ;)
128GB RAM (Micron PC3-12800 1600MHz) ECC - 16 x 8GB
OCZ Vertex 4 512GB SSD
LG 14X Blu-Ray writer
Logitech MX wireless keyboard & mouse
Windows 7 64bit Pro
Total after Misc. shipping expenses and whatnot: $8,462.24


Not a direct like comparison as the in-house build has double the RAM and the SSD. If I were to equalize those components with the HP build, the self-build would come in at about $7033. Which means the HP system comes at about a $1500 premium but has a 3 year parts, labor and on-site warranty. Not to mention I don't have to spend a few hours assembling and 2 days of burn-in/ testing on it. The HP is not a bad deal here at all. Truth be told, I'm placing a Vertex 4 SSD in the HP too. As the system / apps drive and will be using the 2TB that I'm getting with it, along with others I'm adding, to make an internal array.

I didn't factor in video cards as I have a few here and will be doing some tests between the systems. Mostly trying dual GTX580's, a Quadro 6000 + GTX580 combo and also should have a GTX680 here soon too.

[Joseph S.]

Decided to put this up for sale (a bit below cost). Wasting away when I have no footage to do anything with :)

http://www.reduser.net/forum/showthread.php?77977-FS-Latest-and-greatest-workstation-setup-Dual-E5-2690