Completed My 56TB (and growing) file server. (1 Viewer)

[zicoz]

Note: Since I've been building this server over the last year there is probably new hardware out that that is better then the one I've built my server with

As I just finnished my expansion chassis for my new file server I thought I'd do a little writeup of my current file server that I spent the better part of 2011 building. The reason it too so long is that I had to buy parts over several months, and I also had to wait for a drive extender-replacement to be released for WHS 2k11.
Setup:


Main chassis:

• Chassis: Norco RPC-4020 (about $300 now but I had it from my previous file server)
• Motherboard: SuperMicro X9SCM-F (about $300)
• PSU: Corsair AX 850 (about $150)
• CPU: Intel Pentium G630T (about $90)
• Memory: 4GB Corsair (about $50)
• SAS-controller: IBM ServerRaid M1015 x 3 (about $80 a piece)
• 1x SFF-8087 to SFF-8088 PCI bracket with 2 ports similar to this. But different brand. (about $50)
• OS Drive: OCZ Agility 3 60GB SSD (about $100 but I had it already)
• Storage: Samsung 2TB drives x 20

Total: about $1180 + storage



(Photo of the hardware in the main server)

Expander chassis:

• Chassis: Norco RPC-4220 (about $350)
• PSU: Ace Outburst 550W (about $100)
• Power controller: Supermicro Storage On/Off PWR Control Board (about $50)
• SAS expander: Intel RES2SV240 (about $250)
• 1x SFF-8087 to SFF-8088 PCI-bracket with one port. (about $30)
• Storage: Samsung 2TB drives x 8

Total: about $780 + storage


(Photo calcof the hardware in the expander chassis, the SuperMicro Powerboard is the little green card in the top left corner.)
The main chassis is connected to expander chassis with a SFF-8088 cable.

Software:

• OS: Windows Home Server 2k11
• Drive pooling: Drive Bender

Story:


I previously had a 40TB fileserver running Windows Home Server v1, but it was running low on space, and it was lacking expansion posibilities, so with WHS 2k11 coming out I decided it was time to build a new server that was more future proof.
Since there was alot of questions around WHS 2k11 and it's lack of a drive extender technology, I decided to look for a motherboard that had good OS-support, and since I wanted alot of expansion posibilities I also needed alot of PCI-E slots. With those conditions the SuperMicro X8ST3-F was a good choice since it had good OS-support, 4 PCI-E slots and in addition to that it had the onboard LSI 1068e.
The downside to that motherboard is that it has a 1366-socket, which means CPUs are a bit more expensive.

(I have since move to the more modern SuperMicro X9SCM-F)

After getting the motherboard I looked around more at what operating systems were out there, and what storage-posibilities they had.
My starting options were ZFS, Raid or drive pooling, but after some thinking I decided to drop both ZFS and Raid since I don't really need the extra speed I can get from those options compared to drive pooling. And I didn't want to "lose" storage space to raid-overhead.
At this point I had two OS-options 1) Windows Home Server and 2) Amahi Home Server, but after finding out that I had to first install Fedora and then install Amahi ontop of that I went with Windows Home Server 2k11, and wait for drive pooling solutions from 3rd party developers.
As I was assembling my server during the first half of 2011 There were 2 products that were officially under development, Drive Bender and StableBit Drivepool, Drive Bender was the first to market, and Stablebit Drivepool is still not out beta so the choice was made for me.
In December a third option showed up with Ldisk, but that's not completed yet either, I might end up moving to one of those later if they're better but so far I'm happy with Drive Bender.
The story behind the rest of my hardware isn't that complex really.

• The CPU was chosen because it was cheap and had low TDP.
• The IBM ServerRaid M1015 was chosen after looking at the LSI Megaraid Sas 9240-8I, which is basically the same card but more expensive, so thanks to PigLover at ServeTheHome.com I ended up with the M1015 instead.
• The Intel RES2SV240 SAS expander was chosen since it was cheap, supported enough drives and had support for 6Gbps as oposed to the HP expander, and it also has enough poorts to populate a 20-drive chassis.
• The Norco chassis were chosen since they're good value for the money
• The SuperMicro Powerboard was chosen as it was cheap and did the job it was supposed to do.
• The Ace PSU was chosen after getting it recommended from the support people at the store where I got the SuperMicro Powerboard (www.nextron.no) A small sidenote here is that I first tried with with a Seasonic M12II PSU and that did not work.
• The reason why I chose Samsung harddrives for storage was that I've had nothing but good experiences with them over many years, I've had 60+ drives from them in my servers over the years ranging from 500GB to 2T, and only two of them area dead now, one of those fell to the floor, and the other was DoA so it got replaced. But with Seagate buying up Samsungs HDD-division I have to chose between Seagate or WD going forward, something that won't be easy as I don't have the best experiences with either of them.
• The OCZ Agility SSD was chosen since I had it on the shelf and I didn't want the OS-drive to slow down the storage pool.

With this setup I now basically have room for as much storage that 40 HDDs can give me, if I need to expand beyond that I simply build a duplicate of the Expander Chassis and connect that to my Main Server. As of today the Main Server has 1 open SFF-8088 port, but all I need to do is to add another M1015 card and a SFF8087-to-SFF8088 bracket to get another 2 ports on it for a total of 4 Expander Chassis.
Special thanks to Patrick, PigLover and Odditory here at ServeTheHome.com aswell as the people at Nextron for helping me find the right components for this build.

 

[JoeSmith]

holy $hit thats some serious amount of storage. What the hell are you storing on that monster? And i thought my 10TB+ Server was stupid ;-)
Is that thing running 24/7? Must be a crazy energy consumption.

 

[Matthew Houston]

What sort of configuration do you have the drives in? Is it just one JBOD pool or have you split them up into smaller Raid 5 arrays? Lol I thought my 5TB Raid 5 setup was getting pretty big ha ha.

One more thing, how long ago did you purchase the HDD's? It would kill me to have to buy them now. I bought 5 x 2TB drives at $80 each about 6 months ago, not you couldn't get the same drives for less than $150 where I live!

 

[zicoz]

It's a normal diskpool so no raid or mirroring going on. The reason for this is that I've had Raid-configurations fail on me before and I've not been able to Access my data, instead I've got 1:1 backup on old drives that are stored off-site, in addition to that I'm planing to back it all up to Crashplan once I get fibre internett installed.

And I bought all my disks before the prices jumped (luckily). I bought the final 5 disks back in October for 450NOK+vat (about $90) a piece. So you can say it was good timing

 

[Spragleknas]

You be crazy, sir

 

[techchad]

Well done on your storage build! I absolutly love the expandablity options. I've been looking into many many soultions for a central storage server (unRAID, Amahi, FlexRAID...so on and so on). I really dig your solution! I know it's a silly question, but how loud is your primary server. I know those cases are not whisper quiet (nor should they need to be), but I'd prefer not being able to hear it in another room with the door closed.

 

[zicoz]

I've replaced the middle fans in my main chassis from 4x80mm to 3x120mm and I've put some queit fans in it so that chassis is not a problem at all, I've yet to do this in my expander chassis so it still has 4x80mm fans that are really loud, too loud to be honest, but since I don't feel like paying $100 in shipping for the $11 part it takes to convert it to 120mm so I just have to live With it for now. I'm hoping to have a New garage to put all my servers in done by this summer anyhow.

So basically: If you do a Norco 4220 With 120mm fans noise should not be a problem.

 

[zicoz]

holy $hit thats some serious amount of storage. What the hell are you storing on that monster? And i thought my 10TB+ Server was stupid ;-)
Is that thing running 24/7? Must be a crazy energy consumption.

Finally remembered to pick up a wattmeter, and here are the results:

Main chassis without harddrives: about 100 watt
Main chassis with harddrives: about 190 watt
Both chassis + downloadserver: about 270 watt


These are idle meassurements, during boot I meassured almost 350 watt on the main chassis.

So it's not too bad considdering it also works as a heater for the room it stands in.

But I've now managed to get my hands on a Supermicro X9SCM-F motherboard and I might be looking to put that in the main chassis with one of the 35w Sandy-Bridge CPUs over time. I should also look into spinning down the drives that aren't in use.

 

[JoeSmith]

just to give you small comparison number:

Intel Core i5-2500
 
MSI H67 Mainboard
 
8GB-DRR3 Ram
 
320GB System Drive
 
5x2TB Samsung F2
 
TV-Card with 2xDvb-t & 2xDvb-s (that ugly thing alone takes ~10W)
 
Windows Server 2008 R2

The system consumes ~ 45W when all harddrives are idle. Without the harddrives and tv-card i was able to drop the system below 20W (i think that is really an amazing number for a quad core system).
The hds are only used for 1-3h a day so they are set to spin down after 30minutes. i know that it is probably not the best for the drives as start/stop cycles are more dangerous than running all the time but i guess 1-2 stop cycles a day won't break them that fast. Usually i tend to keep them until the bigger drives get cheaper but since the flood that plan hasn't worked out.

If you plan to get a good cpu with low power consumption you might take a look at the new ivy bridge cpus from intel as they are pin compatible to sandy bridge and even use less power.


By spinning down the drives and using a smaller energy hungry cpu you could save some power. In my area 270W running 24/7 would be nearly 500€ / year just on energy costs.

 

[MrCrabs]

What the hell are you storing on that monster?

Well the internet needs to be saved somewhere...