Storage can be very expensive, especially if you want a lot of it, want flexibility and expandability, but it doesn’t have to be!
I’m currently in the process of putting together a new storage system, but because we’re strapped for cash I’m limited to opensource and nearly off the shelf products.
The operating system will be OpenSolaris, the exact kernel is undecided as of yet, hopefully Oracle will come out with an update real soon. Filesystem will be ZFS, although it has issues with tens of millions of files it still offers me flexibility and manageability features which are very useful.
The parts are
Storage head
- Dell R610 w. 2 x E5620, 24GB memory and internal PERC H200 contr.
- LSI-9200-8e PCIe
- 1 x OCZ Vertex 2 Pro 100GB (L2ARC)
- *Perhaps 10GbE PCIe card
SAS2 JBOD
- SuperMicro SC847E26-RJBOD1
- 26 x 2TB SAS2 Seagate Constellation ES 3.5″ 7200rpm
- 2 x Intel X25-E 32GB (mirrored SLOG)
- SAS 2-port internal cascading cable
- 2 x LSISS9252 SAS-to-SATA interposer
Total capacity is currently 52TB if I was to stripe it all, but that wouldn’t give me any fault tolerance, so the 26 disks will be divided into 3 RAIDZ2 vdevs each consisting of 8 disks, the two leftover disks will become hotspares. This gives me roughly 36TB of useable space with room to grow (17 free out of 45 total). While it’s no S7310 system it’s not that far off either, the L2ARC SSD will sit on the storage head as a pass-through disk on the internal controller, and the SLOG SSD’s will sit in the SC JBOD array using LSISS9252 interposer cards.
I’ll post an update on the build process, including photos, once the hardware arrives.
Other posts with similar builds,
http://www.natecarlson.com/2010/05/07/review-supermicros-sc847a-4u-chassis-with-36-drive-bays/
http://hardforum.com/showthread.php?t=1508468
Sounds like a great build! Curious – have you figured out exactly how the interposers mount? I haven’t been able to find any information about that.. so I’m not sure if it’s easy to fit the interposer and drive in a standard hot-swap bay.
Have you considered going with a pair of SAS2 controllers so that your paths will go over separate hardware paths?
Nate, I have absolutely no idea exactly how the interposers mount, it’s quite hard to find relevant information on those suckers. I do know that there are two models, one for 2.5″ drives and one for 3.5″, seeing as my SSD’s are 2.5″ I’ll get those, SuperMicro has a special 2.5″ to 3.5″ HD tray and I’m hoping it simply fits in one of those, neither part costs a fortune so my back up plan is to install all SSDs in my storage head, the Dell R610.
Installing more than one controller has been one of the things I’ve thought hard and long about, and I’ve come to the conclusion that because I don’t need massive bandwidth and because I only have one storage head (for the time being) I won’t really gain much, but that being said I am considering swapping the R610 for a R710 simply to have a few available PCI slots eventhough the R710 only has 2*PCIe x8 and 2*PCIe x4, installing the first controller and a 10GbE card leaves me with nothing but x4 slots left which are of little use. The R610 only has two PCIe slots, but both are x8 and the box is 1U as opposed to the R710 being 2U.
Technically maybe I should go for a different manufacturer, but to be honest I prefer to have as much as possible of the same brand, makes it easier in the long run. I originally wanted Sun/Oracle equipment but Dell is capable of much bigger discounts than Oracle.
Building the “perfect” storage system for as little money as possible isn’t easy
btw, I’d love to hear how you’re getting on with your system.
You don’t mention what this server is going to be used for but.. don’t expect much performance from huge raidz/raidz2 vdevs. If you want performance, go with mirrored disks or small raidz vdevs (3-5 disks).
Giovanni,
It’s going to be used for a backup server, it’ll eventually contain 50-80 million files, most of which will be 1-4K but a fair amount of them will end up being static and never get touched again. The backup software databases unfortunately also resides in the same space as the files (I can’t move them). I shouldn’t need a great amount of bandwidth, but a reasonable amount of IOPS when the database files are queried, the L2ARC should help out nicely there.
I’m also considering using 5 x 5-disk RAIDZ vdevs and one or two hotspares.
Thank you for your post. Did the setup you described work out?
I’m planning to build a similar setup and I’m pretty much stuck on the ZIL SSD/LSISS9252 SAS-to-SATA interposer question. I must have seen every single mention of it on the whole internet, but I’ve yet to find out whether it will actually fit in the SuperMicro 2.5″ to 3.5″ tray. It also seems impossible to actually buy the part from the distribution channel in Europe.
Any help on this would be appreciated.
Hi Clemens,
I’ve yet to order the hardware, there’s not enough power left in the rack so have to make a bunch of changes first, but I’ll hopefully have it all in a month or so, fingers crossed. I too fear that the interposers will be hard to find, they don’t however cost a lot of money so if I can get them in the US then so be it, and if they don’t fit it’s not a lot of money down the drain in the grand scheme of things. I’d prefer to have my ZIL in the SC casing and not in the storage head. I’ll for sure write up more about all of this once I get the hardware, so either check back here or email me at lasseoe {at} domain-in-url.