Terabyte Fileserver

Getting a new terabyte fileserver, the old one with its 2.2 TB was filling up too rapidly, was a surprising easy task. Setting it up was an altogether different matter.

First the mechanics. The same type case I used for the old server (called Videodrome, after the movie) was used again for this new machine, to be called Terabak. It is a huge rackmountable, 4U high and a ridiculous 60 or so cm deep. But that leaves me ample space for my project. The BOM (Bill Of Materials), inspired by my BreedbekPC, the machine used to play hidef content on my 42" 1920x1080 LCD, included the mobo, an MSI K9VGM-V, 2 gig of DDR2 memory, a boxed AMD X2 64 bit CPU (BE-2300, FWIW) and a 80 GB Seagate SATA system drive. To be on the safe side powerwise I got a 750W Chieftec PSU. The storage consists of an Areca ARC-1220 eight-port RAID controller and eight Sumsing... errr... SAMSUNG HD103UJ. While they're all but giving them away I attempted to get a rebate by buying a whole box (20 pcs) of these nice drives. I contacted four retailers but none would oblige. So I ended up buying two batches of four, which has the added boon of being from different batches. The ARC apparently was EOL, as was the Chieftec backplane. The RAID controller eventually was purchased in Germany and the backplanes in Austria.

Plonking everything together was straightforward. The only challenge was mounting the backplanes in the case, which eventually involved drilling holes in the bottom plate and screwing it into the case. I also had to make provisions for the alarm LEDs on the frontpanel, which involved making custom cables with 180 Ohm resistors. Then it was time to boot.

I had chosen CentOS 5.1 as my OS of choice, and thought that getting the RAID controller working would be quite a challenge. Not so. While the driver was available well hidden on the accompanying CD, it was already present with the installation, so that when I inserted the PCI-e card into the motherboard it was immediately functional after booting. This was quite a surprise! Not even the need to do insmod or something. So I continued to install the disks into the backplane. After booting the controller saw the drives, and after selecting 64bit LBA to enable >2 TB size I let it go through its merry initialization routine. This took about three hours. Then I booted into my OS, only to find that fdisk only saw 2 TB! WTF! As it turned out, fdisk does not support drive sizes larger than 2 TB, so I searched around a bit and found a nice how-to by Tilo Sloboda. I had to use parted, and then format the thing with mkfs.ext3. The trick here was to set the volume label to "gpt", or GUID partition table. Whatever you say, boss...

Then trouble started. mkfs.ext3 was still busy doing its thing when the alarm started beeping on the controller card. Disk failure. Huh?!?!? I swapped disks around, and the fault in the BIOS screen stayed in the same location. That means a fault between the controller and backplane, or the backplane itself. After unmantling the whole thing, a lot of work as ease of assembly was not a top priority, I found no fault. I did discover that the disk channels were numbered from right to left! So nice of Chieftec to not mention this in the manual! So it was a disk after all! I put everything back together again and swapped the alarm LED cables. The array had split itself into two volumes #00, one degraded array of seven disks and one incomplete of one disk. Really weird. Rebuilding it fixed that. Then reformatting the drive /dev/sdb1 and all was OK.

Time to fill it up! I fired up Videodrome and started to transfer files with NFS. Then I realized that the ownership was wrong (not really a big problem) but that also the timestamp was deleted. This seemed to be fixable by using cp -pru <path>, but this did not work as expected. What was worse, the windows box that I use as master for my files, had no access using Samba. It transpired that the Linux firewall (iptables) was active, so I could connect. But still no access, due to SELinux. After rebooting (almost never a necessity with Linux!) my windows box finally could connect and start transferring files.

So. I just spent 1k8 euro on a new machine. I must say, compared to Videodrome it flies! I can now max out my 100 Mb network and still only have 10% CPU load! It's very cool to see my win32 machine chugging along at 25% CPU usage and Terabak just sitting there and occasionally showing some disk activity on the SATA backplane.

The images below show some parts of the process building Terabak. Especially the backplane modding was fun!

backplane_bottom_plate	system_disk	mobo
psu1	psu2	resistors	completed
array_front	rack_rear	most_important_cable	rack_front