Getting Linux ATARAID working with a Promise 100 (PDC20276) Raid Controller

Having spent something like a two days playing with this, I now know how to do it (quite easily, as it happens).
<--break-->
First, a note about my system. I'm using a 1U rack mounted system with a GA-81EXRR motherboard, which has a built in Promise 100 Lite (PDC20276 chipset), and two hot swap drive bays (there's a third unused bay which isn't hot swap). The system also has a CDROM and floppy drives.

When using Promise 'RAID' controllers, one should be aware of two basic facts:

1) Promise controllers are quasi-raid. That is, they're part hardware, part software, and so not as good as a proper hardware raid solution. It could be argued they're not as good as a decent software raid solution as well, actually.

2) Promise don't do open source drivers for Linux. They do have a proprietary driver, which internally emulates a SCSI device (a legacy throwback from Windows, which can't do it any other way). This emulation is complex and uses up lots of CPU which brings little benefit to the running of the system. Fortunately, Linux has something called 'ataraid' which does the same thing, but much more efficiently (with a couple of shortcomings too though).

Getting all this to work is a real pain in the neck. I most definitely didn't want to use the Promise drivers, so found a cunning way to avoid them. This method works with new and existing systems with any Linux distribution or kernel version, so actually should be useful for everyone.

The basic recipe is this:

- Switch off the Promise raid in the bios (so it acts just like an ordinary IDE controller)
- Get Linux working beautifully
- Compile a new kernel with Promise support, and the ext3 module compiled in
- Do a bit of configuration work
- Reboot the system, switch raid on again, create a raid mirror, make it build the array
- Reboot and let the system boot with full raid in place!

In a bit more detail:

Switch off the Promise Raid controller
This has to be done in the bios (in mine, it's "raid controller function: raid/ata", selecting ata). If you've got a separate Promise card or something, you could remove it completely, putting a single disk on an ordinary IDE channel.

This stage is important. Even without drivers running in the OS, the Promise controller causes problems. You can sort of understand it, because if you make changes to one disk and not the other, the controller won't handle this properly, and so the array goes out of sync. Anyway, once switched off, everything is pretty straight forward.

Essentially, we're going to get Linux working on just one disk, and then introduce another which will be the mirror of the first.

Get Linux Working Beautifully
This is the easy bit. If you've already got a running system, you probably don't need to do anything. If not, just install Linux as you normally would, adding in any toys you want. I've been using Fedora 1, but it really doesn't matter which distro you use. Just make sure all the disks are properly fsck'ed and everything is working properly.

Compile a New Kernel
You can't escape this bit. You have to create a kernel with Promise support built in. You also have to get basic filesystem support in there too (actually, I'm not 100% certain you do need this, so if you find you don't, let me know!). You can't use the modularised kernels that come with most distributions because the kernel has to have some stuff available to it before it's got access to the filesystem.

Possibly the easiest way to do this is to use a recent RPM kernel from Redhat (if you're using Fedora et. al). Once you've got a decent kernel running, get the source for it (again, an RPM is the easiest way for Redhat). Failing that, download something from http://kernel.org.

Redhat binary kernels come with the configuration used to build them (in /boot/config-[something] ). This is useful, as it means you don't have to answer every question in the kernel configuration tool. I personally got this going with Fedora kernel 2.4.22-1.2188. The changes I made to the config (.config file in the kernel source) from that kernel to the one I have working are as follows:

CONFIG_BLK_DEV_ATARAID_PDC=y
CONFIG_BLK_DEV_ATARAID=y
CONFIG_EXT3_FS=y
CONFIG_BLK_DEV_PDC202XX_NEW=y
CONFIG_BLK_DEV_PDC202XX_OLD=y
CONFIG_BLK_DEV_PDC202XX=y
CONFIG_PDC202XX_FORCE=y

I'm not 100% certain you need ext3 to be compiled in, but since my filesystems are ext3, it doesn't really make too much difference. Either way, compile up a kernel, install it and it's modules and configure your boot loader. You can use LILO or Grub for this, I'm using Grub.

There's no reason you can't reboot your system and try your new kernel out, but without Raid. In fact, it's probably a very sensible thing to do, as it'll rule out basic kernel compile and install problems if you have trouble later on.

Do a bit of Configuration Work
There are two main bits of config you need to do. The first is for your boot loader, and is by far the most complex. The second is /etc/fstab.

To configure your boot loader (Grub in my case, but it's almost the same for Lilo) you need to ascertain an "append line". There's a very good guide floating around, one copy of which is here, but there's a full document here.

Once you have your append line, configure Grub. I'd recommend you keep your old kernel available via your boot loader - whilst it's unlikely it'll boot properly, it might get you out of trouble. Here's my Grub config:

title Fedora RAID
    root (hd0,0)
    kernel /boot/vmlinuz ro root=/dev/ataraid/d0p1 ide2=0xc000,0xc402,11 ide3=0xc800,0xcc02,11
    initrd /boot/initrd-2.4.22-1.2188.nptlcustom.img

...you'll note I've put /dev/ataraid/d0p1 as the root disk. That's going to be correct, once we've got the Promise voodoo out of the way ;-)

Lastly, edit your /etc/fstab file and change all the references to /dev/hd devices to be /dev/ataraid/d instead. For example, my system uses /dev/hde and /dev/hdg as it's two disks. Since I'm not using the other disk yet, it's all on /dev/hde. I have three partitions, / swap and a data disk. I then change fstab like this:

/dev/hde1 becomes /dev/ataraid/d0p1
/dev/hde2 becomes /dev/ataraid/d0p2
/dev/hde3 becomes /dev/ataraid/d0p3

...pretty simple (hopefully you get the idea).

Now might be a good idea to make a boot floppy, by the way.

Reboot The System for Promise Voodoo
We're about to reboot the system, and quite possibly mess things up quite severely. So, a little checklist:

1) Do you have a boot floppy?
2) Do you have original distro install CDs (useful for "linux rescue" or similar!)
3) Have you built a kernel (and modules) and installed it properly?
4) Have you added configuration to your boot loader? (preferably leaving old config where it is!)
5) Have you reconfigured fstab to specify the new raid devices?
6) Have you got nerves of steel?

If "yes" to all of those, then go ahead and reboot your machine. DO NOT LET THE O/S BOOT!!

Firstly, go into the Bios and re-enable Raid functionality for the Promise controller. Once you've done this, you'll have to reset again. This time go into the Promise configuration utility. If any array is shown, delete it. Then, do an "auto configuration" to add a mirror (select 'security' rather than 'performance'). When asked if you'd like to do the data transfer, say yes. Select the first disk as the source disk (after all, it's the one that has Linux on it, right?).

You've got time to get a stiff brandy at this point. You're probably needing it.

Reboot into RAID
Once the Promise data transfer between the two disks is complete, it'll ask you if you'd like to reboot. This is the moment of truth - go for it.

With a bit of luck, you system will boot as it did before, except you'll notice both disk lights blinking rather than just one of them. Once booted, "df" or "mount" will show that /dev/ataraid devices are mounted rather than /dev/hd devices. If so, well done, you've got raid working!

Troubleshooting and stuff to watch out for
If you have any /dev/hd devices mounted, you could run into problems. If they're actually partitions on the raid array, then unmount them immediately. Also, you'd do well to unmount the raid partitions and fsck them. You may be in an "undefined state" here, so be really careful. Under no circumstances should you ever mount /dev/hd devices again! (Linux won't stop you, but you'll probably get data loss if you do it).

If your system didn't boot at all, then you'll need to figure out why. I'm assuming it got as far as the boot loader asking you which OS to boot, and got as far as running a kernel. If not, then try your boot floppy. If the kernel boots but then panics because it can't find "init" then you're in trouble. This problem is because the kernel couldn't find something on the disk. That means it either can't access the raid array (ie. Promise support is missing or not working) or it can't read the filesystem (ext2/ext3 support is missing). Bottom line: Your kernel's not working properly, or you didn't get your append line right, or the "root=" part of your append line isn't correct.

At this point, try booting your old kernel - it'll amost certainly not work, and it'll drop you to a nasty shell during booting. DON'T mount anything read/write - the Raid is half active, so you'll cause data loss. If you want to back track a bit, reboot, switch off Raid in the Bios, boot your old kernel, remount the root disk read/write and fix up /etc/fstab and boot the system as normal. Go back and check your work so far and try again.

If you really get into trouble getting your system back to a one-disk mode, then try your distribution CDs. Fedora has a "linux rescue" feature which mounts your existing system so you can mess with it a bit. Just make sure you have Raid turned off, otherwise you'll get data loss.

Good luck!

Caveats
There are a couple of caveats to all this. Firstly, the open source drivers don't do anything with the drive health lights (if your system has them). So, if a disk fails, you won't be able to see it from across the room.

Secondly, because the Promise raid controllers are crappy quasi-raid, you can cause no end of problems by accidentally mounting parts of the disk without raid control. If this happens because of some mangled boot sequence, or just because you're half asleep then you can get data loss. I think you'll be okay if you mount read only though, but that's probably best avoided too.

The open source drivers use less CPU than the Promise drivers. This is because for all intents and purposes, Promise are crap, and best avoided if possible ;-)

References
In no particular order, but thanks to all the people who contributed to them!
http://www.murty.net/ataraid/existing.html
http://www.fedoraforum.org/forum/showthread.php?threadid=1272
http://www.redhat.com/docs/manuals/linux/RHL-8.0-Manual/custom-guide/ch-custom-kernel.html
http://www.linuxquestions.org/questions/history/174543
http://www.fedoraforum.org/forum/showthread.php?s=&threadid=975
http://www.redhat.com/docs/manuals/linux/RHL-8.0-Manual/custom-guide/s1-custom-kernel-bootloader.html
http://www.tldp.org/HOWTO/ATA-RAID-HOWTO/
http://ttul.org/~rrsadler/linux-promise/
https://listman.redhat.com/archives/ataraid-list/2002-February/msg00035.html
http://www.murty.net/ataraid/
http://www.linuxforums.org/tutorials/ATA-RAID-HOWTO/existing.html