Quote:
I'll try pulling/reinserting drives next.


Manual intervention is required here. The raid tools, specifically mdadm, are crap. But they do work, so one just has to write the magic into a recipe script and then never worry about it again.

I pulled one of the four SATA drives, which had partitions that were members of the RAID1 (root filesystem) and RAID5 (/usr/local filesystem) arrays. There was a pause, while the low level libata tried to recover the failed drive, and then the system just continued working afterward.

But reinserting the drive is a current known hassle. It gets seen as an entirely different device than the one that was removed. So manual intervention is required to delete the old unit from the RAIDs, and then add the new one.

The drive I pulled was /dev/sdb, and then became /dev/sde on reinsertion.
So I had to do this to get it back into the arrays without a reboot:

mdadm /dev/md0 -f /dev/sdb1 -r /dev/sdb1 -a /dev/sde1
mdadm /dev/md1 -f /dev/sdb2 -r /dev/sdb2 -a /dev/sde2

That could easily be scripted.

But there was another wrinkle: after the drive is pulled, /dev/sdb no longer exists.
Well, sort of. It''s "half there", but udevd nuked the /dev/ entries for it.
This prevents actually issuing the above commands.

So the full sequence on a udev system is this:
---------------------------------
(1) removal:
## yank the drive, wait about 30 seconds for the fault-recovery to give up, then:

mdadm /dev/md0 -f /dev/.static/dev/sdb1 -r /dev/.static/dev/sdb1
mdadm /dev/md1 -f /dev/.static/dev/sdb2 -r /dev/.static/dev/sdb2

## reinsert the drive, wait about 30 seconds again, then:

mdadm /dev/md0 -a /dev/sdb1
mdadm /dev/md1 -a /dev/sdb2
---------------------------------
Note that with this method, the new drive gets the same name as it previously had.
But with USB discs coming and going, there's no guarantees, unfortunately.

Also, if the inserted drive is a new one, then it will need to first be partititioned correctly before doing those last two commands. The simple way, given identical drives, is to do this just before those last two commands:

dd if=/dev/sda of=/dev/sdb bs=512 count=1
hdparm -z /dev/sdb

Definitely in need of a higher level tool to simplify this.
Mind you, a simple bash script (20-50 lines?) could totally automate it all.

Cheers