Hello list members, In CentOS-5.5 I'm trying to achieve static assignment of SCSI device names for a bunch of RAID-60 drives on a Supermicro motherboard. The "scsi_id" command identifies all drives ok. The board has one SATA controller and three SAS/SATA controllers ... standard on-board ICH-10 ATA channels, an on-board LSI SAS/SATA controller, and two add-on SAS/SATA contoller cards. There are 13 drives in all, spread across the four controllers, all configured for Linux software RAID. The problem is in management of the drive names and figuring out which drive to pull in case of failure. Unfortunately the BIOS scan detects only the three drives connected to the ICH-10 SATA controller. That's ok because that's where the RAID-1 boot drives are. However, when the kernel starts it assigns those drives last, not first. For this reason I want to use a set of udev rules to assign specific names to the drives plugged into specific ports (to maintain my sanity :-) ). Identifying drives by their ID string (which includes the drive's serial number) and assigning names in the rules works ok. BUT, what happens when I have to swap out a failed drive? The serial number (and possibly model number) changes, and the udev assignment should fail, probably assigning an unexpected /dev/sd? name. RAID rebuild would choke until I change the MD device assignment. Is it possible to assign SCSI drive names by hardware path instead? I especially want the three RAID1+spare boot drives to always be assigned sda/sdb/sdc, because that sorts out other issues I'm having in CentOS-5. In the udev rules file I tried piping the output of "scsi_id -g -i -u -s /block/..." through "cut" to extract the path, but I get no match string when I run "udevtest" against that block device. Does the "PROGRAM==....." clause not recognize the pipe symbol? I tried a little shellscript to provide the RESULT match string, but udevtest didn't like that. Is there a supported way to predictably assign a drive name according to the hardware port it's plugged into ... it would make swapping drives a lot easier, since it becomes 'drive-id-string' agnostic. Better yet, is there any way to tell the kernel the order in which to scan the controllers? I'm also hoping the problem doesn't radically change when I install CentOS-6 on this box. I'm using CentOS-5 just to get practice in using KVM and RAID-60. Thanks for any advice you can offer. Chuck
On 1/27/2011 2:41 PM, Chuck Munro wrote:> > Identifying drives by their ID string (which includes the drive's serial > number) and assigning names in the rules works ok. BUT, what happens > when I have to swap out a failed drive? The serial number (and possibly > model number) changes, and the udev assignment should fail, probably > assigning an unexpected /dev/sd? name. RAID rebuild would choke until I > change the MD device assignment.If you can figure things out for the initial md device creation, subsequent assembly uses the uuid to match up the members and doesn't care if the disks have been moved around either physically or by detection order. And if you are hot-swapping drives and rebuilding manually, you should be able to find the just-assigned name with 'dmesg'. -- Les Mikesell lesmikesell at gmail.com
on 12:41 Thu 27 Jan, Chuck Munro (chuckm at seafoam.net) wrote:> > Hello list members, > > In CentOS-5.5 I'm trying to achieve static assignment of SCSI device > names for a bunch of RAID-60 drives on a Supermicro motherboard. The > "scsi_id" command identifies all drives ok. > > The board has one SATA controller and three SAS/SATA controllers ... > standard on-board ICH-10 ATA channels, an on-board LSI SAS/SATA > controller, and two add-on SAS/SATA contoller cards. There are 13 > drives in all, spread across the four controllers, all configured for > Linux software RAID. > > The problem is in management of the drive names and figuring out which > drive to pull in case of failure. Unfortunately the BIOS scan detects > only the three drives connected to the ICH-10 SATA controller. That's > ok because that's where the RAID-1 boot drives are. However, when the > kernel starts it assigns those drives last, not first. > > For this reason I want to use a set of udev rules to assign specific > names to the drives plugged into specific ports (to maintain my sanity > :-) ). > > Identifying drives by their ID string (which includes the drive's serial > number) and assigning names in the rules works ok. BUT, what happens > when I have to swap out a failed drive? The serial number (and possibly > model number) changes, and the udev assignment should fail, probably > assigning an unexpected /dev/sd? name. RAID rebuild would choke until I > change the MD device assignment. > > Is it possible to assign SCSI drive names by hardware path instead? I > especially want the three RAID1+spare boot drives to always be assigned > sda/sdb/sdc, because that sorts out other issues I'm having in CentOS-5. > > In the udev rules file I tried piping the output of "scsi_id -g -i -u -s > /block/..." through "cut" to extract the path, but I get no match string > when I run "udevtest" against that block device. Does the > "PROGRAM==....." clause not recognize the pipe symbol? I tried a little > shellscript to provide the RESULT match string, but udevtest didn't like > that. > > Is there a supported way to predictably assign a drive name according to > the hardware port it's plugged into ... it would make swapping drives a > lot easier, since it becomes 'drive-id-string' agnostic. Better yet, is > there any way to tell the kernel the order in which to scan the controllers? > > I'm also hoping the problem doesn't radically change when I install > CentOS-6 on this box. I'm using CentOS-5 just to get practice in using > KVM and RAID-60.Though I don't swear to understand it well, it's possible that multipath (device-mapper-multipath) may work in your situation. I've been using it for iSCSI storage, where it provides multipathing capabilities, including performance improvements, HA, and persistent device naming. Whether this applies to hotplugged SCSI devices I'm not so sure, and udev would be my first choice. The multipath documentation is unfortunately atrocious. -- Dr. Ed Morbius Chief Scientist Krell Power Systems Unlimited
> Les Mikesell kindly wrote: >> > >> > Identifying drives by their ID string (which includes the drive's serial >> > number) and assigning names in the rules works ok. BUT, what happens >> > when I have to swap out a failed drive? The serial number (and possibly >> > model number) changes, and the udev assignment should fail, probably >> > assigning an unexpected /dev/sd? name. RAID rebuild would choke until I >> > change the MD device assignment. > If you can figure things out for the initial md device creation, > subsequent assembly uses the uuid to match up the members and doesn't > care if the disks have been moved around either physically or by > detection order. And if you are hot-swapping drives and rebuilding > manually, you should be able to find the just-assigned name with 'dmesg'. >That's how I've been doing it with the two RAID-6 servers I have now, but in my advanced age I'm getting lazy. I did figure out how to tag the drive by path using the '-i' option to scsi_id and filtering by a "RESULT==<path> *" expression. However, something has gone very much astray, because now the rules don't do anything ... the system assigns names in the order scanned despite having added the udev rules before the default '50-udev.rules' is executed. I'm a bit mystified by this. Before I started messing around, filtering by drive id string did work, now it doesn't. Is there a small database somewhere I need to trash? Mike's suggestion is intriguing and makes me wonder if I can manually label a replacement drive with the same UUID as the failed one (assuming I keep records, of course). Since all drives in the array are partitioned exactly the same way, I can have one or two spares on the shelf pre-partitioned, ready to be given a UUID. I don't plan to hot swap. Still, I'd love to be able to use udev rules to assign names, if only to make life easier ... inquiring minds need to know why I suddenly can't get any rules to work. Maybe I need to re-install. I have just downloaded a trial version of RHEL-6 which I should try, to see if it handles this stuff more gracefully. It'll do for experimentation until CentOS-6 arrives. Chuck