Mike Burns
2009-Sep-26 02:33 UTC
[Ovirt-devel] Adding handling for Multipath storage devices
The following patches introduce support for multipath and cciss devices to the ovirt-node and node-image. Comments are appreciated. These patches assume that the 3 patches (2 node, 1 node-image) from Joey are all incorporated. Mike
Michael Burns
2009-Sep-26 02:33 UTC
[Ovirt-devel] [PATCH node-image 1/2] add handling for multipath
This includes adding kernel modules for multipath, the appropriate packages for multipath, and enabling multipathd by default Signed-off-by: Michael Burns <mburns at redhat.com> --- common-blacklist.ks | 2 +- common-install.ks | 6 +++++- common-pkgs.ks | 3 ++- 3 files changed, 8 insertions(+), 3 deletions(-) diff --git a/common-blacklist.ks b/common-blacklist.ks index 0dd0332..81f46d6 100644 --- a/common-blacklist.ks +++ b/common-blacklist.ks @@ -10,7 +10,7 @@ echo "Removing excess RPMs" RPMS="system-config-firewall-tui system-config-network-tui rhpl \ rpm-python kudzu libsemanage-python" -RPMS="$RPMS kpartx mkinitrd isomd5sum dmraid checkpolicy" +RPMS="$RPMS mkinitrd isomd5sum dmraid checkpolicy" # Remove additional RPMs forcefully RPMS="$RPMS gamin pm-utils kbd usermode vbetool ConsoleKit hdparm \ diff --git a/common-install.ks b/common-install.ks index 9bcdaa3..d6620f7 100644 --- a/common-install.ks +++ b/common-install.ks @@ -5,7 +5,7 @@ auth --useshadow --enablemd5 selinux --enforcing firewall --disabled part / --size 650 --fstype ext2 -services --enabled=auditd,ntpd,ntpdate,collectd,iptables,network,rsyslog,libvirt-qpid +services --enabled=auditd,ntpd,ntpdate,collectd,iptables,network,rsyslog,libvirt-qpid,multipathd # This requires a new fixed version of livecd-creator to honor the --append settings. bootloader --timeout=30 --append="console=tty0 console=ttyS0,115200n8" @@ -13,3 +13,7 @@ bootloader --timeout=30 --append="console=tty0 console=ttyS0,115200n8" device virtio_blk device virtio_pci device scsi_wait_scan + +# multipath kmods +device dm-multipath +device dm-round-robin diff --git a/common-pkgs.ks b/common-pkgs.ks index 094e9cb..d0d5170 100644 --- a/common-pkgs.ks +++ b/common-pkgs.ks @@ -55,7 +55,6 @@ newt-python -kbd -usermode -fedora-logos --kpartx -dmraid -gzip -less @@ -72,6 +71,8 @@ isomd5sum irqbalance cpuspeed acpid +device-mapper-multipath +kpartx # workaround for gpxe issue with the virt-preview qemu on F11 host kernel # https://bugzilla.redhat.com/show_bug.cgi?id=512358 etherboot-zroms-kvm -- 1.6.2.5
Michael Burns
2009-Sep-26 02:33 UTC
[Ovirt-devel] [PATCH node 1/6] Fix for detecting HP CCISS devices.
Since HAL doesn't properly identify them, this patch goes to /dev/cciss/ and collects the name of all devices found there. support cciss disks in storage_init parameter cciss disks are separate bus in udev NB: fix is needed for RHEL5 udev rules (rhev-h includes it): https://bugzilla.redhat.com/show_bug.cgi?id=513070#c24 example: /dev/disk/by-id/cciss-3600508b100104a3953545233304c0003 /dev/disk/by-id/cciss-3600508b100104a3953545233304c0003-part1 /dev/disk/by-id/cciss-3600508b100104a3953545233304c0003-part2 udevinfo -n /dev/disk/by-id/cciss-3600508b100104a3953545233304c0003 -q env ID_SERIAL=3600508b100104a3953545233304c0003 ID_BUS=cciss handle cciss and multipath partition in uninstall Signed-off-by: Michael Burns <mburns at redhat.com> --- scripts/ovirt-config-storage | 7 +++++++ scripts/ovirt-config-uninstall | 11 ++++++++++- scripts/ovirt-early | 13 +++++++++---- 3 files changed, 26 insertions(+), 5 deletions(-) diff --git a/scripts/ovirt-config-storage b/scripts/ovirt-config-storage index eed126c..473826f 100755 --- a/scripts/ovirt-config-storage +++ b/scripts/ovirt-config-storage @@ -169,6 +169,13 @@ get_dev_name() devices="$devices $(ls /dev/vd? 2> /dev/null | xargs)" devices=$(echo $devices | tr ' ' '\n' | sort -u | xargs) + # FIXME: workaround for dealing with cciss devices + for dev in $(ls /dev/cciss/ 2>/dev/null); do + if [[ ! $dev =~ p[0-9]+\$ ]]; then + devices="$devices /dev/cciss/$dev" + fi + done + local num_devices=$(echo "$devices" | wc -w) # If there's only one device, use it. case $num_devices in diff --git a/scripts/ovirt-config-uninstall b/scripts/ovirt-config-uninstall index 02828a0..fd65cbb 100755 --- a/scripts/ovirt-config-uninstall +++ b/scripts/ovirt-config-uninstall @@ -31,10 +31,18 @@ if ask_yes_or_no "Do you wish to continue and uninstall this node ([Y]es/[N]o)?" if [ -d /dev/HostVG ]; then log "Uninstalling node" log "Detaching logging" + # multipathd holds all mounts under /var in a private namespace + service multipathd stop unmount_logging log "Removing volume group" wipe_volume_group "HostVG" - partition=$(findfs LABEL=BOOT) + rc=$? + if [ $rc -ne 0 ]; then + log "Aborting" + service multipathd start + exit $rc + fi + partition=$(readlink -f $(findfs LABEL=BOOT)) if [ -n "$partition" ]; then log "Unmounting boot partition" umount $partition @@ -43,6 +51,7 @@ if ask_yes_or_no "Do you wish to continue and uninstall this node ([Y]es/[N]o)?" parted -s $drive "rm 1" parted -s $drive "rm 2" fi + service multipathd restart printf "Finished uninstalling node." else log "There is no installed node instance to remove." diff --git a/scripts/ovirt-early b/scripts/ovirt-early index cdd4afd..8990727 100755 --- a/scripts/ovirt-early +++ b/scripts/ovirt-early @@ -109,7 +109,7 @@ find_disk() { local bus=$1 local serial=$2 local live=$(get_live_disk) - for d in /dev/disk/by-id/{scsi,usb}*; do + for d in /dev/disk/by-id/{scsi,usb,cciss}*; do ID_FS_USAGE eval $(udev_info $d env) # ID_FS_USAGE is set for partitions @@ -132,7 +132,7 @@ find_disk() { start() { # oVirt boot parameters # BOOTIF=link|eth*|<MAC> (appended by pxelinux) - # ovirt_init=[usb|scsi[:serial#]|/dev/...] + # ovirt_init=[usb|scsi|ccis[:serial#]|/dev/...] # ovirt_vol=BOOT_MB:SWAP_MB:ROOT_MB:CONFIG_MB:LOGGING_MB:DATA_MB # ovirt_overcommit=<overcommit_ratio> # ovirt_local_boot @@ -159,7 +159,7 @@ start() { # BOOTIF=eth* e.g. BOOTIF=eth0 - use given interface bootif - # ovirt_init=[usb|scsi[:serial#]|/dev/...] + # ovirt_init=[usb|scsi|cciss[:serial#]|/dev/...] # local installation target disk # usb|scsi - select disk type, as reported by udev ID_BUS # serial# - select exact disk using serial number, as reported by @@ -278,6 +278,11 @@ start() { i=${i#=usb} serial=${i#:} ;; + =cciss*) + bus=cciss + i=${i#cciss} + serial=${i#:} + ;; =/dev/*) bus serial@@ -295,7 +300,7 @@ start() { fi else # 'ovirt_init' without value: grab first disk - init=$(ls -1 /dev/?da 2>/dev/null |head -n1) + init=$(ls -1 /dev/?da /dev/cciss/c?d? 2>/dev/null |head -n1) fi if [ -z "$init" ]; then log "Selected disk $i is not valid." -- 1.6.2.5
Michael Burns
2009-Sep-26 02:33 UTC
[Ovirt-devel] [PATCH node 2/6] add LVM filter support in initrd
If boot parameter filter=* is specified, lvm.conf is generated in the initrd to restrict PV scanning. e.g. filter=/dev/disk/by-id/scsi-3600c0ff000d791aceb63ff4901000000 would generate lvm.conf with: devices { filter = [ "a|^/dev/disk/by-id/scsi-3600c0ff000d791aceb63ff4901000000|", "r|.*|" ] } Parameter is optional, if not present, lvm.conf is not generated and LVM defaults apply (which is to scan all block devices). Since optional, this is a low-risk change. Follow-up patch will actualy modify the installation to apply this parameter in grub.conf. Related: rhbz#510691 Signed-off-by: Alan Pevec <apevec at redhat.com> Signed-off-by: Michael Burns <mburns at redhat.com> --- scripts/ovirt-config-boot | 23 ++++++++++++++++++++++- 1 files changed, 22 insertions(+), 1 deletions(-) diff --git a/scripts/ovirt-config-boot b/scripts/ovirt-config-boot index 2961f76..7c5943b 100755 --- a/scripts/ovirt-config-boot +++ b/scripts/ovirt-config-boot @@ -108,6 +108,19 @@ ovirt_boot_setup() { init_script=sbin/real-init fi sed -i '/^\/sbin\/udev.*trigger/ a \ +filter=\ +for o in `cat /proc/cmdline` ; do\ + case $o in\ + filter=*)\ + filter=${o#filter=}\ + ;;\ + esac\ +done\ +if [ -n "$filter" ]; then\ + mkdir -p /etc/lvm\ + echo "devices { filter = [ \\\"a|^$filter|\\\", \\\"r|.*|\\\" ] }" > /etc/lvm/lvm.conf\ +fi\ + echo SCSI wait for scans\ /sbin/modprobe scsi_wait_scan\ /sbin/modprobe -r scsi_wait_scan\ @@ -138,7 +151,15 @@ set -e\ # lvm is not static in Fedora cp /lib$bit/libreadline.so.5 /lib$bit/libncurses.so.5 lib$bit fi - find $init_script bin/lvm lib$bit -type f | + # add udev rules, so we can use /dev/disk/by-id/* in LVM filter + rules=etc/udev/rules.d + mkdir -p $rules + grep -v -E 'path_id|edd_id|check-cdrom.sh|load_floppy_module.sh|create_floppy_devices' /$rules/50-udev.rules > $rules/50-udev.rules + lud=lib/udev + mkdir -p $lud + cp -pv /$lud/ata_id /$lud/scsi_id /$lud/usb_id $lud + ln -s /sbin/modprobe $lud + find $init_script $scsiwait bin/lvm lib$bit $rules $lud -type f -o -type l | cpio -H newc --quiet -o | gzip -9 | cat $live/$syslinux/initrd0.img - > /boot/initrd0.img -- 1.6.2.5
Michael Burns
2009-Sep-26 02:33 UTC
[Ovirt-devel] [PATCH node-image 2/2] Make multipath work with with r/o root fs
Multipath needs to write to /var/lib/multipath to work correctly with user_friendly names. Signed-off-by: Michael Burns <mburns at redhat.com> --- common-post.ks | 1 + 1 files changed, 1 insertions(+), 0 deletions(-) diff --git a/common-post.ks b/common-post.ks index fcff200..caca19d 100644 --- a/common-post.ks +++ b/common-post.ks @@ -153,6 +153,7 @@ mkdir /live mkdir /boot sed -i '/^files \/etc*/ s/^/#/' /etc/rwtab cat > /etc/rwtab.d/ovirt <<EOF +dirs /var/lib/multipath files /etc files /var/cache/libvirt files /var/cache/hald -- 1.6.2.5
Michael Burns
2009-Sep-26 02:33 UTC
[Ovirt-devel] [PATCH node 3/6] optional mpath parameter
***tech-preview*** "mpath" boot parameter starts multipath in the initrd, enabling usage multipath executables are included in the initrd when installed to the disk and there's related rhev-hypervisor ks change to include multipath kernel modules. To use this, /dev/mpath/... needs to be specified manually in the Storage Configuration menu, after starting multipath manually from Support Menu->Shell Resolves: rhbz#510691 Signed-off-by: Alan Pevec <apevec at redhat.com> Signed-off-by: Michael Burns <mburns at redhat.com> --- scripts/ovirt-config-boot | 35 ++++++++++++++++++++++++++++++++++- 1 files changed, 34 insertions(+), 1 deletions(-) diff --git a/scripts/ovirt-config-boot b/scripts/ovirt-config-boot index 7c5943b..342ebd2 100755 --- a/scripts/ovirt-config-boot +++ b/scripts/ovirt-config-boot @@ -109,11 +109,15 @@ ovirt_boot_setup() { fi sed -i '/^\/sbin\/udev.*trigger/ a \ filter=\ +mpath=\ for o in `cat /proc/cmdline` ; do\ case $o in\ filter=*)\ filter=${o#filter=}\ ;;\ + mpath)\ + mpath=1\ + ;;\ esac\ done\ if [ -n "$filter" ]; then\ @@ -124,6 +128,11 @@ fi\ echo SCSI wait for scans\ /sbin/modprobe scsi_wait_scan\ /sbin/modprobe -r scsi_wait_scan\ +if [ -n "$mpath" ]; then\ + modprobe dm_multipath || :\ + multipath -v3\ + dmsetup ls --target multipath --exec "kpartx -a -p p"\ +fi\ set +e\ while true; do\ echo Scanning logical volumes\ @@ -151,6 +160,18 @@ set -e\ # lvm is not static in Fedora cp /lib$bit/libreadline.so.5 /lib$bit/libncurses.so.5 lib$bit fi + # add multipath executables + mkdir -p sbin + if [ -e /sbin/multipath.static ]; then + cp /sbin/multipath.static sbin/multipath + cp /sbin/scsi_id sbin/scsi_id + cp /sbin/dmsetup.static sbin/dmsetup + cp /sbin/kpartx.static sbin/kpartx + for M in /sbin/mpath_prio_*.static ; do + cp ${M} $/${M%%.static}; + done + fi + # add udev rules, so we can use /dev/disk/by-id/* in LVM filter rules=etc/udev/rules.d mkdir -p $rules @@ -159,7 +180,7 @@ set -e\ mkdir -p $lud cp -pv /$lud/ata_id /$lud/scsi_id /$lud/usb_id $lud ln -s /sbin/modprobe $lud - find $init_script $scsiwait bin/lvm lib$bit $rules $lud -type f -o -type l | + find $init_script $scsiwait bin/lvm lib$bit $rules $lud sbin -type f -o -type l | cpio -H newc --quiet -o | gzip -9 | cat $live/$syslinux/initrd0.img - > /boot/initrd0.img @@ -169,6 +190,18 @@ set -e\ # reorder tty0 to allow both serial and phys console after installation bootparams="ro root=/dev/HostVG/Root roottypefs=ext3 console=tty0 \ $(echo $bootparams | sed s/console=tty0//g)" + # add multipath executables + mkdir -p sbin + if [ -e /sbin/multipath.static ]; then + cp /sbin/multipath.static sbin/multipath + cp /sbin/scsi_id sbin/scsi_id + cp /sbin/dmsetup.static sbin/dmsetup + cp /sbin/kpartx.static sbin/kpartx + for M in /sbin/mpath_prio_*.static ; do + cp ${M} $/${M%%.static}; + done + fi + cat > /boot/grub/grub.conf << EOF default=0 timeout=5 -- 1.6.2.5
default config for HP MSA2[02]12fc|MSA2012i uses prio_callout /bin/true add multipath.conf if present, if not defaults are used (empty config file) use /dev/mapper/* in filter to avoid waiting for udev to create /dev/mpath/* Related: rhbz#510691 Signed-off-by: Michael Burns <mburns at redhat.com> --- scripts/ovirt-config-boot | 23 +++++++++++++++++++---- 1 files changed, 19 insertions(+), 4 deletions(-) diff --git a/scripts/ovirt-config-boot b/scripts/ovirt-config-boot index 342ebd2..36fdf41 100755 --- a/scripts/ovirt-config-boot +++ b/scripts/ovirt-config-boot @@ -147,7 +147,7 @@ set -e\ ' $init_script # fix emergency shell sed -i 's/^ bash$/ bash < \/dev\/console/' $init_script - mkdir -p bin + mkdir -p bin etc sbin bit if [ -e /lib64 ]; then bit=64 @@ -160,8 +160,7 @@ set -e\ # lvm is not static in Fedora cp /lib$bit/libreadline.so.5 /lib$bit/libncurses.so.5 lib$bit fi - # add multipath executables - mkdir -p sbin + # add multipath executables and config if [ -e /sbin/multipath.static ]; then cp /sbin/multipath.static sbin/multipath cp /sbin/scsi_id sbin/scsi_id @@ -170,6 +169,15 @@ set -e\ for M in /sbin/mpath_prio_*.static ; do cp ${M} $/${M%%.static}; done + # default config for HP MSA2[02]12fc|MSA2012i + # prio_callout /bin/true + cp /bin/true bin + if [ -f /etc/multipath.conf ]; then + cp /etc/multipath.conf etc + else + # defualt config + touch etc/multipath.conf + fi fi # add udev rules, so we can use /dev/disk/by-id/* in LVM filter @@ -180,7 +188,7 @@ set -e\ mkdir -p $lud cp -pv /$lud/ata_id /$lud/scsi_id /$lud/usb_id $lud ln -s /sbin/modprobe $lud - find $init_script $scsiwait bin/lvm lib$bit $rules $lud sbin -type f -o -type l | + find $init_script $scsiwait lib$bit $lud bin etc sbin -type f -o -type l | cpio -H newc --quiet -o | gzip -9 | cat $live/$syslinux/initrd0.img - > /boot/initrd0.img @@ -190,6 +198,13 @@ set -e\ # reorder tty0 to allow both serial and phys console after installation bootparams="ro root=/dev/HostVG/Root roottypefs=ext3 console=tty0 \ $(echo $bootparams | sed s/console=tty0//g)" + # append mpath and filter if selected disk is /dev/mpath/* + if [[ "$disk" =~ "^/dev/mpath/" ]]; then + # convert to /dev/mapper/* which is created directly by device-mapper + # to avoid waiting for udev + mpath_id=${disk#/dev/mpath/} + bootparams="$bootparams mpath filter=/dev/mapper/$mpath_id" + fi # add multipath executables mkdir -p sbin if [ -e /sbin/multipath.static ]; then -- 1.6.2.5
Michael Burns
2009-Sep-26 02:33 UTC
[Ovirt-devel] [PATCH node 5/6] Add handling so that install to mpath device works.
Detects if user enters /dev/mpath and changes to /dev/mapper instead runs kpartx instead of blockdev to reload partitions if mpath device used. Runs kpartx after parted commands to reload partitions of mpath device. Update ovirt-config-boot to handle /dev/mapper mpath device. Previously, it was not added to kernel args correctly. rhbz#510691 Signed-off-by: Mike Burns <mburns at redhat.com> Signed-off-by: Michael Burns <mburns at redhat.com> --- scripts/ovirt-config-boot | 5 +++++ scripts/ovirt-config-storage | 16 +++++++++++++++- 2 files changed, 20 insertions(+), 1 deletions(-) diff --git a/scripts/ovirt-config-boot b/scripts/ovirt-config-boot index 36fdf41..87cf832 100755 --- a/scripts/ovirt-config-boot +++ b/scripts/ovirt-config-boot @@ -204,6 +204,11 @@ set -e\ # to avoid waiting for udev mpath_id=${disk#/dev/mpath/} bootparams="$bootparams mpath filter=/dev/mapper/$mpath_id" + elif [[ "$disk" =~ "^/dev/mapper/" ]]; then + mpath_id=${disk#/dev/mapper/} + if [[ -e /dev/mpath/$mpath_id ]]; then + bootparams="$bootparams mpath filter=$disk" + fi fi # add multipath executables mkdir -p sbin diff --git a/scripts/ovirt-config-storage b/scripts/ovirt-config-storage index 473826f..91662c5 100755 --- a/scripts/ovirt-config-storage +++ b/scripts/ovirt-config-storage @@ -335,10 +335,21 @@ perform_partitioning() # begin critical section set -e + # change /dev/mpath device to /dev/mapper + if [[ "$DRIVE" =~ "^/dev/mpath/" ]]; then + log "renaming /dev/mpath to be /dev/mapper" + mpath_id=${DRIVE#/dev/mpath/} + DRIVE=/dev/mapper/$mpath_id + fi + # FIXME: save a backup copy, just in case? log "Wiping old boot sector" dd if=/dev/zero of=$DRIVE bs=1024K count=1 - blockdev --rereadpt $DRIVE + if [[ "$DRIVE" =~ "^/dev/mapper" ]]; then + kpartx -a $DRIVE + else + blockdev --rereadpt $DRIVE + fi partprobe -s $DRIVE MEM_SIZE_MB=$(echo "scale=0; $MEM_SIZE_MB / 1024;" | bc -l) @@ -355,6 +366,9 @@ perform_partitioning() parted $DRIVE -s "set 2 lvm on" parted $DRIVE -s "print" udevadm settle 2> /dev/null || udevsettle + if [[ "$DRIVE" =~ "^/dev/mapper" ]]; then + kpartx -a $DRIVE + fi # sync GPT to the legacy MBR partitions if [ "gpt" == "$LABEL_TYPE" ]; then -- 1.6.2.5
Michael Burns
2009-Sep-26 02:33 UTC
[Ovirt-devel] [PATCH node 6/6] Added new menu for Multipath configuration
Also added new kernel command line arg to setup multipath. New menu takes user through mpath whitelisting and choosing an mpath device for installation. Signed-off-by: Michael Burns <mburns at redhat.com> --- scripts/ovirt-config-boot | 2 +- scripts/ovirt-config-storage | 188 +++++++++++++++++++++++++++++++++++++++- scripts/ovirt-config-uninstall | 8 ++ scripts/ovirt-early | 16 +++- 4 files changed, 207 insertions(+), 7 deletions(-) diff --git a/scripts/ovirt-config-boot b/scripts/ovirt-config-boot index 87cf832..ed2bb9f 100755 --- a/scripts/ovirt-config-boot +++ b/scripts/ovirt-config-boot @@ -218,7 +218,7 @@ set -e\ cp /sbin/dmsetup.static sbin/dmsetup cp /sbin/kpartx.static sbin/kpartx for M in /sbin/mpath_prio_*.static ; do - cp ${M} $/${M%%.static}; + cp ${M} .${M%%.static}; done fi diff --git a/scripts/ovirt-config-storage b/scripts/ovirt-config-storage index 91662c5..c448aee 100755 --- a/scripts/ovirt-config-storage +++ b/scripts/ovirt-config-storage @@ -130,6 +130,172 @@ check_partition_sizes() return $rc } +is_multipath_disabled() +{ + grep -q "^ *devnode \"\*\"" /etc/multipath.conf +} + +multipath_warn() +{ + sp=' ' + w='!!WARNING' + wb="$w"'!!' + w8="$w$w$w$w$w$w$w$w" + printf '%s!!\n' \ + "$w8" \ + "$w8" \ + "$wb$sp$w" \ + "$wb$sp$w" \ + "$wb Multipath Configurations should only be made $w" \ + "$wb on true multipath devices. Configuring any $w" \ + "$wb other devices may cause problems with your $w" \ + "$wb system. $w" \ + "$wb$sp$w" \ + "$wb Proceeding from this menu will erase all $w" \ + "$wb previous multipath configurations. $w" \ + "$wb$sp$w" \ + "$wb$sp$w" \ + "$w8" \ + "$w8" + +} + +abort_multipath() +{ + printf "Cleaning up multipath configuration\n">&2 + sed -i 's/^\s*wwid \*$/ devnode "*"/' /etc/multipath.conf + sed -i 's/^\s*wwid \w.*$//g' /etc/multipath.conf + service multipathd reload >&2 +} + +config_multipath_conf() +{ + ! is_multipath_disabled && abort_multipath + if [[ $OVIRT_MPATH = "ALL" ]]; then + sed -i 's/^\s*devnode ".*$/ #devnode "*"/' /etc/multipath.conf + else + sed -i 's/^\s*devnode ".*$/ wwid */' /etc/multipath.conf + test $(grep ^blacklist_exceptions /etc/multipath.conf | wc -l) = 0 \ + && printf "blacklist_exceptions {\n}" >> /etc/multipath.conf + local wwid="" + for d in $whitelisted + do + d=$(basename $d) + local scsi_id=$(scsi_id --whitelisted --device=/dev/$d) + if [[ -z $scsi_id ]]; then + echo "scsi_id failed for device /dev/$d. Not white listing." >&2 + else + wwid="${wwid} $scsi_id" + fi + done + wwid=$(echo $wwid | sed 's/ /\n/g' | sort -u) + for scsi_id in $wwid + do + sed -i "s/^blacklist_exceptions {$/blacklist_exceptions {\n wwid $scsi_id"/ /etc/multipath.conf + done + for d in $OVIRT_MPATH + do + sed -i "s/^blacklist_exceptions {$/blacklist_exceptions {\n wwid $d"/ /etc/multipath.conf + + done + fi + service multipathd reload >&2 + start_log + echo "Generated multipath.conf" + cat /etc/multipath.conf + echo "Output of multiatph -v6 command" + multipath -v6 + stop_log + return 0 +} + + +generate_whitelist() +{ + whitelisted="" + local devs=$devices + local PS3="Please select devices to add or remove from multipath whitelist: " + while true; do + local choices="" + for d in $devs + do + get_drive_size $d >&2 + if [[ $whitelisted =~ $d ]]; then + choices="$choices $d-whitelisted" + else + choices="$choices $d" + fi + done + + choices="$choices Done Abort" + select whitelist in $choices + do + test "$whitelist" = Abort && return 1 + test "$whitelist" = Done && return 0 + #first check if it is already whitelisted + #and remove from whitelist + if [[ $whitelist =~ "whitelisted" ]]; then + local tmp_whitelist="" + whitelist=${whitelist%-whitelisted} + for d in $whitelisted + do + test $d != $whitelist && tmp_whitelist="$tmp_whitelist $d" + done + whitelisted=$tmp_whitelist + #else add it to whitelist + else + whitelisted="$whitelisted $whitelist" + fi + break + done + done +} + +# Choose a multipathed device for installation +get_multipath_device() +{ + multipath_warn >&2 + local DRIVE_VAR=$1 + generate_whitelist + test $? = 1 && return 1 + config_multipath_conf + local count=0 + echo "Detecting Multipath Devices..." >&2 + while [ $count -lt 60 ] + do + test ! `ls /dev/mapper/mpath* 2> /dev/null | wc -l` = 0 && break + let count=$count+1 + sleep 1 + done + if [ `ls /dev/mapper/mpath* 2> /dev/null | wc -l` = 0 ]; then + echo "No Multipath devices found. Aborting...">&2 + abort_multipath + return 1 + fi + #need to provide a menu to choose a multipath device + while true + do + local choices="" + for dev in $(ls /dev/mapper/mpath* | egrep -v "p[0-9]+$") + do + choices="$choices $dev" + get_drive_size $dev >&2 + done + choices="$choices Abort" + local PS3="Select Multipath Device: " + select mpath_device in $choices + do + if [[ $mpath_device = Abort ]]; then + abort_multipath + return 1 + elif [[ -e $mpath_device ]]; then + eval $DRIVE_VAR=$mpath_device + return 0 + fi + done + done +} + # Find a usable/selected storage device. # If there are none, give a diagnostic and return nonzero. # If there is just one, e.g., /dev/sda, treat it as selected (see below). @@ -140,6 +306,7 @@ check_partition_sizes() get_dev_name() { local udi_list=$(hal-find-by-capability --capability storage) + local DRIVE_VAR=$1 if test -z "$udi_list"; then warn "ERROR: no usable storage devices detected" return 1 @@ -152,6 +319,9 @@ get_dev_name() local block_dev=$(hal-get-property --udi "$d" --key block.device) # Must start with a '/'. case $block_dev in + '') + #if block.device not defined, suppress warning + continue;; *' '*) # we use space as separator warn "block device name '$block_dev' contains space; skipping"; @@ -189,12 +359,13 @@ get_dev_name() for d in $devices; do get_drive_size $d >&2 done - local choices="$devices Abort" + local choices="$devices Multipath Abort" select device in $choices do test "$device" = Abort && return 1 + test "$device" = "Multipath" && get_multipath_device device; test -z "$device" && continue - echo "$device" + eval $DRIVE_VAR=$device return 0 done } @@ -202,7 +373,9 @@ get_dev_name() do_configure() { local name_and_size - DRIVE=$(get_dev_name) || return 0 + get_dev_name DRIVE + test ! -e $DRIVE && echo "Drive '$DRIVE' is not a valid drive" >&2 \ + && return 0 get_drive_size $DRIVE SPACE printf "\n\nPlease configure storage partitions.\n\n" @@ -537,6 +710,15 @@ CONFIG_SIZE=${OVIRT_VOL_CONFIG_SIZE:-$default_config_size} LOGGING_SIZE=${OVIRT_VOL_LOGGING_SIZE:-$default_logging_size} DATA_SIZE=${OVIRT_VOL_DATA_SIZE:-$default_data_size} +if [ -n "$OVIRT_MPATH" ]; then + #if present then setup multipath.conf with value + config_multipath_conf + while true + do + test ! `ls /dev/mapper/mpath* 2> /dev/null | wc -l` = 0 && break + done +fi + if [ -n "$OVIRT_INIT" ]; then # if present, use the drive selected with 'ovirt_init' boot parameter DRIVE=$OVIRT_INIT diff --git a/scripts/ovirt-config-uninstall b/scripts/ovirt-config-uninstall index fd65cbb..20f8c2c 100755 --- a/scripts/ovirt-config-uninstall +++ b/scripts/ovirt-config-uninstall @@ -47,6 +47,14 @@ if ask_yes_or_no "Do you wish to continue and uninstall this node ([Y]es/[N]o)?" log "Unmounting boot partition" umount $partition log "Removing partitions" + eval $(echo $partition | awk ' { + print "drive=" substr($0,1,length($1)-1); + print "drive2=" substr($0,1,length($1)-2); + }') + if [ ! -e "$drive" ]; then + # e.g. c0d0p1 + drive="$drive2" + fi drive=$(echo $partition | awk '{ print substr($0, 1, length($0) - 1) }') parted -s $drive "rm 1" parted -s $drive "rm 2" diff --git a/scripts/ovirt-early b/scripts/ovirt-early index 8990727..5632fb0 100755 --- a/scripts/ovirt-early +++ b/scripts/ovirt-early @@ -148,6 +148,7 @@ start() { # collectd=server[:port] # hostname=fqdn # TBD logrotate maxsize + # mpath format: mpath=wwid:[wwid] # BOOTIF=link|eth*|<MAC> (appended by pxelinux) # network boot interface is assumed to be on management network where @@ -236,6 +237,12 @@ start() { collectd_server collectd_port + # mpath=wwid:[wwid...] + # set a list of comma or colon separated wwids to allow + # for multipath install + # Specify ALL to use all devices + mpath+ # save boot parameters like console= for local disk boot menu bootparams cat /etc/system-release >> $OVIRT_LOGFILE @@ -286,8 +293,7 @@ start() { =/dev/*) bus serial- i=${i#=} - init=$(ls -1 "$i" 2>/dev/null | head -n1) + init=${i#=} ;; *) bus@@ -374,6 +380,10 @@ start() { console=*) bootparams="$bootparams $i" ;; + mpath=*) + i=${i#mpath=} + mpath=$(echo $i|tr ",:;" " ") + ;; esac done @@ -384,7 +394,7 @@ start() { ip_gateway=$gateway fi # save boot parameters as defaults for ovirt-config-* - params="bootif init vol_boot_size vol_swap_size vol_root_size vol_config_size vol_logging_size vol_data_size local_boot standalone overcommit ip_address ip_netmask ip_gateway ipv6 dns ntp vlan ssh_pwauth syslog_server syslog_port collectd_server collectd_port bootparams hostname firstboot" + params="bootif init vol_boot_size vol_swap_size vol_root_size vol_config_size vol_logging_size vol_data_size local_boot standalone overcommit ip_address ip_netmask ip_gateway ipv6 dns ntp vlan ssh_pwauth syslog_server syslog_port collectd_server collectd_port bootparams hostname firstboot mpath" # mount /config unless firstboot is forced if [ "$firstboot" != "1" ]; then mount_config -- 1.6.2.5
Joey Boggs
2009-Oct-01 01:27 UTC
[Ovirt-devel] [PATCH node 6/6] Added new menu for Multipath configuration
Michael Burns wrote:> Also added new kernel command line arg to setup multipath. > New menu takes user through mpath whitelisting and choosing an > mpath device for installation. > > Signed-off-by: Michael Burns <mburns at redhat.com> > --- > scripts/ovirt-config-boot | 2 +- > scripts/ovirt-config-storage | 188 +++++++++++++++++++++++++++++++++++++++- > scripts/ovirt-config-uninstall | 8 ++ > scripts/ovirt-early | 16 +++- > 4 files changed, 207 insertions(+), 7 deletions(-) > > diff --git a/scripts/ovirt-config-boot b/scripts/ovirt-config-boot > index 87cf832..ed2bb9f 100755 > --- a/scripts/ovirt-config-boot > +++ b/scripts/ovirt-config-boot > @@ -218,7 +218,7 @@ set -e\ > cp /sbin/dmsetup.static sbin/dmsetup > cp /sbin/kpartx.static sbin/kpartx > for M in /sbin/mpath_prio_*.static ; do > - cp ${M} $/${M%%.static}; > + cp ${M} .${M%%.static}; > done > fi > > diff --git a/scripts/ovirt-config-storage b/scripts/ovirt-config-storage > index 91662c5..c448aee 100755 > --- a/scripts/ovirt-config-storage > +++ b/scripts/ovirt-config-storage > @@ -130,6 +130,172 @@ check_partition_sizes() > return $rc > } > > +is_multipath_disabled() > +{ > + grep -q "^ *devnode \"\*\"" /etc/multipath.conf > +} > + > +multipath_warn() > +{ > + sp=' ' > + w='!!WARNING' > + wb="$w"'!!' > + w8="$w$w$w$w$w$w$w$w" > + printf '%s!!\n' \ > + "$w8" \ > + "$w8" \ > + "$wb$sp$w" \ > + "$wb$sp$w" \ > + "$wb Multipath Configurations should only be made $w" \ > + "$wb on true multipath devices. Configuring any $w" \ > + "$wb other devices may cause problems with your $w" \ > + "$wb system. $w" \ > + "$wb$sp$w" \ > + "$wb Proceeding from this menu will erase all $w" \ > + "$wb previous multipath configurations. $w" \ > + "$wb$sp$w" \ > + "$wb$sp$w" \ > + "$w8" \ > + "$w8" > + > +} > + > +abort_multipath() > +{ > + printf "Cleaning up multipath configuration\n">&2 > + sed -i 's/^\s*wwid \*$/ devnode "*"/' /etc/multipath.conf > + sed -i 's/^\s*wwid \w.*$//g' /etc/multipath.conf > + service multipathd reload >&2 > +} > + > +config_multipath_conf() > +{ > + ! is_multipath_disabled && abort_multipath > + if [[ $OVIRT_MPATH = "ALL" ]]; then > + sed -i 's/^\s*devnode ".*$/ #devnode "*"/' /etc/multipath.conf > + else > + sed -i 's/^\s*devnode ".*$/ wwid */' /etc/multipath.conf > + test $(grep ^blacklist_exceptions /etc/multipath.conf | wc -l) = 0 \ > + && printf "blacklist_exceptions {\n}" >> /etc/multipath.conf > + local wwid="" > + for d in $whitelisted > + do > + d=$(basename $d) > + local scsi_id=$(scsi_id --whitelisted --device=/dev/$d) > + if [[ -z $scsi_id ]]; then > + echo "scsi_id failed for device /dev/$d. Not white listing." >&2 > + else > + wwid="${wwid} $scsi_id" > + fi > + done > + wwid=$(echo $wwid | sed 's/ /\n/g' | sort -u) > + for scsi_id in $wwid > + do > + sed -i "s/^blacklist_exceptions {$/blacklist_exceptions {\n wwid $scsi_id"/ /etc/multipath.conf > + done > + for d in $OVIRT_MPATH > + do > + sed -i "s/^blacklist_exceptions {$/blacklist_exceptions {\n wwid $d"/ /etc/multipath.conf > + > + done > + fi > + service multipathd reload >&2 > + start_log > + echo "Generated multipath.conf" > + cat /etc/multipath.conf > + echo "Output of multiatph -v6 command" > + multipath -v6 > + stop_log > + return 0 > +} > + > + > +generate_whitelist() > +{ > + whitelisted="" > + local devs=$devices > + local PS3="Please select devices to add or remove from multipath whitelist: " > + while true; do > + local choices="" > + for d in $devs > + do > + get_drive_size $d >&2 > + if [[ $whitelisted =~ $d ]]; then > + choices="$choices $d-whitelisted" > + else > + choices="$choices $d" > + fi > + done > + > + choices="$choices Done Abort" > + select whitelist in $choices > + do > + test "$whitelist" = Abort && return 1 > + test "$whitelist" = Done && return 0 > + #first check if it is already whitelisted > + #and remove from whitelist > + if [[ $whitelist =~ "whitelisted" ]]; then > + local tmp_whitelist="" > + whitelist=${whitelist%-whitelisted} > + for d in $whitelisted > + do > + test $d != $whitelist && tmp_whitelist="$tmp_whitelist $d" > + done > + whitelisted=$tmp_whitelist > + #else add it to whitelist > + else > + whitelisted="$whitelisted $whitelist" > + fi > + break > + done > + done > +} > + > +# Choose a multipathed device for installation > +get_multipath_device() > +{ > + multipath_warn >&2 > + local DRIVE_VAR=$1 > + generate_whitelist > + test $? = 1 && return 1 > + config_multipath_conf > + local count=0 > + echo "Detecting Multipath Devices..." >&2 > + while [ $count -lt 60 ] > + do > + test ! `ls /dev/mapper/mpath* 2> /dev/null | wc -l` = 0 && break > + let count=$count+1 > + sleep 1 > + done > + if [ `ls /dev/mapper/mpath* 2> /dev/null | wc -l` = 0 ]; then > + echo "No Multipath devices found. Aborting...">&2 > + abort_multipath > + return 1 > + fi > + #need to provide a menu to choose a multipath device > + while true > + do > + local choices="" > + for dev in $(ls /dev/mapper/mpath* | egrep -v "p[0-9]+$") > + do > + choices="$choices $dev" > + get_drive_size $dev >&2 > + done > + choices="$choices Abort" > + local PS3="Select Multipath Device: " > + select mpath_device in $choices > + do > + if [[ $mpath_device = Abort ]]; then > + abort_multipath > + return 1 > + elif [[ -e $mpath_device ]]; then > + eval $DRIVE_VAR=$mpath_device > + return 0 > + fi > + done > + done > +} > + > # Find a usable/selected storage device. > # If there are none, give a diagnostic and return nonzero. > # If there is just one, e.g., /dev/sda, treat it as selected (see below). > @@ -140,6 +306,7 @@ check_partition_sizes() > get_dev_name() > { > local udi_list=$(hal-find-by-capability --capability storage) > + local DRIVE_VAR=$1 > if test -z "$udi_list"; then > warn "ERROR: no usable storage devices detected" > return 1 > @@ -152,6 +319,9 @@ get_dev_name() > local block_dev=$(hal-get-property --udi "$d" --key block.device) > # Must start with a '/'. > case $block_dev in > + '') > + #if block.device not defined, suppress warning > + continue;; > *' '*) > # we use space as separator > warn "block device name '$block_dev' contains space; skipping"; > @@ -189,12 +359,13 @@ get_dev_name() > for d in $devices; do > get_drive_size $d >&2 > done > - local choices="$devices Abort" > + local choices="$devices Multipath Abort" > select device in $choices > do > test "$device" = Abort && return 1 > + test "$device" = "Multipath" && get_multipath_device device; > test -z "$device" && continue > - echo "$device" > + eval $DRIVE_VAR=$device > return 0 > done > } >> @@ -202,7 +373,9 @@ get_dev_name() > do_configure() > { > local name_and_size > - DRIVE=$(get_dev_name) || return 0 > + get_dev_name DRIVE > + test ! -e $DRIVE && echo "Drive '$DRIVE' is not a valid drive" >&2 \ > + && return 0 > get_drive_size $DRIVE SPACE > >In this section altering DRIVE= kills me trying to use local disks. Took me awhile to track this one down. Here's how to fix it: local num_devices=$(echo "$devices" | wc -w) # If there's only one device, use it. case $num_devices in 0) warn "ERROR: found no usable block device"; return 1;; 1) echo "$devices"; return 0;; *) ;; # found more than one esac set 1) echo "$devices"; return 0;; to 1) eval $DRIVE_VAR=$devices; return0;;> printf "\n\nPlease configure storage partitions.\n\n" > @@ -537,6 +710,15 @@ CONFIG_SIZE=${OVIRT_VOL_CONFIG_SIZE:-$default_config_size} > LOGGING_SIZE=${OVIRT_VOL_LOGGING_SIZE:-$default_logging_size} > DATA_SIZE=${OVIRT_VOL_DATA_SIZE:-$default_data_size} > > +if [ -n "$OVIRT_MPATH" ]; then > + #if present then setup multipath.conf with value > + config_multipath_conf > + while true > + do > + test ! `ls /dev/mapper/mpath* 2> /dev/null | wc -l` = 0 && break > + done > +fi > + > if [ -n "$OVIRT_INIT" ]; then > # if present, use the drive selected with 'ovirt_init' boot parameter > DRIVE=$OVIRT_INIT > diff --git a/scripts/ovirt-config-uninstall b/scripts/ovirt-config-uninstall > index fd65cbb..20f8c2c 100755 > --- a/scripts/ovirt-config-uninstall > +++ b/scripts/ovirt-config-uninstall > @@ -47,6 +47,14 @@ if ask_yes_or_no "Do you wish to continue and uninstall this node ([Y]es/[N]o)?" > log "Unmounting boot partition" > umount $partition > log "Removing partitions" > + eval $(echo $partition | awk ' { > + print "drive=" substr($0,1,length($1)-1); > + print "drive2=" substr($0,1,length($1)-2); > + }') > + if [ ! -e "$drive" ]; then > + # e.g. c0d0p1 > + drive="$drive2" > + fi > drive=$(echo $partition | awk '{ print substr($0, 1, length($0) - 1) }') > parted -s $drive "rm 1" > parted -s $drive "rm 2" > diff --git a/scripts/ovirt-early b/scripts/ovirt-early > index 8990727..5632fb0 100755 > --- a/scripts/ovirt-early > +++ b/scripts/ovirt-early > @@ -148,6 +148,7 @@ start() { > # collectd=server[:port] > # hostname=fqdn > # TBD logrotate maxsize > + # mpath format: mpath=wwid:[wwid] > > # BOOTIF=link|eth*|<MAC> (appended by pxelinux) > # network boot interface is assumed to be on management network where > @@ -236,6 +237,12 @@ start() { > collectd_server> collectd_port> > + # mpath=wwid:[wwid...] > + # set a list of comma or colon separated wwids to allow > + # for multipath install > + # Specify ALL to use all devices > + mpath> + > # save boot parameters like console= for local disk boot menu > bootparams> cat /etc/system-release >> $OVIRT_LOGFILE > @@ -286,8 +293,7 @@ start() { > =/dev/*) > bus> serial> - i=${i#=} > - init=$(ls -1 "$i" 2>/dev/null | head -n1) > + init=${i#=} > ;; > *) > bus> @@ -374,6 +380,10 @@ start() { > console=*) > bootparams="$bootparams $i" > ;; > + mpath=*) > + i=${i#mpath=} > + mpath=$(echo $i|tr ",:;" " ") > + ;; > esac > done > > @@ -384,7 +394,7 @@ start() { > ip_gateway=$gateway > fi > # save boot parameters as defaults for ovirt-config-* > - params="bootif init vol_boot_size vol_swap_size vol_root_size vol_config_size vol_logging_size vol_data_size local_boot standalone overcommit ip_address ip_netmask ip_gateway ipv6 dns ntp vlan ssh_pwauth syslog_server syslog_port collectd_server collectd_port bootparams hostname firstboot" > + params="bootif init vol_boot_size vol_swap_size vol_root_size vol_config_size vol_logging_size vol_data_size local_boot standalone overcommit ip_address ip_netmask ip_gateway ipv6 dns ntp vlan ssh_pwauth syslog_server syslog_port collectd_server collectd_port bootparams hostname firstboot mpath" > # mount /config unless firstboot is forced > if [ "$firstboot" != "1" ]; then > mount_config >
Joey Boggs
2009-Oct-02 01:58 UTC
[Ovirt-devel] [PATCH node 2/6] add LVM filter support in initrd
> diff --git a/scripts/ovirt-config-boot b/scripts/ovirt-config-boot > index 2961f76..7c5943b 100755 > --- a/scripts/ovirt-config-boot > +++ b/scripts/ovirt-config-boot > @@ -108,6 +108,19 @@ ovirt_boot_setup() { > init_script=sbin/real-init > fi > sed -i '/^\/sbin\/udev.*trigger/ a \ > +filter=\ > +for o in `cat /proc/cmdline` ; do\ > + case $o in\ > + filter=*)\ > + filter=${o#filter=}\ > + ;;\ > + esac\ > +done\ > +if [ -n "$filter" ]; then\ > + mkdir -p /etc/lvm\ > + echo "devices { filter = [ \\\"a|^$filter|\\\", \\\"r|.*|\\\" ] }" > /etc/lvm/lvm.conf\ > +fi\ > + >This extra empty line ^^ causes the install to hang for me on local disk. Take it out and it installs fine. I'm hitting another issue where the rebuilt initrd causes the boot to hang like we've seen before right after starting udev. Been a long night, so I'll keep working on it in the morning. Have you ran into similar issues during testing?