Hi list, I'm having one of those 'I'm stupid' -problems with LVM on CentOS 5.2. I've been working with traditional partitions until now, but I've finally been sold on the theoretical benefits of using LVM, but for now I only have a huge pile of broken filesystems to show for my efforts. My scenario; I attach a disk, either over USB or iSCSI. I create a PV on this device, create a VG using the device, and slice off a tiny LV for tests. [root at kasse ~]# dmesg usb 1-3: new high speed USB device using ehci_hcd and address 7 usb 1-3: configuration #1 chosen from 1 choice scsi27 : SCSI emulation for USB Mass Storage devices usb-storage: device found at 7 usb-storage: waiting for device to settle before scanning Vendor: ST375064 Model: 0A Rev: 3.AA Type: Direct-Access ANSI SCSI revision: 00 SCSI device sdg: 1465149168 512-byte hdwr sectors (750156 MB) sdg: Write Protect is off sdg: Mode Sense: 10 00 00 00 sdg: assuming drive cache: write through SCSI device sdg: 1465149168 512-byte hdwr sectors (750156 MB) sdg: Write Protect is off sdg: Mode Sense: 10 00 00 00 sdg: assuming drive cache: write through sdg: sdg1 sd 27:0:0:0: Attached scsi disk sdg sd 27:0:0:0: Attached scsi generic sg6 type 0 usb-storage: device scan complete [root at kasse ~]# pvcreate /dev/sdg1 Physical volume "/dev/sdg1" successfully created [root at kasse ~]# vgcreate testgroup /dev/sdg1 Volume group "testgroup" successfully created [root at kasse ~]# lvcreate -L 200M testgroup -n testLV Logical volume "testLV" created At this point, I have /dev/testgroup/testLV, which I can stick a fs on, mount, store files on, unmount, remount and recover the files from. Just as I'd expect. [root at kasse ~]# lvdisplay --- Logical volume --- LV Name /dev/testgroup/testLV VG Name testgroup LV UUID 3KjDiZ-gTAD-sC9E-sOSA-772o-41Yk-ZKK7cx LV Write Access read/write LV Status available # open 0 LV Size 200.00 MB Current LE 50 Segments 1 Allocation inherit Read ahead sectors auto - currently set to 256 Block device 253:1 I have not explicitly set the vg active, as I understand that it implicitly will be set active in non-clustered environments like mine. LVM works as I'd expect it to up until here in any case. So - with the LV unmounted, I power my USB-device down, and then back up; [root at kasse ~]# dmesg usb 1-3: USB disconnect, address 7 usb 1-3: new high speed USB device using ehci_hcd and address 8 usb 1-3: configuration #1 chosen from 1 choice scsi28 : SCSI emulation for USB Mass Storage devices usb-storage: device found at 8 usb-storage: waiting for device to settle before scanning Vendor: ST375064 Model: 0A Rev: 3.AA Type: Direct-Access ANSI SCSI revision: 00 SCSI device sdi: 1465149168 512-byte hdwr sectors (750156 MB) sdi: Write Protect is off sdi: Mode Sense: 10 00 00 00 sdi: assuming drive cache: write through SCSI device sdi: 1465149168 512-byte hdwr sectors (750156 MB) sdi: Write Protect is off sdi: Mode Sense: 10 00 00 00 sdi: assuming drive cache: write through sdi: sdi1 sd 28:0:0:0: Attached scsi disk sdi sd 28:0:0:0: Attached scsi generic sg6 type 0 usb-storage: device scan complete (note that my disk is now sdi, not sdg, and connected on scsi28 and not scsi27) If I now try to access the device, all falls appart; [root at kasse ~]# lvdisplay /dev/testgroup/testLV: read failed after 0 of 4096 at 209649664: Input/output error /dev/testgroup/testLV: read failed after 0 of 4096 at 209707008: Input/output error /dev/testgroup/testLV: read failed after 0 of 4096 at 0: Input/output error /dev/testgroup/testLV: read failed after 0 of 4096 at 4096: Input/output error /dev/testgroup/testLV: read failed after 0 of 4096 at 0: Input/output error --- Logical volume --- LV Name /dev/testgroup/testLV VG Name testgroup LV UUID 3KjDiZ-gTAD-sC9E-sOSA-772o-41Yk-ZKK7cx LV Write Access read/write LV Status available # open 0 LV Size 200.00 MB Current LE 50 Segments 1 Allocation inherit Read ahead sectors auto - currently set to 256 Block device 253:1 Trying to mount or otherwise access the LV also shows the problems; scsi 27:0:0:0: rejecting I/O to dead device scsi 27:0:0:0: rejecting I/O to dead device scsi 27:0:0:0: rejecting I/O to dead device scsi 27:0:0:0: rejecting I/O to dead device scsi 27:0:0:0: rejecting I/O to dead device scsi 27:0:0:0: rejecting I/O to dead device printk: 25 messages suppressed. Buffer I/O error on device dm-1, logical block 30 Buffer I/O error on device dm-1, logical block 31 scsi 27:0:0:0: rejecting I/O to dead device Buffer I/O error on device dm-1, logical block 0 Buffer I/O error on device dm-1, logical block 1 Buffer I/O error on device dm-1, logical block 2 Buffer I/O error on device dm-1, logical block 3 Buffer I/O error on device dm-1, logical block 4 Buffer I/O error on device dm-1, logical block 5 Buffer I/O error on device dm-1, logical block 6 Buffer I/O error on device dm-1, logical block 7 scsi 27:0:0:0: rejecting I/O to dead device scsi 27:0:0:0: rejecting I/O to dead device scsi 27:0:0:0: rejecting I/O to dead device scsi 27:0:0:0: rejecting I/O to dead device hfs: unable to find HFS+ superblock So - LVM seems to believe my LV is still attached to scsi27, while it now actually sits on scsi28. What am I missing here? How do I get LVM to use a persistent device instead of a transient path? Yours, -S -- Simen Thoresen, Dolphin ICS Systems Administration and Wulfkit Support
On Sun, 2008-08-31 at 11:19 +0200, Simen Timian Thoresen wrote:> Hi list, > > I'm having one of those 'I'm stupid' -problems with LVM on CentOS 5.2. > I've been working with traditional partitions until now, but I've > finally been sold on the theoretical benefits of using LVM, but for now > I only have a huge pile of broken filesystems to show for my efforts. > > My scenario; > I attach a disk, either over USB or iSCSI. > I create a PV on this device, create a VG using the device, and slice > off a tiny LV for tests.Uh-oh! I hope these are for temporary usage only!> > [root at kasse ~]# dmesg > <snip>> [root at kasse ~]# pvcreate /dev/sdg1 > Physical volume "/dev/sdg1" successfully created > [root at kasse ~]# vgcreate testgroup /dev/sdg1 > Volume group "testgroup" successfully created > [root at kasse ~]# lvcreate -L 200M testgroup -n testLV > Logical volume "testLV" created > > > At this point, I have /dev/testgroup/testLV, which I can stick a fs on, > mount, store files on, unmount, remount and recover the files from. Just > as I'd expect. > > [root at kasse ~]# lvdisplay > --- Logical volume --- > LV Name /dev/testgroup/testLV > VG Name testgroup > LV UUID 3KjDiZ-gTAD-sC9E-sOSA-772o-41Yk-ZKK7cx > LV Write Access read/write > LV Status available > # open 0 > LV Size 200.00 MB > Current LE 50 > Segments 1 > Allocation inherit > Read ahead sectors auto > - currently set to 256 > Block device 253:1 > > > I have not explicitly set the vg active, as I understand that it > implicitly will be set active in non-clustered environments like mine. > LVM works as I'd expect it to up until here in any case. > > So - with the LV unmounted, I power my USB-device down, and then back up; > > [root at kasse ~]# dmesg > <snip>> (note that my disk is now sdi, not sdg, and connected on scsi28 and not > scsi27)By now you realize that is a big problem?> > If I now try to access the device, all falls appart; > [root at kasse ~]# lvdisplay > /dev/testgroup/testLV: read failed after 0 of 4096 at 209649664: > Input/output error > <snip>> --- Logical volume --- > LV Name /dev/testgroup/testLV > VG Name testgroup > LV UUID 3KjDiZ-gTAD-sC9E-sOSA-772o-41Yk-ZKK7cx > LV Write Access read/write > LV Status available > # open 0 > LV Size 200.00 MB > Current LE 50 > Segments 1 > Allocation inherit > Read ahead sectors auto > - currently set to 256 > Block device 253:1 > > > Trying to mount or otherwise access the LV also shows the problems; > > > scsi 27:0:0:0: rejecting I/O to dead device > <snip> > Buffer I/O error on device dm-1, logical block 7 > scsi 27:0:0:0: rejecting I/O to dead device > scsi 27:0:0:0: rejecting I/O to dead device > scsi 27:0:0:0: rejecting I/O to dead device > scsi 27:0:0:0: rejecting I/O to dead device > hfs: unable to find HFS+ superblock > > So - LVM seems to believe my LV is still attached to scsi27, while it > now actually sits on scsi28. > > What am I missing here?Use the pvdisplay command and you'll see. # pvdisplay --- Physical volume --- PV Name /dev/sdb2 VG Name VolGroup00 PV Size 148.95 GB / not usable 11.37 MB Allocatable yes (but full) PE Size (KByte) 32768 Total PE 4766 Free PE 0 Allocated PE 4766 PV UUID eeqb0A-6EJV-VE3Q-paLi-kTXv-S46k-0xaeTy Sit back a moment and think about it. You defined a PV on a specific device. That device is no longer available. LVM has to track the assignment, allocation and use of PVs. It must have a record of it somewhere. # ls -l /etc/lvm total 44 drwx------ 2 root root 4096 May 25 00:41 archive drwx------ 2 root root 4096 May 25 00:41 backup drwx------ 2 root root 4096 Aug 30 08:38 cache -rw-r--r-- 1 root root 15911 May 25 00:41 lvm.conf Use the man command to read up on more stuff, like "man lvm.conf" and "man lvm". Be sure and read some of the other refs at the bottom of the man pages.> > How do I get LVM to use a persistent device instead of a transient path?Normally you give a persistent device instead of a transient path. You didn't do that. Generally speaking, LVM is designed to use fixed devices - those that always come up on the same address. Neither iscsi nor UHB devices do this reliably. However, on a reboot, there is a liklihood that they will be the same devices. I've not tried to configure it on transient devices. There may be a way, but I've not looked into it. If you make them their own VG, rather than adding the PV to another VG, you can use the export and import LVM commands, but then you really accomplish nothing more than what could be done by mounting the underlying devices in existing directory structures. No point in LVM then unless you anticipate having to grow those mounted structures while in use. Even so, from my LFS days, I do recall a lot of flexibility in the udev process. Maybe there is something there that will allow the assigned device addresses to be fixed? Do man for udev, udevd, udevinfo, udevmonitor. I'm guessing that you'll want to run some utility from udev that will (maybe) look at a a volume label (if one if available on the usb device - is there one on the iscsi device?) and assign a predetermined name if a certain label is found. I've never used iscsi, so I'm even less help on that. Maybe there is something in its man pages that allow a persistent device name assignment.> > Yours, > -SSorry I couldn't offer more. -- Bill
Simen Timian Thoresen wrote:> So - with the LV unmounted, I power my USB-device down, and then back up;I believe your problem is you didn't deactivate the logical volume, and export the volume group before disconnecting. lvchange -a n <path to lvm> vgexport -a (don't worry it will only export groups that are ready to be exported) Now you can safely power the device down(provided the VG was successfully exported, all logical volumes must be unmounted and deactivated before it will succeed). when you power it back up run pvscan vgimport -a lvchange -a y <path to lvm> LVM is very useful for volumes that change device names, but you must use it properly otherwise bad things will happen as you saw. If you want to have USB mount automatically it may be better to put a label on the file system, at least in Debian(don't have a CentOS system with a USB disk handy here) the volume is mounted as /media/<disk label> if there is a disk label. You can use the e2label command to label the device. I'd expect CentOS to likely behave similarly to debian in this respect. nate