Hello,
I have a weird problem after adding new PV do LMV volume group.
It seems the error comes out only during boot time. Please read the story.
I have couple of 1U machines. They all have two, four or more Fujitsu-Siemens
SAS 2,5" disks, which are bounded in Raid1 pairs with Linux mdadm.
First pair of disks has always two arrays (md0, md1). Small md0 is used
for booting and the rest - md1 is used as PV for volume group (vg0).
When I need to enlarge the volume, I just add 2 new disks, create
a raid 1 array of their whole space I add it as another mdX to vg0.
That has been working fine since yesterday. I received new disks
branded by Toshiba (Toshiba bought Fujitsu-Simens hdd business)
which were supposed to be added as disk no 3 and 4.
As far as I remember I've done everything in the same way as before:
- They come out as sdc and sdd, so I
- fdisk /dev/sdc, created one primary parition of whole space, type fd
- the same with /dev/sdd
- mdadm --create /dev/md2 -R -l 1 -n 2 /dev/sdc1 /dev/sdd1
- array has been created and syncronized
- pvcreate /dev/md2
- vgextend vg0 /dev/md2
and it looks fine:
----------------------------------------------------------------------
# cat /proc/mdstat
Personalities : [raid1]
md1 : active raid1 sdb2[1] sda2[0]
140488320 blocks [2/2] [UU]
md2 : active raid1 sdd1[1] sdc1[0]
292961216 blocks [2/2] [UU]
md0 : active raid1 sdb1[1] sda1[0]
3148608 blocks [2/2] [UU]
----------------------------------------------------------------------
# pvdisplay
--- Physical volume ---
PV Name /dev/md1
VG Name vg0
PV Size 133.98 GB / not usable 11.62 MB
Allocatable yes
PE Size (KByte) 32768
Total PE 4287
Free PE 1695
Allocated PE 2592
PV UUID AufZRm-QbFC-xRj1-OxwW-Z2w2-qbkM-qzoEcP
--- Physical volume ---
PV Name /dev/md2
VG Name vg0
PV Size 279.39 GB / not usable 14.94 MB
Allocatable yes
PE Size (KByte) 32768
Total PE 8940
Free PE 8940
Allocated PE 0
PV UUID qeDW2q-nq5b-Yh5U-5sKY-7Rkd-1UXh-LAxL8j
----------------------------------------------------------------------
# vgdisplay
--- Volume group ---
VG Name vg0
System ID
Format lvm2
Metadata Areas 2
Metadata Sequence No 1010
VG Access read/write
VG Status resizable
MAX LV 0
Cur LV 8
Open LV 2
Max PV 0
Cur PV 2
Act PV 2
VG Size 413.34 GB
PE Size 32.00 MB
Total PE 13227
Alloc PE / Size 2592 / 81.00 GB
Free PE / Size 10635 / 332.34 GB
VG UUID 5cF1dk-1CMM-qiuf-CyNY-aCmw-8Hx8-4iO12I
----------------------------------------------------------------------
# lvdisplay
--- Logical volume ---
LV Name /dev/vg0/d0v
VG Name vg0
LV UUID q7zmrV-EykH-jPzR-smYJ-eehh-3Gbx-ebu5wK
LV Write Access read/write
LV Status available
# open 1
LV Size 3.00 GB
Current LE 96
Segments 1
Allocation inherit
Read ahead sectors auto
- currently set to 256
Block device 253:0
[...] this part may be skiped as it's very long an irrelevant.
There are no errors here.
----------------------------------------------------------------------
And now please take a look what happens during boot:
[...]
scsi0 : ioc0: LSISAS1078 C2, FwRev=01180400h, Ports=1, MaxQ=276, IRQ=20
mptsas: ioc0: attaching ssp device: fw_channel 0, fw_id 1, phy 0, sas_addr
0x500000e01f975022
Vendor: FUJITSU Model: MBB2147RC Rev: 0105
Type: Direct-Access ANSI SCSI revision: 05
SCSI device sda: 287277984 512-byte hdwr sectors (147086 MB)
sda: Write Protect is off
SCSI device sda: drive cache: write back
SCSI device sda: 287277984 512-byte hdwr sectors (147086 MB)
sda: Write Protect is off
SCSI device sda: drive cache: write back
sda: sda1 sda2
sd 0:0:0:0: Attached scsi disk sda
mptsas: ioc0: attaching ssp device: fw_channel 0, fw_id 2, phy 1, sas_addr
0x500000e01f602f42
Vendor: FUJITSU Model: MBB2147RC Rev: 0105
Type: Direct-Access ANSI SCSI revision: 05
SCSI device sdb: 287277984 512-byte hdwr sectors (147086 MB)
sdb: Write Protect is off
SCSI device sdb: drive cache: write back
SCSI device sdb: 287277984 512-byte hdwr sectors (147086 MB)
sdb: Write Protect is off
SCSI device sdb: drive cache: write back
sdb: sdb1 sdb2
sd 0:0:1:0: Attached scsi disk sdb
mptsas: ioc0: attaching ssp device: fw_channel 0, fw_id 3, phy 2, sas_addr
0x50000392b8028856
Vendor: TOSHIBA Model: MBF2300RC Rev: 0107
Type: Direct-Access ANSI SCSI revision: 05
SCSI device sdc: 585937500 512-byte hdwr sectors (300000 MB)
sdc: Write Protect is off
SCSI device sdc: drive cache: write back
SCSI device sdc: 585937500 512-byte hdwr sectors (300000 MB)
sdc: Write Protect is off
SCSI device sdc: drive cache: write back
sdc: sdc1
sd 0:0:2:0: Attached scsi disk sdc
mptsas: ioc0: attaching ssp device: fw_channel 0, fw_id 4, phy 3, sas_addr
0x50000392b80236da
Vendor: TOSHIBA Model: MBF2300RC Rev: 0107
Type: Direct-Access ANSI SCSI revision: 05
SCSI device sdd: 585937500 512-byte hdwr sectors (300000 MB)
sdd: Write Protect is off
SCSI device sdd: drive cache: write back
SCSI device sdd: 585937500 512-byte hdwr sectors (300000 MB)
sdd: Write Protect is off
SCSI device sdd: drive cache: write back
sdd: sdd1
sd 0:0:3:0: Attached scsi disk sdd
Loading shpchp.ko module
shpchp: Standard Hot Plug PCI Controller Driver version: 0.4
Loading libata.ko module
Loading ata_piix.ko module
GSI 23 sharing vector 0x41 and IRQ 23
ACPI: PCI Interrupt 0000:00:1f.2[B] -> GSI 18 (level, low) -> IRQ 23
ata_piix 0000:00:1f.2: MAP [ P0 P2 P1 P3 ]
scsi1 : ata_piix
scsi2 : ata_piix
ata1: SATA max UDMA/133 cmd 0x3138 ctl 0x314c bmdma 0x3110 irq 23
ata2: SATA max UDMA/133 cmd 0x3130 ctl 0x3148 bmdma 0x3118 irq 23
ata1: SATA link down (SStatus 0 SControl 300)
ata2: SATA link down (SStatus 0 SControl 300)
GSI 24 sharing vector 0x49 and IRQ 24
ACPI: PCI Interrupt 0000:00:1f.5[D] -> GSI 21 (level, low) -> IRQ 24
ata_piix 0000:00:1f.5: MAP [ P0 -- P1 -- ]
scsi3 : ata_piix
scsi4 : ata_piix
ata3: SATA max UDMA/133 cmd 0x3128 ctl 0x3144 bmdma 0x30f0 irq 24
ata4: SATA max UDMA/133 cmd 0x3120 ctl 0x3140 bmdma 0x30f8 irq 24
ata3: SATA link down (SStatus 4 SControl 300)
ata4: SATA link down (SStatus 0 SControl 300)
Loading usb-storage.ko module
Initializing USB Mass Storage driver...
scsi5 : SCSI emulation for USB Mass Storage devices
usbcore: registered new driver usb-storage
Waiting for driver initialization.
USB Mass Storage support registered.
Vendor: TEAC Model: DV-28S-V Rev: 1.0B
Type: CD-ROM ANSI SCSI revision: 00
Loading dm-mod.ko module
device-mapper: uevent: version 1.0.3
device-mapper: ioctl: 4.11.5-ioctl (2007-12-12) initialised: dm-devel at
redhat.com
Loading dm-log.ko module
Loading dm-mirror.ko module
Loading dm-zero.ko module
Loading dm-snapshot.ko module
Loading dm-mem-cache.ko module
Loading dm-region_hash.ko module
Loading dm-message.ko module
Loading dm-raid45.ko module
device-mapper: dm-raid45: initialized v0.2594l
Waiting for driver initialization.
Scanning and configuring dmraid supported devices
md: Autodetecting RAID arrays.
md: autorun ...
md: considering sdd1 ...
md: adding sdd1 ...
md: adding sdc1 ...
md: sdb2 has different UUID to sdd1
md: sdb1 has different UUID to sdd1
md: sda2 has different UUID to sdd1
md: sda1 has different UUID to sdd1
md: created md2
md: bind<sdc1>
md: bind<sdd1>
md: running: <sdd1><sdc1>
raid1: raid set md2 active with 2 out of 2 mirrors
md: considering sdb2 ...
md: adding sdb2 ...
md: sdb1 has different UUID to sdb2
md: adding sda2 ...
md: sda1 has different UUID to sdb2
md: created md1
md: bind<sda2>
md: bind<sdb2>
md: running: <sdb2><sda2>
raid1: raid set md1 active with 2 out of 2 mirrors
md: considering sdb1 ...
md: adding sdb1 ...
md: adding sda1 ...
md: created md0
md: bind<sda1>
md: bind<sdb1>
md: running: <sdb1><sda1>
raid1: raid set md0 active with 2 out of 2 mirrors
md: ... autorun DONE.
md: Autodetecting RAID arrays.
md: autorun ...
md: ... autorun DONE.
------------------------ and come the problem -----------------------
Scanning logical volumes
Reading all physical volumes. This may take a while...
Couldn't find device with uuid qeDW2q-nq5b-Yh5U-5sKY-7Rkd-1UXh-LAxL8j.
Found volume group "vg0" using metadata type lvm2
Activating logical volumes
Couldn't find device with uuid qeDW2q-nq5b-Yh5U-5sKY-7Rkd-1UXh-LAxL8j.
8 logical volume(s) in volume group "vg0" now active
Creating root device.
Mounting root filesystem.
kjournald starting. Commit interval 5 seconds
EXT3-fs: mounted filesystem with ordered data mode.
Setting up other filesystems.
Setting up new root fs
no fstab.sys, mounting internal defaults
Switching to new root and running init.
unmounting old /dev
unmounting old /proc
unmounting old /sys
SELinux: Disabled at runtime.
type=1404 audit(1298476404.427:2): selinux=0 auid=4294967295 ses=4294967295
INIT: version 2.86 booting
Welcome to CentOS release 5.5 (Final)
Press 'I' to enter interactive startup.
Setting clock (utc): Wed Feb 23 16:53:25 CET 2011 [ OK ]
Starting udev: [ OK ]
Loading default keymap (us): [ OK ]
Setting hostname xen13.local: [ OK ]
Setting up Logical Volume Management: 8 logical volume(s) in volume group
"vg0" now
active
[...]
I have no idea were the problem is. I could paste vgck result here
but its long, is it nessesary?
I have yet another machine with the same setup and the same new
Toshiba disks and it shows even some more errors:
Scanning logical volumes
Reading all physical volumes. This may take a while...
Couldn't find device with uuid fNeN0H-zeD1-ems8-TfqW-2pyC-ZBdR-3ni250.
Found volume group "vg0" using metadata type lvm2
Activating logical volumes
Couldn't find device with uuid fNeN0H-zeD1-ems8-TfqW-2pyC-ZBdR-3ni250.
Refusing activation of partial LV d1dc. Use --partial to override.
6 logical volume(s) in volume group "vg0" now active
[...]
but then later:
[...]
Setting up Logical Volume Management: 7 logical volume(s) in volume group
"vg0" now
active
and after boot all 7 logical volumes work fine.
Where could be the problem with these errors?
--
Tomasz
I forgot to mention that after creation of md2 I've added a new line to /etc/mdadm.conf: # cat /etc/mdadm.conf DEVICE partitions MAILADDR root ARRAY /dev/md0 level=raid1 num-devices=2 UUID=91518bd3:79953138:f203f159:f2795fde ARRAY /dev/md1 level=raid1 num-devices=2 UUID=5b6722ca:b325344b:4ab40fc6:c3f1d6b4 ARRAY /dev/md2 level=raid1 num-devices=2 UUID=45e2f59b:1be94f42:5d55c385:da4faf78 # mdadm --examine --scan ARRAY /dev/md0 level=raid1 num-devices=2 UUID=91518bd3:79953138:f203f159:f2795fde ARRAY /dev/md1 level=raid1 num-devices=2 UUID=5b6722ca:b325344b:4ab40fc6:c3f1d6b4 ARRAY /dev/md2 level=raid1 num-devices=2 UUID=45e2f59b:1be94f42:5d55c385:da4faf78 -- Tomasz
Tomasz Nowak
2011-Feb-24 13:31 UTC
[CentOS] [SOLVED] Re: LVM problem after adding new (md) PV
I solved this issue thanks to help on linux-lvm list from a guy from redhat. Initrd image had to be recreated so that new raid devices could be seen before root is mounted. Solution: # mkinitrd /boot/initrd-$(uname -r).img $(uname -r) -- Tomasz
Reasonably Related Threads
- BTRFS thinks device is busy [kernel 3.5.3]
- SCSI/SAS error message by mpt2sas.ko
- anaconda, kickstart, lvm over raid, logvol --grow, centos7 mystery
- can not lvcreate after lvm snapshot and "xm mem-set" and lvremove
- 3.5.0-rc6: btrfs and LVM snapshots -> wrong devicename in /proc/mounts