Madison Kelly
2009-Nov-07 05:26 UTC
[Xen-users] All VMs are "blocked", terrible performance
Hi all, I''ve built up a handful of VMs on my 2-node cluster and all are showing as being in a blocked state. The performance is terrible, too. All VMs are currently on one node (another problem, maybe related? is keeping me from migrating any). My Setup: (each node) 2x Quad Core AMD Opteron 2347 HE 32GB RAM (16GB/CPU) Nodes have a DRBD partition running cluster-aware LVM for all domU VMs. Each VM has it''s own logical volume. The DRBD has a dedicate gigabit link and DRBD is using ''Protocol C'', as required. LVM is set to use ''locking_type=3''. Here''s what I see: # xm list Name ID Mem(MiB) VCPUs State Time(s) Domain-0 0 32544 8 r----- 22400.8 auth01 10 1023 1 -b---- 3659.7 dev01 22 8191 1 -b---- 830.2 fw01 11 1023 1 -b---- 1046.9 res01 23 2047 1 -b---- 812.1 sql01 24 16383 1 -b---- 817.0 web01 20 2047 1 -b---- 1156.3 web02 21 1023 1 -b---- 931.1 When I ran that, all VMs were running yum update (all but two were fresh installs). Any idea what''s causing this and/or why my performance is so bad? Each VM is taking minutes to install each updated RPM. In case it''s related, when I tried to do a live migration of a VM from one node to the other, I got an error saying: VM Disk dom0_vg0/web02.disk.xm not found on the destination node vsh03 However, ''/proc/drbd'' shows both nodes are sync''ed and in Primary/Primary mode. Also, both nodes have identical output from ''lvdisplay'' (all LVs are ''active'') and all LVs were created during the provision on the first node. Kinda stuck here, so any input will be greatly appreciated! Let me know if I can post anything else useful. Madi PS: dom0 and all domU are running CentOS 5.4 x86_64 _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
Stephen Ross
2009-Nov-07 10:14 UTC
Re: [Xen-users] All VMs are "blocked", terrible performance
With regards to your migration problem, it might be worth temporarily turning LVM locking off completely to see if this helps? No idea why performance is so poor though. I''d guess it''s related to the DRBD stuff somehow? How quick is the disc set these VM''s run off? On Sat, Nov 7, 2009 at 5:26 AM, Madison Kelly <linux@alteeve.com> wrote:> Hi all, > > I''ve built up a handful of VMs on my 2-node cluster and all are showing as > being in a blocked state. The performance is terrible, too. All VMs are > currently on one node (another problem, maybe related? is keeping me from > migrating any). > > My Setup: > (each node) > 2x Quad Core AMD Opteron 2347 HE > 32GB RAM (16GB/CPU) > > Nodes have a DRBD partition running cluster-aware LVM for all domU VMs. > Each VM has it''s own logical volume. The DRBD has a dedicate gigabit link > and DRBD is using ''Protocol C'', as required. LVM is set to use > ''locking_type=3''. > > Here''s what I see: > > # xm list > Name ID Mem(MiB) VCPUs State Time(s) > Domain-0 0 32544 8 r----- 22400.8 > auth01 10 1023 1 -b---- 3659.7 > dev01 22 8191 1 -b---- 830.2 > fw01 11 1023 1 -b---- 1046.9 > res01 23 2047 1 -b---- 812.1 > sql01 24 16383 1 -b---- 817.0 > web01 20 2047 1 -b---- 1156.3 > web02 21 1023 1 -b---- 931.1 > > When I ran that, all VMs were running yum update (all but two were fresh > installs). > > Any idea what''s causing this and/or why my performance is so bad? Each VM > is taking minutes to install each updated RPM. > > In case it''s related, when I tried to do a live migration of a VM from one > node to the other, I got an error saying: > > VM Disk dom0_vg0/web02.disk.xm not found on the destination node vsh03 > > However, ''/proc/drbd'' shows both nodes are sync''ed and in Primary/Primary > mode. Also, both nodes have identical output from ''lvdisplay'' (all LVs are > ''active'') and all LVs were created during the provision on the first node. > > Kinda stuck here, so any input will be greatly appreciated! Let me know if > I can post anything else useful. > > Madi > > PS: dom0 and all domU are running CentOS 5.4 x86_64 > > _______________________________________________ > Xen-users mailing list > Xen-users@lists.xensource.com > http://lists.xensource.com/xen-users >_______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users