Unable to start more than 103 VMs. Any ideas? Running Xen Unstable (changeset 15445) on a Unisys ES7000/one on SLES10 Release, 64gb memory, 16 cpu, dom0_mem=512M, xenheap_megabytes=64. VMs are SLES10 para-virtual guests, each on their own lun on san storage. We can start 103 VMs successfully. When starting the 104th, the VM times out waiting for its disk to appear: XENBUS: Timeout connecting to device: device/vbd/2048 (state 6) XENBUS: Timeout connecting to device: device/vif/0 (state 6) XENBUS: Device with no driver: device/console/0 BIOS EDD facility v0.16 2004-Jun-25, 0 devices found EDD information not available. Freeing unused kernel memory: 172k freed Starting udevd Creating devices Loading jbd jbd: no version for "struct_module" found: kernel tainted. Loading ext3 Loading reiserfs Loading jbd Loading ext3 Waiting for device /dev/sda2 to appear: ..............................not found -- exiting to /bin/sh sh: no job control in this shell $ Noteworthy: - domain-0 can see the lun of the failing VM correctly, and there are no errors if we run kpartx and mount the VMs root partition. - we can reproduce the problem on xen-3.1 as well as xen-unstable. - problem still occurs if we split the VMs across 2 bridges - the 104th still fails. - problem also occurs if we start the VMs in any order - once 103 are started, the next one fails. - if we comment out the "vif = " statement in the VM config files, we can start more than 103 brian carb unisys corporation - malvern, pa _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Christian pointed out that this is probably running into the default dynamic irq limit in dom0, which is 256. You should try raising NR_DYNIRQS in include/asm/mach-xen/irq_vectors.h. If you are hitting this limit you should get a warning in dom0¹s dmesg (or /var/log/messages) unless the loglevel is set too low. K. On 3/7/07 22:21, "Carb, Brian A" <Brian.Carb@unisys.com> wrote:> Unable to start more than 103 VMs. Any ideas? > > Running Xen Unstable (changeset 15445) on a Unisys ES7000/one on SLES10 > Release, 64gb memory, 16 cpu, dom0_mem=512M, xenheap_megabytes=64. VMs are > SLES10 para-virtual guests, each on their own lun on san storage. > > We can start 103 VMs successfully. When starting the 104th, the VM times out > waiting for its disk to appear: > XENBUS: Timeout connecting to device: device/vbd/2048 (state 6) > XENBUS: Timeout connecting to device: device/vif/0 (state 6) > XENBUS: Device with no driver: device/console/0 > BIOS EDD facility v0.16 2004-Jun-25, 0 devices found > EDD information not available. > Freeing unused kernel memory: 172k freed > Starting udevd > Creating devices > Loading jbd > jbd: no version for "struct_module" found: kernel tainted. > Loading ext3 > Loading reiserfs > Loading jbd > Loading ext3 > Waiting for device /dev/sda2 to appear: ..............................not > found -- exiting to /bin/sh > sh: no job control in this shell > $ > > Noteworthy: > - domain-0 can see the lun of the failing VM correctly, and there are no > errors if we run kpartx and mount the VMs root partition. > - we can reproduce the problem on xen-3.1 as well as xen-unstable. > - problem still occurs if we split the VMs across 2 bridges - the 104th > still fails. > - problem also occurs if we start the VMs in any order - once 103 are > started, the next one fails. > - if we comment out the "vif = " statement in the VM config files, we can > start more than 103 > brian carb > unisys corporation - malvern, pa > > > _______________________________________________ > Xen-devel mailing list > Xen-devel@lists.xensource.com > http://lists.xensource.com/xen-devel_______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Thanks Keir. When we tried this scaling in December, we were able to start 121 VMs, so I think I was not suspecting the IRQ limit yet at 103. No message was output to the serial console or the xend.log - but you are right, it''s there right under my nose in dmesg: eth1: port 105(vif104.0) entering learning state eth1: topology change detected, propagating eth1: port 105(vif104.0) entering forwarding state vif104.0: no IPv6 routers present blkback: ring-ref 8, event-channel 15, protocol 1 (x86_64-abi) No available IRQ to bind to: increase NR_DYNIRQS. brian carb unisys corporation - malvern, pa ________________________________ From: Keir Fraser [mailto:Keir.Fraser@cl.cam.ac.uk] Sent: Wednesday, July 04, 2007 4:35 AM To: Carb, Brian A; xen-devel@lists.xensource.com Subject: Re: [Xen-devel] Unable to start more than 103 VMs Christian pointed out that this is probably running into the default dynamic irq limit in dom0, which is 256. You should try raising NR_DYNIRQS in include/asm/mach-xen/irq_vectors.h. If you are hitting this limit you should get a warning in dom0''s dmesg (or /var/log/messages) unless the loglevel is set too low. K. On 3/7/07 22:21, "Carb, Brian A" <Brian.Carb@unisys.com> wrote: Unable to start more than 103 VMs. Any ideas? Running Xen Unstable (changeset 15445) on a Unisys ES7000/one on SLES10 Release, 64gb memory, 16 cpu, dom0_mem=512M, xenheap_megabytes=64. VMs are SLES10 para-virtual guests, each on their own lun on san storage. We can start 103 VMs successfully. When starting the 104th, the VM times out waiting for its disk to appear: XENBUS: Timeout connecting to device: device/vbd/2048 (state 6) XENBUS: Timeout connecting to device: device/vif/0 (state 6) XENBUS: Device with no driver: device/console/0 BIOS EDD facility v0.16 2004-Jun-25, 0 devices found EDD information not available. Freeing unused kernel memory: 172k freed Starting udevd Creating devices Loading jbd jbd: no version for "struct_module" found: kernel tainted. Loading ext3 Loading reiserfs Loading jbd Loading ext3 Waiting for device /dev/sda2 to appear: ..............................not found -- exiting to /bin/sh sh: no job control in this shell $ Noteworthy: - domain-0 can see the lun of the failing VM correctly, and there are no errors if we run kpartx and mount the VMs root partition. - we can reproduce the problem on xen-3.1 as well as xen-unstable. - problem still occurs if we split the VMs across 2 bridges - the 104th still fails. - problem also occurs if we start the VMs in any order - once 103 are started, the next one fails. - if we comment out the "vif = " statement in the VM config files, we can start more than 103 brian carb unisys corporation - malvern, pa ________________________________ _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel