PCextreme B.V. - Wido den Hollander
2009-Mar-12 17:47 UTC
[Xen-users] Problem with network for new domU''s (vif vif-247-0: 28 mapping shared-frames)
Hello, Today i ran into a problem with new domU''s on my Xen 3.2 system. On this server i have 78 domU''s running, but when creating domU #79 the network for this domU fails. I allready submitted a bugreport with Ubuntu and please forgive me for just posting a link: https://bugs.launchpad.net/ubuntu/+source/xen-meta/+bug/341846 Since this machine is already in production i can''t simply reboot or build a new kernel. Could somebody give me a clue where to find the solution? - Met vriendelijke groet, Wido den Hollander Hoofd Systeembeheer / CSO Telefoon Support Nederland: 0900 9633 (45 cpm) Telefoon Support België: 0900 70312 (45 cpm) Telefoon Direct: (+31) (0)20 50 60 104 Fax: +31 (0)20 50 60 111 E-mail: support@pcextreme.nl Website: http://www.pcextreme.nl Kennisbank: http://support.pcextreme.nl/ Netwerkstatus: http://nmc.pcextreme.nl _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
PCextreme B.V. - Wido den Hollander
2009-Mar-17 13:59 UTC
[Xen-users] Re: Problem with network for new domU''s (Grant table problem/limit?) (vif vif-247-0: 28 mapping shared-frames)
Hello,
The last few days i did some further research.
It seems i have run into some limit for the Xen grant tables (although,
that was what people told me).
I tried to replicate the error so i set up a new server. Since the
machine wich is having the troubles is a pretty heavy machine (64GB ram)
i could not exactly replicate it.
So i set up a new server with 6GB ram, but with exactly the same
versions of Xen and i was NOT able to reproduce the error.
I created 115 domU''s (with 48MB of memory) and they all had network
connectivity.
So i started reading more about the Grant tables and found out that also
vbd''s are mapped in these tables.
For the test i used this config for the domU:
*****************************************************************
kernel = ''/boot/vmlinuz-2.6.24-22-xen''
ramdisk = ''/boot/initrd.img-2.6.24-22-xen''
memory = 512
vcpus = 1
root = ''/dev/xvda1 ro''
disk = [
''phy:/dev/xen-domains-root/vps78-root,xvda1,w''
''phy:/dev/xen-domains-swap/vps78-swap,xvda2,w''
]
name = ''vps78''
vif = [ ''mac=00:16:3e:12:de:e4,vifname=vps78,bridge=vlanbr710''
]
on_poweroff = ''destroy''
on_reboot = ''restart''
on_crash = ''restart''
extra=''xencons=tty1 rootflags=quota''
*****************************************************************
This did NOT work, it resulted in the following message:
vif vif-279-0: 28 mapping shared-frames 768/769 port 8
After searching in drivers/xen/interface.c i found out that it was bad,
it is a error with code 28.
As i ran out of options i remove the xvda2 line from my domU''s config
and i''ll be damned, the network connectivity was back!
I created a new domU (#80) and this had the same troubles, no network.
But when i removed the xvda2 line (for the swap) the network was OK.
So for the last test i added the swap after it was created:
root@vps-pool-01:~# xm block-attach
vps78 /dev/xen-domains-swap/vps78-swap xvda2 w
root@vps-pool-01:~# dmesg
[2540234.810635] blkback: ring-ref 299, event-channel 8, protocol 1
(x86_64-abi)
[2540234.810813] vbd vbd-280-51714: 28 mapping ring-ref 299 port 8
root@vps-pool-01:~#
As you can see i have the same kind of error, mapping the vbd fails.
Today i ordered a new server wich is exactly the same as the machine
wich is running into this limit. So i will be able to reproduce a exact
setup and hopefully reproduce the errors as well.
I hope to receive this server within one week.
But in the meantime i hope their is somebody who has any clues of what
this could be.
root@vps-pool-01:~# xm info
host : vps-pool-01.xen.pcextreme.nl
release : 2.6.24-22-xen
version : #1 SMP Mon Nov 24 21:35:54 UTC 2008
machine : x86_64
nr_cpus : 8
nr_nodes : 1
cores_per_socket : 4
threads_per_core : 1
cpu_mhz : 2500
hw_caps :
bfebfbff:20100800:00000000:00000140:040ce3bd:00000000:00000001
total_memory : 64766
free_memory : 26797
max_free_memory : 26797
max_para_memory : 26793
max_hvm_memory : 26722
node_to_cpu : node0:0-7
xen_major : 3
xen_minor : 2
xen_extra : .1-rc1-pre
xen_caps : xen-3.0-x86_64 xen-3.0-x86_32p hvm-3.0-x86_32
hvm-3.0-x86_32p hvm-3.0-x86_64
xen_scheduler : credit
xen_pagesize : 4096
platform_params : virt_start=0xffff800000000000
xen_changeset : unavailable
cc_compiler : gcc version 4.2.3 (Ubuntu 4.2.3-2ubuntu7)
cc_compile_by : buildd
cc_compile_domain : buildd
cc_compile_date : Fri Apr 11 01:13:51 UTC 2008
xend_config_format : 4
root@vps-pool-01:~#
Thank you in advance!
-
Met vriendelijke groet,
Wido den Hollander
Hoofd Systeembeheer / CSO
Telefoon Support Nederland: 0900 9633 (45 cpm)
Telefoon Support België: 0900 70312 (45 cpm)
Telefoon Direct: (+31) (0)20 50 60 104
Fax: +31 (0)20 50 60 111
E-mail: support@pcextreme.nl
Website: http://www.pcextreme.nl
Kennisbank: http://support.pcextreme.nl/
Netwerkstatus: http://nmc.pcextreme.nl
On Thu, 2009-03-12 at 18:47 +0100, PCextreme B.V. - Wido den Hollander
wrote:> Hello,
>
> Today i ran into a problem with new domU''s on my Xen 3.2 system.
>
> On this server i have 78 domU''s running, but when creating domU
#79 the
> network for this domU fails.
>
> I allready submitted a bugreport with Ubuntu and please forgive me for
> just posting a link:
> https://bugs.launchpad.net/ubuntu/+source/xen-meta/+bug/341846
>
> Since this machine is already in production i can''t simply reboot
or
> build a new kernel.
>
> Could somebody give me a clue where to find the solution?
>
> -
> Met vriendelijke groet,
>
> Wido den Hollander
> Hoofd Systeembeheer / CSO
> Telefoon Support Nederland: 0900 9633 (45 cpm)
> Telefoon Support België: 0900 70312 (45 cpm)
> Telefoon Direct: (+31) (0)20 50 60 104
> Fax: +31 (0)20 50 60 111
> E-mail: support@pcextreme.nl
> Website: http://www.pcextreme.nl
> Kennisbank: http://support.pcextreme.nl/
> Netwerkstatus: http://nmc.pcextreme.nl
>
_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users
PCextreme B.V. - Wido den Hollander
2009-Mar-18 21:19 UTC
[Xen-users] Re: Problem with network for new domU''s (Grant table problem/limit?) (vif vif-247-0: 28 mapping shared-frames)
Hello, I have been able to reproduce the bug and found out what goes wrong. After creating 79 VM''s with all 2 vbd (root disk and swap) and one vif i ran into the problem again. One error caught my attention (and is only mentioned once!): [ 628.040958] No available IRQ to bind to: increase NR_DYNIRQS. After some searching i found: * https://bugzilla.redhat.com/show_bug.cgi?id=442736 * http://www.nabble.com/Unable-to-start-more-than-103-VMs-td11420853.html I downloaded the source of "linux-image-2.6.24-23-xen" and modified "debian/binary-custom.d/xen/patchset/001-xen-base.patch" on line 87148 Here you should change NR_DYNIRQS from 256 to 1024. When you have a VM with 3 "devices" (in my case 2x vbd, 1x vif) you reserve 3 Dynamic IRQ''s per domU. 79 * 3 = 237 IRQ''s The default limit of 256 is reached since the dom0 also reserves some IRQ''s. You can find out how many IRQ''s you are using with: grep Dynamic-irq /proc/interrupts |wc -l I have now been able to address over 256 IRQ''s and start 90 VM''s without any troubles. I think Ubuntu should raise the limit from 256 to 512 or 1024 OR make it a option wich can be passed to the kernel on boot (like suggested in the RedHat bugreport) I hope this helps more sysadmins in the future! - Met vriendelijke groet, Wido den Hollander Hoofd Systeembeheer / CSO Telefoon Support Nederland: 0900 9633 (45 cpm) Telefoon Support België: 0900 70312 (45 cpm) Telefoon Direct: (+31) (0)20 50 60 104 Fax: +31 (0)20 50 60 111 E-mail: support@pcextreme.nl Website: http://www.pcextreme.nl Kennisbank: http://support.pcextreme.nl/ Netwerkstatus: http://nmc.pcextreme.nl On Tue, 2009-03-17 at 14:59 +0100, PCextreme B.V. - Wido den Hollander wrote:> Hello, > > The last few days i did some further research. > > It seems i have run into some limit for the Xen grant tables (although, > that was what people told me). > > I tried to replicate the error so i set up a new server. Since the > machine wich is having the troubles is a pretty heavy machine (64GB ram) > i could not exactly replicate it. > > So i set up a new server with 6GB ram, but with exactly the same > versions of Xen and i was NOT able to reproduce the error. > > I created 115 domU''s (with 48MB of memory) and they all had network > connectivity. > > So i started reading more about the Grant tables and found out that also > vbd''s are mapped in these tables. > > For the test i used this config for the domU: > > ***************************************************************** > kernel = ''/boot/vmlinuz-2.6.24-22-xen'' > ramdisk = ''/boot/initrd.img-2.6.24-22-xen'' > memory = 512 > vcpus = 1 > > root = ''/dev/xvda1 ro'' > > disk = [ > ''phy:/dev/xen-domains-root/vps78-root,xvda1,w'' > ''phy:/dev/xen-domains-swap/vps78-swap,xvda2,w'' > ] > > name = ''vps78'' > > vif = [ ''mac=00:16:3e:12:de:e4,vifname=vps78,bridge=vlanbr710'' ] > > on_poweroff = ''destroy'' > on_reboot = ''restart'' > on_crash = ''restart'' > > extra=''xencons=tty1 rootflags=quota'' > ***************************************************************** > > This did NOT work, it resulted in the following message: > > vif vif-279-0: 28 mapping shared-frames 768/769 port 8 > > After searching in drivers/xen/interface.c i found out that it was bad, > it is a error with code 28. > > As i ran out of options i remove the xvda2 line from my domU''s config > and i''ll be damned, the network connectivity was back! > > I created a new domU (#80) and this had the same troubles, no network. > But when i removed the xvda2 line (for the swap) the network was OK. > > So for the last test i added the swap after it was created: > > root@vps-pool-01:~# xm block-attach > vps78 /dev/xen-domains-swap/vps78-swap xvda2 w > root@vps-pool-01:~# dmesg > [2540234.810635] blkback: ring-ref 299, event-channel 8, protocol 1 > (x86_64-abi) > [2540234.810813] vbd vbd-280-51714: 28 mapping ring-ref 299 port 8 > root@vps-pool-01:~# > > As you can see i have the same kind of error, mapping the vbd fails. > > Today i ordered a new server wich is exactly the same as the machine > wich is running into this limit. So i will be able to reproduce a exact > setup and hopefully reproduce the errors as well. > > I hope to receive this server within one week. > > But in the meantime i hope their is somebody who has any clues of what > this could be. > > root@vps-pool-01:~# xm info > host : vps-pool-01.xen.pcextreme.nl > release : 2.6.24-22-xen > version : #1 SMP Mon Nov 24 21:35:54 UTC 2008 > machine : x86_64 > nr_cpus : 8 > nr_nodes : 1 > cores_per_socket : 4 > threads_per_core : 1 > cpu_mhz : 2500 > hw_caps : > bfebfbff:20100800:00000000:00000140:040ce3bd:00000000:00000001 > total_memory : 64766 > free_memory : 26797 > max_free_memory : 26797 > max_para_memory : 26793 > max_hvm_memory : 26722 > node_to_cpu : node0:0-7 > xen_major : 3 > xen_minor : 2 > xen_extra : .1-rc1-pre > xen_caps : xen-3.0-x86_64 xen-3.0-x86_32p hvm-3.0-x86_32 > hvm-3.0-x86_32p hvm-3.0-x86_64 > xen_scheduler : credit > xen_pagesize : 4096 > platform_params : virt_start=0xffff800000000000 > xen_changeset : unavailable > cc_compiler : gcc version 4.2.3 (Ubuntu 4.2.3-2ubuntu7) > cc_compile_by : buildd > cc_compile_domain : buildd > cc_compile_date : Fri Apr 11 01:13:51 UTC 2008 > xend_config_format : 4 > root@vps-pool-01:~# > > Thank you in advance! > - > Met vriendelijke groet, > > Wido den Hollander > Hoofd Systeembeheer / CSO > Telefoon Support Nederland: 0900 9633 (45 cpm) > Telefoon Support België: 0900 70312 (45 cpm) > Telefoon Direct: (+31) (0)20 50 60 104 > Fax: +31 (0)20 50 60 111 > E-mail: support@pcextreme.nl > Website: http://www.pcextreme.nl > Kennisbank: http://support.pcextreme.nl/ > Netwerkstatus: http://nmc.pcextreme.nl > > > On Thu, 2009-03-12 at 18:47 +0100, PCextreme B.V. - Wido den Hollander > wrote: > > Hello, > > > > Today i ran into a problem with new domU''s on my Xen 3.2 system. > > > > On this server i have 78 domU''s running, but when creating domU #79 the > > network for this domU fails. > > > > I allready submitted a bugreport with Ubuntu and please forgive me for > > just posting a link: > > https://bugs.launchpad.net/ubuntu/+source/xen-meta/+bug/341846 > > > > Since this machine is already in production i can''t simply reboot or > > build a new kernel. > > > > Could somebody give me a clue where to find the solution? > > > > - > > Met vriendelijke groet, > > > > Wido den Hollander > > Hoofd Systeembeheer / CSO > > Telefoon Support Nederland: 0900 9633 (45 cpm) > > Telefoon Support België: 0900 70312 (45 cpm) > > Telefoon Direct: (+31) (0)20 50 60 104 > > Fax: +31 (0)20 50 60 111 > > E-mail: support@pcextreme.nl > > Website: http://www.pcextreme.nl > > Kennisbank: http://support.pcextreme.nl/ > > Netwerkstatus: http://nmc.pcextreme.nl > >_______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users