Xen users - Jan 2010 - 101th domU fails to start with "SETVCPUCONTEXT failed"

Hello there,


We (a small hosting community) are running a steadily growing number of Xen 
domUs on a quad dualcore Xeon server with 64GB ram. We''ve got 100
running
domUs at the moment. Trying to create a new one results in this error:

Error: (1, ''Internal error'', ''launch_vm:
SETVCPUCONTEXT failed (rc=-1)\n'')

If I shut down another domain, I can create one again, but the limit
seems to be 100 domUs. 
Each domU has one network interface and one or two disk devices (raid-1
md device consisting of two iscsi luns).

Version of Xen is rather old: 3.2.1-amd64 (Debian). 

Can anyone give me hint as to what causes this behaviour? The host
machine should be able to cope with the amount of domUs easily. I''ve
got
a feeling I''m bumping into an obscure sourcecode parameter somewhere.


Thanks in advance,


Lennert


_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

On Sun, Jan 10, 2010 at 12:30:12AM +0100, storm66@club-internet.fr
wrote:
>Le samedi 09 janvier 2010 à 21:57 +0100, Lennert Van Alboom a écrit :
>> Hello there,
>> 
>> 
>> We (a small hosting community) are running a steadily growing number of
Xen
>> domUs on a quad dualcore Xeon server with 64GB ram. We''ve got
100 running
>> domUs at the moment. Trying to create a new one results in this error:
>> 
>> Error: (1, ''Internal error'', ''launch_vm:
SETVCPUCONTEXT failed (rc=-1)\n'')
>> 
>> If I shut down another domain, I can create one again, but the limit
>> seems to be 100 domUs. 
>> Each domU has one network interface and one or two disk devices (raid-1
>> md device consisting of two iscsi luns).
>> 
>> Version of Xen is rather old: 3.2.1-amd64 (Debian). 
>> 
>> Can anyone give me hint as to what causes this behaviour? The host
>> machine should be able to cope with the amount of domUs easily.
I''ve got
>> a feeling I''m bumping into an obscure sourcecode parameter
somewhere.
>> 
>> 
>> Thanks in advance,
>> 
>> 
>> Lennert
>> 
>Hello,
>
>In my thought Xen is heavily using loop devices, I had some problems
>some years ago with a system running XEN 3.0 with a kernel where the
>default loop max number was too low. 
>I didn''t get that problem with newer versions.
>
>
>Regards
>
>JPP
>
Thanks for your reply. We aren''t using loop devices though - only
physical ones (raid-1 md devices). 

I''ve checked
http://lxr.mstier.de/Xen/source/xen_3.2.1/tools/libxc/xc_dom_boot.c?v=3.2.1
for the location where it barfs - it looks to me that the failing
function is do_domctl (line 60).
This do_domctl is defined in
http://lxr.mstier.de/Xen/source/xen_3.2.1/xen/common/domctl.c?v=3.2.1
and it seems a bit over my head...

Anyone have experience with digging in the xen sourcecode that can help
me resolve this issue?


Kind regards,


Lennert

_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

On Sun, Jan 10, 2010 at 12:30:12AM +0100, storm66@club-internet.fr
wrote:
>Le samedi 09 janvier 2010 à 21:57 +0100, Lennert Van Alboom a écrit :
>> Hello there,
>> 
>> 
>> We (a small hosting community) are running a steadily growing number of
Xen
>> domUs on a quad dualcore Xeon server with 64GB ram. We''ve got
100 running
>> domUs at the moment. Trying to create a new one results in this error:
>> 
>> Error: (1, ''Internal error'', ''launch_vm:
SETVCPUCONTEXT failed (rc=-1)\n'')
>> 
>> If I shut down another domain, I can create one again, but the limit
>> seems to be 100 domUs. 
>> Each domU has one network interface and one or two disk devices (raid-1
>> md device consisting of two iscsi luns).
>> 
>> Version of Xen is rather old: 3.2.1-amd64 (Debian). 
>> 
>> Can anyone give me hint as to what causes this behaviour? The host
>> machine should be able to cope with the amount of domUs easily.
I''ve got
>> a feeling I''m bumping into an obscure sourcecode parameter
somewhere.
>> 
>> 
>> Thanks in advance,
>> 
>> 
>> Lennert
>> 
>Hello,
>
>In my thought Xen is heavily using loop devices, I had some problems
>some years ago with a system running XEN 3.0 with a kernel where the
>default loop max number was too low. 
>I didn''t get that problem with newer versions.
>
>
>Regards
>
>JPP
>
>
Thanks for your reply. We''re not using loopbacks though - only physical
devices (raid1 md). 

I''ve been tracking this problem through the xen source, and found this:
The launch_vm code is located in
http://lxr.mstier.de/Xen/source/xen_3.2.1/tools/libxc/xc_dom_boot.c?v=3.2.1#049
; the function that is causing the return code of -1 is do_domctl.

I''m unsure as to where do_domctl is really located - I''ve
found two
locations:
http://lxr.mstier.de/Xen/source/xen_3.2.1/tools/libxc/xc_private.h?v=3.2.1#099
and
http://lxr.mstier.de/Xen/source/xen_3.2.1/xen/common/domctl.c?v=3.2.1#179
. Neither give me much of a clue. 

In case of the first definition, the returncode is given by 
ret = do_xen_hypercall(xc_handle, &hypercall)
which leads me further to 
ioctl(xc_handle, cmd, data);
and after that I''m stumped. 

In the second case, the -1 comes *either* from 
IS_PRIV(current->domain)
failing, causing an EPERM error. This sounds unlikely. The other place
where a -1 could be given is 
ret = xsm_setvcpucontext(d);
which translates back to
xsm_call(setvcpucontext(d));
and there I''m stuck again. I''m totally not capable of
deciphering
xsm_ops and its meanings.


Does anyone with some insight in the innards of Xen have a clue on what
might be causing our issues?


Kind regards,


Lennert



_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

On Mon, Jan 11, 2010 at 8:08 PM, Lennert Van Alboom
<lennert.vanalboom@telenet.be> wrote:
>>> We (a small hosting community) are running a steadily growing
number of
>>> Xen domUs on a quad dualcore Xeon server with 64GB ram.
We''ve got 100
>>> running domUs at the moment. Trying to create a new one results in
this
>>> error:
>>> Version of Xen is rather old: 3.2.1-amd64 (Debian).
>>> Can anyone give me hint as to what causes this behaviour?
IMHO instead of digging into old source code, you''d better try latest
stable version. I was able to start over 300 PV domUs with 16 MBs
memory each for testing purposes, never got the error you mentioned.
It just became so slow at that point so I stopped testing. This was on
RHEL 5.4 with Xen 3.4.1 from Gitco.

-- 
Fajar

_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users