Xen users - Jan 2011 - [Xen-API] Frequent I/O error of HTTP server and memory error on XenServer5.6 FP1

Hi,
     I am using XenServer5.6 FP1 GA release with openvswitch as network 
backend. When I create VM through xapi,  "apache.xmlrpc.XmlRpcException: 
I/O error" happens very frequently. At meanwhile my xencenter lost 
connection to hosts.
     Another thing is when I tried to kill VMs from xencenter, it 
complains "Internal error:Sys_error("Thread.create: Cannot allocate 
memory")", see attached picture.
     does anyone meet similar errors?
     Thank you.


_______________________________________________
xen-api mailing list
xen-api@lists.xensource.com
http://lists.xensource.com/mailman/listinfo/xen-api

Hi,

My best guess is that the "Thread.create: Cannot allocate memory"
means that there''s insufficient virtual address space in the process to
allocate a 10MiB thread stack. I suspect this is being caused by there being too
many threads (approx 300 is the max)

Since xapi currently uses a thread per connection model there are probably too
many open connections -- could you describe the software which is using the
apache xmlrpc client? Is it possible that someone is opening connections but not
closing them quickly enough?

If you install gdb and attach it to the running (child -- the big one) xapi
process it''ll tell you how many threads are present and allow you to
examine the stacks of each one (''thread apply all bt'') -- it
should be easy to spot where the threads are blocked (e.g. in a network recv or
perhaps on a pthread_mutex)

At some point in future we''ll have to switch to a different threading
model and make the connection handling more robust. In the meantime we have to
be a bit careful about connection cleanup :/

Thanks,
Dave
> -----Original Message-----
> From: xen-api-bounces@lists.xensource.com [mailto:xen-api-
> bounces@lists.xensource.com] On Behalf Of Frank
> Sent: 13 January 2011 22:14
> To: xen-api@lists.xensource.com; xen-users@lists.xensource.com
> Subject: [Xen-API] Frequent I/O error of HTTP server and memory error
> on XenServer5.6 FP1
> 
> Hi,
>      I am using XenServer5.6 FP1 GA release with openvswitch as network
> backend. When I create VM through xapi, 
"apache.xmlrpc.XmlRpcException:
> I/O error" happens very frequently. At meanwhile my xencenter lost
> connection to hosts.
>      Another thing is when I tried to kill VMs from xencenter, it
> complains "Internal error:Sys_error("Thread.create: Cannot
allocate
> memory")", see attached picture.
>      does anyone meet similar errors?
>      Thank you.
_______________________________________________
xen-api mailing list
xen-api@lists.xensource.com
http://lists.xensource.com/mailman/listinfo/xen-api

Thank you. Are there many changes from 5.6 to 5.6FP1? with same 
software, xenserver5.6 works fine.
anyway, I will try your method to see how many threads run when the 
problem happens

On 01/13/2011 02:41 PM, Dave Scott wrote:
> Hi,
>
> My best guess is that the "Thread.create: Cannot allocate memory"
means that there''s insufficient virtual address space in the process to
allocate a 10MiB thread stack. I suspect this is being caused by there being too
many threads (approx 300 is the max)
>
> Since xapi currently uses a thread per connection model there are probably
too many open connections -- could you describe the software which is using the
apache xmlrpc client? Is it possible that someone is opening connections but not
closing them quickly enough?
>
> If you install gdb and attach it to the running (child -- the big one) xapi
process it''ll tell you how many threads are present and allow you to
examine the stacks of each one (''thread apply all bt'') -- it
should be easy to spot where the threads are blocked (e.g. in a network recv or
perhaps on a pthread_mutex)
>
> At some point in future we''ll have to switch to a different
threading model and make the connection handling more robust. In the meantime we
have to be a bit careful about connection cleanup :/
>
> Thanks,
> Dave
>
>> -----Original Message-----
>> From: xen-api-bounces@lists.xensource.com [mailto:xen-api-
>> bounces@lists.xensource.com] On Behalf Of Frank
>> Sent: 13 January 2011 22:14
>> To: xen-api@lists.xensource.com; xen-users@lists.xensource.com
>> Subject: [Xen-API] Frequent I/O error of HTTP server and memory error
>> on XenServer5.6 FP1
>>
>> Hi,
>>       I am using XenServer5.6 FP1 GA release with openvswitch as
network
>> backend. When I create VM through xapi, 
"apache.xmlrpc.XmlRpcException:
>> I/O error" happens very frequently. At meanwhile my xencenter lost
>> connection to hosts.
>>       Another thing is when I tried to kill VMs from xencenter, it
>> complains "Internal error:Sys_error("Thread.create: Cannot
allocate
>> memory")", see attached picture.
>>       does anyone meet similar errors?
>>       Thank you.

_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users