Hi,
I am running SLES 10 SP2 with the following Xen packaged with it:
xen_major : 3
xen_minor : 2
xen_extra : .0_16718_16-0.3
After a YEAR of uptime, my server is acting up.
xm commands are not working. xm list stays there for ever.
xm top returns
unable to open xenstore
:Connection refuse
xm dmesg shows the following output:
(XEN) Freed 100kB init memory.
(XEN) instrlen.c:252:d303 Cannot read from address 77b36c4a (eip
77b36c4a, mode 8)
(XEN) instrlen.c:252:d303 Cannot read from address 77b36c4a (eip
77b36c4a, mode 8)
(XEN) instrlen.c:252:d303 Cannot read from address 77b36c4a (eip
77b36c4a, mode 8)
(XEN) instrlen.c:252:d303 Cannot read from address 77b36c4a (eip
77b36c4a, mode 8)
(XEN) instrlen.c:252:d303 Cannot read from address 77b36c4a (eip
77b36c4a, mode 8)
(XEN) instrlen.c:252:d303 Cannot read from address 77b36c4a (eip
77b36c4a, mode 8)
(XEN) instrlen.c:252:d303 Cannot read from address 77b36c4a (eip
77b36c4a, mode 8)
(XEN) instrlen.c:252:d303 Cannot read from address 77b36c4a (eip
77b36c4a, mode 8)
(XEN) instrlen.c:252:d303 Cannot read from address 77b36c4a (eip
77b36c4a, mode 8)
(XEN) instrlen.c:252:d303 Cannot read from address 77b36c4a (eip
77b36c4a, mode 8)
(XEN) sh error: sh_remove_write_access(): can''t remove write access to
mfn 4d60a8: guest has 1 special-use mappings of it
(XEN) domain_crash called from common.c:1850
(XEN) Domain 1129 (vcpu#0) crashed on cpu#2:
(XEN) ----[ Xen-3.2.0_16718_16-0.3 x86_64 debug=n Not tainted ]----
(XEN) CPU: 2
(XEN) RIP: 0008:[<000000008fb5918f>]
(XEN) RFLAGS: 0000000000010286 CONTEXT: hvm
(XEN) rax: 00000000ffacc008 rbx: 0000000000000000 rcx: 0000000083a08200
(XEN) rdx: 0000000000000000 rsi: 000000008187eca0 rdi: 0000000000000001
(XEN) rbp: 000000008a0c0cf8 rsp: 000000008a0c0cdc r8: 0000000000000000
(XEN) r9: 0000000000000000 r10: 0000000000000000 r11: 000000000000000
(XEN) r12: 0000000000000000 r13: 0000000000000000 r14: 0000000000000000
(XEN) r15: 0000000000000000 cr0: 000000008001003b cr4: 00000000000006b9
(XEN) cr3: 000000001f3450a0 cr2: 000000009746300f
(XEN) ds: 0023 es: 0023 fs: 0030 gs: 0000 ss: 0010 cs: 0008
(XEN) printk: 164 messages suppressed.
(XEN) grant_table.c:340:d0 Iomem mapping not permitted
ffffffffffffffff (domain 1202)
(XEN) grant_table.c:340:d0 Iomem mapping not permitted
ffffffffffffffff (domain 1202)
(XEN) grant_table.c:340:d0 Iomem mapping not permitted
ffffffffffffffff (domain 1202)
(XEN) grant_table.c:340:d0 Iomem mapping not permitted
ffffffffffffffff (domain 1202)
(XEN) grant_table.c:340:d0 Iomem mapping not permitted
ffffffffffffffff (domain 1202)
(XEN) grant_table.c:340:d0 Iomem mapping not permitted
ffffffffffffffff (domain 1311)
(XEN) grant_table.c:340:d0 Iomem mapping not permitted
ffffffffffffffff (domain 1311)
(XEN) grant_table.c:340:d0 Iomem mapping not permitted
ffffffffffffffff (domain 1311)
(XEN) instrlen.c:252:d1670 Cannot read from address 7fefd8215f4 (eip
7fefd8215f4, mode 8)
(XEN) instrlen.c:252:d1670 Cannot read from address 7fefd8215f4 (eip
7fefd8215f4, mode 8)
(XEN) instrlen.c:252:d1670 Cannot read from address 7fefd8215f4 (eip
7fefd8215f4, mode 8)
(XEN) instrlen.c:252:d1670 Cannot read from address 7fefd8215f4 (eip
7fefd8215f4, mode 8)
(XEN) instrlen.c:252:d1670 Cannot read from address 7fefd8215f4 (eip
7fefd8215f4, mode 8)
(XEN) instrlen.c:252:d1670 Cannot read from address 7fefd8215f4 (eip
7fefd8215f4, mode 8)
(XEN) instrlen.c:252:d1670 Cannot read from address 7fefd8215f4 (eip
7fefd8215f4, mode 8)
(XEN) instrlen.c:252:d1670 Cannot read from address 7fefd8215f4 (eip
7fefd8215f4, mode 8)
(XEN) instrlen.c:252:d1670 Cannot read from address 7fefd8215f4 (eip
7fefd8215f4, mode 8)
(XEN) instrlen.c:252:d1670 Cannot read from address 7fefd8215f4 (eip
7fefd8215f4, mode 8)
All my domains have crashed but I am not able to find info about the
nature of the crash other than for the above info.
The xend.log does not show errors. However, I see some errors in
xend-debug.log. For some of the domains I see the following:
Traceback (most recent call last):
File "/usr/lib64/python2.4/site-packages/xen/web/httpserver.py",
line 140, in process
resource = self.getResource()
File "/usr/lib64/python2.4/site-packages/xen/web/httpserver.py",
line 172, in getResource
return self.getServer().getResource(self)
File "/usr/lib64/python2.4/site-packages/xen/web/httpserver.py",
line 351, in getResource
return self.root.getRequestResource(req)
File "/usr/lib64/python2.4/site-packages/xen/web/resource.py", line
39, in getRequestResource
return findResource(self, req)
File "/usr/lib64/python2.4/site-packages/xen/web/resource.py", line
26, in findResource
next = resource.getPathResource(pathElement, request)
File "/usr/lib64/python2.4/site-packages/xen/web/resource.py", line
49, in getPathResource
val = self.getChild(path, request)
File "/usr/lib64/python2.4/site-packages/xen/web/SrvDir.py", line
71, in getChild
val = self.get(x)
File
"/usr/lib64/python2.4/site-packages/xen/xend/server/SrvDomainDir.py",
line 52, in get
return self.domain(x)
File
"/usr/lib64/python2.4/site-packages/xen/xend/server/SrvDomainDir.py",
line 44, in domain
dom = self.xd.domain_lookup(x)
File "/usr/lib64/python2.4/site-packages/xen/xend/XendDomain.py",
line 525, in domain_lookup
raise XendInvalidDomain(str(domid))
XendInvalidDomain: <Fault 3: ''549''>
Not sure why the domain was flagged as invalid. Doing a create on it
again went through.
There also a ton of these messages:
Traceback (most recent call last):
File "/usr/lib64/python2.4/SocketServer.py", line 463, in
process_request_thread
self.finish_request(request, client_address)
File "/usr/lib64/python2.4/SocketServer.py", line 254, in
finish_request
self.RequestHandlerClass(request, client_address, self)
File "/usr/lib64/python2.4/site-packages/xen/util/xmlrpclib2.py",
line 105, in <lambda>
(lambda x, y, z:
File "/usr/lib64/python2.4/site-packages/xen/util/xmlrpclib2.py",
line 65, in __init__
server)
File "/usr/lib64/python2.4/SocketServer.py", line 521, in __init__
self.handle()
File "/usr/lib64/python2.4/BaseHTTPServer.py", line 316, in handle
self.handle_one_request()
File "/usr/lib64/python2.4/BaseHTTPServer.py", line 310, in
handle_one_request
method()
File "/usr/lib64/python2.4/site-packages/xen/util/xmlrpclib2.py",
line 82, in do_POST
self.send_response(200)
File "/usr/lib64/python2.4/BaseHTTPServer.py", line 367, in
send_response
self.wfile.write("%s %d %s\r\n" %
File "/usr/lib64/python2.4/socket.py", line 256, in write
self.flush()
File "/usr/lib64/python2.4/socket.py", line 243, in flush
self._sock.sendall(buffer)
error: (32, ''Broken pipe'')
At this point I was forced to reboot the server to get it back and
running. I am not seeing any of the errors as of yet but I am afraid
it will show up after heavy activity of creating and destroying VMs.
Is this is a known bug that has been fixed in upstream Xen?
Thanks,
Aravindh
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel