Hi
I''m using Xen on my AMD64 / Ubuntu 7.10 server which I recently
upgraded, and
my 2 VMs are decidedly flaky. They seem to be blocked for most of the time,
and networking doesn''t work properly. How can I debug this and get them
online reliably? Any tips gratefully received!
* I can log in via console or ssh, can ping the server''s NICs but
can''t ping
the other VM or beyond the machine. From the server I can ping the VMs and
outside.
* Firewall settings on the server are very relaxed at this point.
* The exceptions below about hda1 and hda2 don''t make sense to me,
since the
devices are mounted via /etc/fstab as root and swap.
* /var/log/xen/xend-debug.log reports a broken pipe
Thanks!
M
uname string:
Linux server.home.xx 2.6.19-4-generic-amd64 #2 SMP Sat Oct 6 01:35:53 UTC 2007
x86_64 GNU/Linux
[11:57:03] $ sudo xm list
Name ID Mem(MiB) VCPUs State Time(s)
Domain-0 0 782 1 r----- 487.2
slimserv 5 56 1 ------ 183.3
work 7 128 1 ------ 2.8
[11:57:06] $ sudo xm list
Name ID Mem(MiB) VCPUs State Time(s)
Domain-0 0 782 1 r----- 488.3
slimserv 5 56 1 -b---- 185.6
work 7 128 1 -b---- 7.2
[11:57:20] $ sudo xm list
Name ID Mem(MiB) VCPUs State Time(s)
Domain-0 0 782 1 r----- 521.0
slimserv 5 56 1 ------ 267.3
work 7 128 1 -b---- 7.3
[12:19:09] $
Here is an excerpt from the xend-debug.log
Traceback (most recent call last):
File "SocketServer.py", line 464, in process_request_thread
self.finish_request(request, client_address)
File "SocketServer.py", line 254, in finish_request
self.RequestHandlerClass(request, client_address, self)
File "SocketServer.py", line 522, in __init__
self.handle()
File "BaseHTTPServer.py", line 316, in handle
self.handle_one_request()
File "BaseHTTPServer.py", line 310, in handle_one_request
method()
File "/usr/lib/python2.5/site-packages/xen/util/xmlrpclib2.py", line
66, in
do_POST
self.send_response(200)
File "BaseHTTPServer.py", line 368, in send_response
(self.protocol_version, code, message))
File "socket.py", line 262, in write
self.flush()
File "socket.py", line 249, in flush
self._sock.sendall(buffer)
error: (32, ''Broken pipe'')
Here is a tail excerpt from /var/log/xen/xend.log:
[2007-10-30 11:57:01 xend.XendDomainInfo 16192] DEBUG (XendDomainInfo:190)
XendDomainInfo.create([''vm'', [''name'',
''work''], [''memory'',
''128''],
[''on_poweroff'', ''destroy''],
[''on_reboot'', ''restart''],
[''on_crash'', ''restart''],
[''vcpus'', 1], [''image'',
[''linux'',
[''kernel'',
''/boot/vmlinuz-2.6.19-4-generic-amd64''],
[''ramdisk'',
''/boot/initrd.img-2.6.19-4-generic-amd64''],
[''root'', ''/dev/hda1
ro'']]], [''device'', [''vbd'',
[''uname'',
''file:/var/xen/domains/work/disk.img''],
[''dev'', ''hda1''], [''mode'',
''w'']]], [''device'', [''vbd'',
[''uname'',
''file:/var/xen/domains/work/swap.img''],
[''dev'', ''hda2''],
[''mode'', ''w'']]],
[''device'', [''vif'', [''ip'',
''192.168.2.6''],
[''mac'', ''00:16:3e:b1:d7:91''],
[''vifname'', ''vif2.0'']]]])
[2007-10-30 11:57:01 xend.XendDomainInfo 16192] DEBUG (XendDomainInfo:296)
parseConfig: config is [''vm'', [''name'',
''work''], [''memory'',
''128''],
[''on_poweroff'', ''destroy''],
[''on_reboot'', ''restart''],
[''on_crash'', ''restart''],
[''vcpus'', 1], [''image'',
[''linux'',
[''kernel'',
''/boot/vmlinuz-2.6.19-4-generic-amd64''],
[''ramdisk'',
''/boot/initrd.img-2.6.19-4-generic-amd64''],
[''root'', ''/dev/hda1
ro'']]], [''device'', [''vbd'',
[''uname'',
''file:/var/xen/domains/work/disk.img''],
[''dev'', ''hda1''], [''mode'',
''w'']]], [''device'', [''vbd'',
[''uname'',
''file:/var/xen/domains/work/swap.img''],
[''dev'', ''hda2''],
[''mode'', ''w'']]],
[''device'', [''vif'', [''ip'',
''192.168.2.6''],
[''mac'', ''00:16:3e:b1:d7:91''],
[''vifname'', ''vif2.0'']]]]
[2007-10-30 11:57:01 xend.XendDomainInfo 16192] DEBUG (XendDomainInfo:395)
parseConfig: result is {''shadow_memory'': None,
''uuid'':
None, ''on_crash'': ''restart'',
''on_reboot'': ''restart'',
''localtime'':
None, ''image'': [''linux'',
[''kernel'',
''/boot/vmlinuz-2.6.19-4-generic-amd64''],
[''ramdisk'',
''/boot/initrd.img-2.6.19-4-generic-amd64''],
[''root'', ''/dev/hda1
ro'']], ''on_poweroff'': ''destroy'',
''bootloader_args'': None, ''cpus'':
None, ''name'': ''work'',
''backend'': [], ''vcpus'': 1,
''cpu_weight'':
None, ''features'': None, ''vcpu_avail'': None,
''memory'': 128, ''device'':
[(''vbd'',
[''vbd'', [''uname'',
''file:/var/xen/domains/work/disk.img''],
[''dev'', ''hda1''],
[''mode'', ''w'']]), (''vbd'',
[''vbd'',
[''uname'',
''file:/var/xen/domains/work/swap.img''],
[''dev'', ''hda2''],
[''mode'', ''w'']]), (''vif'',
[''vif'', [''ip'',
''192.168.2.6''],
[''mac'', ''00:16:3e:b1:d7:91''],
[''vifname'', ''vif2.0'']])],
''bootloader'':
None, ''cpu'': None, ''maxmem'': None}
[2007-10-30 11:57:01 xend.XendDomainInfo 16192] DEBUG (XendDomainInfo:1262)
XendDomainInfo.construct: None
[2007-10-30 11:57:01 xend.XendDomainInfo 16192] DEBUG (XendDomainInfo:1294)
XendDomainInfo.initDomain: 7 1.0
[2007-10-30 11:57:01 xend 16192] DEBUG (balloon:127) Balloon: 131680 KiB free;
need 131072; done.
[2007-10-30 11:57:01 xend 16192] INFO (image:136) buildDomain os=linux dom=7
vcpus=1
[2007-10-30 11:57:01 xend 16192] DEBUG (image:199) dom = 7
[2007-10-30 11:57:01 xend 16192] DEBUG (image:200) image
= /boot/vmlinuz-2.6.19-4-generic-amd64
[2007-10-30 11:57:01 xend 16192] DEBUG (image:201) store_evtchn = 1
[2007-10-30 11:57:01 xend 16192] DEBUG (image:202) console_evtchn = 2
[2007-10-30 11:57:01 xend 16192] DEBUG (image:203) cmdline =
root=/dev/hda1 ro
[2007-10-30 11:57:01 xend 16192] DEBUG (image:204) ramdisk
= /boot/initrd.img-2.6.19-4-generic-amd64
[2007-10-30 11:57:01 xend 16192] DEBUG (image:205) vcpus = 1
[2007-10-30 11:57:01 xend 16192] DEBUG (image:206) features [2007-10-30
11:57:02 xend 16192] DEBUG (blkif:24) exception looking up device
number for hda1: [Errno 2] No such file or directory:
''/dev/hda1''
[2007-10-30 11:57:02 xend 16192] DEBUG (DevController:110) DevController:
writing
{''backend-id'': ''0'',
''virtual-device'': ''769'',
''device-type'': ''disk'',
''state'': ''1'', ''backend'':
''/local/domain/0/backend/vbd/7/769''}
to /local/domain/7/device/vbd/769.
[2007-10-30 11:57:02 xend 16192] DEBUG (DevController:112) DevController:
writing
{''domain'': ''work'',
''frontend'':
''/local/domain/7/device/vbd/769'', ''dev'':
''hda1'', ''state'': ''1'',
''params'': ''/var/xen/domains/work/disk.img'',
''mode'': ''w'', ''online'':
''1'', ''frontend-id'': ''7'',
''type'': ''file''}
to /local/domain/0/backend/vbd/7/769.
[2007-10-30 11:57:02 xend 16192] DEBUG (blkif:24) exception looking up device
number for hda2: [Errno 2] No such file or directory:
''/dev/hda2''
[2007-10-30 11:57:02 xend 16192] DEBUG (DevController:110) DevController:
writing
{''backend-id'': ''0'',
''virtual-device'': ''770'',
''device-type'': ''disk'',
''state'': ''1'', ''backend'':
''/local/domain/0/backend/vbd/7/770''}
to /local/domain/7/device/vbd/770.
[2007-10-30 11:57:02 xend 16192] DEBUG (DevController:112) DevController:
writing
{''domain'': ''work'',
''frontend'':
''/local/domain/7/device/vbd/770'', ''dev'':
''hda2'', ''state'': ''1'',
''params'': ''/var/xen/domains/work/swap.img'',
''mode'': ''w'', ''online'':
''1'', ''frontend-id'': ''7'',
''type'': ''file''}
to /local/domain/0/backend/vbd/7/770.
[2007-10-30 11:57:02 xend 16192] DEBUG (DevController:110) DevController:
writing
{''backend-id'': ''0'', ''mac'':
''00:16:3e:b1:d7:91'', ''handle'':
''0'', ''state'': ''1'',
''backend'':
''/local/domain/0/backend/vif/7/0''}
to /local/domain/7/device/vif/0.
[2007-10-30 11:57:02 xend 16192] DEBUG (DevController:112) DevController:
writing
{''domain'': ''work'',
''handle'': ''0'', ''script'':
''/etc/xen/scripts/vif-route'', ''ip'':
''192.168.2.6'', ''state'':
''1'', ''frontend'':
''/local/domain/7/device/vif/0'', ''mac'':
''00:16:3e:b1:d7:91'', ''vifname'':
''vif2.0'', ''online'': ''1'',
''frontend-id'': ''7''}
to /local/domain/0/backend/vif/7/0.
[2007-10-30 11:57:02 xend.XendDomainInfo 16192] DEBUG (XendDomainInfo:713)
Storing VM details:
{''shadow_memory'': ''0'',
''uuid'':
''89708af1-f491-4aed-55e9-1be3acd352b7'',
''on_reboot'': ''restart'',
''start_time'': ''1193741822.28'',
''on_poweroff'': ''destroy'',
''name'': ''work'',
''xend/restart_count'': ''0'',
''vcpus'': ''1'',
''vcpu_avail'': ''1'',
''memory'': ''128'',
''on_crash'': ''restart'',
''image'': "(linux
(kernel /boot/vmlinuz-2.6.19-4-generic-amd64)
(ramdisk /boot/initrd.img-2.6.19-4-generic-amd64) (root ''/dev/hda1
ro''))", ''maxmem'': ''128''}
[2007-10-30 11:57:02 xend.XendDomainInfo 16192] DEBUG (XendDomainInfo:748)
Storing domain details:
{''console/ring-ref'': ''9890'',
''console/port'': ''2'',
''name'': ''work'',
''console/limit'': ''1048576'',
''vm'':
''/vm/89708af1-f491-4aed-55e9-1be3acd352b7'',
''domid'': ''7'',
''cpu/0/availability'': ''online'',
''memory/target'': ''131072'',
''store/ring-ref'': ''9891'',
''store/port'': ''1''}
[2007-10-30 11:57:02 xend.XendDomainInfo 16192] DEBUG (XendDomainInfo:938)
XendDomainInfo.handleShutdownWatch
[2007-10-30 11:57:02 xend 16192] DEBUG (DevController:143) Waiting for devices
vif.
[2007-10-30 11:57:02 xend 16192] DEBUG (DevController:149) Waiting for 0.
[2007-10-30 11:57:02 xend 16192] DEBUG (DevController:464)
hotplugStatusCallback /local/domain/0/backend/vif/7/0/hotplug-status.
[2007-10-30 11:57:02 xend 16192] DEBUG (DevController:464)
hotplugStatusCallback /local/domain/0/backend/vif/7/0/hotplug-status.
[2007-10-30 11:57:02 xend 16192] DEBUG (DevController:478)
hotplugStatusCallback 1.
[2007-10-30 11:57:02 xend 16192] DEBUG (DevController:143) Waiting for devices
usb.
[2007-10-30 11:57:02 xend 16192] DEBUG (DevController:143) Waiting for devices
vbd.
[2007-10-30 11:57:02 xend 16192] DEBUG (DevController:149) Waiting for 769.
[2007-10-30 11:57:02 xend 16192] DEBUG (DevController:464)
hotplugStatusCallback /local/domain/0/backend/vbd/7/769/hotplug-status.
[2007-10-30 11:57:03 xend 16192] DEBUG (DevController:464)
hotplugStatusCallback /local/domain/0/backend/vbd/7/769/hotplug-status.
[2007-10-30 11:57:03 xend 16192] DEBUG (DevController:478)
hotplugStatusCallback 1.
[2007-10-30 11:57:03 xend 16192] DEBUG (DevController:149) Waiting for 770.
[2007-10-30 11:57:03 xend 16192] DEBUG (DevController:464)
hotplugStatusCallback /local/domain/0/backend/vbd/7/770/hotplug-status.
[2007-10-30 11:57:03 xend 16192] DEBUG (DevController:478)
hotplugStatusCallback 1.
[2007-10-30 11:57:03 xend 16192] DEBUG (DevController:143) Waiting for devices
irq.
[2007-10-30 11:57:03 xend 16192] DEBUG (DevController:143) Waiting for devices
pci.
[2007-10-30 11:57:03 xend 16192] DEBUG (DevController:143) Waiting for devices
ioports.
[2007-10-30 11:57:03 xend 16192] DEBUG (DevController:143) Waiting for devices
tap.
[2007-10-30 11:57:03 xend 16192] DEBUG (DevController:143) Waiting for devices
vtpm.
[2007-10-30 11:57:03 xend 16192] INFO (XendDomain:370) Domain work (7)
unpaused.
[12:26:51] $
_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users