Xen devel - Jan 2006 - Infinite loop shutting down xendomains

Pardon me for a long detailed post and for pestering the developer list
but this looks pretty serious.  The bottom line is that, when I attempt
to shut down or reboot my simple xen-2.0.7 server running FC3 with two
domUs, shutting down xendomains appears to send python into an infinite
loop.  This may be due to a serious misconfiguration on my part but it
is certainly a big problem.

If we shutdown the domUs manually first, the system reboots perfectly
fine.

I''ll describe some history and include lots of log information.  The
configuration is slightly unusual.  Both domUs are VPN gateways running
the 2.4.30 kernel with openswan and openvpn.  Because of the openvpn,
but are privileged domains but the only device driver configured in the
kernel is the openvpn tun/tap adapter.

We first noticed problems during shutdown in that the last domU to
shutdown crashed very nastily when closing openvpn.  We could see this
even when manually stopping the domUs.  We surmised that the problem was
we had tun0 and tun1 defined in both domains and killing them in one
must be killing them in another.  We renamed those tun adapters in the
second domain to tun10 and tun11 and that problem went away.

We still noticed very long reboot times.  The hang seemed to be at
killall but the "logjam" seemed to break after xendomains errored out.
We assumed this was because killall killed something that xendomains
needed to shutdown gracefully so we added explicit K scripts in rc0.d
and rc6.d for xend and xendomains.

At first this seemed to work but it must have been only coincidental
because the problem returned.  We then realized we had brain cramped and
shutdown xend before xendomains.  We thought for sure we had found the
problem and put them in the proper order but, alas, the problem did not
go away.

To gain a little more insight into exactly what was happening, we
manually ran each kill script in rc6.d.  When we ran K78xendomains stop,
we saw the problem, or at least many aspect of it.
>From a xen console, we saw that the first domU had cleanly shutdown but
the second one was still listed howbeit with zero memory.

The hard disks were thrashing madly and top showed python consuming 99.9
percent CPU utilization.  Most of the time, during reboot, we do not see
the madly thrashing disks.  They do so for a moment as one might expect
when shutting down the domains but then just blip occasionally
thereafter.

Here is what we see in the logs:
>From messages (ntpd was the last service stopped before xendomains):
Jan 28 22:05:49 xenpub1dc1 ntpd[3652]: ntpd exiting on signal 15
Jan 28 22:05:49 xenpub1dc1 ntpd: ntpd shutdown succeeded
Jan 28 22:06:22 xenpub1dc1 kernel: device vif1.0 left promiscuous mode
Jan 28 22:06:22 xenpub1dc1 kernel: xen-br0: port 2(vif1.0) entering
disabled state
Jan 28 22:06:22 xenpub1dc1 kernel: xen-br0: port 2(vif1.0) entering
disabled state
Jan 28 22:06:22 xenpub1dc1 kernel: device vif1.1 left promiscuous mode
Jan 28 22:06:22 xenpub1dc1 kernel: xen-br0: port 3(vif1.1) entering
disabled state
Jan 28 22:06:22 xenpub1dc1 kernel: xen-br0: port 3(vif1.1) entering
disabled state
Jan 28 22:06:23 xenpub1dc1 net.agent[4714]: remove event not handled
Jan 28 22:06:23 xenpub1dc1 net.agent[4723]: remove event not handled
Jan 28 22:06:23 xenpub1dc1 net.agent[4737]: remove event not handled
Jan 28 22:06:23 xenpub1dc1 net.agent[4746]: remove event not handled
Jan 28 22:06:24 xenpub1dc1 kernel: device vif2.0 left promiscuous mode
Jan 28 22:06:24 xenpub1dc1 kernel: xen-br0: port 4(vif2.0) entering
disabled state
Jan 28 22:06:24 xenpub1dc1 kernel: xen-br0: port 4(vif2.0) entering
disabled state
Jan 28 22:06:24 xenpub1dc1 kernel: device vif2.1 left promiscuous mode
Jan 28 22:06:24 xenpub1dc1 kernel: xen-br1: port 2(vif2.1) entering
disabled state
Jan 28 22:06:24 xenpub1dc1 kernel: xen-br1: port 2(vif2.1) entering
disabled state
Jan 28 22:06:24 xenpub1dc1 net.agent[4774]: remove event not handled
Jan 28 22:06:24 xenpub1dc1 net.agent[4766]: remove event not handled
Jan 28 22:06:24 xenpub1dc1 net.agent[4789]: remove event not handled
Jan 28 22:06:24 xenpub1dc1 net.agent[4798]: remove event not handled

Here is the complete xend-debug.log:
network start bridge=xen-br0 netdev=eth0 antispoof=no
VIRTUAL MEMORY ARRANGEMENT:
 Loaded kernel: c0100000->c0319394
 Init. ramdisk: c031a000->c031a000
 Phys-Mach map: c031a000->c0332000
 Page tables:   c0332000->c0334000
 Start info:    c0334000->c0335000
 Boot stack:    c0335000->c0336000
 TOTAL:         c0000000->c0400000
 ENTRY ADDRESS: c0100000
vif-bridge up vif=vif1.0 domain=niagras1 mac=02:00:00:01:02:00
bridge=xen-br0
vif-bridge up vif=vif1.1 domain=niagras1 mac=aa:00:00:58:4a:a7
bridge=xen-br0
VIRTUAL MEMORY ARRANGEMENT:
 Loaded kernel: c0100000->c0319394
 Init. ramdisk: c031a000->c031a000
 Phys-Mach map: c031a000->c0332000
 Page tables:   c0332000->c0334000
 Start info:    c0334000->c0335000
 Boot stack:    c0335000->c0336000
 TOTAL:         c0000000->c0400000
 ENTRY ADDRESS: c0100000
vif-bridge up vif=vif2.0 domain=vpn1dc1 mac=02:00:00:01:01:00
bridge=xen-br0
vif-bridge up vif=vif2.1 domain=vpn1dc1 mac=02:00:00:01:01:01
bridge=xen-br1
recv_fe_driver_status> {''status'': 1}

recv_fe_driver_status>

recv_fe_interface_connect {''tx_shmem_frame'': 44763,
''rx_shmem_frame'':
44764, ''handle'': 0}

recv_fe_interface_connect {''tx_shmem_frame'': 44767,
''rx_shmem_frame'':
44768, ''handle'': 1}
recv_fe_driver_status> {''status'': 1}

recv_fe_driver_status>

recv_fe_interface_connect {''tx_shmem_frame'': 69339,
''rx_shmem_frame'':
69340, ''handle'': 0}

recv_fe_interface_connect {''tx_shmem_frame'': 69343,
''rx_shmem_frame'':
69344, ''handle'': 1}
{}
{}
VirqClient.virqReceived> 4
vif-bridge down vif=vif2.0 domain=vpn1dc1 mac=02:00:00:01:01:00
bridge=xen-br0
vif-bridge down vif=vif2.1 domain=vpn1dc1 mac=02:00:00:01:01:01
bridge=xen-br1
VirqClient.virqReceived> 4
vif-bridge down vif=vif1.0 domain=niagras1 mac=02:00:00:01:02:00
bridge=xen-br0
vif-bridge down vif=vif1.1 domain=niagras1 mac=aa:00:00:26:24:c4
bridge=xen-br0
Traceback (most recent call last):
  File "/usr/lib/python2.3/site-packages/twisted/protocols/basic.py",
line 229, in dataReceived
    return self.rawDataReceived(data)
  File "/usr/lib/python2.3/site-packages/twisted/protocols/http.py",
line 1015, in rawDataReceived
    self.allContentReceived()
  File "/usr/lib/python2.3/site-packages/twisted/protocols/http.py",
line 1006, in allContentReceived
    req.requestReceived(command, path, version)
  File "/usr/lib/python2.3/site-packages/twisted/protocols/http.py",
line 557, in requestReceived
    self.process()
--- <exception caught here> ---
  File "/usr/lib/python2.3/site-packages/twisted/web/server.py", line
165, in process
    self.render(resrc)
  File "/usr/lib/python2.3/site-packages/twisted/web/server.py", line
172, in render
    body = resrc.render(self)
  File "/usr/lib/python2.3/site-packages/twisted/web/resource.py",
line
201, in render
    return m(request)
  File "/usr/lib/python/xen/xend/server/SrvDomain.py", line 180, in
render_POST
    return self.perform(req)
  File "/usr/lib/python/xen/xend/server/SrvBase.py", line 103, in
perform
    return self._perform(op, op_method, req)
  File "/usr/lib/python/xen/xend/server/SrvBase.py", line 109, in
_perform
    return self._perform_err(err, op, req)
  File "/usr/lib/python/xen/xend/server/SrvBase.py", line 107, in
_perform
    val = op_method(op, req)
  File "/usr/lib/python/xen/xend/server/SrvDomain.py", line 47, in
op_shutdown
    val = fn(req.args, {''dom'': self.dom.id})
  File "/usr/lib/python/xen/xend/Args.py", line 145, in __call__
    return self.call_with_form_args(self.fn, fargs, xargs=xargs)
  File "/usr/lib/python/xen/xend/Args.py", line 121, in
call_with_form_args
    return fn(*params, **keys)
  File "/usr/lib/python/xen/xend/XendDomain.py", line 477, in
domain_shutdown
    val = xend.domain_shutdown(dominfo.id, reason, key)
  File "/usr/lib/python/xen/xend/server/SrvDaemon.py", line 723, in
domain_shutdown
    ctrl = self.domainCF.getController(dom)
  File "/usr/lib/python/xen/xend/server/controller.py", line 323, in
getController
    ctrl = self.createController(dom)
  File "/usr/lib/python/xen/xend/server/domain.py", line 20, in
createController
    return DomainController(self, dom)
  File "/usr/lib/python/xen/xend/server/domain.py", line 38, in
__init__
    self.registerChannel()
  File "/usr/lib/python/xen/xend/server/controller.py", line 220, in
registerChannel
    self.channel = self.channelFactory.domChannel(self.dom)
  File "/usr/lib/python/xen/xend/server/channel.py", line 83, in
domChannel
    remote_port=remote_port)
  File "/usr/lib/python/xen/xend/server/channel.py", line 253, in
__init__
    remote_port=remote_port)
  File "/usr/lib/python/xen/xend/server/channel.py", line 137, in
createPort
    remote_port=int(remote_port))
xen.lowlevel.xu.PortError: Failed to map domain control interface
/usr/lib/python2.3/site-packages/twisted/web/util.py:290: FutureWarning:
hex()/oct() of negative int will return a signed string in Python 2.4
and up
  return ''''''<div
class="instance"><span class="instanceName">%s
instance
@ %s</span>
/usr/lib/python2.3/site-packages/twisted/web/util.py:266: FutureWarning:
hex()/oct() of negative int will return a signed string in Python 2.4
and up
  w(''<div class="dict"><span
class="heading">Dictionary instance @ %
s</span>'' % hex(id(d)))
/usr/lib/python2.3/site-packages/twisted/web/util.py:279: FutureWarning:
hex()/oct() of negative int will return a signed string in Python 2.4
and up
  w(''<div class="list"><span
class="heading">List instance @ %s</span>''
% hex(id(l)))

 Init. ramdisk: c031a000->c031a000
 Phys-Mach map: c031a000->c0332000
 Page tables:   c0332000->c0334000
 Start info:    c0334000->c0335000
 Boot stack:    c0335000->c0336000
 TOTAL:         c0000000->c0400000
 ENTRY ADDRESS: c0100000
vif-bridge up vif=vif7.0 domain=vpn1dc1 mac=02:00:00:01:01:00
bridge=xen-br0
vif-bridge up vif=vif7.1 domain=vpn1dc1 mac=02:00:00:01:01:01
bridge=xen-br1
recv_fe_driver_status> {''status'': 1}

recv_fe_driver_status>

recv_fe_interface_connect {''tx_shmem_frame'': 88257,
''rx_shmem_frame'':
88258, ''handle'': 0}

recv_fe_interface_connect {''tx_shmem_frame'': 88261,
''rx_shmem_frame'':
88262, ''handle'': 1}
VIRTUAL MEMORY ARRANGEMENT:
 Loaded kernel: c0100000->c0319394
 Init. ramdisk: c031a000->c031a000
 Phys-Mach map: c031a000->c0332000
 Page tables:   c0332000->c0334000
 Start info:    c0334000->c0335000
 Boot stack:    c0335000->c0336000
 TOTAL:         c0000000->c0400000
 ENTRY ADDRESS: c0100000
vif-bridge up vif=vif8.0 domain=niagras1 mac=02:00:00:01:02:00
bridge=xen-br0
vif-bridge up vif=vif8.1 domain=niagras1 mac=aa:00:00:53:7a:5b
bridge=xen-br0
recv_fe_driver_status> {''status'': 1}

recv_fe_driver_status>

recv_fe_interface_connect {''tx_shmem_frame'': 185025,
''rx_shmem_frame'':
185026, ''handle'': 0}

recv_fe_interface_connect {''tx_shmem_frame'': 185029,
''rx_shmem_frame'':
185030, ''handle'': 1}
{}
{}
VirqClient.virqReceived> 4
vif-bridge down vif=vif8.0 domain=niagras1 mac=02:00:00:01:02:00
bridge=xen-br0
vif-bridge down vif=vif8.1 domain=niagras1 mac=aa:00:00:53:7a:5b
bridge=xen-br0
VirqClient.virqReceived> 4
vif-bridge down vif=vif7.0 domain=vpn1dc1 mac=02:00:00:01:01:00
bridge=xen-br0
vif-bridge down vif=vif7.1 domain=vpn1dc1 mac=02:00:00:01:01:01
bridge=xen-br1
-bridge down vif=vif4.1 domain=vpn1dc1 mac=02:00:00:01:01:01
bridge=xen-br1
interface vif4.1 does not exist!
VIRTUAL MEMORY ARRANGEMENT:
 Loaded kernel: c0100000->c02c9df4
 Init. ramdisk: c02ca000->c02ca000
 Phys-Mach map: c02ca000->c02e2000
 Page tables:   c02e2000->c02e4000
 Start info:    c02e4000->c02e5000
 Boot stack:    c02e5000->c02e6000
 TOTAL:         c0000000->c0400000
 ENTRY ADDRESS: c0100000
vif-bridge up vif=vif5.0 domain=vpn1dc1 mac=02:00:00:01:01:00
bridge=xen-br0
interface vif5.0 does not exist!
vif-bridge up vif=vif5.1 domain=vpn1dc1 mac=02:00:00:01:01:01
bridge=xen-br1
interface vif5.1 does not exist!
recv_fe_driver_status> {''status'': 1}

recv_fe_driver_status>

recv_fe_interface_connect {''tx_shmem_frame'': 202459,
''rx_shmem_frame'':
202460, ''handle'': 0}

recv_fe_interface_connect {''tx_shmem_frame'': 202463,
''rx_shmem_frame'':
202464, ''handle'': 1}
VirqClient.virqReceived> 4
vif-bridge down vif=vif5.0 domain=vpn1dc1 mac=02:00:00:01:01:00
bridge=xen-br0
interface vif5.0 does not exist!
vif-bridge down vif=vif5.1 domain=vpn1dc1 mac=02:00:00:01:01:01
bridge=xen-br1
interface vif5.1 does not exist!

Here is the information from xend.log:
Jan 28 22:05:49 xenpub1dc1 ntpd[3652]: ntpd exiting on signal 15
Jan 28 22:05:49 xenpub1dc1 ntpd: ntpd shutdown succeeded
Jan 28 22:06:22 xenpub1dc1 kernel: device vif1.0 left promiscuous mode
Jan 28 22:06:22 xenpub1dc1 kernel: xen-br0: port 2(vif1.0) entering
disabled state
Jan 28 22:06:22 xenpub1dc1 kernel: xen-br0: port 2(vif1.0) entering
disabled state
Jan 28 22:06:22 xenpub1dc1 kernel: device vif1.1 left promiscuous mode
Jan 28 22:06:22 xenpub1dc1 kernel: xen-br0: port 3(vif1.1) entering
disabled state
Jan 28 22:06:22 xenpub1dc1 kernel: xen-br0: port 3(vif1.1) entering
disabled state
Jan 28 22:06:23 xenpub1dc1 net.agent[4714]: remove event not handled
Jan 28 22:06:23 xenpub1dc1 net.agent[4723]: remove event not handled
Jan 28 22:06:23 xenpub1dc1 net.agent[4737]: remove event not handled
Jan 28 22:06:23 xenpub1dc1 net.agent[4746]: remove event not handled
Jan 28 22:06:24 xenpub1dc1 kernel: device vif2.0 left promiscuous mode
Jan 28 22:06:24 xenpub1dc1 kernel: xen-br0: port 4(vif2.0) entering
disabled state
Jan 28 22:06:24 xenpub1dc1 kernel: xen-br0: port 4(vif2.0) entering
disabled state
Jan 28 22:06:24 xenpub1dc1 kernel: device vif2.1 left promiscuous mode
Jan 28 22:06:24 xenpub1dc1 kernel: xen-br1: port 2(vif2.1) entering
disabled state
Jan 28 22:06:24 xenpub1dc1 kernel: xen-br1: port 2(vif2.1) entering
disabled state
Jan 28 22:06:24 xenpub1dc1 net.agent[4774]: remove event not handled
Jan 28 22:06:24 xenpub1dc1 net.agent[4766]: remove event not handled
Jan 28 22:06:24 xenpub1dc1 net.agent[4789]: remove event not handled
Jan 28 22:06:24 xenpub1dc1 net.agent[4798]: remove event not handled
Jan 28 22:18:03 xenpub1dc1 shutdown: shutting down for system reboot
Jan 28 22:18:03 xenpub1dc1 init: Switching to runlevel: 6
Jan 28 22:18:03 xenpub1dc1 login(pam_unix)[3781]: session closed for
user root
Jan 28 22:18:03 xenpub1dc1 login(pam_unix)[3779]: session closed for
user root
Jan 28 22:18:03 xenpub1dc1 login(pam_unix)[3780]: session closed for
user root
Jan 28 22:18:04 xenpub1dc1 xendomains: Shutting down all Xen domains:
Jan 28 22:18:05 xenpub1dc1 xendomains: Error: Internal Server Error
Jan 28 22:18:05 xenpub1dc1 xendomains:  failed
Jan 28 22:18:05 xenpub1dc1 xendomains:
Jan 28 22:18:05 xenpub1dc1 rc: Stopping xendomains:  failed
Jan 28 22:18:05 xenpub1dc1 kernel: Kernel logging (proc) stopped.
Jan 28 22:18:05 xenpub1dc1 kernel: Kernel log daemon terminating.
Jan 28 22:18:06 xenpub1dc1 syslog: klogd shutdown succeeded
Jan 28 22:18:06 xenpub1dc1 exiting on signal 15
Jan 28 22:20:06 xenpub1dc1 syslogd 1.4.1: restart.
Jan 28 22:20:06 xenpub1dc1 syslog: syslogd startup succeeded
Jan 28 22:20:06 xenpub1dc1 kernel: klogd 1.4.1, log source = /proc/kmsg
started.
Jan 28 22:20:06 xenpub1dc1 kernel: Linux version 2.6.11.12-xen0
(root@xenpub1dc1.atlasgroup.net) (gcc version 3.4.4 20050721 (Red Hat
3.4.4-2)) #4 Thu Jan
File messages not changed so no update needed.
[root@xenpub1dc1 log]# ls
acpid            boot.log.1  cron.2   mail       messages    prelink.log
samba     spooler    wtmp            xfrd.log
anaconda.log     boot.log.2  cups     maillog    messages.1  rpmpkgs
secure    spooler.1  xend-debug.log  Xorg.0.log
anaconda.syslog  cron        dmesg    maillog.1  messages.2  rpmpkgs.1
secure.1  spooler.2  xend.log        Xorg.0.log.old
boot.log         cron.1      lastlog  maillog.2  ppp         rpmpkgs.2
secure.2  vbox       xend.log.1      yum.log
[root@xenpub1dc1 log]# joe xend-debug.log
Processing ''/etc/joe/joerc''...done
Processing ''/etc/joe/joerc''...done









































    I    xend-debug.log
Row 171  Col 1   11:02  Ctrl-K H for help
recv_fe_driver_status>

recv_fe_interface_connect {''tx_shmem_frame'': 185025,
''rx_shmem_frame'':
185026, ''handle'': 0}

recv_fe_interface_connect {''tx_shmem_frame'': 185029,
''rx_shmem_frame'':
185030, ''handle'': 1}
{}
{}
VirqClient.virqReceived> 4
vif-bridge down vif=vif8.0 domain=niagras1 mac=02:00:00:01:02:00
bridge=xen-br0
vif-bridge down vif=vif8.1 domain=niagras1 mac=aa:00:00:53:7a:5b
bridge=xen-br0
VirqClient.virqReceived> 4
vif-bridge down vif=vif7.0 domain=vpn1dc1 mac=02:00:00:01:01:00
bridge=xen-br0
vif-bridge down vif=vif7.1 domain=vpn1dc1 mac=02:00:00:01:01:01
bridge=xen-br1
-bridge down vif=vif4.1 domain=vpn1dc1 mac=02:00:00:01:01:01
bridge=xen-br1
interface vif4.1 does not exist!
VIRTUAL MEMORY ARRANGEMENT:
 Loaded kernel: c0100000->c02c9df4
 Init. ramdisk: c02ca000->c02ca000
 Phys-Mach map: c02ca000->c02e2000
 Page tables:   c02e2000->c02e4000
 Start info:    c02e4000->c02e5000
 Boot stack:    c02e5000->c02e6000
 TOTAL:         c0000000->c0400000
 ENTRY ADDRESS: c0100000
vif-bridge up vif=vif5.0 domain=vpn1dc1 mac=02:00:00:01:01:00
bridge=xen-br0
interface vif5.0 does not exist!
vif-bridge up vif=vif5.1 domain=vpn1dc1 mac=02:00:00:01:01:01
bridge=xen-br1
interface vif5.1 does not exist!
recv_fe_driver_status> {''status'': 1}

recv_fe_driver_status>

recv_fe_interface_connect {''tx_shmem_frame'': 202459,
''rx_shmem_frame'':
202460, ''handle'': 0}

recv_fe_interface_connect {''tx_shmem_frame'': 202463,
''rx_shmem_frame'':
202464, ''handle'': 1}
VirqClient.virqReceived> 4
vif-bridge down vif=vif5.0 domain=vpn1dc1 mac=02:00:00:01:01:00
bridge=xen-br0
interface vif5.0 does not exist!
vif-bridge down vif=vif5.1 domain=vpn1dc1 mac=02:00:00:01:01:01
bridge=xen-br1
interface vif5.1 does not exist!

File xend-debug.log not changed so no update needed.
[root@xenpub1dc1 log]# ls
acpid            boot.log.1  cron.2   mail       messages    prelink.log
samba     spooler    wtmp            xfrd.log
anaconda.log     boot.log.2  cups     maillog    messages.1  rpmpkgs
secure    spooler.1  xend-debug.log  Xorg.0.log
anaconda.syslog  cron        dmesg    maillog.1  messages.2  rpmpkgs.1
secure.1  spooler.2  xend.log        Xorg.0.log.old
boot.log         cron.1      lastlog  maillog.2  ppp         rpmpkgs.2
secure.2  vbox       xend.log.1      yum.log
[root@xenpub1dc1 log]# joe xend.log
Processing ''/etc/joe/joerc''...done
Processing ''/etc/joe/joerc''...done









































    I    xend.log
Row 6079 Col 1   11:04  Ctrl-K H for help
[2006-01-28 22:06:09 xend] INFO (XendRoot:113) EVENT>
xend.domain.shutdown [''vpn1dc1'', ''2'',
''halt'']
[2006-01-28 22:06:09 xend] INFO (XendRoot:113) EVENT>
xend.domain.shutdown [''niagras1'', ''1'',
''halt'']
[2006-01-28 22:06:22 xend] DEBUG (XendDomain:244) XendDomain>reap>
domain died name=niagras1 id=1
[2006-01-28 22:06:22 xend] DEBUG (XendDomain:247) XendDomain>reap>
shutdown id=1 reason=poweroff
[2006-01-28 22:06:22 xend] DEBUG (XendDomain:487)
domain_restart_schedule> 1 poweroff 0
[2006-01-28 22:06:22 xend] INFO (XendDomain:568) Destroying domain:
name=niagras1
[2006-01-28 22:06:22 xend] DEBUG (XendDomainInfo:665) Destroying vifs
for domain 1
[2006-01-28 22:06:22 xend] DEBUG (netif:305) Destroying vif domain=1
vif=0
[2006-01-28 22:06:22 xend] DEBUG (netif:305) Destroying vif domain=1
vif=1
[2006-01-28 22:06:22 xend] DEBUG (XendDomainInfo:674) Destroying vbds
for domain 1
[2006-01-28 22:06:22 xend] DEBUG (blkif:552) Destroying blkif domain=1
[2006-01-28 22:06:22 xend] DEBUG (blkif:408) Destroying vbd domain=1
idx=0
[2006-01-28 22:06:22 xend] DEBUG (blkif:408) Destroying vbd domain=1
idx=1
[2006-01-28 22:06:22 xend] DEBUG (XendDomainInfo:634) Closing console,
domain 1
[2006-01-28 22:06:22 xend] DEBUG (XendDomainInfo:622) Closing channel to
domain 1
[2006-01-28 22:06:22 xend] INFO (XendRoot:113) EVENT> xend.virq 4
[2006-01-28 22:06:22 xend] DEBUG (blkif:363) Unbinding vbd (type phy)
from VG1/niagras1root
[2006-01-28 22:06:22 xend] DEBUG (blkif:363) Unbinding vbd (type phy)
from VG1/niagras1swap
[2006-01-28 22:06:22 xend] INFO (console:58) Console disconnected 15
127.0.0.1 3484
[2006-01-28 22:06:22 xend] INFO (XendRoot:113) EVENT> xend.domain.exit
[''niagras1'', ''1'',
''poweroff'']
[2006-01-28 22:06:22 xend] INFO (XendRoot:113) EVENT>
xend.domain.destroy [''niagras1'', ''1'']
[2006-01-28 22:06:22 xend] INFO (XendRoot:113) EVENT>
xend.console.disconnect [15, ''127.0.0.1'', 3484]
[2006-01-28 22:06:23 xend] INFO (XendRoot:113) EVENT> xend.domain.died
[''niagras1'', ''1'']
[2006-01-28 22:06:24 xend] DEBUG (XendDomain:244) XendDomain>reap>
domain died name=vpn1dc1 id=2
[2006-01-28 22:06:24 xend] DEBUG (XendDomain:247) XendDomain>reap>
shutdown id=2 reason=poweroff
[2006-01-28 22:06:24 xend] DEBUG (XendDomain:487)
domain_restart_schedule> 2 poweroff 0
[2006-01-28 22:06:24 xend] INFO (XendDomain:568) Destroying domain:
name=vpn1dc1
[2006-01-28 22:06:24 xend] DEBUG (XendDomainInfo:665) Destroying vifs
for domain 2
[2006-01-28 22:06:24 xend] DEBUG (netif:305) Destroying vif domain=2
vif=0
[2006-01-28 22:06:24 xend] DEBUG (netif:305) Destroying vif domain=2
vif=1
[2006-01-28 22:06:24 xend] DEBUG (XendDomainInfo:674) Destroying vbds
for domain 2
[2006-01-28 22:06:24 xend] DEBUG (blkif:552) Destroying blkif domain=2
[2006-01-28 22:06:24 xend] DEBUG (blkif:408) Destroying vbd domain=2
idx=0
[2006-01-28 22:06:24 xend] DEBUG (blkif:408) Destroying vbd domain=2
idx=1
[2006-01-28 22:06:24 xend] DEBUG (XendDomainInfo:634) Closing console,
domain 2
[2006-01-28 22:06:24 xend] DEBUG (XendDomainInfo:622) Closing channel to
domain 2
[2006-01-28 22:06:24 xend] INFO (XendRoot:113) EVENT> xend.virq 4
[2006-01-28 22:06:24 xend] DEBUG (blkif:363) Unbinding vbd (type phy)
from VG1/vpn1dc1root
[2006-01-28 22:06:24 xend] DEBUG (blkif:363) Unbinding vbd (type phy)
from VG1/vpn1dc1swap
[2006-01-28 22:06:24 xend] INFO (console:58) Console disconnected 16
127.0.0.1 3986
[2006-01-28 22:06:24 xend] INFO (XendRoot:113) EVENT> xend.domain.exit
[''vpn1dc1'', ''2'',
''poweroff'']
Jan 28 22:05:49 xenpub1dc1 ntpd[3652]: ntpd exiting on signal 15
Jan 28 22:05:49 xenpub1dc1 ntpd: ntpd shutdown succeeded
Jan 28 22:06:22 xenpub1dc1 kernel: device vif1.0 left promiscuous mode
Jan 28 22:06:22 xenpub1dc1 kernel: xen-br0: port 2(vif1.0) entering
disabled state
Jan 28 22:06:22 xenpub1dc1 kernel: xen-br0: port 2(vif1.0) entering
disabled state
Jan 28 22:06:22 xenpub1dc1 kernel: device vif1.1 left promiscuous mode
Jan 28 22:06:22 xenpub1dc1 kernel: xen-br0: port 3(vif1.1) entering
disabled state
Jan 28 22:06:22 xenpub1dc1 kernel: xen-br0: port 3(vif1.1) entering
disabled state
Jan 28 22:06:23 xenpub1dc1 net.agent[4714]: remove event not handled
Jan 28 22:06:23 xenpub1dc1 net.agent[4723]: remove event not handled
Jan 28 22:06:23 xenpub1dc1 net.agent[4737]: remove event not handled
Jan 28 22:06:23 xenpub1dc1 net.agent[4746]: remove event not handled
Jan 28 22:06:24 xenpub1dc1 kernel: device vif2.0 left promiscuous mode
Jan 28 22:06:24 xenpub1dc1 kernel: xen-br0: port 4(vif2.0) entering
disabled state
Jan 28 22:06:24 xenpub1dc1 kernel: xen-br0: port 4(vif2.0) entering
disabled state
Jan 28 22:06:24 xenpub1dc1 kernel: device vif2.1 left promiscuous mode
Jan 28 22:06:24 xenpub1dc1 kernel: xen-br1: port 2(vif2.1) entering
disabled state
Jan 28 22:06:24 xenpub1dc1 kernel: xen-br1: port 2(vif2.1) entering
disabled state
Jan 28 22:06:24 xenpub1dc1 net.agent[4774]: remove event not handled
Jan 28 22:06:24 xenpub1dc1 net.agent[4766]: remove event not handled
Jan 28 22:06:24 xenpub1dc1 net.agent[4789]: remove event not handled
Jan 28 22:06:24 xenpub1dc1 net.agent[4798]: remove event not handled
Jan 28 22:18:03 xenpub1dc1 shutdown: shutting down for system reboot
Jan 28 22:18:03 xenpub1dc1 init: Switching to runlevel: 6
Jan 28 22:18:03 xenpub1dc1 login(pam_unix)[3781]: session closed for
user root
Jan 28 22:18:03 xenpub1dc1 login(pam_unix)[3779]: session closed for
user root
Jan 28 22:18:03 xenpub1dc1 login(pam_unix)[3780]: session closed for
user root
Jan 28 22:18:04 xenpub1dc1 xendomains: Shutting down all Xen domains:
Jan 28 22:18:05 xenpub1dc1 xendomains: Error: Internal Server Error
Jan 28 22:18:05 xenpub1dc1 xendomains:  failed
Jan 28 22:18:05 xenpub1dc1 xendomains:
Jan 28 22:18:05 xenpub1dc1 rc: Stopping xendomains:  failed
Jan 28 22:18:05 xenpub1dc1 kernel: Kernel logging (proc) stopped.
Jan 28 22:18:05 xenpub1dc1 kernel: Kernel log daemon terminating.
Jan 28 22:18:06 xenpub1dc1 syslog: klogd shutdown succeeded
Jan 28 22:18:06 xenpub1dc1 exiting on signal 15
Jan 28 22:20:06 xenpub1dc1 syslogd 1.4.1: restart.
Jan 28 22:20:06 xenpub1dc1 syslog: syslogd startup succeeded
Jan 28 22:20:06 xenpub1dc1 kernel: klogd 1.4.1, log source = /proc/kmsg
started.
Jan 28 22:20:06 xenpub1dc1 kernel: Linux version 2.6.11.12-xen0
(root@xenpub1dc1.atlasgroup.net) (gcc version 3.4.4 20050721 (Red Hat
3.4.4-2)) #4 Thu Jan
File messages not changed so no update needed.
[root@xenpub1dc1 log]# ls
acpid            boot.log.1  cron.2   mail       messages    prelink.log
samba     spooler    wtmp            xfrd.log
anaconda.log     boot.log.2  cups     maillog    messages.1  rpmpkgs
secure    spooler.1  xend-debug.log  Xorg.0.log
anaconda.syslog  cron        dmesg    maillog.1  messages.2  rpmpkgs.1
secure.1  spooler.2  xend.log        Xorg.0.log.old
boot.log         cron.1      lastlog  maillog.2  ppp         rpmpkgs.2
secure.2  vbox       xend.log.1      yum.log
[root@xenpub1dc1 log]# joe xend-debug.log
Processing ''/etc/joe/joerc''...done
Processing ''/etc/joe/joerc''...done









































    I    xend-debug.log
Row 171  Col 1   11:02  Ctrl-K H for help
recv_fe_driver_status>

recv_fe_interface_connect {''tx_shmem_frame'': 185025,
''rx_shmem_frame'':
185026, ''handle'': 0}

recv_fe_interface_connect {''tx_shmem_frame'': 185029,
''rx_shmem_frame'':
185030, ''handle'': 1}
{}
{}
VirqClient.virqReceived> 4
vif-bridge down vif=vif8.0 domain=niagras1 mac=02:00:00:01:02:00
bridge=xen-br0
vif-bridge down vif=vif8.1 domain=niagras1 mac=aa:00:00:53:7a:5b
bridge=xen-br0
VirqClient.virqReceived> 4
vif-bridge down vif=vif7.0 domain=vpn1dc1 mac=02:00:00:01:01:00
bridge=xen-br0
vif-bridge down vif=vif7.1 domain=vpn1dc1 mac=02:00:00:01:01:01
bridge=xen-br1
-bridge down vif=vif4.1 domain=vpn1dc1 mac=02:00:00:01:01:01
bridge=xen-br1
interface vif4.1 does not exist!
VIRTUAL MEMORY ARRANGEMENT:
 Loaded kernel: c0100000->c02c9df4
 Init. ramdisk: c02ca000->c02ca000
 Phys-Mach map: c02ca000->c02e2000
 Page tables:   c02e2000->c02e4000
 Start info:    c02e4000->c02e5000
 Boot stack:    c02e5000->c02e6000
 TOTAL:         c0000000->c0400000
 ENTRY ADDRESS: c0100000
vif-bridge up vif=vif5.0 domain=vpn1dc1 mac=02:00:00:01:01:00
bridge=xen-br0
interface vif5.0 does not exist!
vif-bridge up vif=vif5.1 domain=vpn1dc1 mac=02:00:00:01:01:01
bridge=xen-br1
interface vif5.1 does not exist!
recv_fe_driver_status> {''status'': 1}

recv_fe_driver_status>

recv_fe_interface_connect {''tx_shmem_frame'': 202459,
''rx_shmem_frame'':
202460, ''handle'': 0}

recv_fe_interface_connect {''tx_shmem_frame'': 202463,
''rx_shmem_frame'':
202464, ''handle'': 1}
VirqClient.virqReceived> 4
vif-bridge down vif=vif5.0 domain=vpn1dc1 mac=02:00:00:01:01:00
bridge=xen-br0
interface vif5.0 does not exist!
vif-bridge down vif=vif5.1 domain=vpn1dc1 mac=02:00:00:01:01:01
bridge=xen-br1
interface vif5.1 does not exist!

File xend-debug.log not changed so no update needed.
[root@xenpub1dc1 log]# ls
acpid            boot.log.1  cron.2   mail       messages    prelink.log
samba     spooler    wtmp            xfrd.log
anaconda.log     boot.log.2  cups     maillog    messages.1  rpmpkgs
secure    spooler.1  xend-debug.log  Xorg.0.log
anaconda.syslog  cron        dmesg    maillog.1  messages.2  rpmpkgs.1
secure.1  spooler.2  xend.log        Xorg.0.log.old
boot.log         cron.1      lastlog  maillog.2  ppp         rpmpkgs.2
secure.2  vbox       xend.log.1      yum.log
[root@xenpub1dc1 log]# joe xend.log
Processing ''/etc/joe/joerc''...done
Processing ''/etc/joe/joerc''...done









































    I    xend.log
Row 6077 Col 91  11:06  Ctrl-K H for help
[2006-01-28 22:06:24 xend] DEBUG (blkif:363) Unbinding vbd (type phy)
from VG1/vpn1dc1swap
[2006-01-28 22:06:24 xend] INFO (console:58) Console disconnected 16
127.0.0.1 3986
[2006-01-28 22:06:24 xend] INFO (XendRoot:113) EVENT> xend.domain.exit
[''vpn1dc1'', ''2'',
''poweroff'']
[2006-01-28 22:06:24 xend] INFO (XendRoot:113) EVENT>
xend.domain.destroy [''vpn1dc1'', ''2'']
[2006-01-28 22:06:24 xend] INFO (XendRoot:113) EVENT>
xend.console.disconnect [16, ''127.0.0.1'', 3986]
[2006-01-28 22:06:26 xend] DEBUG (XendDomain:244) XendDomain>reap>
domain died name=vpn1dc1 id=2
[2006-01-28 22:06:26 xend] DEBUG (XendDomain:247) XendDomain>reap>
shutdown id=2 reason=poweroff
[2006-01-28 22:06:26 xend] DEBUG (XendDomain:487)
domain_restart_schedule> 2 poweroff 0
[2006-01-28 22:06:26 xend] INFO (XendDomain:568) Destroying domain:
name=vpn1dc1
[2006-01-28 22:06:26 xend] DEBUG (XendDomainInfo:634) Closing console,
domain 2
[2006-01-28 22:06:26 xend] INFO (XendRoot:113) EVENT> xend.domain.exit
[''vpn1dc1'', ''2'',
''poweroff'']
[2006-01-28 22:06:26 xend] INFO (XendRoot:113) EVENT>
xend.domain.destroy [''vpn1dc1'', ''2'']
[2006-01-28 22:06:28 xend] DEBUG (XendDomain:244) XendDomain>reap>
domain died name=vpn1dc1 id=2
[2006-01-28 22:06:28 xend] DEBUG (XendDomain:247) XendDomain>reap>
shutdown id=2 reason=poweroff
[2006-01-28 22:06:28 xend] DEBUG (XendDomain:487)
domain_restart_schedule> 2 poweroff 0
[2006-01-28 22:06:28 xend] INFO (XendDomain:568) Destroying domain:
name=vpn1dc1
[2006-01-28 22:06:28 xend] DEBUG (XendDomainInfo:634) Closing console,
domain 2
[2006-01-28 22:06:28 xend] INFO (XendRoot:113) EVENT> xend.domain.exit
[''vpn1dc1'', ''2'',
''poweroff'']
[2006-01-28 22:06:28 xend] INFO (XendRoot:113) EVENT>
xend.domain.destroy [''vpn1dc1'', ''2'']
[2006-01-28 22:06:30 xend] DEBUG (XendDomain:244) XendDomain>reap>
domain died name=vpn1dc1 id=2
[2006-01-28 22:06:30 xend] DEBUG (XendDomain:247) XendDomain>reap>
shutdown id=2 reason=poweroff
[2006-01-28 22:06:30 xend] DEBUG (XendDomain:487)
domain_restart_schedule> 2 poweroff 0
[2006-01-28 22:06:30 xend] INFO (XendDomain:568) Destroying domain:
name=vpn1dc1
[2006-01-28 22:06:30 xend] DEBUG (XendDomainInfo:634) Closing console,
domain 2
[2006-01-28 22:06:30 xend] INFO (XendRoot:113) EVENT> xend.domain.exit
[''vpn1dc1'', ''2'',
''poweroff'']
[2006-01-28 22:06:30 xend] INFO (XendRoot:113) EVENT>
xend.domain.destroy [''vpn1dc1'', ''2'']
[2006-01-28 22:06:32 xend] DEBUG (XendDomain:244) XendDomain>reap>
domain died name=vpn1dc1 id=2
[2006-01-28 22:06:32 xend] DEBUG (XendDomain:247) XendDomain>reap>
shutdown id=2 reason=poweroff
[2006-01-28 22:06:32 xend] DEBUG (XendDomain:487)
domain_restart_schedule> 2 poweroff 0
[2006-01-28 22:06:32 xend] INFO (XendDomain:568) Destroying domain:
name=vpn1dc1
[2006-01-28 22:06:32 xend] DEBUG (XendDomainInfo:634) Closing console,
domain 2
[2006-01-28 22:06:32 xend] INFO (XendRoot:113) EVENT> xend.domain.exit
[''vpn1dc1'', ''2'',
''poweroff'']
[2006-01-28 22:06:32 xend] INFO (XendRoot:113) EVENT>
xend.domain.destroy [''vpn1dc1'', ''2'']
[2006-01-28 22:06:34 xend] DEBUG (XendDomain:244) XendDomain>reap>
domain died name=vpn1dc1 id=2
[2006-01-28 22:06:34 xend] DEBUG (XendDomain:247) XendDomain>reap>
shutdown id=2 reason=poweroff
[2006-01-28 22:06:34 xend] DEBUG (XendDomain:487)
domain_restart_schedule> 2 poweroff 0
[2006-01-28 22:06:34 xend] INFO (XendDomain:568) Destroying domain:
name=vpn1dc1
[2006-01-28 22:06:34 xend] DEBUG (XendDomainInfo:634) Closing console,
domain 2
[2006-01-28 22:06:34 xend] INFO (XendRoot:113) EVENT> xend.domain.exit
[''vpn1dc1'', ''2'',
''poweroff'']
[2006-01-28 22:06:34 xend] INFO (XendRoot:113) EVENT>
xend.domain.destroy [''vpn1dc1'', ''2'']
[2006-01-28 22:06:36 xend] DEBUG (XendDomain:244) XendDomain>reap>
domain died name=vpn1dc1 id=2  
Jan 28 22:05:49 xenpub1dc1 ntpd[3652]: ntpd exiting on signal 15
Jan 28 22:05:49 xenpub1dc1 ntpd: ntpd shutdown succeeded
Jan 28 22:06:22 xenpub1dc1 kernel: device vif1.0 left promiscuous mode
Jan 28 22:06:22 xenpub1dc1 kernel: xen-br0: port 2(vif1.0) entering
disabled state
Jan 28 22:06:22 xenpub1dc1 kernel: xen-br0: port 2(vif1.0) entering
disabled state
Jan 28 22:06:22 xenpub1dc1 kernel: device vif1.1 left promiscuous mode
Jan 28 22:06:22 xenpub1dc1 kernel: xen-br0: port 3(vif1.1) entering
disabled state
Jan 28 22:06:22 xenpub1dc1 kernel: xen-br0: port 3(vif1.1) entering
disabled state
Jan 28 22:06:23 xenpub1dc1 net.agent[4714]: remove event not handled
Jan 28 22:06:23 xenpub1dc1 net.agent[4723]: remove event not handled
Jan 28 22:06:23 xenpub1dc1 net.agent[4737]: remove event not handled
Jan 28 22:06:23 xenpub1dc1 net.agent[4746]: remove event not handled
Jan 28 22:06:24 xenpub1dc1 kernel: device vif2.0 left promiscuous mode
Jan 28 22:06:24 xenpub1dc1 kernel: xen-br0: port 4(vif2.0) entering
disabled state
Jan 28 22:06:24 xenpub1dc1 kernel: xen-br0: port 4(vif2.0) entering
disabled state
Jan 28 22:06:24 xenpub1dc1 kernel: device vif2.1 left promiscuous mode
Jan 28 22:06:24 xenpub1dc1 kernel: xen-br1: port 2(vif2.1) entering
disabled state
Jan 28 22:06:24 xenpub1dc1 kernel: xen-br1: port 2(vif2.1) entering
disabled state
Jan 28 22:06:24 xenpub1dc1 net.agent[4774]: remove event not handled
Jan 28 22:06:24 xenpub1dc1 net.agent[4766]: remove event not handled
Jan 28 22:06:24 xenpub1dc1 net.agent[4789]: remove event not handled
Jan 28 22:06:24 xenpub1dc1 net.agent[4798]: remove event not handled
Jan 28 22:18:03 xenpub1dc1 shutdown: shutting down for system reboot
Jan 28 22:18:03 xenpub1dc1 init: Switching to runlevel: 6
Jan 28 22:18:03 xenpub1dc1 login(pam_unix)[3781]: session closed for
user root
Jan 28 22:18:03 xenpub1dc1 login(pam_unix)[3779]: session closed for
user root
Jan 28 22:18:03 xenpub1dc1 login(pam_unix)[3780]: session closed for
user root
Jan 28 22:18:04 xenpub1dc1 xendomains: Shutting down all Xen domains:
Jan 28 22:18:05 xenpub1dc1 xendomains: Error: Internal Server Error
Jan 28 22:18:05 xenpub1dc1 xendomains:  failed
Jan 28 22:18:05 xenpub1dc1 xendomains:
Jan 28 22:18:05 xenpub1dc1 rc: Stopping xendomains:  failed
Jan 28 22:18:05 xenpub1dc1 kernel: Kernel logging (proc) stopped.
Jan 28 22:18:05 xenpub1dc1 kernel: Kernel log daemon terminating.
Jan 28 22:18:06 xenpub1dc1 syslog: klogd shutdown succeeded
Jan 28 22:18:06 xenpub1dc1 exiting on signal 15
Jan 28 22:20:06 xenpub1dc1 syslogd 1.4.1: restart.
Jan 28 22:20:06 xenpub1dc1 syslog: syslogd startup succeeded
Jan 28 22:20:06 xenpub1dc1 kernel: klogd 1.4.1, log source = /proc/kmsg
started.
Jan 28 22:20:06 xenpub1dc1 kernel: Linux version 2.6.11.12-xen0
(root@xenpub1dc1.atlasgroup.net) (gcc version 3.4.4 20050721 (Red Hat
3.4.4-2)) #4 Thu Jan
File messages not changed so no update needed.
[root@xenpub1dc1 log]# ls
acpid            boot.log.1  cron.2   mail       messages    prelink.log
samba     spooler    wtmp            xfrd.log
anaconda.log     boot.log.2  cups     maillog    messages.1  rpmpkgs
secure    spooler.1  xend-debug.log  Xorg.0.log
anaconda.syslog  cron        dmesg    maillog.1  messages.2  rpmpkgs.1
secure.1  spooler.2  xend.log        Xorg.0.log.old
boot.log         cron.1      lastlog  maillog.2  ppp         rpmpkgs.2
secure.2  vbox       xend.log.1      yum.log
[root@xenpub1dc1 log]# joe xend-debug.log
Processing ''/etc/joe/joerc''...done
Processing ''/etc/joe/joerc''...done









































    I    xend-debug.log
Row 171  Col 1   11:02  Ctrl-K H for help
recv_fe_driver_status>

recv_fe_interface_connect {''tx_shmem_frame'': 185025,
''rx_shmem_frame'':
185026, ''handle'': 0}

recv_fe_interface_connect {''tx_shmem_frame'': 185029,
''rx_shmem_frame'':
185030, ''handle'': 1}
{}
{}
VirqClient.virqReceived> 4
vif-bridge down vif=vif8.0 domain=niagras1 mac=02:00:00:01:02:00
bridge=xen-br0
vif-bridge down vif=vif8.1 domain=niagras1 mac=aa:00:00:53:7a:5b
bridge=xen-br0
VirqClient.virqReceived> 4
vif-bridge down vif=vif7.0 domain=vpn1dc1 mac=02:00:00:01:01:00
bridge=xen-br0
vif-bridge down vif=vif7.1 domain=vpn1dc1 mac=02:00:00:01:01:01
bridge=xen-br1
-bridge down vif=vif4.1 domain=vpn1dc1 mac=02:00:00:01:01:01
bridge=xen-br1
interface vif4.1 does not exist!
VIRTUAL MEMORY ARRANGEMENT:
 Loaded kernel: c0100000->c02c9df4
 Init. ramdisk: c02ca000->c02ca000
 Phys-Mach map: c02ca000->c02e2000
 Page tables:   c02e2000->c02e4000
 Start info:    c02e4000->c02e5000
 Boot stack:    c02e5000->c02e6000
 TOTAL:         c0000000->c0400000
 ENTRY ADDRESS: c0100000
vif-bridge up vif=vif5.0 domain=vpn1dc1 mac=02:00:00:01:01:00
bridge=xen-br0
interface vif5.0 does not exist!
vif-bridge up vif=vif5.1 domain=vpn1dc1 mac=02:00:00:01:01:01
bridge=xen-br1
interface vif5.1 does not exist!
recv_fe_driver_status> {''status'': 1}

recv_fe_driver_status>

recv_fe_interface_connect {''tx_shmem_frame'': 202459,
''rx_shmem_frame'':
202460, ''handle'': 0}

recv_fe_interface_connect {''tx_shmem_frame'': 202463,
''rx_shmem_frame'':
202464, ''handle'': 1}
VirqClient.virqReceived> 4
vif-bridge down vif=vif5.0 domain=vpn1dc1 mac=02:00:00:01:01:00
bridge=xen-br0
interface vif5.0 does not exist!
vif-bridge down vif=vif5.1 domain=vpn1dc1 mac=02:00:00:01:01:01
bridge=xen-br1
interface vif5.1 does not exist!

File xend-debug.log not changed so no update needed.
[root@xenpub1dc1 log]# ls
acpid            boot.log.1  cron.2   mail       messages    prelink.log
samba     spooler    wtmp            xfrd.log
anaconda.log     boot.log.2  cups     maillog    messages.1  rpmpkgs
secure    spooler.1  xend-debug.log  Xorg.0.log
anaconda.syslog  cron        dmesg    maillog.1  messages.2  rpmpkgs.1
secure.1  spooler.2  xend.log        Xorg.0.log.old
boot.log         cron.1      lastlog  maillog.2  ppp         rpmpkgs.2
secure.2  vbox       xend.log.1      yum.log
[root@xenpub1dc1 log]# joe xend.log
Processing ''/etc/joe/joerc''...done
Processing ''/etc/joe/joerc''...done









































    I    xend.log
Row 6157 Col 100 11:07  Ctrl-K H for help
[2006-01-28 22:06:36 xend] DEBUG (XendDomain:244) XendDomain>reap>
domain died name=vpn1dc1 id=2
[2006-01-28 22:06:36 xend] DEBUG (XendDomain:247) XendDomain>reap>
shutdown id=2 reason=poweroff
[2006-01-28 22:06:36 xend] DEBUG (XendDomain:487)
domain_restart_schedule> 2 poweroff 0
[2006-01-28 22:06:36 xend] INFO (XendDomain:568) Destroying domain:
name=vpn1dc1
[2006-01-28 22:06:36 xend] DEBUG (XendDomainInfo:634) Closing console,
domain 2
[2006-01-28 22:06:36 xend] INFO (XendRoot:113) EVENT> xend.domain.exit
[''vpn1dc1'', ''2'',
''poweroff'']
[2006-01-28 22:06:36 xend] INFO (XendRoot:113) EVENT>
xend.domain.destroy [''vpn1dc1'', ''2'']
[2006-01-28 22:06:38 xend] DEBUG (XendDomain:244) XendDomain>reap>
domain died name=vpn1dc1 id=2
[2006-01-28 22:06:38 xend] DEBUG (XendDomain:247) XendDomain>reap>
shutdown id=2 reason=poweroff
[2006-01-28 22:06:38 xend] DEBUG (XendDomain:487)
domain_restart_schedule> 2 poweroff 0
[2006-01-28 22:06:38 xend] INFO (XendDomain:568) Destroying domain:
name=vpn1dc1
[2006-01-28 22:06:38 xend] DEBUG (XendDomainInfo:634) Closing console,
domain 2
[2006-01-28 22:06:38 xend] INFO (XendRoot:113) EVENT> xend.domain.exit
[''vpn1dc1'', ''2'',
''poweroff'']
[2006-01-28 22:06:38 xend] INFO (XendRoot:113) EVENT>
xend.domain.destroy [''vpn1dc1'', ''2'']
[2006-01-28 22:06:40 xend] DEBUG (XendDomain:244) XendDomain>reap>
domain died name=vpn1dc1 id=2
[2006-01-28 22:06:40 xend] DEBUG (XendDomain:247) XendDomain>reap>
shutdown id=2 reason=poweroff
[2006-01-28 22:06:40 xend] DEBUG (XendDomain:487)
domain_restart_schedule> 2 poweroff 0
[2006-01-28 22:06:40 xend] INFO (XendDomain:568) Destroying domain:
name=vpn1dc1
[2006-01-28 22:06:40 xend] DEBUG (XendDomainInfo:634) Closing console,
domain 2
[2006-01-28 22:06:40 xend] INFO (XendRoot:113) EVENT> xend.domain.exit
[''vpn1dc1'', ''2'',
''poweroff'']
[2006-01-28 22:06:40 xend] INFO (XendRoot:113) EVENT>
xend.domain.destroy [''vpn1dc1'', ''2'']
[2006-01-28 22:06:42 xend] DEBUG (XendDomain:244) XendDomain>reap>
domain died name=vpn1dc1 id=2
[2006-01-28 22:06:42 xend] DEBUG (XendDomain:247) XendDomain>reap>
shutdown id=2 reason=poweroff
[2006-01-28 22:06:42 xend] DEBUG (XendDomain:487)
domain_restart_schedule> 2 poweroff 0
[2006-01-28 22:06:42 xend] INFO (XendDomain:568) Destroying domain:
name=vpn1dc1
[2006-01-28 22:06:42 xend] DEBUG (XendDomainInfo:634) Closing console,
domain 2
[2006-01-28 22:06:42 xend] INFO (XendRoot:113) EVENT> xend.domain.exit
[''vpn1dc1'', ''2'',
''poweroff'']
[2006-01-28 22:06:42 xend] INFO (XendRoot:113) EVENT>
xend.domain.destroy [''vpn1dc1'', ''2'']
[2006-01-28 22:06:44 xend] DEBUG (XendDomain:244) XendDomain>reap>
domain died name=vpn1dc1 id=2
[2006-01-28 22:06:44 xend] DEBUG (XendDomain:247) XendDomain>reap>
shutdown id=2 reason=poweroff
[2006-01-28 22:06:44 xend] DEBUG (XendDomain:487)
domain_restart_schedule> 2 poweroff 0
[2006-01-28 22:06:44 xend] INFO (XendDomain:568) Destroying domain:
name=vpn1dc1
[2006-01-28 22:06:44 xend] DEBUG (XendDomainInfo:634) Closing console,
domain 2
[2006-01-28 22:06:44 xend] INFO (XendRoot:113) EVENT> xend.domain.exit
[''vpn1dc1'', ''2'',
''poweroff'']
[2006-01-28 22:06:44 xend] INFO (XendRoot:113) EVENT>
xend.domain.destroy [''vpn1dc1'', ''2'']
[2006-01-28 22:06:46 xend] DEBUG (XendDomain:244) XendDomain>reap>
domain died name=vpn1dc1 id=2
[2006-01-28 22:06:46 xend] DEBUG (XendDomain:247) XendDomain>reap>
shutdown id=2 reason=poweroff
[2006-01-28 22:06:46 xend] DEBUG (XendDomain:487)
domain_restart_schedule> 2 poweroff 0
[2006-01-28 22:06:46 xend] INFO (XendDomain:568) Destroying domain:
name=vpn1dc1
[2006-01-28 22:06:46 xend] DEBUG (XendDomainInfo:634) Closing console,
domain 2
[2006-01-28 22:06:46 xend] INFO (XendRoot:113) EVENT> xend.domain.exit
[''vpn1dc1'', ''2'',
''poweroff'']
[2006-01-28 22:06:46 xend] INFO (XendRoot:113) EVENT>
xend.domain.destroy [''vpn1dc1'', ''2'']
[2006-01-28 22:06:48 xend] DEBUG (XendDomain:244) XendDomain>reap>
domain died name=vpn1dc1 id=2
[2006-01-28 22:06:48 xend] DEBUG (XendDomain:247) XendDomain>reap>
shutdown id=2 reason=poweroff
[2006-01-28 22:06:48 xend] DEBUG (XendDomain:487)
domain_restart_schedule> 2 poweroff 0
[2006-01-28 22:06:48 xend] INFO (XendDomain:568) Destroying domain:
name=vpn1dc1
[2006-01-28 22:06:48 xend] DEBUG (XendDomainInfo:634) Closing console,
domain 2
[2006-01-28 22:06:48 xend] INFO (XendRoot:113) EVENT> xend.domain.exit
[''vpn1dc1'', ''2'',
''poweroff'']
[2006-01-28 22:06:48 xend] INFO (XendRoot:113) EVENT>
xend.domain.destroy [''vpn1dc1'', ''2'']
[2006-01-28 22:06:50 xend] DEBUG (XendDomain:244) XendDomain>reap>
domain died name=vpn1dc1 id=2
[2006-01-28 22:06:50 xend] DEBUG (XendDomain:247) XendDomain>reap>
shutdown id=2 reason=poweroff
[2006-01-28 22:06:50 xend] DEBUG (XendDomain:487)
domain_restart_schedule> 2 poweroff 0
[2006-01-28 22:06:50 xend] INFO (XendDomain:568) Destroying domain:
name=vpn1dc1
[2006-01-28 22:06:50 xend] DEBUG (XendDomainInfo:634) Closing console,
domain 2
[2006-01-28 22:06:50 xend] INFO (XendRoot:113) EVENT> xend.domain.exit
[''vpn1dc1'', ''2'',
''poweroff'']
[2006-01-28 22:06:50 xend] INFO (XendRoot:113) EVENT>
xend.domain.destroy [''vpn1dc1'', ''2'']
[2006-01-28 22:06:52 xend] DEBUG (XendDomain:244) XendDomain>reap>
domain died name=vpn1dc1 id=2
[2006-01-28 22:06:52 xend] DEBUG (XendDomain:247) XendDomain>reap>
shutdown id=2 reason=poweroff
[2006-01-28 22:06:52 xend] DEBUG (XendDomain:487)
domain_restart_schedule> 2 poweroff 0
[2006-01-28 22:06:52 xend] INFO (XendDomain:568) Destroying domain:
name=vpn1dc1
[2006-01-28 22:06:52 xend] DEBUG (XendDomainInfo:634) Closing console,
domain 2
[2006-01-28 22:06:52 xend] INFO (XendRoot:113) EVENT> xend.domain.exit
[''vpn1dc1'', ''2'',
''poweroff'']
[2006-01-28 22:06:52 xend] INFO (XendRoot:113) EVENT>
xend.domain.destroy [''vpn1dc1'', ''2'']
[2006-01-28 22:06:54 xend] DEBUG (XendDomain:244) XendDomain>reap>
domain died name=vpn1dc1 id=2
[2006-01-28 22:06:54 xend] DEBUG (XendDomain:247) XendDomain>reap>
shutdown id=2 reason=poweroff
[2006-01-28 22:06:54 xend] DEBUG (XendDomain:487)
domain_restart_schedule> 2 poweroff 0
[2006-01-28 22:06:54 xend] INFO (XendDomain:568) Destroying domain:
name=vpn1dc1
[2006-01-28 22:06:54 xend] DEBUG (XendDomainInfo:634) Closing console,
domain 2
[2006-01-28 22:06:54 xend] INFO (XendRoot:113) EVENT> xend.domain.exit
[''vpn1dc1'', ''2'',
''poweroff'']
[2006-01-28 22:06:54 xend] INFO (XendRoot:113) EVENT>
xend.domain.destroy [''vpn1dc1'', ''2'']
[2006-01-28 22:06:56 xend] DEBUG (XendDomain:244) XendDomain>reap>
domain died name=vpn1dc1 id=2
[2006-01-28 22:06:56 xend] DEBUG (XendDomain:247) XendDomain>reap>
shutdown id=2 reason=poweroff
[2006-01-28 22:06:56 xend] DEBUG (XendDomain:487)
domain_restart_schedule> 2 poweroff 0
[2006-01-28 22:06:56 xend] INFO (XendDomain:568) Destroying domain:
name=vpn1dc1
[2006-01-28 22:06:56 xend] DEBUG (XendDomainInfo:634) Closing console,
domain 2
[2006-01-28 22:06:56 xend] INFO (XendRoot:113) EVENT> xend.domain.exit
[''vpn1dc1'', ''2'',
''poweroff'']
[2006-01-28 22:06:56 xend] INFO (XendRoot:113) EVENT>
xend.domain.destroy [''vpn1dc1'', ''2'']
[2006-01-28 22:06:57 xend] DEBUG (XendDomain:244) XendDomain>reap>
domain died name=vpn1dc1 id=2
[2006-01-28 22:06:57 xend] DEBUG (XendDomain:247) XendDomain>reap>
shutdown id=2 reason=poweroff
[2006-01-28 22:06:57 xend] DEBUG (XendDomain:487)
domain_restart_schedule> 2 poweroff 0
[2006-01-28 22:06:57 xend] INFO (XendDomain:568) Destroying domain:
name=vpn1dc1 
etc., etc., etc.

Here are the configuration files for the two domUs:
#  -*- mode: python; -*-
#===========================================================================#
Python configuration setup for ''xm create''.
# This script sets the parameters used when a domain is created using
''xm create''.
# You use a separate script for each domain you want to create, or 
# you can set the parameters for the domain on the xm command line.
#===========================================================================
#----------------------------------------------------------------------------
# Kernel image file.
kernel = "/boot/vmlinuz-2.4.30-xenUVPN"

# Optional ramdisk.
#ramdisk = "/boot/initrd.gz"

# The domain build function. Default is ''linux''.
#builder=''linux''

# Initial memory allocation (in megabytes) for the new domain.
memory = 96

# A name for your domain. All domains must have different names.
name = "niagras1"

# Which CPU to start domain on? 
#cpu = -1   # leave to Xen to pick

#----------------------------------------------------------------------------
# Define network interfaces.

# Number of network interfaces. Default is 1.
nics=2

# Optionally define mac and/or bridge for the network interfaces.
# Random MACs are assigned if not given.
#vif = [ ''mac=aa:00:00:00:00:11, bridge=xen-br0'' ]

vif = [''mac=02:00:00:01:02:00, bridge=xen-br0'' ]

#----------------------------------------------------------------------------
# Define the disk devices you want the domain to have access to, and
# what you want them accessible as.
# Each disk entry is of the form phy:UNAME,DEV,MODE
# where UNAME is the device, DEV is the device name the domain will see,
# and MODE is r for read-only, w for read-write.

#disk = [ ''phy:hda1,hda1,w'' ]

disk = [ ''phy:VG1/niagras1root,sda1,w'',
''phy:VG1/niagras1swap,sda2,w'' ]

#----------------------------------------------------------------------------
# Set the kernel command line for the new domain.
# You only need to define the IP parameters and hostname if the
domain''s
# IP config doesn''t, e.g. in ifcfg-eth0 or via DHCP.
# You can use ''extra'' to set the runlevel and custom
environment
# variables used by custom rc scripts (e.g. VMID=, usr= ).

# Set if you want dhcp to allocate the IP address.
#dhcp="dhcp"
# Set netmask.
#netmask# Set default gateway.
#gateway# Set the hostname.
#hostname= "vm%d" % vmid

# Set root device.
root = "/dev/sda1 ro"

# Root device for nfs.
#root = "/dev/nfs"
# The nfs server.
#nfs_server = ''169.254.1.0''  
# Root directory on the nfs server.
#nfs_root   = ''/full/path/to/root/directory''

# Sets runlevel 4.
#extra = "4"

#----------------------------------------------------------------------------
# Set according to whether you want the domain restarted when it exits.
# The default is ''onreboot'', which restarts the domain when it
shuts
down
# with exit code reboot.
# Other values are ''always'', and ''never''.

#restart = ''onreboot''

#===========================================================================

#  -*- mode: python; -*-
#===========================================================================#
Python configuration setup for ''xm create''.
# This script sets the parameters used when a domain is created using
''xm create''.
# You use a separate script for each domain you want to create, or 
# you can set the parameters for the domain on the xm command line.
#===========================================================================
#----------------------------------------------------------------------------
# Kernel image file.
kernel = "/boot/vmlinuz-2.4.30-xenUVPN"

# Optional ramdisk.
#ramdisk = "/boot/initrd.gz"

# The domain build function. Default is ''linux''.
#builder=''linux''

# Initial memory allocation (in megabytes) for the new domain.
memory = 96

# A name for your domain. All domains must have different names.
name = "vpn1dc1"

# Which CPU to start domain on? 
#cpu = -1   # leave to Xen to pick

#----------------------------------------------------------------------------
# Define network interfaces.

# Number of network interfaces. Default is 1.
nics=2

# Optionally define mac and/or bridge for the network interfaces.
# Random MACs are assigned if not given.
#vif = [ ''mac=aa:00:00:00:00:11, bridge=xen-br0'' ]

vif = [''mac=02:00:00:01:01:00, bridge=xen-br0'',
''mac=02:00:00:01:01:01,
bridge=xen-br1'' ]

#----------------------------------------------------------------------------
# Define the disk devices you want the domain to have access to, and
# what you want them accessible as.
# Each disk entry is of the form phy:UNAME,DEV,MODE
# where UNAME is the device, DEV is the device name the domain will see,
# and MODE is r for read-only, w for read-write.

#disk = [ ''phy:hda1,hda1,w'' ]

disk = [ ''phy:VG1/vpn1dc1root,sda1,w'',
''phy:VG1/vpn1dc1swap,sda2,w'' ]

#----------------------------------------------------------------------------
# Set the kernel command line for the new domain.
# You only need to define the IP parameters and hostname if the
domain''s
# IP config doesn''t, e.g. in ifcfg-eth0 or via DHCP.
# You can use ''extra'' to set the runlevel and custom
environment
# variables used by custom rc scripts (e.g. VMID=, usr= ).

# Set if you want dhcp to allocate the IP address.
#dhcp="dhcp"
# Set netmask.
#netmask# Set default gateway.
#gateway# Set the hostname.
#hostname= "vm%d" % vmid

# Set root device.
root = "/dev/sda1 ro"

# Root device for nfs.
#root = "/dev/nfs"
# The nfs server.
#nfs_server = ''169.254.1.0''  
# Root directory on the nfs server.
#nfs_root   = ''/full/path/to/root/directory''

# Sets runlevel 4.
#extra = "4"

#----------------------------------------------------------------------------
# Set according to whether you want the domain restarted when it exits.
# The default is ''onreboot'', which restarts the domain when it
shuts
down
# with exit code reboot.
# Other values are ''always'', and ''never''.

#restart = ''onreboot''

#===========================================================================

I''m stumped but it sure looks serious.  Have I produced some dreadful
misconfiguration? This host is being shipped to a remote data center on
Monday and goes into production in a week.  Any help would be greatly
appreciated.  Thanks - John
-- 
John A. Sullivan III
Open Source Development Corporation
+1 207-985-7880
jsullivan@opensourcedevel.com

Financially sustainable open source development
http://www.opensourcedevel.com
-- 
John A. Sullivan III
Strategist
Senior Consultant
Atlas Group
+1 207-985-7880
jsullivan@atlasgroup.net


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Pardon me for a long detailed post and for pestering the developer list
but this looks pretty serious.  The bottom line is that, when I attempt
to shut down or reboot my simple xen-2.0.7 server running FC3 with two
domUs, shutting down xendomains appears to send python into an infinite
loop.  This may be due to a serious misconfiguration on my part but it
is certainly a big problem.

If we shutdown the domUs manually first, the system reboots perfectly
fine.

I''ll describe some history and include lots of log information.  The
configuration is slightly unusual.  Both domUs are VPN gateways running
the 2.4.30 kernel with openswan and openvpn.  Because of the openvpn,
but are privileged domains but the only device driver configured in the
kernel is the openvpn tun/tap adapter.

Since we have three interfaces, we slightly modified the network script.
It is included below.

We first noticed problems during shutdown in that the last domU to
shutdown crashed very nastily when closing openvpn.  We could see this
even when manually stopping the domUs.  We surmised that the problem was
we had tun0 and tun1 defined in both domains and killing them in one
must be killing them in another.  We renamed those tun adapters in the
second domain to tun10 and tun11 and that problem went away.

We still noticed very long reboot times.  The hang seemed to be at
killall but the "logjam" seemed to break after xendomains errored out.
We assumed this was because killall killed something that xendomains
needed to shutdown gracefully so we added explicit K scripts in rc0.d
and rc6.d for xend and xendomains.

At first this seemed to work but it must have been only coincidental
because the problem returned.  We then realized we had brain cramped and
shutdown xend before xendomains.  We thought for sure we had found the
problem and put them in the proper order but, alas, the problem did not
go away.

To gain a little more insight into exactly what was happening, we
manually ran each kill script in rc6.d.  When we ran K78xendomains stop,
we saw the problem, or at least many aspect of it.
>From a xen console, we saw that the first domU had cleanly shutdown but
the second one was still listed howbeit with zero memory.

The hard disks were thrashing madly and top showed python consuming 99.9
percent CPU utilization.  Most of the time, during reboot, we do not see
the madly thrashing disks.  They do so for a moment as one might expect
when shutting down the domains but then just blip occasionally
thereafter.

Here is what we see in the logs:
>From messages (ntpd was the last service stopped before xendomains):
Jan 28 22:05:49 xenpub1dc1 ntpd[3652]: ntpd exiting on signal 15
Jan 28 22:05:49 xenpub1dc1 ntpd: ntpd shutdown succeeded
Jan 28 22:06:22 xenpub1dc1 kernel: device vif1.0 left promiscuous mode
Jan 28 22:06:22 xenpub1dc1 kernel: xen-br0: port 2(vif1.0) entering
disabled state
Jan 28 22:06:22 xenpub1dc1 kernel: xen-br0: port 2(vif1.0) entering
disabled state
Jan 28 22:06:22 xenpub1dc1 kernel: device vif1.1 left promiscuous mode
Jan 28 22:06:22 xenpub1dc1 kernel: xen-br0: port 3(vif1.1) entering
disabled state
Jan 28 22:06:22 xenpub1dc1 kernel: xen-br0: port 3(vif1.1) entering
disabled state
Jan 28 22:06:23 xenpub1dc1 net.agent[4714]: remove event not handled
Jan 28 22:06:23 xenpub1dc1 net.agent[4723]: remove event not handled
Jan 28 22:06:23 xenpub1dc1 net.agent[4737]: remove event not handled
Jan 28 22:06:23 xenpub1dc1 net.agent[4746]: remove event not handled
Jan 28 22:06:24 xenpub1dc1 kernel: device vif2.0 left promiscuous mode
Jan 28 22:06:24 xenpub1dc1 kernel: xen-br0: port 4(vif2.0) entering
disabled state
Jan 28 22:06:24 xenpub1dc1 kernel: xen-br0: port 4(vif2.0) entering
disabled state
Jan 28 22:06:24 xenpub1dc1 kernel: device vif2.1 left promiscuous mode
Jan 28 22:06:24 xenpub1dc1 kernel: xen-br1: port 2(vif2.1) entering
disabled state
Jan 28 22:06:24 xenpub1dc1 kernel: xen-br1: port 2(vif2.1) entering
disabled state
Jan 28 22:06:24 xenpub1dc1 net.agent[4774]: remove event not handled
Jan 28 22:06:24 xenpub1dc1 net.agent[4766]: remove event not handled
Jan 28 22:06:24 xenpub1dc1 net.agent[4789]: remove event not handled
Jan 28 22:06:24 xenpub1dc1 net.agent[4798]: remove event not handled

Here is the complete xend-debug.log:
network start bridge=xen-br0 netdev=eth0 antispoof=no
VIRTUAL MEMORY ARRANGEMENT:
 Loaded kernel: c0100000->c0319394
 Init. ramdisk: c031a000->c031a000
 Phys-Mach map: c031a000->c0332000
 Page tables:   c0332000->c0334000
 Start info:    c0334000->c0335000
 Boot stack:    c0335000->c0336000
 TOTAL:         c0000000->c0400000
 ENTRY ADDRESS: c0100000
vif-bridge up vif=vif1.0 domain=niagras1 mac=02:00:00:01:02:00
bridge=xen-br0
vif-bridge up vif=vif1.1 domain=niagras1 mac=aa:00:00:58:4a:a7
bridge=xen-br0
VIRTUAL MEMORY ARRANGEMENT:
 Loaded kernel: c0100000->c0319394
 Init. ramdisk: c031a000->c031a000
 Phys-Mach map: c031a000->c0332000
 Page tables:   c0332000->c0334000
 Start info:    c0334000->c0335000
 Boot stack:    c0335000->c0336000
 TOTAL:         c0000000->c0400000
 ENTRY ADDRESS: c0100000
vif-bridge up vif=vif2.0 domain=vpn1dc1 mac=02:00:00:01:01:00
bridge=xen-br0
vif-bridge up vif=vif2.1 domain=vpn1dc1 mac=02:00:00:01:01:01
bridge=xen-br1
recv_fe_driver_status> {''status'': 1}

recv_fe_driver_status>

recv_fe_interface_connect {''tx_shmem_frame'': 44763,
''rx_shmem_frame'':
44764, ''handle'': 0}

recv_fe_interface_connect {''tx_shmem_frame'': 44767,
''rx_shmem_frame'':
44768, ''handle'': 1}
recv_fe_driver_status> {''status'': 1}

recv_fe_driver_status>

recv_fe_interface_connect {''tx_shmem_frame'': 69339,
''rx_shmem_frame'':
69340, ''handle'': 0}

recv_fe_interface_connect {''tx_shmem_frame'': 69343,
''rx_shmem_frame'':
69344, ''handle'': 1}
{}
{}
VirqClient.virqReceived> 4
vif-bridge down vif=vif2.0 domain=vpn1dc1 mac=02:00:00:01:01:00
bridge=xen-br0
vif-bridge down vif=vif2.1 domain=vpn1dc1 mac=02:00:00:01:01:01
bridge=xen-br1
VirqClient.virqReceived> 4
vif-bridge down vif=vif1.0 domain=niagras1 mac=02:00:00:01:02:00
bridge=xen-br0
vif-bridge down vif=vif1.1 domain=niagras1 mac=aa:00:00:26:24:c4
bridge=xen-br0
Traceback (most recent call last):
  File "/usr/lib/python2.3/site-packages/twisted/protocols/basic.py",
line 229, in dataReceived
    return self.rawDataReceived(data)
  File "/usr/lib/python2.3/site-packages/twisted/protocols/http.py",
line 1015, in rawDataReceived
    self.allContentReceived()
  File "/usr/lib/python2.3/site-packages/twisted/protocols/http.py",
line 1006, in allContentReceived
    req.requestReceived(command, path, version)
  File "/usr/lib/python2.3/site-packages/twisted/protocols/http.py",
line 557, in requestReceived
    self.process()
--- <exception caught here> ---
  File "/usr/lib/python2.3/site-packages/twisted/web/server.py", line
165, in process
    self.render(resrc)
  File "/usr/lib/python2.3/site-packages/twisted/web/server.py", line
172, in render
    body = resrc.render(self)
  File "/usr/lib/python2.3/site-packages/twisted/web/resource.py",
line
201, in render
    return m(request)
  File "/usr/lib/python/xen/xend/server/SrvDomain.py", line 180, in
render_POST
    return self.perform(req)
  File "/usr/lib/python/xen/xend/server/SrvBase.py", line 103, in
perform
    return self._perform(op, op_method, req)
  File "/usr/lib/python/xen/xend/server/SrvBase.py", line 109, in
_perform
    return self._perform_err(err, op, req)
  File "/usr/lib/python/xen/xend/server/SrvBase.py", line 107, in
_perform
    val = op_method(op, req)
  File "/usr/lib/python/xen/xend/server/SrvDomain.py", line 47, in
op_shutdown
    val = fn(req.args, {''dom'': self.dom.id})
  File "/usr/lib/python/xen/xend/Args.py", line 145, in __call__
    return self.call_with_form_args(self.fn, fargs, xargs=xargs)
  File "/usr/lib/python/xen/xend/Args.py", line 121, in
call_with_form_args
    return fn(*params, **keys)
  File "/usr/lib/python/xen/xend/XendDomain.py", line 477, in
domain_shutdown
    val = xend.domain_shutdown(dominfo.id, reason, key)
  File "/usr/lib/python/xen/xend/server/SrvDaemon.py", line 723, in
domain_shutdown
    ctrl = self.domainCF.getController(dom)
  File "/usr/lib/python/xen/xend/server/controller.py", line 323, in
getController
    ctrl = self.createController(dom)
  File "/usr/lib/python/xen/xend/server/domain.py", line 20, in
createController
    return DomainController(self, dom)
  File "/usr/lib/python/xen/xend/server/domain.py", line 38, in
__init__
    self.registerChannel()
  File "/usr/lib/python/xen/xend/server/controller.py", line 220, in
registerChannel
    self.channel = self.channelFactory.domChannel(self.dom)
  File "/usr/lib/python/xen/xend/server/channel.py", line 83, in
domChannel
    remote_port=remote_port)
  File "/usr/lib/python/xen/xend/server/channel.py", line 253, in
__init__
    remote_port=remote_port)
  File "/usr/lib/python/xen/xend/server/channel.py", line 137, in
createPort
    remote_port=int(remote_port))
xen.lowlevel.xu.PortError: Failed to map domain control interface
/usr/lib/python2.3/site-packages/twisted/web/util.py:290: FutureWarning:
hex()/oct() of negative int will return a signed string in Python 2.4
and up
  return ''''''<div
class="instance"><span class="instanceName">%s
instance
@ %s</span>
/usr/lib/python2.3/site-packages/twisted/web/util.py:266: FutureWarning:
hex()/oct() of negative int will return a signed string in Python 2.4
and up
  w(''<div class="dict"><span
class="heading">Dictionary instance @ %
s</span>'' % hex(id(d)))
/usr/lib/python2.3/site-packages/twisted/web/util.py:279: FutureWarning:
hex()/oct() of negative int will return a signed string in Python 2.4
and up
  w(''<div class="list"><span
class="heading">List instance @ %s</span>''
% hex(id(l)))

 Init. ramdisk: c031a000->c031a000
 Phys-Mach map: c031a000->c0332000
 Page tables:   c0332000->c0334000
 Start info:    c0334000->c0335000
 Boot stack:    c0335000->c0336000
 TOTAL:         c0000000->c0400000
 ENTRY ADDRESS: c0100000
vif-bridge up vif=vif7.0 domain=vpn1dc1 mac=02:00:00:01:01:00
bridge=xen-br0
vif-bridge up vif=vif7.1 domain=vpn1dc1 mac=02:00:00:01:01:01
bridge=xen-br1
recv_fe_driver_status> {''status'': 1}

recv_fe_driver_status>

recv_fe_interface_connect {''tx_shmem_frame'': 88257,
''rx_shmem_frame'':
88258, ''handle'': 0}

recv_fe_interface_connect {''tx_shmem_frame'': 88261,
''rx_shmem_frame'':
88262, ''handle'': 1}
VIRTUAL MEMORY ARRANGEMENT:
 Loaded kernel: c0100000->c0319394
 Init. ramdisk: c031a000->c031a000
 Phys-Mach map: c031a000->c0332000
 Page tables:   c0332000->c0334000
 Start info:    c0334000->c0335000
 Boot stack:    c0335000->c0336000
 TOTAL:         c0000000->c0400000
 ENTRY ADDRESS: c0100000
vif-bridge up vif=vif8.0 domain=niagras1 mac=02:00:00:01:02:00
bridge=xen-br0
vif-bridge up vif=vif8.1 domain=niagras1 mac=aa:00:00:53:7a:5b
bridge=xen-br0
recv_fe_driver_status> {''status'': 1}

recv_fe_driver_status>

recv_fe_interface_connect {''tx_shmem_frame'': 185025,
''rx_shmem_frame'':
185026, ''handle'': 0}

recv_fe_interface_connect {''tx_shmem_frame'': 185029,
''rx_shmem_frame'':
185030, ''handle'': 1}
{}
{}
VirqClient.virqReceived> 4
vif-bridge down vif=vif8.0 domain=niagras1 mac=02:00:00:01:02:00
bridge=xen-br0
vif-bridge down vif=vif8.1 domain=niagras1 mac=aa:00:00:53:7a:5b
bridge=xen-br0
VirqClient.virqReceived> 4
vif-bridge down vif=vif7.0 domain=vpn1dc1 mac=02:00:00:01:01:00
bridge=xen-br0
vif-bridge down vif=vif7.1 domain=vpn1dc1 mac=02:00:00:01:01:01
bridge=xen-br1
-bridge down vif=vif4.1 domain=vpn1dc1 mac=02:00:00:01:01:01
bridge=xen-br1
interface vif4.1 does not exist!
VIRTUAL MEMORY ARRANGEMENT:
 Loaded kernel: c0100000->c02c9df4
 Init. ramdisk: c02ca000->c02ca000
 Phys-Mach map: c02ca000->c02e2000
 Page tables:   c02e2000->c02e4000
 Start info:    c02e4000->c02e5000
 Boot stack:    c02e5000->c02e6000
 TOTAL:         c0000000->c0400000
 ENTRY ADDRESS: c0100000
vif-bridge up vif=vif5.0 domain=vpn1dc1 mac=02:00:00:01:01:00
bridge=xen-br0
interface vif5.0 does not exist!
vif-bridge up vif=vif5.1 domain=vpn1dc1 mac=02:00:00:01:01:01
bridge=xen-br1
interface vif5.1 does not exist!
recv_fe_driver_status> {''status'': 1}

recv_fe_driver_status>

recv_fe_interface_connect {''tx_shmem_frame'': 202459,
''rx_shmem_frame'':
202460, ''handle'': 0}

recv_fe_interface_connect {''tx_shmem_frame'': 202463,
''rx_shmem_frame'':
202464, ''handle'': 1}
VirqClient.virqReceived> 4
vif-bridge down vif=vif5.0 domain=vpn1dc1 mac=02:00:00:01:01:00
bridge=xen-br0
interface vif5.0 does not exist!
vif-bridge down vif=vif5.1 domain=vpn1dc1 mac=02:00:00:01:01:01
bridge=xen-br1
interface vif5.1 does not exist!

Here is the information from xend.log:
Jan 28 22:05:49 xenpub1dc1 ntpd[3652]: ntpd exiting on signal 15
Jan 28 22:05:49 xenpub1dc1 ntpd: ntpd shutdown succeeded
Jan 28 22:06:22 xenpub1dc1 kernel: device vif1.0 left promiscuous mode
Jan 28 22:06:22 xenpub1dc1 kernel: xen-br0: port 2(vif1.0) entering
disabled state
Jan 28 22:06:22 xenpub1dc1 kernel: xen-br0: port 2(vif1.0) entering
disabled state
Jan 28 22:06:22 xenpub1dc1 kernel: device vif1.1 left promiscuous mode
Jan 28 22:06:22 xenpub1dc1 kernel: xen-br0: port 3(vif1.1) entering
disabled state
Jan 28 22:06:22 xenpub1dc1 kernel: xen-br0: port 3(vif1.1) entering
disabled state
Jan 28 22:06:23 xenpub1dc1 net.agent[4714]: remove event not handled
Jan 28 22:06:23 xenpub1dc1 net.agent[4723]: remove event not handled
Jan 28 22:06:23 xenpub1dc1 net.agent[4737]: remove event not handled
Jan 28 22:06:23 xenpub1dc1 net.agent[4746]: remove event not handled
Jan 28 22:06:24 xenpub1dc1 kernel: device vif2.0 left promiscuous mode
Jan 28 22:06:24 xenpub1dc1 kernel: xen-br0: port 4(vif2.0) entering
disabled state
Jan 28 22:06:24 xenpub1dc1 kernel: xen-br0: port 4(vif2.0) entering
disabled state
Jan 28 22:06:24 xenpub1dc1 kernel: device vif2.1 left promiscuous mode
Jan 28 22:06:24 xenpub1dc1 kernel: xen-br1: port 2(vif2.1) entering
disabled state
Jan 28 22:06:24 xenpub1dc1 kernel: xen-br1: port 2(vif2.1) entering
disabled state
Jan 28 22:06:24 xenpub1dc1 net.agent[4774]: remove event not handled
Jan 28 22:06:24 xenpub1dc1 net.agent[4766]: remove event not handled
Jan 28 22:06:24 xenpub1dc1 net.agent[4789]: remove event not handled
Jan 28 22:06:24 xenpub1dc1 net.agent[4798]: remove event not handled
Jan 28 22:18:03 xenpub1dc1 shutdown: shutting down for system reboot
Jan 28 22:18:03 xenpub1dc1 init: Switching to runlevel: 6
Jan 28 22:18:03 xenpub1dc1 login(pam_unix)[3781]: session closed for
user root
Jan 28 22:18:03 xenpub1dc1 login(pam_unix)[3779]: session closed for
user root
Jan 28 22:18:03 xenpub1dc1 login(pam_unix)[3780]: session closed for
user root
Jan 28 22:18:04 xenpub1dc1 xendomains: Shutting down all Xen domains:
Jan 28 22:18:05 xenpub1dc1 xendomains: Error: Internal Server Error
Jan 28 22:18:05 xenpub1dc1 xendomains:  failed
Jan 28 22:18:05 xenpub1dc1 xendomains:
Jan 28 22:18:05 xenpub1dc1 rc: Stopping xendomains:  failed
Jan 28 22:18:05 xenpub1dc1 kernel: Kernel logging (proc) stopped.
Jan 28 22:18:05 xenpub1dc1 kernel: Kernel log daemon terminating.
Jan 28 22:18:06 xenpub1dc1 syslog: klogd shutdown succeeded
Jan 28 22:18:06 xenpub1dc1 exiting on signal 15
Jan 28 22:20:06 xenpub1dc1 syslogd 1.4.1: restart.
Jan 28 22:20:06 xenpub1dc1 syslog: syslogd startup succeeded
Jan 28 22:20:06 xenpub1dc1 kernel: klogd 1.4.1, log source = /proc/kmsg
started.
Jan 28 22:20:06 xenpub1dc1 kernel: Linux version 2.6.11.12-xen0
(root@xenpub1dc1.atlasgroup.net) (gcc version 3.4.4 20050721 (Red Hat
3.4.4-2)) #4 Thu Jan
File messages not changed so no update needed.
[root@xenpub1dc1 log]# ls
acpid            boot.log.1  cron.2   mail       messages    prelink.log
samba     spooler    wtmp            xfrd.log
anaconda.log     boot.log.2  cups     maillog    messages.1  rpmpkgs
secure    spooler.1  xend-debug.log  Xorg.0.log
anaconda.syslog  cron        dmesg    maillog.1  messages.2  rpmpkgs.1
secure.1  spooler.2  xend.log        Xorg.0.log.old
boot.log         cron.1      lastlog  maillog.2  ppp         rpmpkgs.2
secure.2  vbox       xend.log.1      yum.log
[root@xenpub1dc1 log]# joe xend-debug.log
Processing ''/etc/joe/joerc''...done
Processing ''/etc/joe/joerc''...done









































    I    xend-debug.log
Row 171  Col 1   11:02  Ctrl-K H for help
recv_fe_driver_status>

recv_fe_interface_connect {''tx_shmem_frame'': 185025,
''rx_shmem_frame'':
185026, ''handle'': 0}

recv_fe_interface_connect {''tx_shmem_frame'': 185029,
''rx_shmem_frame'':
185030, ''handle'': 1}
{}
{}
VirqClient.virqReceived> 4
vif-bridge down vif=vif8.0 domain=niagras1 mac=02:00:00:01:02:00
bridge=xen-br0
vif-bridge down vif=vif8.1 domain=niagras1 mac=aa:00:00:53:7a:5b
bridge=xen-br0
VirqClient.virqReceived> 4
vif-bridge down vif=vif7.0 domain=vpn1dc1 mac=02:00:00:01:01:00
bridge=xen-br0
vif-bridge down vif=vif7.1 domain=vpn1dc1 mac=02:00:00:01:01:01
bridge=xen-br1
-bridge down vif=vif4.1 domain=vpn1dc1 mac=02:00:00:01:01:01
bridge=xen-br1
interface vif4.1 does not exist!
VIRTUAL MEMORY ARRANGEMENT:
 Loaded kernel: c0100000->c02c9df4
 Init. ramdisk: c02ca000->c02ca000
 Phys-Mach map: c02ca000->c02e2000
 Page tables:   c02e2000->c02e4000
 Start info:    c02e4000->c02e5000
 Boot stack:    c02e5000->c02e6000
 TOTAL:         c0000000->c0400000
 ENTRY ADDRESS: c0100000
vif-bridge up vif=vif5.0 domain=vpn1dc1 mac=02:00:00:01:01:00
bridge=xen-br0
interface vif5.0 does not exist!
vif-bridge up vif=vif5.1 domain=vpn1dc1 mac=02:00:00:01:01:01
bridge=xen-br1
interface vif5.1 does not exist!
recv_fe_driver_status> {''status'': 1}

recv_fe_driver_status>

recv_fe_interface_connect {''tx_shmem_frame'': 202459,
''rx_shmem_frame'':
202460, ''handle'': 0}

recv_fe_interface_connect {''tx_shmem_frame'': 202463,
''rx_shmem_frame'':
202464, ''handle'': 1}
VirqClient.virqReceived> 4
vif-bridge down vif=vif5.0 domain=vpn1dc1 mac=02:00:00:01:01:00
bridge=xen-br0
interface vif5.0 does not exist!
vif-bridge down vif=vif5.1 domain=vpn1dc1 mac=02:00:00:01:01:01
bridge=xen-br1
interface vif5.1 does not exist!

File xend-debug.log not changed so no update needed.
[root@xenpub1dc1 log]# ls
acpid            boot.log.1  cron.2   mail       messages    prelink.log
samba     spooler    wtmp            xfrd.log
anaconda.log     boot.log.2  cups     maillog    messages.1  rpmpkgs
secure    spooler.1  xend-debug.log  Xorg.0.log
anaconda.syslog  cron        dmesg    maillog.1  messages.2  rpmpkgs.1
secure.1  spooler.2  xend.log        Xorg.0.log.old
boot.log         cron.1      lastlog  maillog.2  ppp         rpmpkgs.2
secure.2  vbox       xend.log.1      yum.log
[root@xenpub1dc1 log]# joe xend.log
Processing ''/etc/joe/joerc''...done
Processing ''/etc/joe/joerc''...done









































    I    xend.log
Row 6079 Col 1   11:04  Ctrl-K H for help
[2006-01-28 22:06:09 xend] INFO (XendRoot:113) EVENT>
xend.domain.shutdown [''vpn1dc1'', ''2'',
''halt'']
[2006-01-28 22:06:09 xend] INFO (XendRoot:113) EVENT>
xend.domain.shutdown [''niagras1'', ''1'',
''halt'']
[2006-01-28 22:06:22 xend] DEBUG (XendDomain:244) XendDomain>reap>
domain died name=niagras1 id=1
[2006-01-28 22:06:22 xend] DEBUG (XendDomain:247) XendDomain>reap>
shutdown id=1 reason=poweroff
[2006-01-28 22:06:22 xend] DEBUG (XendDomain:487)
domain_restart_schedule> 1 poweroff 0
[2006-01-28 22:06:22 xend] INFO (XendDomain:568) Destroying domain:
name=niagras1
[2006-01-28 22:06:22 xend] DEBUG (XendDomainInfo:665) Destroying vifs
for domain 1
[2006-01-28 22:06:22 xend] DEBUG (netif:305) Destroying vif domain=1
vif=0
[2006-01-28 22:06:22 xend] DEBUG (netif:305) Destroying vif domain=1
vif=1
[2006-01-28 22:06:22 xend] DEBUG (XendDomainInfo:674) Destroying vbds
for domain 1
[2006-01-28 22:06:22 xend] DEBUG (blkif:552) Destroying blkif domain=1
[2006-01-28 22:06:22 xend] DEBUG (blkif:408) Destroying vbd domain=1
idx=0
[2006-01-28 22:06:22 xend] DEBUG (blkif:408) Destroying vbd domain=1
idx=1
[2006-01-28 22:06:22 xend] DEBUG (XendDomainInfo:634) Closing console,
domain 1
[2006-01-28 22:06:22 xend] DEBUG (XendDomainInfo:622) Closing channel to
domain 1
[2006-01-28 22:06:22 xend] INFO (XendRoot:113) EVENT> xend.virq 4
[2006-01-28 22:06:22 xend] DEBUG (blkif:363) Unbinding vbd (type phy)
from VG1/niagras1root
[2006-01-28 22:06:22 xend] DEBUG (blkif:363) Unbinding vbd (type phy)
from VG1/niagras1swap
[2006-01-28 22:06:22 xend] INFO (console:58) Console disconnected 15
127.0.0.1 3484
[2006-01-28 22:06:22 xend] INFO (XendRoot:113) EVENT> xend.domain.exit
[''niagras1'', ''1'',
''poweroff'']
[2006-01-28 22:06:22 xend] INFO (XendRoot:113) EVENT>
xend.domain.destroy [''niagras1'', ''1'']
[2006-01-28 22:06:22 xend] INFO (XendRoot:113) EVENT>
xend.console.disconnect [15, ''127.0.0.1'', 3484]
[2006-01-28 22:06:23 xend] INFO (XendRoot:113) EVENT> xend.domain.died
[''niagras1'', ''1'']
[2006-01-28 22:06:24 xend] DEBUG (XendDomain:244) XendDomain>reap>
domain died name=vpn1dc1 id=2
[2006-01-28 22:06:24 xend] DEBUG (XendDomain:247) XendDomain>reap>
shutdown id=2 reason=poweroff
[2006-01-28 22:06:24 xend] DEBUG (XendDomain:487)
domain_restart_schedule> 2 poweroff 0
[2006-01-28 22:06:24 xend] INFO (XendDomain:568) Destroying domain:
name=vpn1dc1
[2006-01-28 22:06:24 xend] DEBUG (XendDomainInfo:665) Destroying vifs
for domain 2
[2006-01-28 22:06:24 xend] DEBUG (netif:305) Destroying vif domain=2
vif=0
[2006-01-28 22:06:24 xend] DEBUG (netif:305) Destroying vif domain=2
vif=1
[2006-01-28 22:06:24 xend] DEBUG (XendDomainInfo:674) Destroying vbds
for domain 2
[2006-01-28 22:06:24 xend] DEBUG (blkif:552) Destroying blkif domain=2
[2006-01-28 22:06:24 xend] DEBUG (blkif:408) Destroying vbd domain=2
idx=0
[2006-01-28 22:06:24 xend] DEBUG (blkif:408) Destroying vbd domain=2
idx=1
[2006-01-28 22:06:24 xend] DEBUG (XendDomainInfo:634) Closing console,
domain 2
[2006-01-28 22:06:24 xend] DEBUG (XendDomainInfo:622) Closing channel to
domain 2
[2006-01-28 22:06:24 xend] INFO (XendRoot:113) EVENT> xend.virq 4
[2006-01-28 22:06:24 xend] DEBUG (blkif:363) Unbinding vbd (type phy)
from VG1/vpn1dc1root
[2006-01-28 22:06:24 xend] DEBUG (blkif:363) Unbinding vbd (type phy)
from VG1/vpn1dc1swap
[2006-01-28 22:06:24 xend] INFO (console:58) Console disconnected 16
127.0.0.1 3986
[2006-01-28 22:06:24 xend] INFO (XendRoot:113) EVENT> xend.domain.exit
[''vpn1dc1'', ''2'',
''poweroff'']
Jan 28 22:05:49 xenpub1dc1 ntpd[3652]: ntpd exiting on signal 15
Jan 28 22:05:49 xenpub1dc1 ntpd: ntpd shutdown succeeded
Jan 28 22:06:22 xenpub1dc1 kernel: device vif1.0 left promiscuous mode
Jan 28 22:06:22 xenpub1dc1 kernel: xen-br0: port 2(vif1.0) entering
disabled state
Jan 28 22:06:22 xenpub1dc1 kernel: xen-br0: port 2(vif1.0) entering
disabled state
Jan 28 22:06:22 xenpub1dc1 kernel: device vif1.1 left promiscuous mode
Jan 28 22:06:22 xenpub1dc1 kernel: xen-br0: port 3(vif1.1) entering
disabled state
Jan 28 22:06:22 xenpub1dc1 kernel: xen-br0: port 3(vif1.1) entering
disabled state
Jan 28 22:06:23 xenpub1dc1 net.agent[4714]: remove event not handled
Jan 28 22:06:23 xenpub1dc1 net.agent[4723]: remove event not handled
Jan 28 22:06:23 xenpub1dc1 net.agent[4737]: remove event not handled
Jan 28 22:06:23 xenpub1dc1 net.agent[4746]: remove event not handled
Jan 28 22:06:24 xenpub1dc1 kernel: device vif2.0 left promiscuous mode
Jan 28 22:06:24 xenpub1dc1 kernel: xen-br0: port 4(vif2.0) entering
disabled state
Jan 28 22:06:24 xenpub1dc1 kernel: xen-br0: port 4(vif2.0) entering
disabled state
Jan 28 22:06:24 xenpub1dc1 kernel: device vif2.1 left promiscuous mode
Jan 28 22:06:24 xenpub1dc1 kernel: xen-br1: port 2(vif2.1) entering
disabled state
Jan 28 22:06:24 xenpub1dc1 kernel: xen-br1: port 2(vif2.1) entering
disabled state
Jan 28 22:06:24 xenpub1dc1 net.agent[4774]: remove event not handled
Jan 28 22:06:24 xenpub1dc1 net.agent[4766]: remove event not handled
Jan 28 22:06:24 xenpub1dc1 net.agent[4789]: remove event not handled
Jan 28 22:06:24 xenpub1dc1 net.agent[4798]: remove event not handled
Jan 28 22:18:03 xenpub1dc1 shutdown: shutting down for system reboot
Jan 28 22:18:03 xenpub1dc1 init: Switching to runlevel: 6
Jan 28 22:18:03 xenpub1dc1 login(pam_unix)[3781]: session closed for
user root
Jan 28 22:18:03 xenpub1dc1 login(pam_unix)[3779]: session closed for
user root
Jan 28 22:18:03 xenpub1dc1 login(pam_unix)[3780]: session closed for
user root
Jan 28 22:18:04 xenpub1dc1 xendomains: Shutting down all Xen domains:
Jan 28 22:18:05 xenpub1dc1 xendomains: Error: Internal Server Error
Jan 28 22:18:05 xenpub1dc1 xendomains:  failed
Jan 28 22:18:05 xenpub1dc1 xendomains:
Jan 28 22:18:05 xenpub1dc1 rc: Stopping xendomains:  failed
Jan 28 22:18:05 xenpub1dc1 kernel: Kernel logging (proc) stopped.
Jan 28 22:18:05 xenpub1dc1 kernel: Kernel log daemon terminating.
Jan 28 22:18:06 xenpub1dc1 syslog: klogd shutdown succeeded
Jan 28 22:18:06 xenpub1dc1 exiting on signal 15
Jan 28 22:20:06 xenpub1dc1 syslogd 1.4.1: restart.
Jan 28 22:20:06 xenpub1dc1 syslog: syslogd startup succeeded
Jan 28 22:20:06 xenpub1dc1 kernel: klogd 1.4.1, log source = /proc/kmsg
started.
Jan 28 22:20:06 xenpub1dc1 kernel: Linux version 2.6.11.12-xen0
(root@xenpub1dc1.atlasgroup.net) (gcc version 3.4.4 20050721 (Red Hat
3.4.4-2)) #4 Thu Jan
File messages not changed so no update needed.
[root@xenpub1dc1 log]# ls
acpid            boot.log.1  cron.2   mail       messages    prelink.log
samba     spooler    wtmp            xfrd.log
anaconda.log     boot.log.2  cups     maillog    messages.1  rpmpkgs
secure    spooler.1  xend-debug.log  Xorg.0.log
anaconda.syslog  cron        dmesg    maillog.1  messages.2  rpmpkgs.1
secure.1  spooler.2  xend.log        Xorg.0.log.old
boot.log         cron.1      lastlog  maillog.2  ppp         rpmpkgs.2
secure.2  vbox       xend.log.1      yum.log
[root@xenpub1dc1 log]# joe xend-debug.log
Processing ''/etc/joe/joerc''...done
Processing ''/etc/joe/joerc''...done









































    I    xend-debug.log
Row 171  Col 1   11:02  Ctrl-K H for help
recv_fe_driver_status>

recv_fe_interface_connect {''tx_shmem_frame'': 185025,
''rx_shmem_frame'':
185026, ''handle'': 0}

recv_fe_interface_connect {''tx_shmem_frame'': 185029,
''rx_shmem_frame'':
185030, ''handle'': 1}
{}
{}
VirqClient.virqReceived> 4
vif-bridge down vif=vif8.0 domain=niagras1 mac=02:00:00:01:02:00
bridge=xen-br0
vif-bridge down vif=vif8.1 domain=niagras1 mac=aa:00:00:53:7a:5b
bridge=xen-br0
VirqClient.virqReceived> 4
vif-bridge down vif=vif7.0 domain=vpn1dc1 mac=02:00:00:01:01:00
bridge=xen-br0
vif-bridge down vif=vif7.1 domain=vpn1dc1 mac=02:00:00:01:01:01
bridge=xen-br1
-bridge down vif=vif4.1 domain=vpn1dc1 mac=02:00:00:01:01:01
bridge=xen-br1
interface vif4.1 does not exist!
VIRTUAL MEMORY ARRANGEMENT:
 Loaded kernel: c0100000->c02c9df4
 Init. ramdisk: c02ca000->c02ca000
 Phys-Mach map: c02ca000->c02e2000
 Page tables:   c02e2000->c02e4000
 Start info:    c02e4000->c02e5000
 Boot stack:    c02e5000->c02e6000
 TOTAL:         c0000000->c0400000
 ENTRY ADDRESS: c0100000
vif-bridge up vif=vif5.0 domain=vpn1dc1 mac=02:00:00:01:01:00
bridge=xen-br0
interface vif5.0 does not exist!
vif-bridge up vif=vif5.1 domain=vpn1dc1 mac=02:00:00:01:01:01
bridge=xen-br1
interface vif5.1 does not exist!
recv_fe_driver_status> {''status'': 1}

recv_fe_driver_status>

recv_fe_interface_connect {''tx_shmem_frame'': 202459,
''rx_shmem_frame'':
202460, ''handle'': 0}

recv_fe_interface_connect {''tx_shmem_frame'': 202463,
''rx_shmem_frame'':
202464, ''handle'': 1}
VirqClient.virqReceived> 4
vif-bridge down vif=vif5.0 domain=vpn1dc1 mac=02:00:00:01:01:00
bridge=xen-br0
interface vif5.0 does not exist!
vif-bridge down vif=vif5.1 domain=vpn1dc1 mac=02:00:00:01:01:01
bridge=xen-br1
interface vif5.1 does not exist!

File xend-debug.log not changed so no update needed.
[root@xenpub1dc1 log]# ls
acpid            boot.log.1  cron.2   mail       messages    prelink.log
samba     spooler    wtmp            xfrd.log
anaconda.log     boot.log.2  cups     maillog    messages.1  rpmpkgs
secure    spooler.1  xend-debug.log  Xorg.0.log
anaconda.syslog  cron        dmesg    maillog.1  messages.2  rpmpkgs.1
secure.1  spooler.2  xend.log        Xorg.0.log.old
boot.log         cron.1      lastlog  maillog.2  ppp         rpmpkgs.2
secure.2  vbox       xend.log.1      yum.log
[root@xenpub1dc1 log]# joe xend.log
Processing ''/etc/joe/joerc''...done
Processing ''/etc/joe/joerc''...done









































    I    xend.log
Row 6077 Col 91  11:06  Ctrl-K H for help
[2006-01-28 22:06:24 xend] DEBUG (blkif:363) Unbinding vbd (type phy)
from VG1/vpn1dc1swap
[2006-01-28 22:06:24 xend] INFO (console:58) Console disconnected 16
127.0.0.1 3986
[2006-01-28 22:06:24 xend] INFO (XendRoot:113) EVENT> xend.domain.exit
[''vpn1dc1'', ''2'',
''poweroff'']
[2006-01-28 22:06:24 xend] INFO (XendRoot:113) EVENT>
xend.domain.destroy [''vpn1dc1'', ''2'']
[2006-01-28 22:06:24 xend] INFO (XendRoot:113) EVENT>
xend.console.disconnect [16, ''127.0.0.1'', 3986]
[2006-01-28 22:06:26 xend] DEBUG (XendDomain:244) XendDomain>reap>
domain died name=vpn1dc1 id=2
[2006-01-28 22:06:26 xend] DEBUG (XendDomain:247) XendDomain>reap>
shutdown id=2 reason=poweroff
[2006-01-28 22:06:26 xend] DEBUG (XendDomain:487)
domain_restart_schedule> 2 poweroff 0
[2006-01-28 22:06:26 xend] INFO (XendDomain:568) Destroying domain:
name=vpn1dc1
[2006-01-28 22:06:26 xend] DEBUG (XendDomainInfo:634) Closing console,
domain 2
[2006-01-28 22:06:26 xend] INFO (XendRoot:113) EVENT> xend.domain.exit
[''vpn1dc1'', ''2'',
''poweroff'']
[2006-01-28 22:06:26 xend] INFO (XendRoot:113) EVENT>
xend.domain.destroy [''vpn1dc1'', ''2'']
[2006-01-28 22:06:28 xend] DEBUG (XendDomain:244) XendDomain>reap>
domain died name=vpn1dc1 id=2
[2006-01-28 22:06:28 xend] DEBUG (XendDomain:247) XendDomain>reap>
shutdown id=2 reason=poweroff
[2006-01-28 22:06:28 xend] DEBUG (XendDomain:487)
domain_restart_schedule> 2 poweroff 0
[2006-01-28 22:06:28 xend] INFO (XendDomain:568) Destroying domain:
name=vpn1dc1
[2006-01-28 22:06:28 xend] DEBUG (XendDomainInfo:634) Closing console,
domain 2
[2006-01-28 22:06:28 xend] INFO (XendRoot:113) EVENT> xend.domain.exit
[''vpn1dc1'', ''2'',
''poweroff'']
[2006-01-28 22:06:28 xend] INFO (XendRoot:113) EVENT>
xend.domain.destroy [''vpn1dc1'', ''2'']
[2006-01-28 22:06:30 xend] DEBUG (XendDomain:244) XendDomain>reap>
domain died name=vpn1dc1 id=2
[2006-01-28 22:06:30 xend] DEBUG (XendDomain:247) XendDomain>reap>
shutdown id=2 reason=poweroff
[2006-01-28 22:06:30 xend] DEBUG (XendDomain:487)
domain_restart_schedule> 2 poweroff 0
[2006-01-28 22:06:30 xend] INFO (XendDomain:568) Destroying domain:
name=vpn1dc1
[2006-01-28 22:06:30 xend] DEBUG (XendDomainInfo:634) Closing console,
domain 2
[2006-01-28 22:06:30 xend] INFO (XendRoot:113) EVENT> xend.domain.exit
[''vpn1dc1'', ''2'',
''poweroff'']
[2006-01-28 22:06:30 xend] INFO (XendRoot:113) EVENT>
xend.domain.destroy [''vpn1dc1'', ''2'']
[2006-01-28 22:06:32 xend] DEBUG (XendDomain:244) XendDomain>reap>
domain died name=vpn1dc1 id=2
[2006-01-28 22:06:32 xend] DEBUG (XendDomain:247) XendDomain>reap>
shutdown id=2 reason=poweroff
[2006-01-28 22:06:32 xend] DEBUG (XendDomain:487)
domain_restart_schedule> 2 poweroff 0
[2006-01-28 22:06:32 xend] INFO (XendDomain:568) Destroying domain:
name=vpn1dc1
[2006-01-28 22:06:32 xend] DEBUG (XendDomainInfo:634) Closing console,
domain 2
[2006-01-28 22:06:32 xend] INFO (XendRoot:113) EVENT> xend.domain.exit
[''vpn1dc1'', ''2'',
''poweroff'']
[2006-01-28 22:06:32 xend] INFO (XendRoot:113) EVENT>
xend.domain.destroy [''vpn1dc1'', ''2'']
[2006-01-28 22:06:34 xend] DEBUG (XendDomain:244) XendDomain>reap>
domain died name=vpn1dc1 id=2
[2006-01-28 22:06:34 xend] DEBUG (XendDomain:247) XendDomain>reap>
shutdown id=2 reason=poweroff
[2006-01-28 22:06:34 xend] DEBUG (XendDomain:487)
domain_restart_schedule> 2 poweroff 0
[2006-01-28 22:06:34 xend] INFO (XendDomain:568) Destroying domain:
name=vpn1dc1
[2006-01-28 22:06:34 xend] DEBUG (XendDomainInfo:634) Closing console,
domain 2
[2006-01-28 22:06:34 xend] INFO (XendRoot:113) EVENT> xend.domain.exit
[''vpn1dc1'', ''2'',
''poweroff'']
[2006-01-28 22:06:34 xend] INFO (XendRoot:113) EVENT>
xend.domain.destroy [''vpn1dc1'', ''2'']
[2006-01-28 22:06:36 xend] DEBUG (XendDomain:244) XendDomain>reap>
domain died name=vpn1dc1 id=2  
Jan 28 22:05:49 xenpub1dc1 ntpd[3652]: ntpd exiting on signal 15
Jan 28 22:05:49 xenpub1dc1 ntpd: ntpd shutdown succeeded
Jan 28 22:06:22 xenpub1dc1 kernel: device vif1.0 left promiscuous mode
Jan 28 22:06:22 xenpub1dc1 kernel: xen-br0: port 2(vif1.0) entering
disabled state
Jan 28 22:06:22 xenpub1dc1 kernel: xen-br0: port 2(vif1.0) entering
disabled state
Jan 28 22:06:22 xenpub1dc1 kernel: device vif1.1 left promiscuous mode
Jan 28 22:06:22 xenpub1dc1 kernel: xen-br0: port 3(vif1.1) entering
disabled state
Jan 28 22:06:22 xenpub1dc1 kernel: xen-br0: port 3(vif1.1) entering
disabled state
Jan 28 22:06:23 xenpub1dc1 net.agent[4714]: remove event not handled
Jan 28 22:06:23 xenpub1dc1 net.agent[4723]: remove event not handled
Jan 28 22:06:23 xenpub1dc1 net.agent[4737]: remove event not handled
Jan 28 22:06:23 xenpub1dc1 net.agent[4746]: remove event not handled
Jan 28 22:06:24 xenpub1dc1 kernel: device vif2.0 left promiscuous mode
Jan 28 22:06:24 xenpub1dc1 kernel: xen-br0: port 4(vif2.0) entering
disabled state
Jan 28 22:06:24 xenpub1dc1 kernel: xen-br0: port 4(vif2.0) entering
disabled state
Jan 28 22:06:24 xenpub1dc1 kernel: device vif2.1 left promiscuous mode
Jan 28 22:06:24 xenpub1dc1 kernel: xen-br1: port 2(vif2.1) entering
disabled state
Jan 28 22:06:24 xenpub1dc1 kernel: xen-br1: port 2(vif2.1) entering
disabled state
Jan 28 22:06:24 xenpub1dc1 net.agent[4774]: remove event not handled
Jan 28 22:06:24 xenpub1dc1 net.agent[4766]: remove event not handled
Jan 28 22:06:24 xenpub1dc1 net.agent[4789]: remove event not handled
Jan 28 22:06:24 xenpub1dc1 net.agent[4798]: remove event not handled
Jan 28 22:18:03 xenpub1dc1 shutdown: shutting down for system reboot
Jan 28 22:18:03 xenpub1dc1 init: Switching to runlevel: 6
Jan 28 22:18:03 xenpub1dc1 login(pam_unix)[3781]: session closed for
user root
Jan 28 22:18:03 xenpub1dc1 login(pam_unix)[3779]: session closed for
user root
Jan 28 22:18:03 xenpub1dc1 login(pam_unix)[3780]: session closed for
user root
Jan 28 22:18:04 xenpub1dc1 xendomains: Shutting down all Xen domains:
Jan 28 22:18:05 xenpub1dc1 xendomains: Error: Internal Server Error
Jan 28 22:18:05 xenpub1dc1 xendomains:  failed
Jan 28 22:18:05 xenpub1dc1 xendomains:
Jan 28 22:18:05 xenpub1dc1 rc: Stopping xendomains:  failed
Jan 28 22:18:05 xenpub1dc1 kernel: Kernel logging (proc) stopped.
Jan 28 22:18:05 xenpub1dc1 kernel: Kernel log daemon terminating.
Jan 28 22:18:06 xenpub1dc1 syslog: klogd shutdown succeeded
Jan 28 22:18:06 xenpub1dc1 exiting on signal 15
Jan 28 22:20:06 xenpub1dc1 syslogd 1.4.1: restart.
Jan 28 22:20:06 xenpub1dc1 syslog: syslogd startup succeeded
Jan 28 22:20:06 xenpub1dc1 kernel: klogd 1.4.1, log source = /proc/kmsg
started.
Jan 28 22:20:06 xenpub1dc1 kernel: Linux version 2.6.11.12-xen0
(root@xenpub1dc1.atlasgroup.net) (gcc version 3.4.4 20050721 (Red Hat
3.4.4-2)) #4 Thu Jan
File messages not changed so no update needed.
[root@xenpub1dc1 log]# ls
acpid            boot.log.1  cron.2   mail       messages    prelink.log
samba     spooler    wtmp            xfrd.log
anaconda.log     boot.log.2  cups     maillog    messages.1  rpmpkgs
secure    spooler.1  xend-debug.log  Xorg.0.log
anaconda.syslog  cron        dmesg    maillog.1  messages.2  rpmpkgs.1
secure.1  spooler.2  xend.log        Xorg.0.log.old
boot.log         cron.1      lastlog  maillog.2  ppp         rpmpkgs.2
secure.2  vbox       xend.log.1      yum.log
[root@xenpub1dc1 log]# joe xend-debug.log
Processing ''/etc/joe/joerc''...done
Processing ''/etc/joe/joerc''...done









































    I    xend-debug.log
Row 171  Col 1   11:02  Ctrl-K H for help
recv_fe_driver_status>

recv_fe_interface_connect {''tx_shmem_frame'': 185025,
''rx_shmem_frame'':
185026, ''handle'': 0}

recv_fe_interface_connect {''tx_shmem_frame'': 185029,
''rx_shmem_frame'':
185030, ''handle'': 1}
{}
{}
VirqClient.virqReceived> 4
vif-bridge down vif=vif8.0 domain=niagras1 mac=02:00:00:01:02:00
bridge=xen-br0
vif-bridge down vif=vif8.1 domain=niagras1 mac=aa:00:00:53:7a:5b
bridge=xen-br0
VirqClient.virqReceived> 4
vif-bridge down vif=vif7.0 domain=vpn1dc1 mac=02:00:00:01:01:00
bridge=xen-br0
vif-bridge down vif=vif7.1 domain=vpn1dc1 mac=02:00:00:01:01:01
bridge=xen-br1
-bridge down vif=vif4.1 domain=vpn1dc1 mac=02:00:00:01:01:01
bridge=xen-br1
interface vif4.1 does not exist!
VIRTUAL MEMORY ARRANGEMENT:
 Loaded kernel: c0100000->c02c9df4
 Init. ramdisk: c02ca000->c02ca000
 Phys-Mach map: c02ca000->c02e2000
 Page tables:   c02e2000->c02e4000
 Start info:    c02e4000->c02e5000
 Boot stack:    c02e5000->c02e6000
 TOTAL:         c0000000->c0400000
 ENTRY ADDRESS: c0100000
vif-bridge up vif=vif5.0 domain=vpn1dc1 mac=02:00:00:01:01:00
bridge=xen-br0
interface vif5.0 does not exist!
vif-bridge up vif=vif5.1 domain=vpn1dc1 mac=02:00:00:01:01:01
bridge=xen-br1
interface vif5.1 does not exist!
recv_fe_driver_status> {''status'': 1}

recv_fe_driver_status>

recv_fe_interface_connect {''tx_shmem_frame'': 202459,
''rx_shmem_frame'':
202460, ''handle'': 0}

recv_fe_interface_connect {''tx_shmem_frame'': 202463,
''rx_shmem_frame'':
202464, ''handle'': 1}
VirqClient.virqReceived> 4
vif-bridge down vif=vif5.0 domain=vpn1dc1 mac=02:00:00:01:01:00
bridge=xen-br0
interface vif5.0 does not exist!
vif-bridge down vif=vif5.1 domain=vpn1dc1 mac=02:00:00:01:01:01
bridge=xen-br1
interface vif5.1 does not exist!

File xend-debug.log not changed so no update needed.
[root@xenpub1dc1 log]# ls
acpid            boot.log.1  cron.2   mail       messages    prelink.log
samba     spooler    wtmp            xfrd.log
anaconda.log     boot.log.2  cups     maillog    messages.1  rpmpkgs
secure    spooler.1  xend-debug.log  Xorg.0.log
anaconda.syslog  cron        dmesg    maillog.1  messages.2  rpmpkgs.1
secure.1  spooler.2  xend.log        Xorg.0.log.old
boot.log         cron.1      lastlog  maillog.2  ppp         rpmpkgs.2
secure.2  vbox       xend.log.1      yum.log
[root@xenpub1dc1 log]# joe xend.log
Processing ''/etc/joe/joerc''...done
Processing ''/etc/joe/joerc''...done









































    I    xend.log
Row 6157 Col 100 11:07  Ctrl-K H for help
[2006-01-28 22:06:36 xend] DEBUG (XendDomain:244) XendDomain>reap>
domain died name=vpn1dc1 id=2
[2006-01-28 22:06:36 xend] DEBUG (XendDomain:247) XendDomain>reap>
shutdown id=2 reason=poweroff
[2006-01-28 22:06:36 xend] DEBUG (XendDomain:487)
domain_restart_schedule> 2 poweroff 0
[2006-01-28 22:06:36 xend] INFO (XendDomain:568) Destroying domain:
name=vpn1dc1
[2006-01-28 22:06:36 xend] DEBUG (XendDomainInfo:634) Closing console,
domain 2
[2006-01-28 22:06:36 xend] INFO (XendRoot:113) EVENT> xend.domain.exit
[''vpn1dc1'', ''2'',
''poweroff'']
[2006-01-28 22:06:36 xend] INFO (XendRoot:113) EVENT>
xend.domain.destroy [''vpn1dc1'', ''2'']
[2006-01-28 22:06:38 xend] DEBUG (XendDomain:244) XendDomain>reap>
domain died name=vpn1dc1 id=2
[2006-01-28 22:06:38 xend] DEBUG (XendDomain:247) XendDomain>reap>
shutdown id=2 reason=poweroff
[2006-01-28 22:06:38 xend] DEBUG (XendDomain:487)
domain_restart_schedule> 2 poweroff 0
[2006-01-28 22:06:38 xend] INFO (XendDomain:568) Destroying domain:
name=vpn1dc1
[2006-01-28 22:06:38 xend] DEBUG (XendDomainInfo:634) Closing console,
domain 2
[2006-01-28 22:06:38 xend] INFO (XendRoot:113) EVENT> xend.domain.exit
[''vpn1dc1'', ''2'',
''poweroff'']
[2006-01-28 22:06:38 xend] INFO (XendRoot:113) EVENT>
xend.domain.destroy [''vpn1dc1'', ''2'']
[2006-01-28 22:06:40 xend] DEBUG (XendDomain:244) XendDomain>reap>
domain died name=vpn1dc1 id=2
[2006-01-28 22:06:40 xend] DEBUG (XendDomain:247) XendDomain>reap>
shutdown id=2 reason=poweroff
[2006-01-28 22:06:40 xend] DEBUG (XendDomain:487)
domain_restart_schedule> 2 poweroff 0
[2006-01-28 22:06:40 xend] INFO (XendDomain:568) Destroying domain:
name=vpn1dc1
[2006-01-28 22:06:40 xend] DEBUG (XendDomainInfo:634) Closing console,
domain 2
[2006-01-28 22:06:40 xend] INFO (XendRoot:113) EVENT> xend.domain.exit
[''vpn1dc1'', ''2'',
''poweroff'']
[2006-01-28 22:06:40 xend] INFO (XendRoot:113) EVENT>
xend.domain.destroy [''vpn1dc1'', ''2'']
[2006-01-28 22:06:42 xend] DEBUG (XendDomain:244) XendDomain>reap>
domain died name=vpn1dc1 id=2
[2006-01-28 22:06:42 xend] DEBUG (XendDomain:247) XendDomain>reap>
shutdown id=2 reason=poweroff
[2006-01-28 22:06:42 xend] DEBUG (XendDomain:487)
domain_restart_schedule> 2 poweroff 0
[2006-01-28 22:06:42 xend] INFO (XendDomain:568) Destroying domain:
name=vpn1dc1
[2006-01-28 22:06:42 xend] DEBUG (XendDomainInfo:634) Closing console,
domain 2
[2006-01-28 22:06:42 xend] INFO (XendRoot:113) EVENT> xend.domain.exit
[''vpn1dc1'', ''2'',
''poweroff'']
[2006-01-28 22:06:42 xend] INFO (XendRoot:113) EVENT>
xend.domain.destroy [''vpn1dc1'', ''2'']
[2006-01-28 22:06:44 xend] DEBUG (XendDomain:244) XendDomain>reap>
domain died name=vpn1dc1 id=2
[2006-01-28 22:06:44 xend] DEBUG (XendDomain:247) XendDomain>reap>
shutdown id=2 reason=poweroff
[2006-01-28 22:06:44 xend] DEBUG (XendDomain:487)
domain_restart_schedule> 2 poweroff 0
[2006-01-28 22:06:44 xend] INFO (XendDomain:568) Destroying domain:
name=vpn1dc1
[2006-01-28 22:06:44 xend] DEBUG (XendDomainInfo:634) Closing console,
domain 2
[2006-01-28 22:06:44 xend] INFO (XendRoot:113) EVENT> xend.domain.exit
[''vpn1dc1'', ''2'',
''poweroff'']
[2006-01-28 22:06:44 xend] INFO (XendRoot:113) EVENT>
xend.domain.destroy [''vpn1dc1'', ''2'']
[2006-01-28 22:06:46 xend] DEBUG (XendDomain:244) XendDomain>reap>
domain died name=vpn1dc1 id=2
[2006-01-28 22:06:46 xend] DEBUG (XendDomain:247) XendDomain>reap>
shutdown id=2 reason=poweroff
[2006-01-28 22:06:46 xend] DEBUG (XendDomain:487)
domain_restart_schedule> 2 poweroff 0
[2006-01-28 22:06:46 xend] INFO (XendDomain:568) Destroying domain:
name=vpn1dc1
[2006-01-28 22:06:46 xend] DEBUG (XendDomainInfo:634) Closing console,
domain 2
[2006-01-28 22:06:46 xend] INFO (XendRoot:113) EVENT> xend.domain.exit
[''vpn1dc1'', ''2'',
''poweroff'']
[2006-01-28 22:06:46 xend] INFO (XendRoot:113) EVENT>
xend.domain.destroy [''vpn1dc1'', ''2'']
[2006-01-28 22:06:48 xend] DEBUG (XendDomain:244) XendDomain>reap>
domain died name=vpn1dc1 id=2
[2006-01-28 22:06:48 xend] DEBUG (XendDomain:247) XendDomain>reap>
shutdown id=2 reason=poweroff
[2006-01-28 22:06:48 xend] DEBUG (XendDomain:487)
domain_restart_schedule> 2 poweroff 0
[2006-01-28 22:06:48 xend] INFO (XendDomain:568) Destroying domain:
name=vpn1dc1
[2006-01-28 22:06:48 xend] DEBUG (XendDomainInfo:634) Closing console,
domain 2
[2006-01-28 22:06:48 xend] INFO (XendRoot:113) EVENT> xend.domain.exit
[''vpn1dc1'', ''2'',
''poweroff'']
[2006-01-28 22:06:48 xend] INFO (XendRoot:113) EVENT>
xend.domain.destroy [''vpn1dc1'', ''2'']
[2006-01-28 22:06:50 xend] DEBUG (XendDomain:244) XendDomain>reap>
domain died name=vpn1dc1 id=2
[2006-01-28 22:06:50 xend] DEBUG (XendDomain:247) XendDomain>reap>
shutdown id=2 reason=poweroff
[2006-01-28 22:06:50 xend] DEBUG (XendDomain:487)
domain_restart_schedule> 2 poweroff 0
[2006-01-28 22:06:50 xend] INFO (XendDomain:568) Destroying domain:
name=vpn1dc1
[2006-01-28 22:06:50 xend] DEBUG (XendDomainInfo:634) Closing console,
domain 2
[2006-01-28 22:06:50 xend] INFO (XendRoot:113) EVENT> xend.domain.exit
[''vpn1dc1'', ''2'',
''poweroff'']
[2006-01-28 22:06:50 xend] INFO (XendRoot:113) EVENT>
xend.domain.destroy [''vpn1dc1'', ''2'']
[2006-01-28 22:06:52 xend] DEBUG (XendDomain:244) XendDomain>reap>
domain died name=vpn1dc1 id=2
[2006-01-28 22:06:52 xend] DEBUG (XendDomain:247) XendDomain>reap>
shutdown id=2 reason=poweroff
[2006-01-28 22:06:52 xend] DEBUG (XendDomain:487)
domain_restart_schedule> 2 poweroff 0
[2006-01-28 22:06:52 xend] INFO (XendDomain:568) Destroying domain:
name=vpn1dc1
[2006-01-28 22:06:52 xend] DEBUG (XendDomainInfo:634) Closing console,
domain 2
[2006-01-28 22:06:52 xend] INFO (XendRoot:113) EVENT> xend.domain.exit
[''vpn1dc1'', ''2'',
''poweroff'']
[2006-01-28 22:06:52 xend] INFO (XendRoot:113) EVENT>
xend.domain.destroy [''vpn1dc1'', ''2'']
[2006-01-28 22:06:54 xend] DEBUG (XendDomain:244) XendDomain>reap>
domain died name=vpn1dc1 id=2
[2006-01-28 22:06:54 xend] DEBUG (XendDomain:247) XendDomain>reap>
shutdown id=2 reason=poweroff
[2006-01-28 22:06:54 xend] DEBUG (XendDomain:487)
domain_restart_schedule> 2 poweroff 0
[2006-01-28 22:06:54 xend] INFO (XendDomain:568) Destroying domain:
name=vpn1dc1
[2006-01-28 22:06:54 xend] DEBUG (XendDomainInfo:634) Closing console,
domain 2
[2006-01-28 22:06:54 xend] INFO (XendRoot:113) EVENT> xend.domain.exit
[''vpn1dc1'', ''2'',
''poweroff'']
[2006-01-28 22:06:54 xend] INFO (XendRoot:113) EVENT>
xend.domain.destroy [''vpn1dc1'', ''2'']
[2006-01-28 22:06:56 xend] DEBUG (XendDomain:244) XendDomain>reap>
domain died name=vpn1dc1 id=2
[2006-01-28 22:06:56 xend] DEBUG (XendDomain:247) XendDomain>reap>
shutdown id=2 reason=poweroff
[2006-01-28 22:06:56 xend] DEBUG (XendDomain:487)
domain_restart_schedule> 2 poweroff 0
[2006-01-28 22:06:56 xend] INFO (XendDomain:568) Destroying domain:
name=vpn1dc1
[2006-01-28 22:06:56 xend] DEBUG (XendDomainInfo:634) Closing console,
domain 2
[2006-01-28 22:06:56 xend] INFO (XendRoot:113) EVENT> xend.domain.exit
[''vpn1dc1'', ''2'',
''poweroff'']
[2006-01-28 22:06:56 xend] INFO (XendRoot:113) EVENT>
xend.domain.destroy [''vpn1dc1'', ''2'']
[2006-01-28 22:06:57 xend] DEBUG (XendDomain:244) XendDomain>reap>
domain died name=vpn1dc1 id=2
[2006-01-28 22:06:57 xend] DEBUG (XendDomain:247) XendDomain>reap>
shutdown id=2 reason=poweroff
[2006-01-28 22:06:57 xend] DEBUG (XendDomain:487)
domain_restart_schedule> 2 poweroff 0
[2006-01-28 22:06:57 xend] INFO (XendDomain:568) Destroying domain:
name=vpn1dc1 
etc., etc., etc.

Here are the configuration files for the two domUs:
#  -*- mode: python; -*-
#===========================================================================#
Python configuration setup for ''xm create''.
# This script sets the parameters used when a domain is created using
''xm create''.
# You use a separate script for each domain you want to create, or 
# you can set the parameters for the domain on the xm command line.
#===========================================================================
#----------------------------------------------------------------------------
# Kernel image file.
kernel = "/boot/vmlinuz-2.4.30-xenUVPN"

# Optional ramdisk.
#ramdisk = "/boot/initrd.gz"

# The domain build function. Default is ''linux''.
#builder=''linux''

# Initial memory allocation (in megabytes) for the new domain.
memory = 96

# A name for your domain. All domains must have different names.
name = "niagras1"

# Which CPU to start domain on? 
#cpu = -1   # leave to Xen to pick

#----------------------------------------------------------------------------
# Define network interfaces.

# Number of network interfaces. Default is 1.
nics=2

# Optionally define mac and/or bridge for the network interfaces.
# Random MACs are assigned if not given.
#vif = [ ''mac=aa:00:00:00:00:11, bridge=xen-br0'' ]

vif = [''mac=02:00:00:01:02:00, bridge=xen-br0'' ]

#----------------------------------------------------------------------------
# Define the disk devices you want the domain to have access to, and
# what you want them accessible as.
# Each disk entry is of the form phy:UNAME,DEV,MODE
# where UNAME is the device, DEV is the device name the domain will see,
# and MODE is r for read-only, w for read-write.

#disk = [ ''phy:hda1,hda1,w'' ]

disk = [ ''phy:VG1/niagras1root,sda1,w'',
''phy:VG1/niagras1swap,sda2,w'' ]

#----------------------------------------------------------------------------
# Set the kernel command line for the new domain.
# You only need to define the IP parameters and hostname if the
domain''s
# IP config doesn''t, e.g. in ifcfg-eth0 or via DHCP.
# You can use ''extra'' to set the runlevel and custom
environment
# variables used by custom rc scripts (e.g. VMID=, usr= ).

# Set if you want dhcp to allocate the IP address.
#dhcp="dhcp"
# Set netmask.
#netmask# Set default gateway.
#gateway# Set the hostname.
#hostname= "vm%d" % vmid

# Set root device.
root = "/dev/sda1 ro"

# Root device for nfs.
#root = "/dev/nfs"
# The nfs server.
#nfs_server = ''169.254.1.0''  
# Root directory on the nfs server.
#nfs_root   = ''/full/path/to/root/directory''

# Sets runlevel 4.
#extra = "4"

#----------------------------------------------------------------------------
# Set according to whether you want the domain restarted when it exits.
# The default is ''onreboot'', which restarts the domain when it
shuts
down
# with exit code reboot.
# Other values are ''always'', and ''never''.

#restart = ''onreboot''

#===========================================================================

#  -*- mode: python; -*-
#===========================================================================#
Python configuration setup for ''xm create''.
# This script sets the parameters used when a domain is created using
''xm create''.
# You use a separate script for each domain you want to create, or 
# you can set the parameters for the domain on the xm command line.
#===========================================================================
#----------------------------------------------------------------------------
# Kernel image file.
kernel = "/boot/vmlinuz-2.4.30-xenUVPN"

# Optional ramdisk.
#ramdisk = "/boot/initrd.gz"

# The domain build function. Default is ''linux''.
#builder=''linux''

# Initial memory allocation (in megabytes) for the new domain.
memory = 96

# A name for your domain. All domains must have different names.
name = "vpn1dc1"

# Which CPU to start domain on? 
#cpu = -1   # leave to Xen to pick

#----------------------------------------------------------------------------
# Define network interfaces.

# Number of network interfaces. Default is 1.
nics=2

# Optionally define mac and/or bridge for the network interfaces.
# Random MACs are assigned if not given.
#vif = [ ''mac=aa:00:00:00:00:11, bridge=xen-br0'' ]

vif = [''mac=02:00:00:01:01:00, bridge=xen-br0'',
''mac=02:00:00:01:01:01,
bridge=xen-br1'' ]

#----------------------------------------------------------------------------
# Define the disk devices you want the domain to have access to, and
# what you want them accessible as.
# Each disk entry is of the form phy:UNAME,DEV,MODE
# where UNAME is the device, DEV is the device name the domain will see,
# and MODE is r for read-only, w for read-write.

#disk = [ ''phy:hda1,hda1,w'' ]

disk = [ ''phy:VG1/vpn1dc1root,sda1,w'',
''phy:VG1/vpn1dc1swap,sda2,w'' ]

#----------------------------------------------------------------------------
# Set the kernel command line for the new domain.
# You only need to define the IP parameters and hostname if the
domain''s
# IP config doesn''t, e.g. in ifcfg-eth0 or via DHCP.
# You can use ''extra'' to set the runlevel and custom
environment
# variables used by custom rc scripts (e.g. VMID=, usr= ).

# Set if you want dhcp to allocate the IP address.
#dhcp="dhcp"
# Set netmask.
#netmask# Set default gateway.
#gateway# Set the hostname.
#hostname= "vm%d" % vmid

# Set root device.
root = "/dev/sda1 ro"

# Root device for nfs.
#root = "/dev/nfs"
# The nfs server.
#nfs_server = ''169.254.1.0''  
# Root directory on the nfs server.
#nfs_root   = ''/full/path/to/root/directory''

# Sets runlevel 4.
#extra = "4"

#----------------------------------------------------------------------------
# Set according to whether you want the domain restarted when it exits.
# The default is ''onreboot'', which restarts the domain when it
shuts
down
# with exit code reboot.
# Other values are ''always'', and ''never''.

#restart = ''onreboot''

#===========================================================================

Here is the network script:

# Exit if anything goes wrong.
set -e 

# First arg is the operation.
OP=$1
shift

# Pull variables in args in to environment.
for arg ; do export "${arg}" ; done

bridge=${bridge:-xen-br0}
netdev=${netdev:-eth0}
antispoof=${antispoof:-yes}
SCRIPTDIR=/etc/xen/scripts

echo "network $OP bridge=$bridge netdev=$netdev antispoof=$antispoof"

# Usage: transfer_addrs src dst
# Copy all IP addresses (including aliases) from device $src to device $dst.
transfer_addrs () {
    local src=$1
    local dst=$2
    # Don''t bother if $dst already has IP addresses.
    if ip addr show dev ${dst} | egrep -q ''^ *inet '' ; then
        return
    fi
    # Address lines start with ''inet'' and have the device in
them.
    # Replace ''inet'' with ''ip addr add'' and
change the device name $src
    # to ''dev $src''. Remove netmask as we''ll add
routes later.
    ip addr show dev ${src} | egrep ''^ *inet '' | sed -e "
s/inet/ip addr add/
s@\([0-9]\+\.[0-9]\+\.[0-9]\+\.[0-9]\+\)/[0-9]\+@\1@
s/${src}/dev ${dst}/
" | sh -e
}

# Usage: del_addrs src
del_addrs () {
    local src=$1
    ip addr show dev ${src} | egrep ''^ *inet '' | sed -e "
s/inet/ip addr del/
s@\([0-9]\+\.[0-9]\+\.[0-9]\+\.[0-9]\+\)/[0-9]\+@\1@
s/${src}/dev ${src}/
" | sh -e
}

# Usage: transfer_routes src dst
# Get all IP routes to device $src, delete them, and
# add the same routes to device $dst.
# The original routes have to be deleted, otherwise adding them
# for $dst fails (duplicate routes).
transfer_routes () {
    local src=$1
    local dst=$2
    # List all routes and grep the ones with $src in.
    # Stick ''ip route del'' on the front to delete.
    # Change $src to $dst and use ''ip route add'' to add.
    ip route list | grep ${src} | sed -e "
h
s/^/ip route del /
P
g
s/${src}/${dst}/
s/^/ip route add /
P
d
" | sh -e
}

# Usage: create_bridge bridge
create_bridge () {
    local bridge=$1

    # Don''t create the bridge if it already exists.
    if ! brctl show | grep -q ${bridge} ; then
        brctl addbr ${bridge}
        brctl stp ${bridge} off
        brctl setfd ${bridge} 0
    fi
    ifconfig ${bridge} up
}

# Usage: add_to_bridge bridge dev
add_to_bridge () {
    local bridge=$1
    local dev=$2
    # Don''t add $dev to $bridge if it''s already on a bridge.
    if ! brctl show | grep -q ${dev} ; then
        brctl addif ${bridge} ${dev}
    fi
}

# Usage: antispoofing dev bridge
# Set the default forwarding policy for $dev to drop.
# Allow forwarding to the bridge.
antispoofing () {
    local dev=$1
    local bridge=$2

    iptables -P FORWARD DROP
    iptables -A FORWARD -m physdev --physdev-in ${dev} -j ACCEPT
}

# Usage: show_status dev bridge
# Print ifconfig and routes.
show_status () {
    local dev=$1
    local bridge=$2
    
    echo
''============================================================''
    ifconfig ${dev}
    ifconfig ${bridge}
    echo '' ''
    ip route list
    echo '' ''
    route -n
    echo
''============================================================''
}

op_start () {
    if [ "${bridge}" == "null" ] ; then
        return
    fi
# Create the bridge and give it the interface IP addresses.
# Move the interface routes onto the bridge.
    while read bridge netdev
	do
            create_bridge ${bridge}
            if [ -n "$(ip add ls dev $netdev | grep ''inet
'')" ]; then
                transfer_addrs ${netdev} ${bridge}
                transfer_routes ${netdev} ${bridge}
            fi
# Don''t add $dev to $bridge if it''s already on a bridge.
            if ! brctl show | grep -q ${netdev} ; then
                brctl addif ${bridge} ${netdev}
            fi
            
            if [ ${antispoof} == ''yes'' ] ; then
                antispoofing ${netdev} ${bridge}
            fi
    done < ${SCRIPTDIR}/bridges.txt
}

op_stop () {
    if [ "${bridge}" == "null" ] ; then
        return
    fi
# Remove the interface from the bridge.
# Move the routes back to the interface.
    while read bridge netdev
    do
        brctl delif ${bridge} ${netdev}              
        if [ -n "$(ip add ls dev $netdev | grep ''inet
'')" ]; then
            transfer_routes ${bridge} ${netdev}
        fi
    done < ${SCRIPTDIR}/bridges.txt
# It''s not our place to be enabling forwarding...
}

case ${OP} in
  start)
        op_start
        ;;
    
    stop)
        op_stop
        ;;

    status)
        show_status ${netdev} ${bridge}
       ;;

    *)
       echo ''Unknown command: '' ${OP}
       echo ''Valid commands are: start, stop, status''
       exit 1
esac

Here is the bridges.txt file:
xen-br0 eth0
xen-br1 eth1
xen-br2 eth2


I''m stumped but it sure looks serious.  Have I produced some dreadful
misconfiguration? This host is being shipped to a remote data center on
Monday and goes into production in a week.  Any help would be greatly
appreciated.  Thanks - John

-- 
John A. Sullivan III
Open Source Development Corporation
+1 207-985-7880
jsullivan@opensourcedevel.com

Financially sustainable open source development
http://www.opensourcedevel.com


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

On Sat, 2006-01-28 at 23:44 -0500, John A. Sullivan III
wrote:
> Pardon me for a long detailed post and for pestering the developer list
> but this looks pretty serious.  The bottom line is that, when I attempt
> to shut down or reboot my simple xen-2.0.7 server running FC3 with two
> domUs, shutting down xendomains appears to send python into an infinite
> loop.  This may be due to a serious misconfiguration on my part but it
> is certainly a big problem.
> 
> If we shutdown the domUs manually first, the system reboots perfectly
> fine.
> 
> I''ll describe some history and include lots of log information. 
The
> configuration is slightly unusual.  Both domUs are VPN gateways running
> the 2.4.30 kernel with openswan and openvpn.  Because of the openvpn,
> but are privileged domains but the only device driver configured in the
> kernel is the openvpn tun/tap adapter.
> 
> Since we have three interfaces, we slightly modified the network script.
> It is included below.
> 
> We first noticed problems during shutdown in that the last domU to
> shutdown crashed very nastily when closing openvpn.  We could see this
> even when manually stopping the domUs.  We surmised that the problem was
> we had tun0 and tun1 defined in both domains and killing them in one
> must be killing them in another.  We renamed those tun adapters in the
> second domain to tun10 and tun11 and that problem went away.
> 
> We still noticed very long reboot times.  The hang seemed to be at
> killall but the "logjam" seemed to break after xendomains errored
out.
> We assumed this was because killall killed something that xendomains
> needed to shutdown gracefully so we added explicit K scripts in rc0.d
> and rc6.d for xend and xendomains.
> 
> At first this seemed to work but it must have been only coincidental
> because the problem returned.  We then realized we had brain cramped and
> shutdown xend before xendomains.  We thought for sure we had found the
> problem and put them in the proper order but, alas, the problem did not
> go away.
> 
> To gain a little more insight into exactly what was happening, we
> manually ran each kill script in rc6.d.  When we ran K78xendomains stop,
> we saw the problem, or at least many aspect of it.
> 
> >From a xen console, we saw that the first domU had cleanly shutdown but
> the second one was still listed howbeit with zero memory.
> 
> The hard disks were thrashing madly and top showed python consuming 99.9
> percent CPU utilization.  Most of the time, during reboot, we do not see
> the madly thrashing disks.  They do so for a moment as one might expect
> when shutting down the domains but then just blip occasionally
> thereafter.
> 
> Here is what we see in the logs:
> 
> >From messages (ntpd was the last service stopped before xendomains):
> Jan 28 22:05:49 xenpub1dc1 ntpd[3652]: ntpd exiting on signal 15
> Jan 28 22:05:49 xenpub1dc1 ntpd: ntpd shutdown succeeded
> Jan 28 22:06:22 xenpub1dc1 kernel: device vif1.0 left promiscuous mode
> Jan 28 22:06:22 xenpub1dc1 kernel: xen-br0: port 2(vif1.0) entering
> disabled state
> Jan 28 22:06:22 xenpub1dc1 kernel: xen-br0: port 2(vif1.0) entering
> disabled state
> Jan 28 22:06:22 xenpub1dc1 kernel: device vif1.1 left promiscuous mode
> Jan 28 22:06:22 xenpub1dc1 kernel: xen-br0: port 3(vif1.1) entering
> disabled state
> Jan 28 22:06:22 xenpub1dc1 kernel: xen-br0: port 3(vif1.1) entering
> disabled state
> Jan 28 22:06:23 xenpub1dc1 net.agent[4714]: remove event not handled
> Jan 28 22:06:23 xenpub1dc1 net.agent[4723]: remove event not handled
> Jan 28 22:06:23 xenpub1dc1 net.agent[4737]: remove event not handled
> Jan 28 22:06:23 xenpub1dc1 net.agent[4746]: remove event not handled
> Jan 28 22:06:24 xenpub1dc1 kernel: device vif2.0 left promiscuous mode
> Jan 28 22:06:24 xenpub1dc1 kernel: xen-br0: port 4(vif2.0) entering
> disabled state
> Jan 28 22:06:24 xenpub1dc1 kernel: xen-br0: port 4(vif2.0) entering
> disabled state
> Jan 28 22:06:24 xenpub1dc1 kernel: device vif2.1 left promiscuous mode
> Jan 28 22:06:24 xenpub1dc1 kernel: xen-br1: port 2(vif2.1) entering
> disabled state
> Jan 28 22:06:24 xenpub1dc1 kernel: xen-br1: port 2(vif2.1) entering
> disabled state
> Jan 28 22:06:24 xenpub1dc1 net.agent[4774]: remove event not handled
> Jan 28 22:06:24 xenpub1dc1 net.agent[4766]: remove event not handled
> Jan 28 22:06:24 xenpub1dc1 net.agent[4789]: remove event not handled
> Jan 28 22:06:24 xenpub1dc1 net.agent[4798]: remove event not handled
> 
> Here is the complete xend-debug.log:
> network start bridge=xen-br0 netdev=eth0 antispoof=no
> VIRTUAL MEMORY ARRANGEMENT:
>  Loaded kernel: c0100000->c0319394
>  Init. ramdisk: c031a000->c031a000
>  Phys-Mach map: c031a000->c0332000
>  Page tables:   c0332000->c0334000
>  Start info:    c0334000->c0335000
>  Boot stack:    c0335000->c0336000
>  TOTAL:         c0000000->c0400000
>  ENTRY ADDRESS: c0100000
> vif-bridge up vif=vif1.0 domain=niagras1 mac=02:00:00:01:02:00
> bridge=xen-br0
> vif-bridge up vif=vif1.1 domain=niagras1 mac=aa:00:00:58:4a:a7
> bridge=xen-br0
> VIRTUAL MEMORY ARRANGEMENT:
>  Loaded kernel: c0100000->c0319394
>  Init. ramdisk: c031a000->c031a000
>  Phys-Mach map: c031a000->c0332000
>  Page tables:   c0332000->c0334000
>  Start info:    c0334000->c0335000
>  Boot stack:    c0335000->c0336000
>  TOTAL:         c0000000->c0400000
>  ENTRY ADDRESS: c0100000
> vif-bridge up vif=vif2.0 domain=vpn1dc1 mac=02:00:00:01:01:00
> bridge=xen-br0
> vif-bridge up vif=vif2.1 domain=vpn1dc1 mac=02:00:00:01:01:01
> bridge=xen-br1
> recv_fe_driver_status> {''status'': 1}
> 
> recv_fe_driver_status>
> 
> recv_fe_interface_connect {''tx_shmem_frame'': 44763,
''rx_shmem_frame'':
> 44764, ''handle'': 0}
> 
> recv_fe_interface_connect {''tx_shmem_frame'': 44767,
''rx_shmem_frame'':
> 44768, ''handle'': 1}
> recv_fe_driver_status> {''status'': 1}
> 
> recv_fe_driver_status>
> 
> recv_fe_interface_connect {''tx_shmem_frame'': 69339,
''rx_shmem_frame'':
> 69340, ''handle'': 0}
> 
> recv_fe_interface_connect {''tx_shmem_frame'': 69343,
''rx_shmem_frame'':
> 69344, ''handle'': 1}
> {}
> {}
> VirqClient.virqReceived> 4
> vif-bridge down vif=vif2.0 domain=vpn1dc1 mac=02:00:00:01:01:00
> bridge=xen-br0
> vif-bridge down vif=vif2.1 domain=vpn1dc1 mac=02:00:00:01:01:01
> bridge=xen-br1
> VirqClient.virqReceived> 4
> vif-bridge down vif=vif1.0 domain=niagras1 mac=02:00:00:01:02:00
> bridge=xen-br0
> vif-bridge down vif=vif1.1 domain=niagras1 mac=aa:00:00:26:24:c4
> bridge=xen-br0
> Traceback (most recent call last):
>   File
"/usr/lib/python2.3/site-packages/twisted/protocols/basic.py",
> line 229, in dataReceived
>     return self.rawDataReceived(data)
>   File
"/usr/lib/python2.3/site-packages/twisted/protocols/http.py",
> line 1015, in rawDataReceived
>     self.allContentReceived()
>   File
"/usr/lib/python2.3/site-packages/twisted/protocols/http.py",
> line 1006, in allContentReceived
>     req.requestReceived(command, path, version)
>   File
"/usr/lib/python2.3/site-packages/twisted/protocols/http.py",
> line 557, in requestReceived
>     self.process()
> --- <exception caught here> ---
>   File "/usr/lib/python2.3/site-packages/twisted/web/server.py",
line
> 165, in process
>     self.render(resrc)
>   File "/usr/lib/python2.3/site-packages/twisted/web/server.py",
line
> 172, in render
>     body = resrc.render(self)
>   File
"/usr/lib/python2.3/site-packages/twisted/web/resource.py", line
> 201, in render
>     return m(request)
>   File "/usr/lib/python/xen/xend/server/SrvDomain.py", line 180,
in
> render_POST
>     return self.perform(req)
>   File "/usr/lib/python/xen/xend/server/SrvBase.py", line 103, in
> perform
>     return self._perform(op, op_method, req)
>   File "/usr/lib/python/xen/xend/server/SrvBase.py", line 109, in
> _perform
>     return self._perform_err(err, op, req)
>   File "/usr/lib/python/xen/xend/server/SrvBase.py", line 107, in
> _perform
>     val = op_method(op, req)
>   File "/usr/lib/python/xen/xend/server/SrvDomain.py", line 47,
in
> op_shutdown
>     val = fn(req.args, {''dom'': self.dom.id})
>   File "/usr/lib/python/xen/xend/Args.py", line 145, in __call__
>     return self.call_with_form_args(self.fn, fargs, xargs=xargs)
>   File "/usr/lib/python/xen/xend/Args.py", line 121, in
> call_with_form_args
>     return fn(*params, **keys)
>   File "/usr/lib/python/xen/xend/XendDomain.py", line 477, in
> domain_shutdown
>     val = xend.domain_shutdown(dominfo.id, reason, key)
>   File "/usr/lib/python/xen/xend/server/SrvDaemon.py", line 723,
in
> domain_shutdown
>     ctrl = self.domainCF.getController(dom)
>   File "/usr/lib/python/xen/xend/server/controller.py", line 323,
in
> getController
>     ctrl = self.createController(dom)
>   File "/usr/lib/python/xen/xend/server/domain.py", line 20, in
> createController
>     return DomainController(self, dom)
>   File "/usr/lib/python/xen/xend/server/domain.py", line 38, in
__init__
>     self.registerChannel()
>   File "/usr/lib/python/xen/xend/server/controller.py", line 220,
in
> registerChannel
>     self.channel = self.channelFactory.domChannel(self.dom)
>   File "/usr/lib/python/xen/xend/server/channel.py", line 83, in
> domChannel
>     remote_port=remote_port)
>   File "/usr/lib/python/xen/xend/server/channel.py", line 253, in
> __init__
>     remote_port=remote_port)
>   File "/usr/lib/python/xen/xend/server/channel.py", line 137, in
> createPort
>     remote_port=int(remote_port))
> xen.lowlevel.xu.PortError: Failed to map domain control interface
> /usr/lib/python2.3/site-packages/twisted/web/util.py:290: FutureWarning:
> hex()/oct() of negative int will return a signed string in Python 2.4
> and up
>   return ''''''<div
class="instance"><span class="instanceName">%s
instance
> @ %s</span>
> /usr/lib/python2.3/site-packages/twisted/web/util.py:266: FutureWarning:
> hex()/oct() of negative int will return a signed string in Python 2.4
> and up
>   w(''<div class="dict"><span
class="heading">Dictionary instance @ %
> s</span>'' % hex(id(d)))
> /usr/lib/python2.3/site-packages/twisted/web/util.py:279: FutureWarning:
> hex()/oct() of negative int will return a signed string in Python 2.4
> and up
>   w(''<div class="list"><span
class="heading">List instance @ %s</span>''
> % hex(id(l)))
> 
>  Init. ramdisk: c031a000->c031a000
>  Phys-Mach map: c031a000->c0332000
>  Page tables:   c0332000->c0334000
>  Start info:    c0334000->c0335000
>  Boot stack:    c0335000->c0336000
>  TOTAL:         c0000000->c0400000
>  ENTRY ADDRESS: c0100000
> vif-bridge up vif=vif7.0 domain=vpn1dc1 mac=02:00:00:01:01:00
> bridge=xen-br0
> vif-bridge up vif=vif7.1 domain=vpn1dc1 mac=02:00:00:01:01:01
> bridge=xen-br1
> recv_fe_driver_status> {''status'': 1}
> 
> recv_fe_driver_status>
> 
> recv_fe_interface_connect {''tx_shmem_frame'': 88257,
''rx_shmem_frame'':
> 88258, ''handle'': 0}
> 
> recv_fe_interface_connect {''tx_shmem_frame'': 88261,
''rx_shmem_frame'':
> 88262, ''handle'': 1}
> VIRTUAL MEMORY ARRANGEMENT:
>  Loaded kernel: c0100000->c0319394
>  Init. ramdisk: c031a000->c031a000
>  Phys-Mach map: c031a000->c0332000
>  Page tables:   c0332000->c0334000
>  Start info:    c0334000->c0335000
>  Boot stack:    c0335000->c0336000
>  TOTAL:         c0000000->c0400000
>  ENTRY ADDRESS: c0100000
> vif-bridge up vif=vif8.0 domain=niagras1 mac=02:00:00:01:02:00
> bridge=xen-br0
> vif-bridge up vif=vif8.1 domain=niagras1 mac=aa:00:00:53:7a:5b
> bridge=xen-br0
> recv_fe_driver_status> {''status'': 1}
> 
> recv_fe_driver_status>
> 
> recv_fe_interface_connect {''tx_shmem_frame'': 185025,
''rx_shmem_frame'':
> 185026, ''handle'': 0}
> 
> recv_fe_interface_connect {''tx_shmem_frame'': 185029,
''rx_shmem_frame'':
> 185030, ''handle'': 1}
> {}
> {}
> VirqClient.virqReceived> 4
> vif-bridge down vif=vif8.0 domain=niagras1 mac=02:00:00:01:02:00
> bridge=xen-br0
> vif-bridge down vif=vif8.1 domain=niagras1 mac=aa:00:00:53:7a:5b
> bridge=xen-br0
> VirqClient.virqReceived> 4
> vif-bridge down vif=vif7.0 domain=vpn1dc1 mac=02:00:00:01:01:00
> bridge=xen-br0
> vif-bridge down vif=vif7.1 domain=vpn1dc1 mac=02:00:00:01:01:01
> bridge=xen-br1
> -bridge down vif=vif4.1 domain=vpn1dc1 mac=02:00:00:01:01:01
> bridge=xen-br1
> interface vif4.1 does not exist!
> VIRTUAL MEMORY ARRANGEMENT:
>  Loaded kernel: c0100000->c02c9df4
>  Init. ramdisk: c02ca000->c02ca000
>  Phys-Mach map: c02ca000->c02e2000
>  Page tables:   c02e2000->c02e4000
>  Start info:    c02e4000->c02e5000
>  Boot stack:    c02e5000->c02e6000
>  TOTAL:         c0000000->c0400000
>  ENTRY ADDRESS: c0100000
> vif-bridge up vif=vif5.0 domain=vpn1dc1 mac=02:00:00:01:01:00
> bridge=xen-br0
> interface vif5.0 does not exist!
> vif-bridge up vif=vif5.1 domain=vpn1dc1 mac=02:00:00:01:01:01
> bridge=xen-br1
> interface vif5.1 does not exist!
> recv_fe_driver_status> {''status'': 1}
> 
> recv_fe_driver_status>
> 
> recv_fe_interface_connect {''tx_shmem_frame'': 202459,
''rx_shmem_frame'':
> 202460, ''handle'': 0}
> 
> recv_fe_interface_connect {''tx_shmem_frame'': 202463,
''rx_shmem_frame'':
> 202464, ''handle'': 1}
> VirqClient.virqReceived> 4
> vif-bridge down vif=vif5.0 domain=vpn1dc1 mac=02:00:00:01:01:00
> bridge=xen-br0
> interface vif5.0 does not exist!
> vif-bridge down vif=vif5.1 domain=vpn1dc1 mac=02:00:00:01:01:01
> bridge=xen-br1
> interface vif5.1 does not exist!
> 
> Here is the information from xend.log:
> Jan 28 22:05:49 xenpub1dc1 ntpd[3652]: ntpd exiting on signal 15
> Jan 28 22:05:49 xenpub1dc1 ntpd: ntpd shutdown succeeded
> Jan 28 22:06:22 xenpub1dc1 kernel: device vif1.0 left promiscuous mode
> Jan 28 22:06:22 xenpub1dc1 kernel: xen-br0: port 2(vif1.0) entering
> disabled state
> Jan 28 22:06:22 xenpub1dc1 kernel: xen-br0: port 2(vif1.0) entering
> disabled state
> Jan 28 22:06:22 xenpub1dc1 kernel: device vif1.1 left promiscuous mode
> Jan 28 22:06:22 xenpub1dc1 kernel: xen-br0: port 3(vif1.1) entering
> disabled state
> Jan 28 22:06:22 xenpub1dc1 kernel: xen-br0: port 3(vif1.1) entering
> disabled state
> Jan 28 22:06:23 xenpub1dc1 net.agent[4714]: remove event not handled
> Jan 28 22:06:23 xenpub1dc1 net.agent[4723]: remove event not handled
> Jan 28 22:06:23 xenpub1dc1 net.agent[4737]: remove event not handled
> Jan 28 22:06:23 xenpub1dc1 net.agent[4746]: remove event not handled
> Jan 28 22:06:24 xenpub1dc1 kernel: device vif2.0 left promiscuous mode
> Jan 28 22:06:24 xenpub1dc1 kernel: xen-br0: port 4(vif2.0) entering
> disabled state
> Jan 28 22:06:24 xenpub1dc1 kernel: xen-br0: port 4(vif2.0) entering
> disabled state
> Jan 28 22:06:24 xenpub1dc1 kernel: device vif2.1 left promiscuous mode
> Jan 28 22:06:24 xenpub1dc1 kernel: xen-br1: port 2(vif2.1) entering
> disabled state
> Jan 28 22:06:24 xenpub1dc1 kernel: xen-br1: port 2(vif2.1) entering
> disabled state
> Jan 28 22:06:24 xenpub1dc1 net.agent[4774]: remove event not handled
> Jan 28 22:06:24 xenpub1dc1 net.agent[4766]: remove event not handled
> Jan 28 22:06:24 xenpub1dc1 net.agent[4789]: remove event not handled
> Jan 28 22:06:24 xenpub1dc1 net.agent[4798]: remove event not handled
> Jan 28 22:18:03 xenpub1dc1 shutdown: shutting down for system reboot
> Jan 28 22:18:03 xenpub1dc1 init: Switching to runlevel: 6
> Jan 28 22:18:03 xenpub1dc1 login(pam_unix)[3781]: session closed for
> user root
> Jan 28 22:18:03 xenpub1dc1 login(pam_unix)[3779]: session closed for
> user root
> Jan 28 22:18:03 xenpub1dc1 login(pam_unix)[3780]: session closed for
> user root
> Jan 28 22:18:04 xenpub1dc1 xendomains: Shutting down all Xen domains:
> Jan 28 22:18:05 xenpub1dc1 xendomains: Error: Internal Server Error
> Jan 28 22:18:05 xenpub1dc1 xendomains:  failed
> Jan 28 22:18:05 xenpub1dc1 xendomains:
> Jan 28 22:18:05 xenpub1dc1 rc: Stopping xendomains:  failed
> Jan 28 22:18:05 xenpub1dc1 kernel: Kernel logging (proc) stopped.
> Jan 28 22:18:05 xenpub1dc1 kernel: Kernel log daemon terminating.
> Jan 28 22:18:06 xenpub1dc1 syslog: klogd shutdown succeeded
> Jan 28 22:18:06 xenpub1dc1 exiting on signal 15
> Jan 28 22:20:06 xenpub1dc1 syslogd 1.4.1: restart.
> Jan 28 22:20:06 xenpub1dc1 syslog: syslogd startup succeeded
> Jan 28 22:20:06 xenpub1dc1 kernel: klogd 1.4.1, log source = /proc/kmsg
> started.
> Jan 28 22:20:06 xenpub1dc1 kernel: Linux version 2.6.11.12-xen0
> (root@xenpub1dc1.atlasgroup.net) (gcc version 3.4.4 20050721 (Red Hat
> 3.4.4-2)) #4 Thu Jan
> File messages not changed so no update needed.
> [root@xenpub1dc1 log]# ls
> acpid            boot.log.1  cron.2   mail       messages    prelink.log
> samba     spooler    wtmp            xfrd.log
> anaconda.log     boot.log.2  cups     maillog    messages.1  rpmpkgs
> secure    spooler.1  xend-debug.log  Xorg.0.log
> anaconda.syslog  cron        dmesg    maillog.1  messages.2  rpmpkgs.1
> secure.1  spooler.2  xend.log        Xorg.0.log.old
> boot.log         cron.1      lastlog  maillog.2  ppp         rpmpkgs.2
> secure.2  vbox       xend.log.1      yum.log
> [root@xenpub1dc1 log]# joe xend-debug.log
> Processing ''/etc/joe/joerc''...done
> Processing ''/etc/joe/joerc''...done
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
>     I    xend-debug.log
> Row 171  Col 1   11:02  Ctrl-K H for help
> recv_fe_driver_status>
> 
> recv_fe_interface_connect {''tx_shmem_frame'': 185025,
''rx_shmem_frame'':
> 185026, ''handle'': 0}
> 
> recv_fe_interface_connect {''tx_shmem_frame'': 185029,
''rx_shmem_frame'':
> 185030, ''handle'': 1}
> {}
> {}
> VirqClient.virqReceived> 4
> vif-bridge down vif=vif8.0 domain=niagras1 mac=02:00:00:01:02:00
> bridge=xen-br0
> vif-bridge down vif=vif8.1 domain=niagras1 mac=aa:00:00:53:7a:5b
> bridge=xen-br0
> VirqClient.virqReceived> 4
> vif-bridge down vif=vif7.0 domain=vpn1dc1 mac=02:00:00:01:01:00
> bridge=xen-br0
> vif-bridge down vif=vif7.1 domain=vpn1dc1 mac=02:00:00:01:01:01
> bridge=xen-br1
> -bridge down vif=vif4.1 domain=vpn1dc1 mac=02:00:00:01:01:01
> bridge=xen-br1
> interface vif4.1 does not exist!
> VIRTUAL MEMORY ARRANGEMENT:
>  Loaded kernel: c0100000->c02c9df4
>  Init. ramdisk: c02ca000->c02ca000
>  Phys-Mach map: c02ca000->c02e2000
>  Page tables:   c02e2000->c02e4000
>  Start info:    c02e4000->c02e5000
>  Boot stack:    c02e5000->c02e6000
>  TOTAL:         c0000000->c0400000
>  ENTRY ADDRESS: c0100000
> vif-bridge up vif=vif5.0 domain=vpn1dc1 mac=02:00:00:01:01:00
> bridge=xen-br0
> interface vif5.0 does not exist!
> vif-bridge up vif=vif5.1 domain=vpn1dc1 mac=02:00:00:01:01:01
> bridge=xen-br1
> interface vif5.1 does not exist!
> recv_fe_driver_status> {''status'': 1}
> 
> recv_fe_driver_status>
> 
> recv_fe_interface_connect {''tx_shmem_frame'': 202459,
''rx_shmem_frame'':
> 202460, ''handle'': 0}
> 
> recv_fe_interface_connect {''tx_shmem_frame'': 202463,
''rx_shmem_frame'':
> 202464, ''handle'': 1}
> VirqClient.virqReceived> 4
> vif-bridge down vif=vif5.0 domain=vpn1dc1 mac=02:00:00:01:01:00
> bridge=xen-br0
> interface vif5.0 does not exist!
> vif-bridge down vif=vif5.1 domain=vpn1dc1 mac=02:00:00:01:01:01
> bridge=xen-br1
> interface vif5.1 does not exist!
> 
> File xend-debug.log not changed so no update needed.
> [root@xenpub1dc1 log]# ls
> acpid            boot.log.1  cron.2   mail       messages    prelink.log
> samba     spooler    wtmp            xfrd.log
> anaconda.log     boot.log.2  cups     maillog    messages.1  rpmpkgs
> secure    spooler.1  xend-debug.log  Xorg.0.log
> anaconda.syslog  cron        dmesg    maillog.1  messages.2  rpmpkgs.1
> secure.1  spooler.2  xend.log        Xorg.0.log.old
> boot.log         cron.1      lastlog  maillog.2  ppp         rpmpkgs.2
> secure.2  vbox       xend.log.1      yum.log
> [root@xenpub1dc1 log]# joe xend.log
> Processing ''/etc/joe/joerc''...done
> Processing ''/etc/joe/joerc''...done
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
>     I    xend.log
> Row 6079 Col 1   11:04  Ctrl-K H for help
> [2006-01-28 22:06:09 xend] INFO (XendRoot:113) EVENT>
> xend.domain.shutdown [''vpn1dc1'', ''2'',
''halt'']
> [2006-01-28 22:06:09 xend] INFO (XendRoot:113) EVENT>
> xend.domain.shutdown [''niagras1'', ''1'',
''halt'']
> [2006-01-28 22:06:22 xend] DEBUG (XendDomain:244) XendDomain>reap>
> domain died name=niagras1 id=1
> [2006-01-28 22:06:22 xend] DEBUG (XendDomain:247) XendDomain>reap>
> shutdown id=1 reason=poweroff
> [2006-01-28 22:06:22 xend] DEBUG (XendDomain:487)
> domain_restart_schedule> 1 poweroff 0
> [2006-01-28 22:06:22 xend] INFO (XendDomain:568) Destroying domain:
> name=niagras1
> [2006-01-28 22:06:22 xend] DEBUG (XendDomainInfo:665) Destroying vifs
> for domain 1
> [2006-01-28 22:06:22 xend] DEBUG (netif:305) Destroying vif domain=1
> vif=0
> [2006-01-28 22:06:22 xend] DEBUG (netif:305) Destroying vif domain=1
> vif=1
> [2006-01-28 22:06:22 xend] DEBUG (XendDomainInfo:674) Destroying vbds
> for domain 1
> [2006-01-28 22:06:22 xend] DEBUG (blkif:552) Destroying blkif domain=1
> [2006-01-28 22:06:22 xend] DEBUG (blkif:408) Destroying vbd domain=1
> idx=0
> [2006-01-28 22:06:22 xend] DEBUG (blkif:408) Destroying vbd domain=1
> idx=1
> [2006-01-28 22:06:22 xend] DEBUG (XendDomainInfo:634) Closing console,
> domain 1
> [2006-01-28 22:06:22 xend] DEBUG (XendDomainInfo:622) Closing channel to
> domain 1
> [2006-01-28 22:06:22 xend] INFO (XendRoot:113) EVENT> xend.virq 4
> [2006-01-28 22:06:22 xend] DEBUG (blkif:363) Unbinding vbd (type phy)
> from VG1/niagras1root
> [2006-01-28 22:06:22 xend] DEBUG (blkif:363) Unbinding vbd (type phy)
> from VG1/niagras1swap
> [2006-01-28 22:06:22 xend] INFO (console:58) Console disconnected 15
> 127.0.0.1 3484
> [2006-01-28 22:06:22 xend] INFO (XendRoot:113) EVENT> xend.domain.exit
> [''niagras1'', ''1'',
''poweroff'']
> [2006-01-28 22:06:22 xend] INFO (XendRoot:113) EVENT>
> xend.domain.destroy [''niagras1'', ''1'']
> [2006-01-28 22:06:22 xend] INFO (XendRoot:113) EVENT>
> xend.console.disconnect [15, ''127.0.0.1'', 3484]
> [2006-01-28 22:06:23 xend] INFO (XendRoot:113) EVENT> xend.domain.died
> [''niagras1'', ''1'']
> [2006-01-28 22:06:24 xend] DEBUG (XendDomain:244) XendDomain>reap>
> domain died name=vpn1dc1 id=2
> [2006-01-28 22:06:24 xend] DEBUG (XendDomain:247) XendDomain>reap>
> shutdown id=2 reason=poweroff
> [2006-01-28 22:06:24 xend] DEBUG (XendDomain:487)
> domain_restart_schedule> 2 poweroff 0
> [2006-01-28 22:06:24 xend] INFO (XendDomain:568) Destroying domain:
> name=vpn1dc1
> [2006-01-28 22:06:24 xend] DEBUG (XendDomainInfo:665) Destroying vifs
> for domain 2
> [2006-01-28 22:06:24 xend] DEBUG (netif:305) Destroying vif domain=2
> vif=0
> [2006-01-28 22:06:24 xend] DEBUG (netif:305) Destroying vif domain=2
> vif=1
> [2006-01-28 22:06:24 xend] DEBUG (XendDomainInfo:674) Destroying vbds
> for domain 2
> [2006-01-28 22:06:24 xend] DEBUG (blkif:552) Destroying blkif domain=2
> [2006-01-28 22:06:24 xend] DEBUG (blkif:408) Destroying vbd domain=2
> idx=0
> [2006-01-28 22:06:24 xend] DEBUG (blkif:408) Destroying vbd domain=2
> idx=1
> [2006-01-28 22:06:24 xend] DEBUG (XendDomainInfo:634) Closing console,
> domain 2
> [2006-01-28 22:06:24 xend] DEBUG (XendDomainInfo:622) Closing channel to
> domain 2
> [2006-01-28 22:06:24 xend] INFO (XendRoot:113) EVENT> xend.virq 4
> [2006-01-28 22:06:24 xend] DEBUG (blkif:363) Unbinding vbd (type phy)
> from VG1/vpn1dc1root
> [2006-01-28 22:06:24 xend] DEBUG (blkif:363) Unbinding vbd (type phy)
> from VG1/vpn1dc1swap
> [2006-01-28 22:06:24 xend] INFO (console:58) Console disconnected 16
> 127.0.0.1 3986
> [2006-01-28 22:06:24 xend] INFO (XendRoot:113) EVENT> xend.domain.exit
> [''vpn1dc1'', ''2'',
''poweroff'']
> Jan 28 22:05:49 xenpub1dc1 ntpd[3652]: ntpd exiting on signal 15
> Jan 28 22:05:49 xenpub1dc1 ntpd: ntpd shutdown succeeded
> Jan 28 22:06:22 xenpub1dc1 kernel: device vif1.0 left promiscuous mode
> Jan 28 22:06:22 xenpub1dc1 kernel: xen-br0: port 2(vif1.0) entering
> disabled state
> Jan 28 22:06:22 xenpub1dc1 kernel: xen-br0: port 2(vif1.0) entering
> disabled state
> Jan 28 22:06:22 xenpub1dc1 kernel: device vif1.1 left promiscuous mode
> Jan 28 22:06:22 xenpub1dc1 kernel: xen-br0: port 3(vif1.1) entering
> disabled state
> Jan 28 22:06:22 xenpub1dc1 kernel: xen-br0: port 3(vif1.1) entering
> disabled state
> Jan 28 22:06:23 xenpub1dc1 net.agent[4714]: remove event not handled
> Jan 28 22:06:23 xenpub1dc1 net.agent[4723]: remove event not handled
> Jan 28 22:06:23 xenpub1dc1 net.agent[4737]: remove event not handled
> Jan 28 22:06:23 xenpub1dc1 net.agent[4746]: remove event not handled
> Jan 28 22:06:24 xenpub1dc1 kernel: device vif2.0 left promiscuous mode
> Jan 28 22:06:24 xenpub1dc1 kernel: xen-br0: port 4(vif2.0) entering
> disabled state
> Jan 28 22:06:24 xenpub1dc1 kernel: xen-br0: port 4(vif2.0) entering
> disabled state
> Jan 28 22:06:24 xenpub1dc1 kernel: device vif2.1 left promiscuous mode
> Jan 28 22:06:24 xenpub1dc1 kernel: xen-br1: port 2(vif2.1) entering
> disabled state
> Jan 28 22:06:24 xenpub1dc1 kernel: xen-br1: port 2(vif2.1) entering
> disabled state
> Jan 28 22:06:24 xenpub1dc1 net.agent[4774]: remove event not handled
> Jan 28 22:06:24 xenpub1dc1 net.agent[4766]: remove event not handled
> Jan 28 22:06:24 xenpub1dc1 net.agent[4789]: remove event not handled
> Jan 28 22:06:24 xenpub1dc1 net.agent[4798]: remove event not handled
> Jan 28 22:18:03 xenpub1dc1 shutdown: shutting down for system reboot
> Jan 28 22:18:03 xenpub1dc1 init: Switching to runlevel: 6
> Jan 28 22:18:03 xenpub1dc1 login(pam_unix)[3781]: session closed for
> user root
> Jan 28 22:18:03 xenpub1dc1 login(pam_unix)[3779]: session closed for
> user root
> Jan 28 22:18:03 xenpub1dc1 login(pam_unix)[3780]: session closed for
> user root
> Jan 28 22:18:04 xenpub1dc1 xendomains: Shutting down all Xen domains:
> Jan 28 22:18:05 xenpub1dc1 xendomains: Error: Internal Server Error
> Jan 28 22:18:05 xenpub1dc1 xendomains:  failed
> Jan 28 22:18:05 xenpub1dc1 xendomains:
> Jan 28 22:18:05 xenpub1dc1 rc: Stopping xendomains:  failed
> Jan 28 22:18:05 xenpub1dc1 kernel: Kernel logging (proc) stopped.
> Jan 28 22:18:05 xenpub1dc1 kernel: Kernel log daemon terminating.
> Jan 28 22:18:06 xenpub1dc1 syslog: klogd shutdown succeeded
> Jan 28 22:18:06 xenpub1dc1 exiting on signal 15
> Jan 28 22:20:06 xenpub1dc1 syslogd 1.4.1: restart.
> Jan 28 22:20:06 xenpub1dc1 syslog: syslogd startup succeeded
> Jan 28 22:20:06 xenpub1dc1 kernel: klogd 1.4.1, log source = /proc/kmsg
> started.
> Jan 28 22:20:06 xenpub1dc1 kernel: Linux version 2.6.11.12-xen0
> (root@xenpub1dc1.atlasgroup.net) (gcc version 3.4.4 20050721 (Red Hat
> 3.4.4-2)) #4 Thu Jan
> File messages not changed so no update needed.
> [root@xenpub1dc1 log]# ls
> acpid            boot.log.1  cron.2   mail       messages    prelink.log
> samba     spooler    wtmp            xfrd.log
> anaconda.log     boot.log.2  cups     maillog    messages.1  rpmpkgs
> secure    spooler.1  xend-debug.log  Xorg.0.log
> anaconda.syslog  cron        dmesg    maillog.1  messages.2  rpmpkgs.1
> secure.1  spooler.2  xend.log        Xorg.0.log.old
> boot.log         cron.1      lastlog  maillog.2  ppp         rpmpkgs.2
> secure.2  vbox       xend.log.1      yum.log
> [root@xenpub1dc1 log]# joe xend-debug.log
> Processing ''/etc/joe/joerc''...done
> Processing ''/etc/joe/joerc''...done
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
>     I    xend-debug.log
> Row 171  Col 1   11:02  Ctrl-K H for help
> recv_fe_driver_status>
> 
> recv_fe_interface_connect {''tx_shmem_frame'': 185025,
''rx_shmem_frame'':
> 185026, ''handle'': 0}
> 
> recv_fe_interface_connect {''tx_shmem_frame'': 185029,
''rx_shmem_frame'':
> 185030, ''handle'': 1}
> {}
> {}
> VirqClient.virqReceived> 4
> vif-bridge down vif=vif8.0 domain=niagras1 mac=02:00:00:01:02:00
> bridge=xen-br0
> vif-bridge down vif=vif8.1 domain=niagras1 mac=aa:00:00:53:7a:5b
> bridge=xen-br0
> VirqClient.virqReceived> 4
> vif-bridge down vif=vif7.0 domain=vpn1dc1 mac=02:00:00:01:01:00
> bridge=xen-br0
> vif-bridge down vif=vif7.1 domain=vpn1dc1 mac=02:00:00:01:01:01
> bridge=xen-br1
> -bridge down vif=vif4.1 domain=vpn1dc1 mac=02:00:00:01:01:01
> bridge=xen-br1
> interface vif4.1 does not exist!
> VIRTUAL MEMORY ARRANGEMENT:
>  Loaded kernel: c0100000->c02c9df4
>  Init. ramdisk: c02ca000->c02ca000
>  Phys-Mach map: c02ca000->c02e2000
>  Page tables:   c02e2000->c02e4000
>  Start info:    c02e4000->c02e5000
>  Boot stack:    c02e5000->c02e6000
>  TOTAL:         c0000000->c0400000
>  ENTRY ADDRESS: c0100000
> vif-bridge up vif=vif5.0 domain=vpn1dc1 mac=02:00:00:01:01:00
> bridge=xen-br0
> interface vif5.0 does not exist!
> vif-bridge up vif=vif5.1 domain=vpn1dc1 mac=02:00:00:01:01:01
> bridge=xen-br1
> interface vif5.1 does not exist!
> recv_fe_driver_status> {''status'': 1}
> 
> recv_fe_driver_status>
> 
> recv_fe_interface_connect {''tx_shmem_frame'': 202459,
''rx_shmem_frame'':
> 202460, ''handle'': 0}
> 
> recv_fe_interface_connect {''tx_shmem_frame'': 202463,
''rx_shmem_frame'':
> 202464, ''handle'': 1}
> VirqClient.virqReceived> 4
> vif-bridge down vif=vif5.0 domain=vpn1dc1 mac=02:00:00:01:01:00
> bridge=xen-br0
> interface vif5.0 does not exist!
> vif-bridge down vif=vif5.1 domain=vpn1dc1 mac=02:00:00:01:01:01
> bridge=xen-br1
> interface vif5.1 does not exist!
> 
> File xend-debug.log not changed so no update needed.
> [root@xenpub1dc1 log]# ls
> acpid            boot.log.1  cron.2   mail       messages    prelink.log
> samba     spooler    wtmp            xfrd.log
> anaconda.log     boot.log.2  cups     maillog    messages.1  rpmpkgs
> secure    spooler.1  xend-debug.log  Xorg.0.log
> anaconda.syslog  cron        dmesg    maillog.1  messages.2  rpmpkgs.1
> secure.1  spooler.2  xend.log        Xorg.0.log.old
> boot.log         cron.1      lastlog  maillog.2  ppp         rpmpkgs.2
> secure.2  vbox       xend.log.1      yum.log
> [root@xenpub1dc1 log]# joe xend.log
> Processing ''/etc/joe/joerc''...done
> Processing ''/etc/joe/joerc''...done
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
>     I    xend.log
> Row 6077 Col 91  11:06  Ctrl-K H for help
> [2006-01-28 22:06:24 xend] DEBUG (blkif:363) Unbinding vbd (type phy)
> from VG1/vpn1dc1swap
> [2006-01-28 22:06:24 xend] INFO (console:58) Console disconnected 16
> 127.0.0.1 3986
> [2006-01-28 22:06:24 xend] INFO (XendRoot:113) EVENT> xend.domain.exit
> [''vpn1dc1'', ''2'',
''poweroff'']
> [2006-01-28 22:06:24 xend] INFO (XendRoot:113) EVENT>
> xend.domain.destroy [''vpn1dc1'', ''2'']
> [2006-01-28 22:06:24 xend] INFO (XendRoot:113) EVENT>
> xend.console.disconnect [16, ''127.0.0.1'', 3986]
> [2006-01-28 22:06:26 xend] DEBUG (XendDomain:244) XendDomain>reap>
> domain died name=vpn1dc1 id=2
> [2006-01-28 22:06:26 xend] DEBUG (XendDomain:247) XendDomain>reap>
> shutdown id=2 reason=poweroff
> [2006-01-28 22:06:26 xend] DEBUG (XendDomain:487)
> domain_restart_schedule> 2 poweroff 0
> [2006-01-28 22:06:26 xend] INFO (XendDomain:568) Destroying domain:
> name=vpn1dc1
> [2006-01-28 22:06:26 xend] DEBUG (XendDomainInfo:634) Closing console,
> domain 2
> [2006-01-28 22:06:26 xend] INFO (XendRoot:113) EVENT> xend.domain.exit
> [''vpn1dc1'', ''2'',
''poweroff'']
> [2006-01-28 22:06:26 xend] INFO (XendRoot:113) EVENT>
> xend.domain.destroy [''vpn1dc1'', ''2'']
> [2006-01-28 22:06:28 xend] DEBUG (XendDomain:244) XendDomain>reap>
> domain died name=vpn1dc1 id=2
> [2006-01-28 22:06:28 xend] DEBUG (XendDomain:247) XendDomain>reap>
> shutdown id=2 reason=poweroff
> [2006-01-28 22:06:28 xend] DEBUG (XendDomain:487)
> domain_restart_schedule> 2 poweroff 0
> [2006-01-28 22:06:28 xend] INFO (XendDomain:568) Destroying domain:
> name=vpn1dc1
> [2006-01-28 22:06:28 xend] DEBUG (XendDomainInfo:634) Closing console,
> domain 2
> [2006-01-28 22:06:28 xend] INFO (XendRoot:113) EVENT> xend.domain.exit
> [''vpn1dc1'', ''2'',
''poweroff'']
> [2006-01-28 22:06:28 xend] INFO (XendRoot:113) EVENT>
> xend.domain.destroy [''vpn1dc1'', ''2'']
> [2006-01-28 22:06:30 xend] DEBUG (XendDomain:244) XendDomain>reap>
> domain died name=vpn1dc1 id=2
> [2006-01-28 22:06:30 xend] DEBUG (XendDomain:247) XendDomain>reap>
> shutdown id=2 reason=poweroff
> [2006-01-28 22:06:30 xend] DEBUG (XendDomain:487)
> domain_restart_schedule> 2 poweroff 0
> [2006-01-28 22:06:30 xend] INFO (XendDomain:568) Destroying domain:
> name=vpn1dc1
> [2006-01-28 22:06:30 xend] DEBUG (XendDomainInfo:634) Closing console,
> domain 2
> [2006-01-28 22:06:30 xend] INFO (XendRoot:113) EVENT> xend.domain.exit
> [''vpn1dc1'', ''2'',
''poweroff'']
> [2006-01-28 22:06:30 xend] INFO (XendRoot:113) EVENT>
> xend.domain.destroy [''vpn1dc1'', ''2'']
> [2006-01-28 22:06:32 xend] DEBUG (XendDomain:244) XendDomain>reap>
> domain died name=vpn1dc1 id=2
> [2006-01-28 22:06:32 xend] DEBUG (XendDomain:247) XendDomain>reap>
> shutdown id=2 reason=poweroff
> [2006-01-28 22:06:32 xend] DEBUG (XendDomain:487)
> domain_restart_schedule> 2 poweroff 0
> [2006-01-28 22:06:32 xend] INFO (XendDomain:568) Destroying domain:
> name=vpn1dc1
> [2006-01-28 22:06:32 xend] DEBUG (XendDomainInfo:634) Closing console,
> domain 2
> [2006-01-28 22:06:32 xend] INFO (XendRoot:113) EVENT> xend.domain.exit
> [''vpn1dc1'', ''2'',
''poweroff'']
> [2006-01-28 22:06:32 xend] INFO (XendRoot:113) EVENT>
> xend.domain.destroy [''vpn1dc1'', ''2'']
> [2006-01-28 22:06:34 xend] DEBUG (XendDomain:244) XendDomain>reap>
> domain died name=vpn1dc1 id=2
> [2006-01-28 22:06:34 xend] DEBUG (XendDomain:247) XendDomain>reap>
> shutdown id=2 reason=poweroff
> [2006-01-28 22:06:34 xend] DEBUG (XendDomain:487)
> domain_restart_schedule> 2 poweroff 0
> [2006-01-28 22:06:34 xend] INFO (XendDomain:568) Destroying domain:
> name=vpn1dc1
> [2006-01-28 22:06:34 xend] DEBUG (XendDomainInfo:634) Closing console,
> domain 2
> [2006-01-28 22:06:34 xend] INFO (XendRoot:113) EVENT> xend.domain.exit
> [''vpn1dc1'', ''2'',
''poweroff'']
> [2006-01-28 22:06:34 xend] INFO (XendRoot:113) EVENT>
> xend.domain.destroy [''vpn1dc1'', ''2'']
> [2006-01-28 22:06:36 xend] DEBUG (XendDomain:244) XendDomain>reap>
> domain died name=vpn1dc1 id=2  
> Jan 28 22:05:49 xenpub1dc1 ntpd[3652]: ntpd exiting on signal 15
> Jan 28 22:05:49 xenpub1dc1 ntpd: ntpd shutdown succeeded
> Jan 28 22:06:22 xenpub1dc1 kernel: device vif1.0 left promiscuous mode
> Jan 28 22:06:22 xenpub1dc1 kernel: xen-br0: port 2(vif1.0) entering
> disabled state
> Jan 28 22:06:22 xenpub1dc1 kernel: xen-br0: port 2(vif1.0) entering
> disabled state
> Jan 28 22:06:22 xenpub1dc1 kernel: device vif1.1 left promiscuous mode
> Jan 28 22:06:22 xenpub1dc1 kernel: xen-br0: port 3(vif1.1) entering
> disabled state
> Jan 28 22:06:22 xenpub1dc1 kernel: xen-br0: port 3(vif1.1) entering
> disabled state
> Jan 28 22:06:23 xenpub1dc1 net.agent[4714]: remove event not handled
> Jan 28 22:06:23 xenpub1dc1 net.agent[4723]: remove event not handled
> Jan 28 22:06:23 xenpub1dc1 net.agent[4737]: remove event not handled
> Jan 28 22:06:23 xenpub1dc1 net.agent[4746]: remove event not handled
> Jan 28 22:06:24 xenpub1dc1 kernel: device vif2.0 left promiscuous mode
> Jan 28 22:06:24 xenpub1dc1 kernel: xen-br0: port 4(vif2.0) entering
> disabled state
> Jan 28 22:06:24 xenpub1dc1 kernel: xen-br0: port 4(vif2.0) entering
> disabled state
> Jan 28 22:06:24 xenpub1dc1 kernel: device vif2.1 left promiscuous mode
> Jan 28 22:06:24 xenpub1dc1 kernel: xen-br1: port 2(vif2.1) entering
> disabled state
> Jan 28 22:06:24 xenpub1dc1 kernel: xen-br1: port 2(vif2.1) entering
> disabled state
> Jan 28 22:06:24 xenpub1dc1 net.agent[4774]: remove event not handled
> Jan 28 22:06:24 xenpub1dc1 net.agent[4766]: remove event not handled
> Jan 28 22:06:24 xenpub1dc1 net.agent[4789]: remove event not handled
> Jan 28 22:06:24 xenpub1dc1 net.agent[4798]: remove event not handled
> Jan 28 22:18:03 xenpub1dc1 shutdown: shutting down for system reboot
> Jan 28 22:18:03 xenpub1dc1 init: Switching to runlevel: 6
> Jan 28 22:18:03 xenpub1dc1 login(pam_unix)[3781]: session closed for
> user root
> Jan 28 22:18:03 xenpub1dc1 login(pam_unix)[3779]: session closed for
> user root
> Jan 28 22:18:03 xenpub1dc1 login(pam_unix)[3780]: session closed for
> user root
> Jan 28 22:18:04 xenpub1dc1 xendomains: Shutting down all Xen domains:
> Jan 28 22:18:05 xenpub1dc1 xendomains: Error: Internal Server Error
> Jan 28 22:18:05 xenpub1dc1 xendomains:  failed
> Jan 28 22:18:05 xenpub1dc1 xendomains:
> Jan 28 22:18:05 xenpub1dc1 rc: Stopping xendomains:  failed
> Jan 28 22:18:05 xenpub1dc1 kernel: Kernel logging (proc) stopped.
> Jan 28 22:18:05 xenpub1dc1 kernel: Kernel log daemon terminating.
> Jan 28 22:18:06 xenpub1dc1 syslog: klogd shutdown succeeded
> Jan 28 22:18:06 xenpub1dc1 exiting on signal 15
> Jan 28 22:20:06 xenpub1dc1 syslogd 1.4.1: restart.
> Jan 28 22:20:06 xenpub1dc1 syslog: syslogd startup succeeded
> Jan 28 22:20:06 xenpub1dc1 kernel: klogd 1.4.1, log source = /proc/kmsg
> started.
> Jan 28 22:20:06 xenpub1dc1 kernel: Linux version 2.6.11.12-xen0
> (root@xenpub1dc1.atlasgroup.net) (gcc version 3.4.4 20050721 (Red Hat
> 3.4.4-2)) #4 Thu Jan
> File messages not changed so no update needed.
> [root@xenpub1dc1 log]# ls
> acpid            boot.log.1  cron.2   mail       messages    prelink.log
> samba     spooler    wtmp            xfrd.log
> anaconda.log     boot.log.2  cups     maillog    messages.1  rpmpkgs
> secure    spooler.1  xend-debug.log  Xorg.0.log
> anaconda.syslog  cron        dmesg    maillog.1  messages.2  rpmpkgs.1
> secure.1  spooler.2  xend.log        Xorg.0.log.old
> boot.log         cron.1      lastlog  maillog.2  ppp         rpmpkgs.2
> secure.2  vbox       xend.log.1      yum.log
> [root@xenpub1dc1 log]# joe xend-debug.log
> Processing ''/etc/joe/joerc''...done
> Processing ''/etc/joe/joerc''...done
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
>     I    xend-debug.log
> Row 171  Col 1   11:02  Ctrl-K H for help
> recv_fe_driver_status>
> 
> recv_fe_interface_connect {''tx_shmem_frame'': 185025,
''rx_shmem_frame'':
> 185026, ''handle'': 0}
> 
> recv_fe_interface_connect {''tx_shmem_frame'': 185029,
''rx_shmem_frame'':
> 185030, ''handle'': 1}
> {}
> {}
> VirqClient.virqReceived> 4
> vif-bridge down vif=vif8.0 domain=niagras1 mac=02:00:00:01:02:00
> bridge=xen-br0
> vif-bridge down vif=vif8.1 domain=niagras1 mac=aa:00:00:53:7a:5b
> bridge=xen-br0
> VirqClient.virqReceived> 4
> vif-bridge down vif=vif7.0 domain=vpn1dc1 mac=02:00:00:01:01:00
> bridge=xen-br0
> vif-bridge down vif=vif7.1 domain=vpn1dc1 mac=02:00:00:01:01:01
> bridge=xen-br1
> -bridge down vif=vif4.1 domain=vpn1dc1 mac=02:00:00:01:01:01
> bridge=xen-br1
> interface vif4.1 does not exist!
> VIRTUAL MEMORY ARRANGEMENT:
>  Loaded kernel: c0100000->c02c9df4
>  Init. ramdisk: c02ca000->c02ca000
>  Phys-Mach map: c02ca000->c02e2000
>  Page tables:   c02e2000->c02e4000
>  Start info:    c02e4000->c02e5000
>  Boot stack:    c02e5000->c02e6000
>  TOTAL:         c0000000->c0400000
>  ENTRY ADDRESS: c0100000
> vif-bridge up vif=vif5.0 domain=vpn1dc1 mac=02:00:00:01:01:00
> bridge=xen-br0
> interface vif5.0 does not exist!
> vif-bridge up vif=vif5.1 domain=vpn1dc1 mac=02:00:00:01:01:01
> bridge=xen-br1
> interface vif5.1 does not exist!
> recv_fe_driver_status> {''status'': 1}
> 
> recv_fe_driver_status>
> 
> recv_fe_interface_connect {''tx_shmem_frame'': 202459,
''rx_shmem_frame'':
> 202460, ''handle'': 0}
> 
> recv_fe_interface_connect {''tx_shmem_frame'': 202463,
''rx_shmem_frame'':
> 202464, ''handle'': 1}
> VirqClient.virqReceived> 4
> vif-bridge down vif=vif5.0 domain=vpn1dc1 mac=02:00:00:01:01:00
> bridge=xen-br0
> interface vif5.0 does not exist!
> vif-bridge down vif=vif5.1 domain=vpn1dc1 mac=02:00:00:01:01:01
> bridge=xen-br1
> interface vif5.1 does not exist!
> 
> File xend-debug.log not changed so no update needed.
> [root@xenpub1dc1 log]# ls
> acpid            boot.log.1  cron.2   mail       messages    prelink.log
> samba     spooler    wtmp            xfrd.log
> anaconda.log     boot.log.2  cups     maillog    messages.1  rpmpkgs
> secure    spooler.1  xend-debug.log  Xorg.0.log
> anaconda.syslog  cron        dmesg    maillog.1  messages.2  rpmpkgs.1
> secure.1  spooler.2  xend.log        Xorg.0.log.old
> boot.log         cron.1      lastlog  maillog.2  ppp         rpmpkgs.2
> secure.2  vbox       xend.log.1      yum.log
> [root@xenpub1dc1 log]# joe xend.log
> Processing ''/etc/joe/joerc''...done
> Processing ''/etc/joe/joerc''...done
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
>     I    xend.log
> Row 6157 Col 100 11:07  Ctrl-K H for help
> [2006-01-28 22:06:36 xend] DEBUG (XendDomain:244) XendDomain>reap>
> domain died name=vpn1dc1 id=2
> [2006-01-28 22:06:36 xend] DEBUG (XendDomain:247) XendDomain>reap>
> shutdown id=2 reason=poweroff
> [2006-01-28 22:06:36 xend] DEBUG (XendDomain:487)
> domain_restart_schedule> 2 poweroff 0
> [2006-01-28 22:06:36 xend] INFO (XendDomain:568) Destroying domain:
> name=vpn1dc1
> [2006-01-28 22:06:36 xend] DEBUG (XendDomainInfo:634) Closing console,
> domain 2
> [2006-01-28 22:06:36 xend] INFO (XendRoot:113) EVENT> xend.domain.exit
> [''vpn1dc1'', ''2'',
''poweroff'']
> [2006-01-28 22:06:36 xend] INFO (XendRoot:113) EVENT>
> xend.domain.destroy [''vpn1dc1'', ''2'']
> [2006-01-28 22:06:38 xend] DEBUG (XendDomain:244) XendDomain>reap>
> domain died name=vpn1dc1 id=2
> [2006-01-28 22:06:38 xend] DEBUG (XendDomain:247) XendDomain>reap>
> shutdown id=2 reason=poweroff
> [2006-01-28 22:06:38 xend] DEBUG (XendDomain:487)
> domain_restart_schedule> 2 poweroff 0
> [2006-01-28 22:06:38 xend] INFO (XendDomain:568) Destroying domain:
> name=vpn1dc1
> [2006-01-28 22:06:38 xend] DEBUG (XendDomainInfo:634) Closing console,
> domain 2
> [2006-01-28 22:06:38 xend] INFO (XendRoot:113) EVENT> xend.domain.exit
> [''vpn1dc1'', ''2'',
''poweroff'']
> [2006-01-28 22:06:38 xend] INFO (XendRoot:113) EVENT>
> xend.domain.destroy [''vpn1dc1'', ''2'']
> [2006-01-28 22:06:40 xend] DEBUG (XendDomain:244) XendDomain>reap>
> domain died name=vpn1dc1 id=2
> [2006-01-28 22:06:40 xend] DEBUG (XendDomain:247) XendDomain>reap>
> shutdown id=2 reason=poweroff
> [2006-01-28 22:06:40 xend] DEBUG (XendDomain:487)
> domain_restart_schedule> 2 poweroff 0
> [2006-01-28 22:06:40 xend] INFO (XendDomain:568) Destroying domain:
> name=vpn1dc1
> [2006-01-28 22:06:40 xend] DEBUG (XendDomainInfo:634) Closing console,
> domain 2
> [2006-01-28 22:06:40 xend] INFO (XendRoot:113) EVENT> xend.domain.exit
> [''vpn1dc1'', ''2'',
''poweroff'']
> [2006-01-28 22:06:40 xend] INFO (XendRoot:113) EVENT>
> xend.domain.destroy [''vpn1dc1'', ''2'']
> [2006-01-28 22:06:42 xend] DEBUG (XendDomain:244) XendDomain>reap>
> domain died name=vpn1dc1 id=2
> [2006-01-28 22:06:42 xend] DEBUG (XendDomain:247) XendDomain>reap>
> shutdown id=2 reason=poweroff
> [2006-01-28 22:06:42 xend] DEBUG (XendDomain:487)
> domain_restart_schedule> 2 poweroff 0
> [2006-01-28 22:06:42 xend] INFO (XendDomain:568) Destroying domain:
> name=vpn1dc1
> [2006-01-28 22:06:42 xend] DEBUG (XendDomainInfo:634) Closing console,
> domain 2
> [2006-01-28 22:06:42 xend] INFO (XendRoot:113) EVENT> xend.domain.exit
> [''vpn1dc1'', ''2'',
''poweroff'']
> [2006-01-28 22:06:42 xend] INFO (XendRoot:113) EVENT>
> xend.domain.destroy [''vpn1dc1'', ''2'']
> [2006-01-28 22:06:44 xend] DEBUG (XendDomain:244) XendDomain>reap>
> domain died name=vpn1dc1 id=2
> [2006-01-28 22:06:44 xend] DEBUG (XendDomain:247) XendDomain>reap>
> shutdown id=2 reason=poweroff
> [2006-01-28 22:06:44 xend] DEBUG (XendDomain:487)
> domain_restart_schedule> 2 poweroff 0
> [2006-01-28 22:06:44 xend] INFO (XendDomain:568) Destroying domain:
> name=vpn1dc1
> [2006-01-28 22:06:44 xend] DEBUG (XendDomainInfo:634) Closing console,
> domain 2
> [2006-01-28 22:06:44 xend] INFO (XendRoot:113) EVENT> xend.domain.exit
> [''vpn1dc1'', ''2'',
''poweroff'']
> [2006-01-28 22:06:44 xend] INFO (XendRoot:113) EVENT>
> xend.domain.destroy [''vpn1dc1'', ''2'']
> [2006-01-28 22:06:46 xend] DEBUG (XendDomain:244) XendDomain>reap>
> domain died name=vpn1dc1 id=2
> [2006-01-28 22:06:46 xend] DEBUG (XendDomain:247) XendDomain>reap>
> shutdown id=2 reason=poweroff
> [2006-01-28 22:06:46 xend] DEBUG (XendDomain:487)
> domain_restart_schedule> 2 poweroff 0
> [2006-01-28 22:06:46 xend] INFO (XendDomain:568) Destroying domain:
> name=vpn1dc1
> [2006-01-28 22:06:46 xend] DEBUG (XendDomainInfo:634) Closing console,
> domain 2
> [2006-01-28 22:06:46 xend] INFO (XendRoot:113) EVENT> xend.domain.exit
> [''vpn1dc1'', ''2'',
''poweroff'']
> [2006-01-28 22:06:46 xend] INFO (XendRoot:113) EVENT>
> xend.domain.destroy [''vpn1dc1'', ''2'']
> [2006-01-28 22:06:48 xend] DEBUG (XendDomain:244) XendDomain>reap>
> domain died name=vpn1dc1 id=2
> [2006-01-28 22:06:48 xend] DEBUG (XendDomain:247) XendDomain>reap>
> shutdown id=2 reason=poweroff
> [2006-01-28 22:06:48 xend] DEBUG (XendDomain:487)
> domain_restart_schedule> 2 poweroff 0
> [2006-01-28 22:06:48 xend] INFO (XendDomain:568) Destroying domain:
> name=vpn1dc1
> [2006-01-28 22:06:48 xend] DEBUG (XendDomainInfo:634) Closing console,
> domain 2
> [2006-01-28 22:06:48 xend] INFO (XendRoot:113) EVENT> xend.domain.exit
> [''vpn1dc1'', ''2'',
''poweroff'']
> [2006-01-28 22:06:48 xend] INFO (XendRoot:113) EVENT>
> xend.domain.destroy [''vpn1dc1'', ''2'']
> [2006-01-28 22:06:50 xend] DEBUG (XendDomain:244) XendDomain>reap>
> domain died name=vpn1dc1 id=2
> [2006-01-28 22:06:50 xend] DEBUG (XendDomain:247) XendDomain>reap>
> shutdown id=2 reason=poweroff
> [2006-01-28 22:06:50 xend] DEBUG (XendDomain:487)
> domain_restart_schedule> 2 poweroff 0
> [2006-01-28 22:06:50 xend] INFO (XendDomain:568) Destroying domain:
> name=vpn1dc1
> [2006-01-28 22:06:50 xend] DEBUG (XendDomainInfo:634) Closing console,
> domain 2
> [2006-01-28 22:06:50 xend] INFO (XendRoot:113) EVENT> xend.domain.exit
> [''vpn1dc1'', ''2'',
''poweroff'']
> [2006-01-28 22:06:50 xend] INFO (XendRoot:113) EVENT>
> xend.domain.destroy [''vpn1dc1'', ''2'']
> [2006-01-28 22:06:52 xend] DEBUG (XendDomain:244) XendDomain>reap>
> domain died name=vpn1dc1 id=2
> [2006-01-28 22:06:52 xend] DEBUG (XendDomain:247) XendDomain>reap>
> shutdown id=2 reason=poweroff
> [2006-01-28 22:06:52 xend] DEBUG (XendDomain:487)
> domain_restart_schedule> 2 poweroff 0
> [2006-01-28 22:06:52 xend] INFO (XendDomain:568) Destroying domain:
> name=vpn1dc1
> [2006-01-28 22:06:52 xend] DEBUG (XendDomainInfo:634) Closing console,
> domain 2
> [2006-01-28 22:06:52 xend] INFO (XendRoot:113) EVENT> xend.domain.exit
> [''vpn1dc1'', ''2'',
''poweroff'']
> [2006-01-28 22:06:52 xend] INFO (XendRoot:113) EVENT>
> xend.domain.destroy [''vpn1dc1'', ''2'']
> [2006-01-28 22:06:54 xend] DEBUG (XendDomain:244) XendDomain>reap>
> domain died name=vpn1dc1 id=2
> [2006-01-28 22:06:54 xend] DEBUG (XendDomain:247) XendDomain>reap>
> shutdown id=2 reason=poweroff
> [2006-01-28 22:06:54 xend] DEBUG (XendDomain:487)
> domain_restart_schedule> 2 poweroff 0
> [2006-01-28 22:06:54 xend] INFO (XendDomain:568) Destroying domain:
> name=vpn1dc1
> [2006-01-28 22:06:54 xend] DEBUG (XendDomainInfo:634) Closing console,
> domain 2
> [2006-01-28 22:06:54 xend] INFO (XendRoot:113) EVENT> xend.domain.exit
> [''vpn1dc1'', ''2'',
''poweroff'']
> [2006-01-28 22:06:54 xend] INFO (XendRoot:113) EVENT>
> xend.domain.destroy [''vpn1dc1'', ''2'']
> [2006-01-28 22:06:56 xend] DEBUG (XendDomain:244) XendDomain>reap>
> domain died name=vpn1dc1 id=2
> [2006-01-28 22:06:56 xend] DEBUG (XendDomain:247) XendDomain>reap>
> shutdown id=2 reason=poweroff
> [2006-01-28 22:06:56 xend] DEBUG (XendDomain:487)
> domain_restart_schedule> 2 poweroff 0
> [2006-01-28 22:06:56 xend] INFO (XendDomain:568) Destroying domain:
> name=vpn1dc1
> [2006-01-28 22:06:56 xend] DEBUG (XendDomainInfo:634) Closing console,
> domain 2
> [2006-01-28 22:06:56 xend] INFO (XendRoot:113) EVENT> xend.domain.exit
> [''vpn1dc1'', ''2'',
''poweroff'']
> [2006-01-28 22:06:56 xend] INFO (XendRoot:113) EVENT>
> xend.domain.destroy [''vpn1dc1'', ''2'']
> [2006-01-28 22:06:57 xend] DEBUG (XendDomain:244) XendDomain>reap>
> domain died name=vpn1dc1 id=2
> [2006-01-28 22:06:57 xend] DEBUG (XendDomain:247) XendDomain>reap>
> shutdown id=2 reason=poweroff
> [2006-01-28 22:06:57 xend] DEBUG (XendDomain:487)
> domain_restart_schedule> 2 poweroff 0
> [2006-01-28 22:06:57 xend] INFO (XendDomain:568) Destroying domain:
> name=vpn1dc1 
> etc., etc., etc.
> 
> Here are the configuration files for the two domUs:
> #  -*- mode: python; -*-
>
#===========================================================================>
# Python configuration setup for ''xm create''.
> # This script sets the parameters used when a domain is created using
> ''xm create''.
> # You use a separate script for each domain you want to create, or 
> # you can set the parameters for the domain on the xm command line.
>
#===========================================================================>
>
#----------------------------------------------------------------------------
> # Kernel image file.
> kernel = "/boot/vmlinuz-2.4.30-xenUVPN"
> 
> # Optional ramdisk.
> #ramdisk = "/boot/initrd.gz"
> 
> # The domain build function. Default is ''linux''.
> #builder=''linux''
> 
> # Initial memory allocation (in megabytes) for the new domain.
> memory = 96
> 
> # A name for your domain. All domains must have different names.
> name = "niagras1"
> 
> # Which CPU to start domain on? 
> #cpu = -1   # leave to Xen to pick
> 
>
#----------------------------------------------------------------------------
> # Define network interfaces.
> 
> # Number of network interfaces. Default is 1.
> nics=2
> 
> # Optionally define mac and/or bridge for the network interfaces.
> # Random MACs are assigned if not given.
> #vif = [ ''mac=aa:00:00:00:00:11, bridge=xen-br0'' ]
> 
> vif = [''mac=02:00:00:01:02:00, bridge=xen-br0'' ]
> 
>
#----------------------------------------------------------------------------
> # Define the disk devices you want the domain to have access to, and
> # what you want them accessible as.
> # Each disk entry is of the form phy:UNAME,DEV,MODE
> # where UNAME is the device, DEV is the device name the domain will see,
> # and MODE is r for read-only, w for read-write.
> 
> #disk = [ ''phy:hda1,hda1,w'' ]
> 
> disk = [ ''phy:VG1/niagras1root,sda1,w'',
''phy:VG1/niagras1swap,sda2,w'' ]
> 
>
#----------------------------------------------------------------------------
> # Set the kernel command line for the new domain.
> # You only need to define the IP parameters and hostname if the
domain''s
> # IP config doesn''t, e.g. in ifcfg-eth0 or via DHCP.
> # You can use ''extra'' to set the runlevel and custom
environment
> # variables used by custom rc scripts (e.g. VMID=, usr= ).
> 
> # Set if you want dhcp to allocate the IP address.
> #dhcp="dhcp"
> # Set netmask.
> #netmask> # Set default gateway.
> #gateway> # Set the hostname.
> #hostname= "vm%d" % vmid
> 
> # Set root device.
> root = "/dev/sda1 ro"
> 
> # Root device for nfs.
> #root = "/dev/nfs"
> # The nfs server.
> #nfs_server = ''169.254.1.0''  
> # Root directory on the nfs server.
> #nfs_root   = ''/full/path/to/root/directory''
> 
> # Sets runlevel 4.
> #extra = "4"
> 
>
#----------------------------------------------------------------------------
> # Set according to whether you want the domain restarted when it exits.
> # The default is ''onreboot'', which restarts the domain
when it shuts
> down
> # with exit code reboot.
> # Other values are ''always'', and
''never''.
> 
> #restart = ''onreboot''
> 
>
#===========================================================================>
> 
> #  -*- mode: python; -*-
>
#===========================================================================>
# Python configuration setup for ''xm create''.
> # This script sets the parameters used when a domain is created using
> ''xm create''.
> # You use a separate script for each domain you want to create, or 
> # you can set the parameters for the domain on the xm command line.
>
#===========================================================================>
>
#----------------------------------------------------------------------------
> # Kernel image file.
> kernel = "/boot/vmlinuz-2.4.30-xenUVPN"
> 
> # Optional ramdisk.
> #ramdisk = "/boot/initrd.gz"
> 
> # The domain build function. Default is ''linux''.
> #builder=''linux''
> 
> # Initial memory allocation (in megabytes) for the new domain.
> memory = 96
> 
> # A name for your domain. All domains must have different names.
> name = "vpn1dc1"
> 
> # Which CPU to start domain on? 
> #cpu = -1   # leave to Xen to pick
> 
>
#----------------------------------------------------------------------------
> # Define network interfaces.
> 
> # Number of network interfaces. Default is 1.
> nics=2
> 
> # Optionally define mac and/or bridge for the network interfaces.
> # Random MACs are assigned if not given.
> #vif = [ ''mac=aa:00:00:00:00:11, bridge=xen-br0'' ]
> 
> vif = [''mac=02:00:00:01:01:00, bridge=xen-br0'',
''mac=02:00:00:01:01:01,
> bridge=xen-br1'' ]
> 
>
#----------------------------------------------------------------------------
> # Define the disk devices you want the domain to have access to, and
> # what you want them accessible as.
> # Each disk entry is of the form phy:UNAME,DEV,MODE
> # where UNAME is the device, DEV is the device name the domain will see,
> # and MODE is r for read-only, w for read-write.
> 
> #disk = [ ''phy:hda1,hda1,w'' ]
> 
> disk = [ ''phy:VG1/vpn1dc1root,sda1,w'',
''phy:VG1/vpn1dc1swap,sda2,w'' ]
> 
>
#----------------------------------------------------------------------------
> # Set the kernel command line for the new domain.
> # You only need to define the IP parameters and hostname if the
domain''s
> # IP config doesn''t, e.g. in ifcfg-eth0 or via DHCP.
> # You can use ''extra'' to set the runlevel and custom
environment
> # variables used by custom rc scripts (e.g. VMID=, usr= ).
> 
> # Set if you want dhcp to allocate the IP address.
> #dhcp="dhcp"
> # Set netmask.
> #netmask> # Set default gateway.
> #gateway> # Set the hostname.
> #hostname= "vm%d" % vmid
> 
> # Set root device.
> root = "/dev/sda1 ro"
> 
> # Root device for nfs.
> #root = "/dev/nfs"
> # The nfs server.
> #nfs_server = ''169.254.1.0''  
> # Root directory on the nfs server.
> #nfs_root   = ''/full/path/to/root/directory''
> 
> # Sets runlevel 4.
> #extra = "4"
> 
>
#----------------------------------------------------------------------------
> # Set according to whether you want the domain restarted when it exits.
> # The default is ''onreboot'', which restarts the domain
when it shuts
> down
> # with exit code reboot.
> # Other values are ''always'', and
''never''.
> 
> #restart = ''onreboot''
> 
>
#===========================================================================>
> 
> Here is the network script:
> 
> # Exit if anything goes wrong.
> set -e 
> 
> # First arg is the operation.
> OP=$1
> shift
> 
> # Pull variables in args in to environment.
> for arg ; do export "${arg}" ; done
> 
> bridge=${bridge:-xen-br0}
> netdev=${netdev:-eth0}
> antispoof=${antispoof:-yes}
> SCRIPTDIR=/etc/xen/scripts
> 
> echo "network $OP bridge=$bridge netdev=$netdev
antispoof=$antispoof"
> 
> # Usage: transfer_addrs src dst
> # Copy all IP addresses (including aliases) from device $src to device
$dst.
> transfer_addrs () {
>     local src=$1
>     local dst=$2
>     # Don''t bother if $dst already has IP addresses.
>     if ip addr show dev ${dst} | egrep -q ''^ *inet '' ;
then
>         return
>     fi
>     # Address lines start with ''inet'' and have the device
in them.
>     # Replace ''inet'' with ''ip addr add''
and change the device name $src
>     # to ''dev $src''. Remove netmask as we''ll add
routes later.
>     ip addr show dev ${src} | egrep ''^ *inet '' | sed -e
"
> s/inet/ip addr add/
> s@\([0-9]\+\.[0-9]\+\.[0-9]\+\.[0-9]\+\)/[0-9]\+@\1@
> s/${src}/dev ${dst}/
> " | sh -e
> }
> 
> # Usage: del_addrs src
> del_addrs () {
>     local src=$1
>     ip addr show dev ${src} | egrep ''^ *inet '' | sed -e
"
> s/inet/ip addr del/
> s@\([0-9]\+\.[0-9]\+\.[0-9]\+\.[0-9]\+\)/[0-9]\+@\1@
> s/${src}/dev ${src}/
> " | sh -e
> }
> 
> # Usage: transfer_routes src dst
> # Get all IP routes to device $src, delete them, and
> # add the same routes to device $dst.
> # The original routes have to be deleted, otherwise adding them
> # for $dst fails (duplicate routes).
> transfer_routes () {
>     local src=$1
>     local dst=$2
>     # List all routes and grep the ones with $src in.
>     # Stick ''ip route del'' on the front to delete.
>     # Change $src to $dst and use ''ip route add'' to add.
>     ip route list | grep ${src} | sed -e "
> h
> s/^/ip route del /
> P
> g
> s/${src}/${dst}/
> s/^/ip route add /
> P
> d
> " | sh -e
> }
> 
> # Usage: create_bridge bridge
> create_bridge () {
>     local bridge=$1
> 
>     # Don''t create the bridge if it already exists.
>     if ! brctl show | grep -q ${bridge} ; then
>         brctl addbr ${bridge}
>         brctl stp ${bridge} off
>         brctl setfd ${bridge} 0
>     fi
>     ifconfig ${bridge} up
> }
> 
> # Usage: add_to_bridge bridge dev
> add_to_bridge () {
>     local bridge=$1
>     local dev=$2
>     # Don''t add $dev to $bridge if it''s already on a
bridge.
>     if ! brctl show | grep -q ${dev} ; then
>         brctl addif ${bridge} ${dev}
>     fi
> }
> 
> # Usage: antispoofing dev bridge
> # Set the default forwarding policy for $dev to drop.
> # Allow forwarding to the bridge.
> antispoofing () {
>     local dev=$1
>     local bridge=$2
> 
>     iptables -P FORWARD DROP
>     iptables -A FORWARD -m physdev --physdev-in ${dev} -j ACCEPT
> }
> 
> # Usage: show_status dev bridge
> # Print ifconfig and routes.
> show_status () {
>     local dev=$1
>     local bridge=$2
>     
>     echo
''============================================================''
>     ifconfig ${dev}
>     ifconfig ${bridge}
>     echo '' ''
>     ip route list
>     echo '' ''
>     route -n
>     echo
''============================================================''
> }
> 
> op_start () {
>     if [ "${bridge}" == "null" ] ; then
>         return
>     fi
> # Create the bridge and give it the interface IP addresses.
> # Move the interface routes onto the bridge.
>     while read bridge netdev
> 	do
>             create_bridge ${bridge}
>             if [ -n "$(ip add ls dev $netdev | grep ''inet
'')" ]; then
>                 transfer_addrs ${netdev} ${bridge}
>                 transfer_routes ${netdev} ${bridge}
>             fi
> # Don''t add $dev to $bridge if it''s already on a bridge.
>             if ! brctl show | grep -q ${netdev} ; then
>                 brctl addif ${bridge} ${netdev}
>             fi
>             
>             if [ ${antispoof} == ''yes'' ] ; then
>                 antispoofing ${netdev} ${bridge}
>             fi
>     done < ${SCRIPTDIR}/bridges.txt
> }
>
> op_stop () {
>     if [ "${bridge}" == "null" ] ; then
>         return
>     fi
> # Remove the interface from the bridge.
> # Move the routes back to the interface.
>     while read bridge netdev
>     do
>         brctl delif ${bridge} ${netdev}              
>         if [ -n "$(ip add ls dev $netdev | grep ''inet
'')" ]; then
>             transfer_routes ${bridge} ${netdev}
>         fi
>     done < ${SCRIPTDIR}/bridges.txt
> # It''s not our place to be enabling forwarding...
> }
> 
> case ${OP} in
>   start)
>         op_start
>         ;;
>     
>     stop)
>         op_stop
>         ;;
> 
>     status)
>         show_status ${netdev} ${bridge}
>        ;;
> 
>     *)
>        echo ''Unknown command: '' ${OP}
>        echo ''Valid commands are: start, stop, status''
>        exit 1
> esac
> 
> Here is the bridges.txt file:
> xen-br0 eth0
> xen-br1 eth1
> xen-br2 eth2
> 
> 
> I''m stumped but it sure looks serious.  Have I produced some
dreadful
> misconfiguration? This host is being shipped to a remote data center on
> Monday and goes into production in a week.  Any help would be greatly
> appreciated.  Thanks - John
Perhaps I should also mention that there is no address assigned to eth0
and eth2 and thus no address for xen-br0 and xen-br2 -
John
> 
-- 
John A. Sullivan III
Open Source Development Corporation
+1 207-985-7880
jsullivan@opensourcedevel.com

Financially sustainable open source development
http://www.opensourcedevel.com


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel