We moved some servers from one DC to another DC. The IP addresses and
routing stayed the same (DC handled this) but now XEN doesn''t work on
one of the servers.
When I logged in, XEN wasn''t running so I manually started it with
"/etc/init.d/xend restart" but then there''s no network bridge
at all.
Manually starting the network bridge gave me this error:
root@usaxen03:[~]$ /etc/xen/scripts/network-bridge start
Link veth0 is missing.
This may be because you have reached the limit of the number of interfaces
that the loopback driver supports. If the loopback driver is a module, you
may raise this limit by passing it as a parameter (nloopbacks=<N>); if the
driver is compiled statically into the kernel, then you may set the parameter
using loopback.nloopbacks=<N> on the domain 0 kernel command line.
root@usaxen03:[~]$ uname -a
Linux usaxen03 2.6.18-348.12.1.el5xen #1 SMP Wed Jul 10 01:13:54 EDT
2013 x86_64 x86_64 x86_64 GNU/Linux
root@usaxen03:[~]$ more /etc/redhat-release
Scientific Linux SL release 5.7 (Boron)
root@usaxen03:[~]$
I get the same error in the logs:
root@usaxen03:[~]$ tail -f /var/log/xen/xend-debug.log
using loopback.nloopbacks=<N> on the domain 0 kernel command line.
Link veth0 is missing.
This may be because you have reached the limit of the number of interfaces
that the loopback driver supports. If the loopback driver is a module, you
may raise this limit by passing it as a parameter (nloopbacks=<N>); if the
driver is compiled statically into the kernel, then you may set the parameter
using loopback.nloopbacks=<N> on the domain 0 kernel command line.
root@usaxen03:[~]$ tail -f /var/log/xen/xend.log
[2013-07-19 08:15:13 xend.XendDomainInfo 3805] DEBUG
(XendDomainInfo:1335) XendDomainInfo.handleShutdownWatch
[2013-07-19 08:19:21 xend 3804] INFO (SrvDaemon:190) Xend stopped due
to signal 15.
[2013-07-19 08:19:21 xend 4199] INFO (SrvDaemon:283) Xend Daemon started
[2013-07-19 08:19:21 xend 4199] INFO (SrvDaemon:287) Xend changeset:
unavailable.
[2013-07-19 08:19:21 xend.XendDomainInfo 4199] DEBUG
(XendDomainInfo:283) XendDomainInfo.recreate({''paused'': 0,
''cpu_time'':
24358134692L, ''ssidref'': 0, ''vm'': 0,
''shutdown_reason'': 0, ''dying'': 0,
''mem_kb'': 524288L, ''domid'': 0,
''max_vcpu_id'': 7, ''crashed'': 0,
''running'': 1, ''maxmem_kb'': 17179869180L,
''shutdon'': 0, ''online_vcpus'':
1, ''handle'': [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0],
''blocked'': 0})
[2013-07-19 08:19:21 xend.XendDomainInfo 4199] INFO
(XendDomainInfo:295) Recreating domain 0, UUID
00000000-0000-0000-0000-000000000000.
[2013-07-19 08:19:21 xend.XendDomainInfo 4199] DEBUG
(XendDomainInfo:1439) Setting maximum memory value of domain Domain-0
(0) to 16375 MiB.
[2013-07-19 08:19:21 xend 4199] DEBUG (XendDomain:166) number of vcpus
to use is 1
[2013-07-19 08:19:21 xend 4199] INFO (SrvServer:116) unix
path=/var/lib/xend/xend-socket
[2013-07-19 08:19:21 xend.XendDomainInfo 4199] DEBUG
(XendDomainInfo:1335) XendDomainInfo.handleShutdownWatch
Does anyone know what could cause this, or how to fix it?
--
Kind Regards
Rudi Ahlers
SoftDux
Website: http://www.SoftDux.com
Technical Blog: http://Blog.SoftDux.com
Cell: 082 554 7532
Fax: 086 268 8492
On Fri, 2013-07-19 at 08:13 +0200, Rudi Ahlers wrote:> We moved some servers from one DC to another DC. The IP addresses and > routing stayed the same (DC handled this) but now XEN doesn''t work on > one of the servers. > > > When I logged in, XEN wasn''t running so I manually started it with > "/etc/init.d/xend restart" but then there''s no network bridge at all.Was Xen supposed to run automatically? Did you investigate this? Perhaps this was a symptom of a larger problem? You don''t say which version of Xen you are running, did you also start the xencommons service which is needed from somewhere around Xen 4.1 onwards?> Link veth0 is missing. > This may be because you have reached the limit of the number of interfaces > that the loopback driver supports. If the loopback driver is a module, you > may raise this limit by passing it as a parameter (nloopbacks=<N>); if the > driver is compiled statically into the kernel, then you may set the parameter > using loopback.nloopbacks=<N> on the domain 0 kernel command line.[...]> Does anyone know what could cause this, or how to fix it?Did you try the advice given in the error message? Ian.
Please keep the list in the loop. Re-adding CC. On Fri, 2013-07-19 at 09:51 +0200, Rudi Ahlers wrote:> Yes, this server has been running XEN for about 3 years now. It was > rebooted 87 days ago, if I remember correctly for some updates. This > is an active XEN server with 3 XEN domU virtual machines service > websites to clients on the internet.It seems entirely possible to me that some reconfiguration in the previous 87 days wasn''t applied in a way which would survive a reboot.> I''ve managed to "fix it" by manually creating the bridge, but then I > loose network connectivity to the remote server, and I don''t have a > KVM connection to it. ... yet at least.I think that needs to be a priority for you unless there is a second Ethernet device available (in which case you may be able to fix eth0 while logged in via eth1).> These commands re-create the bridge, but then the host node''s > networking doesn''t work, although the VM''s come up about 10 minutes > later. The thing is, I can''t see what''s going on after running "brctl > addif xenbr0 eth0". The firewall is disabled right now, so it''s not > blocking anything. > > brctl addbr xenbr0 > brctl stp xenbr0 off > brctl setfd xenbr0 0 > ip link set xenbr0 up > brctl addif xenbr0 eth0None of these will transfer the ip configuration from eth0 to xenbr0, which is necessary for host networking to work. If you are using a CentOS like environment then this should all have been done by your ifcfg scripts, or perhaps the network-bridge stuff if you are still using that. Have you compared this system''s configuration with that of another working server?> I have googled that error, but the results I got all related to people > who just setup XEN and the bridge wasn''t setup in the config. The > problem is, that it was setup, a few years ago already so the steps of > adding the bridge to the bridge-network script doesn''t apply here.It sounds to me very much like someone has broken this configuration on this server and nobody noticed until it was rebooted. I would suggest starting from that assumption and double checking everything going through as you would an initial setup. Ian.
On Fri, Jul 19, 2013 at 10:09 AM, Ian Campbell <Ian.Campbell@citrix.com> wrote:> Please keep the list in the loop. Re-adding CC.Sorry, forgot to hit "Reply all".> > On Fri, 2013-07-19 at 09:51 +0200, Rudi Ahlers wrote: >> Yes, this server has been running XEN for about 3 years now. It was >> rebooted 87 days ago, if I remember correctly for some updates. This >> is an active XEN server with 3 XEN domU virtual machines service >> websites to clients on the internet. > > It seems entirely possible to me that some reconfiguration in the > previous 87 days wasn''t applied in a way which would survive a reboot.I''ve always relied on "yum update" for updates and haven''t run into this kind of problem before. So, could it be that there''s an upstream bug, either in Scientifc Linux, or XEN''s rpm''s that could cause this? I don''t know where to search, since when I searched on the error I got a lot of older posts relating to setting up XEN from scratch.> >> I''ve managed to "fix it" by manually creating the bridge, but then I >> loose network connectivity to the remote server, and I don''t have a >> KVM connection to it. ... yet at least. > > I think that needs to be a priority for you unless there is a second > Ethernet device available (in which case you may be able to fix eth0 > while logged in via eth1). >I''m still waiting for the DC to setup the KVM. They had to move "the whole DC to a new building", including networking gear and a boat load full of servers. And they''re still busy moving everything, so for now I have limited access and limited tools at my disposal. I''ll see if I can get another LAN cable hooked upto eth1 in the mean time.>> These commands re-create the bridge, but then the host node''s >> networking doesn''t work, although the VM''s come up about 10 minutes >> later. The thing is, I can''t see what''s going on after running "brctl >> addif xenbr0 eth0". The firewall is disabled right now, so it''s not >> blocking anything. >> >> brctl addbr xenbr0 >> brctl stp xenbr0 off >> brctl setfd xenbr0 0 >> ip link set xenbr0 up >> brctl addif xenbr0 eth0 > > None of these will transfer the ip configuration from eth0 to xenbr0, > which is necessary for host networking to work.Ok, so what else should I have done in order to transfer the IP configuration from eth0 to xenbr0? Right now I can ping the server but it doesn''t respond to SSH, webmin (used for other purposes, locked to internal IP''s) or any of the other network services> > If you are using a CentOS like environment then this should all have > been done by your ifcfg scripts, or perhaps the network-bridge stuff if > you are still using that. > > Have you compared this system''s configuration with that of another > working server? > >> I have googled that error, but the results I got all related to people >> who just setup XEN and the bridge wasn''t setup in the config. The >> problem is, that it was setup, a few years ago already so the steps of >> adding the bridge to the bridge-network script doesn''t apply here. > > It sounds to me very much like someone has broken this configuration on > this server and nobody noticed until it was rebooted. I would suggest > starting from that assumption and double checking everything going > through as you would an initial setup. > > Ian.No one really works on the host servers, apart from doing a yum update from time to time and I can''t see any reason why anyone would have touched the XEN configuration or setup at all. "why fix it if it aint broken"? -- Kind Regards Rudi Ahlers SoftDux Website: http://www.SoftDux.com Technical Blog: http://Blog.SoftDux.com Cell: 082 554 7532 Fax: 086 268 8492
On Fri, Jul 19, 2013 at 10:48 AM, Rudi Ahlers <Rudi@softdux.com> wrote:> On Fri, Jul 19, 2013 at 10:09 AM, Ian Campbell <Ian.Campbell@citrix.com> wrote: >> Please keep the list in the loop. Re-adding CC. > > Sorry, forgot to hit "Reply all". > > >> >> On Fri, 2013-07-19 at 09:51 +0200, Rudi Ahlers wrote: >>> Yes, this server has been running XEN for about 3 years now. It was >>> rebooted 87 days ago, if I remember correctly for some updates. This >>> is an active XEN server with 3 XEN domU virtual machines service >>> websites to clients on the internet. >> >> It seems entirely possible to me that some reconfiguration in the >> previous 87 days wasn''t applied in a way which would survive a reboot. > > I''ve always relied on "yum update" for updates and haven''t run into > this kind of problem before. So, could it be that there''s an upstream > bug, either in Scientifc Linux, or XEN''s rpm''s that could cause this? > I don''t know where to search, since when I searched on the error I got > a lot of older posts relating to setting up XEN from scratch. > >> >>> I''ve managed to "fix it" by manually creating the bridge, but then I >>> loose network connectivity to the remote server, and I don''t have a >>> KVM connection to it. ... yet at least. >> >> I think that needs to be a priority for you unless there is a second >> Ethernet device available (in which case you may be able to fix eth0 >> while logged in via eth1). >> > > I''m still waiting for the DC to setup the KVM. They had to move "the > whole DC to a new building", including networking gear and a boat load > full of servers. And they''re still busy moving everything, so for now > I have limited access and limited tools at my disposal. > > I''ll see if I can get another LAN cable hooked upto eth1 in the mean time. > >>> These commands re-create the bridge, but then the host node''s >>> networking doesn''t work, although the VM''s come up about 10 minutes >>> later. The thing is, I can''t see what''s going on after running "brctl >>> addif xenbr0 eth0". The firewall is disabled right now, so it''s not >>> blocking anything. >>> >>> brctl addbr xenbr0 >>> brctl stp xenbr0 off >>> brctl setfd xenbr0 0 >>> ip link set xenbr0 up >>> brctl addif xenbr0 eth0 >> >> None of these will transfer the ip configuration from eth0 to xenbr0, >> which is necessary for host networking to work. > > Ok, so what else should I have done in order to transfer the IP > configuration from eth0 to xenbr0? > Right now I can ping the server but it doesn''t respond to SSH, webmin > (used for other purposes, locked to internal IP''s) or any of the other > network services > >> >> If you are using a CentOS like environment then this should all have >> been done by your ifcfg scripts, or perhaps the network-bridge stuff if >> you are still using that. >> >> Have you compared this system''s configuration with that of another >> working server? >> >>> I have googled that error, but the results I got all related to people >>> who just setup XEN and the bridge wasn''t setup in the config. The >>> problem is, that it was setup, a few years ago already so the steps of >>> adding the bridge to the bridge-network script doesn''t apply here. >> >> It sounds to me very much like someone has broken this configuration on >> this server and nobody noticed until it was rebooted. I would suggest >> starting from that assumption and double checking everything going >> through as you would an initial setup. >> >> Ian. > > No one really works on the host servers, apart from doing a yum update > from time to time and I can''t see any reason why anyone would have > touched the XEN configuration or setup at all. "why fix it if it aint > broken"? > >Can anyone help me with this please? I have compared this setup to a working server but can''t see why the network bridge isn''t working -- Kind Regards Rudi Ahlers SoftDux Website: http://www.SoftDux.com Technical Blog: http://Blog.SoftDux.com Cell: 082 554 7532 Fax: 086 268 8492