Hi,
With current unstable (hg7500) I have the problem that networking for
domU''s doesn''t work reliable. Seems to be more or less
random, havn''t
seen any pattern yet, sometimes it works normally and sometimes it
doesn''t.
In case it does _not_ work it looks like this (domain 1 runs fine,
domain 3 is broken):
The xen-backend device is present in sysfs:
master-xen root ~# ls /sys/devices/xen-backend
. .. vbd-1-768 vbd-1-832 vbd-3-51712 vif-1-0 vif-3-0
The vif network interface is missing:
master-xen root ~# ip link ls | grep vif
3: vif0.0: <BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue
6: vif1.0: <BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue
/etc/xen/scripts/vif-bridge is not called according to /var/log/messages:
master-xen root /var/log# grep vif-bridge /var/log/messages
Oct 27 15:22:50 master-xen logger: /etc/xen/scripts/vif-bridge: up
XENBUS_PATH=backend/vif/1/0
The backend entries in xenstore look just fine:
master-xen root ~# xenstore-dump backend/vif
1/
0/
bridge xenbr0
mac aa:00:57:4f:25:a7
handle 0
script /etc/xen/scripts/vif-bridge
frontend-id 1
domain debian
frontend /local/domain/1/device/vif/0
3/
0/
bridge xenbr0
mac aa:00:ca:d7:83:e1
handle 0
script /etc/xen/scripts/vif-bridge
frontend-id 3
domain gentoo
frontend /local/domain/3/device/vif/0
ideas anyone?
cheers,
Gerd
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel
On Thu, Oct 27, 2005 at 03:57:50PM +0200, Gerd Knorr wrote:> With current unstable (hg7500) I have the problem that networking for > domU''s doesn''t work reliable. Seems to be more or less random, havn''t > seen any pattern yet, sometimes it works normally and sometimes it doesn''t.There does seem to be a problem with the hotplug scripts either not running at all or bailing out early. Please could you put some tracing into the relevant scripts -- /etc/hotplug/xen-backend.agent, /etc/hotplug.d/default/default.hotplug, and / or /etc/udev/rules.d/xen-backend.rules, depending on your system, and see if you can figure out whether the hotplug event is firing at all, and if it is, where the script bails out. To have got the device in /sys/devices/xen-backend tells us that it''s managed almost everything before the hotplug scripts get started, so this shouldn''t be difficult to track down. Thanks, Ewan. _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
> There does seem to be a problem with the hotplug scripts either not running at > all or bailing out early.Hmm, no, seems to be a missing hotplug event. I''ve added a debug line to xen-backend.rules: SUBSYSTEM=="xen-backend", RUN+="/bin/logger -p daemon.debug -- xen-backend: $env{ACTION} %k" A successfull start looks like this: Oct 27 17:34:50 master-xen logger: xen-backend: add vbd-19-768 Oct 27 17:34:50 master-xen logger: xen-backend: add vbd-19-832 Oct 27 17:34:50 master-xen logger: xen-backend: add vif-19-0 Oct 27 17:34:50 master-xen logger: xen-backend: online vif-19-0 With broken network the "online" event is missing: Oct 27 17:35:31 master-xen logger: xen-backend: add vbd-20-832 Oct 27 17:35:31 master-xen logger: xen-backend: add vbd-20-768 Oct 27 17:35:31 master-xen logger: xen-backend: add vif-20-0 Oh, and btw, reboot doesn''t work all the time either. Sometimes the domain comes up again as it should, and sometimes it doesn''t. cheers, Gerd _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
On Thu, Oct 27, 2005 at 05:43:38PM +0200, Gerd Knorr wrote:> Oh, and btw, reboot doesn''t work all the time either. Sometimes the > domain comes up again as it should, and sometimes it doesn''t.That one really is odd. Could you post your /var/log/xend.log when that happens? Ewan. _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Ewan Mellor wrote:> On Thu, Oct 27, 2005 at 05:43:38PM +0200, Gerd Knorr wrote: > > That one really is odd. Could you post your /var/log/xend.log when that > happens?Attached. The "error: (12, ''Cannot allocate memory'')" line looks strange. The domain has 128 MB assigned. Now (after the failed reboot) it looks like this: master-xen root ~# xm info | grep memory memory : 983 free_memory : 132 Should be enougth memory. I also have "(dom0-min-mem 256)" configured for xend, so it should happily take away memory from dom0 in case it needs more ... cheers, Gerd _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Hi,>> That one really is odd. Could you post your /var/log/xend.log when that >> happens? > > Attached. The "error: (12, ''Cannot allocate memory'')" line looks > strange. [ ... ]Hmm, there seems to be more than one bug. I''ve manually decreased the amount of memory for domain 0 so there is more free memory, which seems to sucessfully workaround the memory allocation issue. Now I got this one: [2005-10-28 11:55:39 xend.XendDomainInfo] INFO (XendDomainInfo:741) Domain has shutdown: name=debian id=9 reason=reboot. [2005-10-28 11:55:39 xend.XendDomainInfo] ERROR (XendDomainInfo:1278) 1130493287.56 [2005-10-28 11:55:39 xend.XendDomainInfo] DEBUG (XendDomainInfo:1156) XendDomainInfo.destroy: domid=9 [2005-10-28 11:55:39 xend.XendDomainInfo] DEBUG (XendDomainInfo:1164) XendDomainInfo.destroyDomain(9) cheers, Gerd _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
> [2005-10-28 10:55:49 xend.XendDomainInfo] ERROR (XendDomainInfo:1307) Failed to restart domain 1. > Traceback (most recent call last): > File "//usr/lib/python/xen/xend/XendDomainInfo.py", line 1300, in restart > new_dom = xd.domain_create(config) > File "//usr/lib/python/xen/xend/XendDomain.py", line 215, in domain_create > dominfo = XendDomainInfo.create(config) > File "//usr/lib/python/xen/xend/XendDomainInfo.py", line 152, in create > vm.initDomain() > File "//usr/lib/python/xen/xend/XendDomainInfo.py", line 1093, in initDomain > xc.domain_memory_increase_reservation(self.domid, m, 0, 0) > error: (12, ''Cannot allocate memory'')Oh, xend-debug.log has something as well: Failed allocation for dom 2: 32768 pages order 0 addr_bits 0 cheers, Gerd _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
On Fri, Oct 28, 2005 at 11:59:47AM +0200, Gerd Knorr wrote:> Hi, > > >>That one really is odd. Could you post your /var/log/xend.log when that > >>happens? > > > >Attached. The "error: (12, ''Cannot allocate memory'')" line looks > >strange. [ ... ] > > Hmm, there seems to be more than one bug. I''ve manually decreased the > amount of memory for domain 0 so there is more free memory, which seems > to sucessfully workaround the memory allocation issue. Now I got this one: > > [2005-10-28 11:55:39 xend.XendDomainInfo] INFO (XendDomainInfo:741) > Domain has shutdown: name=debian id=9 reason=reboot. > [2005-10-28 11:55:39 xend.XendDomainInfo] ERROR (XendDomainInfo:1278) > 1130493287.56 > [2005-10-28 11:55:39 xend.XendDomainInfo] DEBUG (XendDomainInfo:1156) > XendDomainInfo.destroy: domid=9 > [2005-10-28 11:55:39 xend.XendDomainInfo] DEBUG (XendDomainInfo:1164) > XendDomainInfo.destroyDomain(9)I''m not sure about your Cannot allocate memory error -- I''ll look into that. The rebooting code in Xend, however, is just plain wrong. I''ll fix that today. Thanks for your help, Gerd. Ewan. _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
> Hmm, no, seems to be a missing hotplug event.Sticked some printk''s into the net backend, looks like a xenbus issue: domU boot with broken network: netback_probe: add backend watch netback_probe: add frontend watch netback_probe: done frontend_changed: called frontend_changed: called frontend_changed: called frontend_changed: called domU boot with working network: netback_probe: add backend watch netback_probe: add frontend watch netback_probe: done backend_changed: called backend_changed: hotplug online netback_hotplug: called frontend_changed: called frontend_changed: called device vif2.0 entered promiscuous mode xenbr0: port 3(vif2.0) entering learning state xenbr0: topology change detected, propagating xenbr0: port 3(vif2.0) entering forwarding state frontend_changed: called frontend_changed: called frontend_changed: called Note that backend_changed is never called in the first case. cheers, Gerd _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel