Ted Kaczmarek
2005-Oct-22 02:26 UTC
[Xen-devel] Network still broken, new issue as well , 7468:17a9f111fa93
xen_changeset : Fri Oct 21 13:51:42 2005 +0100 7468:17a9f111fa93 Tyan 2462 SMP Athlon - FC4 dom0 Not one vif get mapped to the bridge, but the bridge otherwise appear to work fine with other nodes on the broadcast domain. I am also seeing a new issue as well. I see 2 vifs for a domU that has nics = 1 , this happens on 4 out of 10 domU''s. grepping my configs make me think that something has changed the way they are interpreted, all the the domU''s using mac, bridge have 2 vifs, where the one that ony have mac declared have 1 vif. grep nics idom* idom1:nics=1 idom10:nics = 1 idom2:nics = 1 idom3:nics = 1 idom4:nics = 1 idom5:nics = 1 idom6:nics = 1 idom7:nics = 1 idom8:nics = 1 idom9:nics = 1 grep vif idom* idom1:vif = [''mac=AA:00:00:00:00:17'',''bridge=xen-br0''] idom1:#vif = [''mac=AA:00:00:11:11:12'',''bridge=xen-br0'', ''mac=AA:00:00:11:11:22'',''bridge=xen-br1'', ''mac=AA:00:00:11:12:22'',''bridge=xen-br2''] idom1:#vif = [''mac=AA:00:00:11:11:11'' , ''bridge=xen-br0'' , ''bridge=xen-br1'' , ''bridge=xen-br2''] idom1:#vif = [''mac=AA:00:00:11:11:11'' , ''bridge=xen-br0'' , ''bridge=xen-br1'' ] idom1:#vif = [''mac=AA:00:00:11:11:11'',''bridge=xen-br0'' , ''mac=AA:00:00:11:11:12'', ''bridge=xen-br1''] idom1:#vif = [''mac=AA:00:00:11:11:11'' ''mac=AA:00:00:11:11:12''] BAD, eth1 won''t come up idom10:#vif = [ ''mac=aa:00:66:00:01:08, bridge=xen-br0'', ''mac=aa:00:66:01:01:08, bridge=xen-br1'', ''mac=aa:00:66:02:01:08, bridge=xen-br2'' ] idom10:vif = [ ''mac=AA:00:00:00:01:08, bridge=xen-br0'' ] idom2:vif = [''mac=AA:00:00:00:01:01''] idom3:vif = [''mac=AA:00:00:00:01:02'' , ''bridge=xen-br0''] idom4:vif = [''mac=AA:00:00:00:01:03'' , ''bridge=xen-br0''] idom5:vif = [''mac=AA:00:00"00"01:04'' , ''bridge=xen-br0''] idom6:vif = [ ''mac=AA:00:00:00:01:05'' ] idom7:vif = [ ''mac=AA:00:00:00:01:07'' ] idom8:vif = [ ''mac=AA:00:00:00:01:08'' ] idom9:vif = [ ''mac=AA:00:00:00:01:09'' ] list_vm_nets_backends idom1 (backend/local/domain/0/backend/vif/1/0) (backend/local/domain/0/backend/vif/1/1) idom2 (backend/local/domain/0/backend/vif/2/0) idom3 (backend/local/domain/0/backend/vif/11/0) (backend/local/domain/0/backend/vif/11/1) idom4 (backend/local/domain/0/backend/vif/4/0) (backend/local/domain/0/backend/vif/4/1) idom5 (backend/local/domain/0/backend/vif/5/0) (backend/local/domain/0/backend/vif/5/1) idom6 (backend/local/domain/0/backend/vif/6/0) idom7 (backend/local/domain/0/backend/vif/7/0) idom8 (backend/local/domain/0/backend/vif/8/0) idom9 (backend/local/domain/0/backend/vif/9/0) idom10 (backend/local/domain/0/backend/vif/10/0) I have to get caught up on the archives as I know lots of network related changes are in progress. Would be great if we could get a little change summary, I have tried to throw some kind of user help doc together but you guys are like rabbits :-) IMHO the configs should start to get stricter as well, right now one can get away with spaces anywhere it seems, and until today it didn''t matter if I specified mac or mac/bridge. Same goes for many other options such as cpu = -1, etc.. In the past I have always seen this come back to bite you in the a@@. If the rules for the config get tighter, their is less room for error across the board. Regards, Ted _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Ewan Mellor
2005-Oct-25 17:40 UTC
Re: [Xen-devel] Network still broken, new issue as well , 7468:17a9f111fa93
On Fri, Oct 21, 2005 at 10:26:30PM -0400, Ted Kaczmarek wrote:> I see 2 vifs for a domU that has nics = 1 , this happens on 4 out of 10 > domU''s. grepping my configs make me think that something has changed the > way they are interpreted, all the the domU''s using mac, bridge have 2 > vifs, where the one that ony have mac declared have 1 vif. > > [Snip nics] > > grep vif idom*> idom1:vif = [''mac=AA:00:00:00:00:17'',''bridge=xen-br0''] > [Snip] > idom10:vif = [ ''mac=AA:00:00:00:01:08, bridge=xen-br0'' ]Here, idom1 is declared using two strings, but idom10 is declared using just one. I''m guessing a little here, but from reading the code it looks like this means that idom1 has two vifs, one with a default bridge and the given MAC, and one with a random MAC and the given bridge, whereas idom10 has one vif, with the given MAC and bridge. I think this is fairly sensible behaviour. Whether it is a regression or not, I''m not sure. It might actually have been a long-standing bug that has been masked by other long standing bugs, and now we''re doing things as intended. The vifs= setting seems to only apply if the number of entries in vif= is less than the number specified in vifs=. In that case, the rest of the vifs are created, with default details. This is fairly barbaric, as you''ve found out. We probably ought to have a "extra_vifs" setting or something, to make it clear that they are separate from those configured using vif= and to make it clear that this number is not a limit, to avoid the kind of confusion you''ve suffered. That said, I''m pushing back on configuration file format changes at the moment, because we are trying to stabilise for release, and though the format is pretty hideous, many people have stable configurations now, and I don''t want to break them without strong justification.> idom5:vif = [''mac=AA:00:00"00"01:04'' , ''bridge=xen-br0'']This MAC has quotation marks in it. Lord only knows what that does, but it can''t be doing what you want, so I''d fix that ;-) Regarding your other network problems, if you are still suffering once you''ve changed your configuration, then please submit a full set of logs. The scripts involved have changed recently, as you know, but they are very difficult to test, because we don''t have a great variation of network topologies available. I''d be looking for ifconfig brctl show cat /var/log/syslog cat /var/log/debug cat /var/log/xend.log Ewan. _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Ted Kaczmarek
2005-Oct-25 20:52 UTC
Re: [Xen-devel] Network still broken, new issue as well , 7468:17a9f111fa93
I have to run a very old changeset just to boot my SMP dom0, opened up a bug on this a few days ago. Ever since 2.6.12.5 version started showing up. Have tried many changesets with no joy. On my UP machine things behave better in general, matter of fast I have a domU up for almost 5 days now with only minor issues on the UP P4 test box, but the networking testing is limited by the APCI issues. http://bugzilla.xensource.com/bugzilla/show_bug.cgi?id=359 I can understand the issues with configs and the points are valid. My experience in the past has taught me that stricter configs actually lead to less human error, less bugs in the code, and less ghost chasing by all parties involved, which in turn leads to higher productivity in general. I intentionally throw lots of variations in my configs for the purpose of learning and testing, it is a double edged sword to be sure. Regards, Ted On Tue, 2005-10-25 at 18:40 +0100, Ewan Mellor wrote:> On Fri, Oct 21, 2005 at 10:26:30PM -0400, Ted Kaczmarek wrote: > > > I see 2 vifs for a domU that has nics = 1 , this happens on 4 out of 10 > > domU''s. grepping my configs make me think that something has changed the > > way they are interpreted, all the the domU''s using mac, bridge have 2 > > vifs, where the one that ony have mac declared have 1 vif. > > > > [Snip nics] > > > > grep vif idom* > > > idom1:vif = [''mac=AA:00:00:00:00:17'',''bridge=xen-br0''] > > [Snip] > > idom10:vif = [ ''mac=AA:00:00:00:01:08, bridge=xen-br0'' ] > > Here, idom1 is declared using two strings, but idom10 is declared using just > one. I''m guessing a little here, but from reading the code it looks like this > means that idom1 has two vifs, one with a default bridge and the given MAC, > and one with a random MAC and the given bridge, whereas idom10 has one vif, > with the given MAC and bridge. > > I think this is fairly sensible behaviour. Whether it is a regression or not, > I''m not sure. It might actually have been a long-standing bug that has been > masked by other long standing bugs, and now we''re doing things as intended. > > The vifs= setting seems to only apply if the number of entries in vif= is less > than the number specified in vifs=. In that case, the rest of the vifs are > created, with default details. This is fairly barbaric, as you''ve found out. > We probably ought to have a "extra_vifs" setting or something, to make it > clear that they are separate from those configured using vif= and to make it > clear that this number is not a limit, to avoid the kind of confusion you''ve > suffered. > > That said, I''m pushing back on configuration file format changes at the > moment, because we are trying to stabilise for release, and though the format > is pretty hideous, many people have stable configurations now, and I don''t > want to break them without strong justification. > > > idom5:vif = [''mac=AA:00:00"00"01:04'' , ''bridge=xen-br0''] > > This MAC has quotation marks in it. Lord only knows what that does, but it > can''t be doing what you want, so I''d fix that ;-) > > > Regarding your other network problems, if you are still suffering once you''ve > changed your configuration, then please submit a full set of logs. The > scripts involved have changed recently, as you know, but they are very > difficult to test, because we don''t have a great variation of network > topologies available. I''d be looking for > > ifconfig > brctl show > cat /var/log/syslog > cat /var/log/debug > cat /var/log/xend.log > > Ewan._______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Paul Larson
2005-Oct-25 22:28 UTC
Re: [Xen-devel] Network still broken, new issue as well , 7468:17a9f111fa93
Ewan Mellor wrote:> Regarding your other network problems, if you are still suffering once you''ve > changed your configuration, then please submit a full set of logs. The > scripts involved have changed recently, as you know, but they are very > difficult to test, because we don''t have a great variation of network > topologies available. I''d be looking for > > ifconfig > brctl show > cat /var/log/syslog > cat /var/log/debug > cat /var/log/xend.log >Please see bug #363 that''s currently biting several of us with systems which, for various reasons, use eth1 instead of eth0. I hope you don''t mind, I attached logfiles and command output to the bug rather than spamming here with it. We have always had to workaround this it seems, but were hoping the new script would fix it. Instead, with the new script we just had to find a new workaround. Basicially, on these systems it assumes eth0 (which we don''t use), so we never get networking on domU. Thanks, Paul Larson _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Ewan Mellor
2005-Oct-25 23:01 UTC
Re: [Xen-devel] Network still broken, new issue as well , 7468:17a9f111fa93
On Tue, Oct 25, 2005 at 05:28:17PM -0500, Paul Larson wrote:> Please see bug #363 that''s currently biting several of us with systems > which, for various reasons, use eth1 instead of eth0. I hope you don''t > mind, I attached logfiles and command output to the bug rather than > spamming here with it.I don''t mind that at all, no. In fact, it''s much better for me if the logs are on the bug, and drawing attention to a new bug on the list is good too.> We have always had to workaround this it seems, > but were hoping the new script would fix it. Instead, with the new > script we just had to find a new workaround. > > Basicially, on these systems it assumes eth0 (which we don''t use), so we > never get networking on domU.I think you want to set, in /etc/xen/xend-config.sxp, (network-script blade-network) and then create a file /etc/xen/scripts/blade-network that looks like: #!/bin/sh dir=$(dirname "$0") exec "$dir/network-bridge" "$@" "netdev=eth1" Don''t forget to chmod +x /etc/xen/scripts/blade-network, too. Hopefully, that will do the trick. Let me know how you get on. Ewan. _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Michael Lessard
2005-Oct-26 00:10 UTC
Re: [Xen-devel] Network still broken, new issue as well , 7468:17a9f111fa93
On 10/25/05, Ewan Mellor <ewan@xensource.com> wrote:> > On Tue, Oct 25, 2005 at 05:28:17PM -0500, Paul Larson wrote: > > > Please see bug #363 that''s currently biting several of us with systems > > which, for various reasons, use eth1 instead of eth0. I hope you don''t > > mind, I attached logfiles and command output to the bug rather than > > spamming here with it. > > I don''t mind that at all, no. In fact, it''s much better for me if the logs > are on the bug, and drawing attention to a new bug on the list is good > too. > > > We have always had to workaround this it seems, > > but were hoping the new script would fix it. Instead, with the new > > script we just had to find a new workaround. > > > > Basicially, on these systems it assumes eth0 (which we don''t use), so we > > never get networking on domU. > > I think you want to set, in /etc/xen/xend-config.sxp, > > (network-script blade-network) > > and then create a file /etc/xen/scripts/blade-network that looks like: > > #!/bin/sh > > dir=$(dirname "$0") > exec "$dir/network-bridge" "$@" "netdev=eth1" > > Don''t forget to chmod +x /etc/xen/scripts/blade-network, too. > > Hopefully, that will do the trick. Let me know how you get on. > > Ewan. > > _______________________________________________ > Xen-devel mailing list > Xen-devel@lists.xensource.com > http://lists.xensource.com/xen-devel >Bigelow:/etc/xen/scripts# ./blade-network start /etc/xen/scripts/network-bridge: line 211: /sbin/getcfg: No such file or directory I use Debian Sid .... _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Ewan Mellor
2005-Oct-26 07:21 UTC
Re: [Xen-devel] Network still broken, new issue as well , 7468:17a9f111fa93
On Tue, Oct 25, 2005 at 08:10:16PM -0400, Michael Lessard wrote:> Bigelow:/etc/xen/scripts# ./blade-network start > /etc/xen/scripts/network-bridge: line 211: /sbin/getcfg: No such file or > directory > > I use Debian Sid ....There''s a patch for that on the way. For now, you could just comment out that whole line -- it''s SuSE-specific. Did the script work anyway? For me, that error is not a fatal one, and the configuration still succeeds. Ewan. _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Michael Lessard
2005-Oct-26 12:14 UTC
Re: [Xen-devel] Network still broken, new issue as well , 7468:17a9f111fa93
On 10/26/05, Ewan Mellor <ewan@xensource.com> wrote:> > On Tue, Oct 25, 2005 at 08:10:16PM -0400, Michael Lessard wrote: > > > Bigelow:/etc/xen/scripts# ./blade-network start > > /etc/xen/scripts/network-bridge: line 211: /sbin/getcfg: No such file or > > directory > > > > I use Debian Sid .... > > There''s a patch for that on the way. For now, you could just comment out > that > whole line -- it''s SuSE-specific. > > Did the script work anyway? For me, that error is not a fatal one, and the > configuration still succeeds. > > Ewan.I commend out the line, now when i start the script no error reported but i couldn''t ping dom0 from domU, with eth1 bridge too :( _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Ewan Mellor
2005-Oct-26 12:26 UTC
Re: [Xen-devel] Network still broken, new issue as well , 7468:17a9f111fa93
On Wed, Oct 26, 2005 at 08:14:41AM -0400, Michael Lessard wrote:> On 10/26/05, Ewan Mellor <ewan@xensource.com> wrote: > > On Tue, Oct 25, 2005 at 08:10:16PM -0400, Michael Lessard wrote: > > > Bigelow:/etc/xen/scripts# ./blade-network start > > /etc/xen/scripts/network-bridge: line 211: /sbin/getcfg: No such file > or > > directory > > > > I use Debian Sid .... > > There''s a patch for that on the way. For now, you could just comment > out that > whole line -- it''s SuSE-specific. > > Did the script work anyway? For me, that error is not a fatal one, and > the > configuration still succeeds. > > Ewan. > > I commend out the line, now when i start the script no error reported > but i couldn''t ping dom0 from domU, with eth1 bridge too :(So could you repost the output from brctl show and ifconfig, just as you did for bug #363 before? Also, it would be worth having route and iptables -L. It would also be helpful to know whether you can get packets anywhere -- dom0 to network, network to dom0, domU to network, etc, and if you can figure out where they are getting lost. Cheers, Ewan. _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Michael Lessard
2005-Oct-26 15:35 UTC
Re: [Xen-devel] Network still broken, new issue as well , 7468:17a9f111fa93
On 10/26/05, Ewan Mellor <ewan@xensource.com> wrote:> > On Wed, Oct 26, 2005 at 08:14:41AM -0400, Michael Lessard wrote: > > > On 10/26/05, Ewan Mellor <ewan@xensource.com> wrote: > > > > On Tue, Oct 25, 2005 at 08:10:16PM -0400, Michael Lessard wrote: > > > > > Bigelow:/etc/xen/scripts# ./blade-network start > > > /etc/xen/scripts/network-bridge: line 211: /sbin/getcfg: No such file > > or > > > directory > > > > > > I use Debian Sid .... > > > > There''s a patch for that on the way. For now, you could just comment > > out that > > whole line -- it''s SuSE-specific. > > > > Did the script work anyway? For me, that error is not a fatal one, and > > the > > configuration still succeeds. > > > > Ewan. > > > > I commend out the line, now when i start the script no error reported > > but i couldn''t ping dom0 from domU, with eth1 bridge too :( > > So could you repost the output from brctl show and ifconfig, just as you > did > for bug #363 before? Also, it would be worth having route and iptables -L. > > It would also be helpful to know whether you can get packets anywhere -- > dom0 > to network, network to dom0, domU to network, etc, and if you can figure > out > where they are getting lost. > > Cheers, > > Ewan. >I have two network cards on my computer : eth0 connect to -> FW (DMZ 192.168.100.254 <http://192.168.100.254>) eth1 connect to -> FW (internal network 192.168.10.254<http://192.168.10.254> ) On my DOM0 i haved configure network eth0 : up (no ip) eth1 : 192.168.10.20 <http://192.168.10.20> On my DOMU i haved configure network eth1 : 192.168.100.1 <http://192.168.100.1> ON DOM0 #brctl show xenbr0 8000.feffffffffff no vif8.0 # iptables -L Chain INPUT (policy ACCEPT) target prot opt source destination Chain FORWARD (policy ACCEPT) target prot opt source destination ACCEPT all -- anywhere anywhere PHYSDEV match --physdev-in vif1.0 ACCEPT all -- anywhere anywhere PHYSDEV match --physdev-in vif8.0 Chain OUTPUT (policy ACCEPT) target prot opt source destination # ifconfig eth0 Link encap:Ethernet HWaddr 00:08:54:2F:EA:38 inet addr:192.168.100.5 <http://192.168.100.5> Bcast:192.168.100.255<http://192.168.100.255>Mask: 255.255.255.0 <http://255.255.255.0> UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 RX packets:13 errors:0 dropped:0 overruns:0 frame:0 TX packets:87 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:1000 RX bytes:894 (894.0 b) TX bytes:16522 (16.1 KiB) Interrupt:17 Base address:0xec00 eth1 Link encap:Ethernet HWaddr 00:08:54:2F:EA:32 inet addr:192.168.10.20 <http://192.168.10.20> Bcast:192.168.10.255<http://192.168.10.255>Mask: 255.255.255.0 <http://255.255.255.0> UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 RX packets:7597 errors:0 dropped:0 overruns:0 frame:0 TX packets:4755 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:1000 RX bytes:963119 (940.5 KiB) TX bytes:667126 (651.4 KiB) Interrupt:18 Base address:0xe800 lo Link encap:Local Loopback inet addr:127.0.0.1 <http://127.0.0.1> Mask:255.0.0.0 <http://255.0.0.0> UP LOOPBACK RUNNING MTU:16436 Metric:1 RX packets:76 errors:0 dropped:0 overruns:0 frame:0 TX packets:76 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:0 RX bytes:5064 (4.9 KiB) TX bytes:5064 (4.9 KiB) vif8.0 Link encap:Ethernet HWaddr FE:FF:FF:FF:FF:FF UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 RX packets:72 errors:0 dropped:0 overruns:0 frame:0 TX packets:0 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:0 RX bytes:3024 (2.9 KiB) TX bytes:0 (0.0 b) xenbr0 Link encap:Ethernet HWaddr FE:FF:FF:FF:FF:FF UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 RX packets:72 errors:0 dropped:0 overruns:0 frame:0 TX packets:0 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:0 RX bytes:2016 (1.9 KiB) TX bytes:0 (0.0 b) I haved put ip on eth0 to test connectivty from etho on DOM0 and its working ... I could ping FW from DOM0 but i could''nt ping DOM0 from DOMU _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel