Hi,
Based on other postings describing how to set up bridging for multiple
NICs, I''ve successfully bridged multiple physical devices on my Xen
system. However, I''m observing some strange network behavior.
First, let me describe my setup. I have three NICs, eth0, eth1, eth2.
I modified xend-config.sxp as follows:
#(network-script network-bridge)
(network-script my-network-bridge)
my-network-bridge is as follows:
#!/bin/bash
dir=$(dirname "$0")
"$dir/network-bridge" "$@" vifnum=0 netdev=eth0
bridge=xenwan0
"$dir/network-bridge" "$@" vifnum=1 netdev=eth1
bridge=xenlan1
"$dir/network-bridge" "$@" vifnum=2 netdev=eth2
bridge=xenlan2
# EOF
I did not change vif-bridge. I couldn''t determine whether that was
necessary or not.
''brctl show'' gives:
bridge name bridge id STP enabled interfaces
virbr0 8000.000000000000 yes
xenlan1 8000.feffffffffff no peth1
vif0.1
xenlan2 8000.feffffffffff no peth2
vif0.2
xenwan0 8000.feffffffffff no peth0
vif0.0
I have static IP addresses assigned to each NIC. Simple network
activity like ping or ssh works on all three physical links. However,
when I started an nuttcp server on this machine (''nuttcp -S''),
running
an nuttcp client from another machine only works when connecting to eth0
on the server; if I attempt to connect an nuttcp client to eth1 or eth2
on the Xen machine, I get:
nuttcp-t: v6.1.2: Error: server not ACKing data
Similarly, if I run an NFS server from the Xen machine, file transfer
to/from other machines is extremely slow, except when mounting from the
eth0 connection. Clearly, something is going wrong with the
bridging/routing on my Xen machine.
I have not started any DomU guests on this machine. If I turn off
network bridging in the Xen configuration, then everything returns to
normal (I can run nuttcp/NFS across all three links at normal speed), so
something with the bridging and renaming of the physical links is
affecting network behavior.
I''m running CentOS 5.3, Xen 3.0.3. Can anyone tell me if I''ve
configured something wrong?
Thanks,
Jason
_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users
On Thu, Sep 3, 2009 at 8:57 AM, Jason Kwon<jason.kwon@ericsson.com> wrote:> I have static IP addresses assigned to each NIC. Simple network > activity like ping or ssh works on all three physical links.What does your network setup looks like? Are all three NICs on the same network? Is the client on the same networks as one of the three NICs?> I have not started any DomU guests on this machine. If I turn off > network bridging in the Xen configuration, then everything returns to > normal (I can run nuttcp/NFS across all three links at normal speed), so > something with the bridging and renaming of the physical links is > affecting network behavior.You can try creating bridges manually (or using /etc/sysconfig/network-scripts/ifcfg-*) and see if it makes a difference. -- Fajar _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
Fajar A. Nugraha wrote:> On Thu, Sep 3, 2009 at 8:57 AM, Jason Kwon<jason.kwon@ericsson.com> wrote: > >> I have static IP addresses assigned to each NIC. Simple network >> activity like ping or ssh works on all three physical links. > > What does your network setup looks like? Are all three NICs on the > same network? Is the client on the same networks as one of the three > NICs?eth0 is on one network (let''s call it network A). eth1 and eth2 are two ports of a dual-port 10GbE NIC, and they are directly connected to the client machine (I assigned class C addresses to eth1/eth2). The client can reach the Xen server either through network A, or through the two direct 10GbE links.> >> I have not started any DomU guests on this machine. If I turn off >> network bridging in the Xen configuration, then everything returns to >> normal (I can run nuttcp/NFS across all three links at normal speed), so >> something with the bridging and renaming of the physical links is >> affecting network behavior. > > You can try creating bridges manually (or using > /etc/sysconfig/network-scripts/ifcfg-*) and see if it makes a > difference.Thanks, I''ll try that next. Jason _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
Jason Kwon wrote:> Fajar A. Nugraha wrote: >> On Thu, Sep 3, 2009 at 8:57 AM, Jason Kwon<jason.kwon@ericsson.com> >> wrote: >> >>> I have static IP addresses assigned to each NIC. Simple network >>> activity like ping or ssh works on all three physical links. >> >> What does your network setup looks like? Are all three NICs on the >> same network? Is the client on the same networks as one of the three >> NICs? > > eth0 is on one network (let''s call it network A). eth1 and eth2 are two > ports of a dual-port 10GbE NIC, and they are directly connected to the > client machine (I assigned class C addresses to eth1/eth2). The client > can reach the Xen server either through network A, or through the two > direct 10GbE links. > >> >>> I have not started any DomU guests on this machine. If I turn off >>> network bridging in the Xen configuration, then everything returns to >>> normal (I can run nuttcp/NFS across all three links at normal speed), so >>> something with the bridging and renaming of the physical links is >>> affecting network behavior. >> >> You can try creating bridges manually (or using >> /etc/sysconfig/network-scripts/ifcfg-*) and see if it makes a >> difference. > > Thanks, I''ll try that next. > > Jason > > > _______________________________________________ > Xen-users mailing list > Xen-users@lists.xensource.com > http://lists.xensource.com/xen-usersI''ve done a few more experiments, and it appears the problem has to do with bridging the 10GbE link, and not due to the bridging of multiple NICs. When I disabled all NICs except one 10GbE link on the Xen host, I still had problems. I used the following setup: Machine A: Xen CentOS 5.3 host, single 10GbE NIC Machine B: Xen Fedora 10 guest, running on Machine A Machine C: non-Xen CentOS 5.3 host, 10GbE NIC I have a direct fiber link between Machine A and Machine C, and class C addresses assigned to all three machines. If I bridge the 10GbE link on Machine A using network-bridge, I see the following results: Machine A nuttcp client connecting to Machine C nuttcp server - Works. Machine B nuttcp client connecting to Machine C nuttcp server - Works. Machine C nuttcp client connecting to Machine A nuttcp server - Fails. Machine C nuttcp client connecting to Machine B nuttcp server - Fails. If instead of using network-bridge, I manually create a bridge on Machine A for the 10GbE interface, I get the following behavior: Machine A nuttcp client connecting to Machine C nuttcp server - Works. Machine B nuttcp client connecting to Machine C nuttcp server - Works. Machine C nuttcp client connecting to Machine A nuttcp server - Works. Machine C nuttcp client connecting to Machine B nuttcp server - Fails. In this case, physical host-to-physical host traffic works in both directions. In the network-bridge case, host-to-host traffic only works in one direction. In both cases, Xen guest traffic only works in the outbound direction. When nuttcp fails, it gives the "server not ACKing data" error message mentioned before. Is it not possible to bridge a direct-connected link such as what I''ve got? Traffic size also seems to matter, because ping and ssh work across all links; larger operations like NFS file copy and nuttcp fail. Thanks, Jason _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users