Xen users - Apr 2006 - routing in xen 3.0 domU: icmp gets routed, but tcp/ip only partially

Hello folks,

I have a really strange routing problem that no amount of
googling and experimenting has been able to solve. Then
again, I''m new to Xen and "advanced" networking, so I could
be missing something very basic.

Summary: an unprivileged domU with PCI frontend for a NIC
is used as a router; icmp gets routed, but tcp/ip only
partially. I''m using a xen-unstable snapshot, dated
March 31, running on Ubuntu Dapper Drake.

Here''s the network topology:

    +---------------+
    |      ext      |
    | (192.168.2.1) |
    +---------------+
            |
            |
         (eth2)
     +-------------+               +---------------+
     |  fw  (domU) | (eth1) -+-----|  dmz (domU)   |
     +-------------+         |     | (172.17.17.2) |
         (eth0)            xenbr1  +---------------+
            |
            +-- xenbr0
            |
    +---------------+
    |  xeno (dom0)  |
    |   (10.0.0.1)  |
    +---------------+


The domU host "fw" has the e100 (Intel EtherPro 100) driver
loaded for eth2 (the PCI device was hidden from dom0). Hosts
"xeno" and "fw" are connected to bridge "xenbr0",
hosts
"dmz" and "dmz" are connected through bridge
"xenbr1".

Note that there''s no firewall installed (yet); it''s just
plain routing at the moment. All hosts have inetd running,
with services "daytime" and "echo" active; these services
are great diagnostics; they''re simple and when "echo" works
then more complicated things like ssh will too.

What works?
 * ping from everywhere to everywhere (traceroute too)
 * full access from everywhere to "fw"
 * full access from "fw" to everywhere
 * full access from "dmz" to "xeno"
 * from "dmz": `telnet ext daytime`
 * from "xeno": `telnet ext daytime`

If I disable ip_forwarding on "fw" then it''s not possible to
connect from "dmz" to "xeno" or vice versa; so traffic
really
is going through "fw".

What does NOT work?
 * from "ext": `telnet dmz daytime`
 * from "ext": `telnet dmz echo`
 * from "dmz": `telnet ext echo`
 * from "xeno": `telnet ext echo`
In all these cases I get connected, but no output; however, I
do get output when I connect to a specific interface on "fw"
(iow, if host "fw0" is the ip-address of eth0 on "fw" then
`telnet fw0 echo` works fine from "ext").

It''s as if no IP data (as opposed to syn/ack) wants to go from
"fw" to "ext"?

Routes defined on all hosts:
(192.168.1.1 is the gateway connected to my ADSL modem)

host "fw"
Destination     Gateway         Genmask         Flags  Iface
172.17.17.0     0.0.0.0         255.255.255.0   U      eth1
10.0.0.0        0.0.0.0         255.255.0.0     U      eth0
192.168.0.0     0.0.0.0         255.255.0.0     U      eth2
0.0.0.0         192.168.1.1     0.0.0.0         UG     eth2

host "xeno"
Destination     Gateway         Genmask         Flags  Iface
172.17.18.0     0.0.0.0         255.255.255.0   U      xenbr1
10.0.0.0        0.0.0.0         255.255.0.0     U      eth2
0.0.0.0         10.0.1.1        0.0.0.0         UG     eth2

host "dmz"
Destination     Gateway         Genmask         Flags  Iface
172.17.17.0     0.0.0.0         255.255.255.0   U      eth0
0.0.0.0         172.17.17.1     0.0.0.0         UG     eth0

host "ext" (not xen, separate machine on my LAN)
Destination     Gateway         Genmask         Flags  Iface
172.17.17.0     192.168.8.1     255.255.255.0   UG     eth0
192.168.0.0     0.0.0.0         255.255.0.0     U      eth0
10.0.0.0        192.168.8.1     255.0.0.0       UG     eth0
0.0.0.0         192.168.1.1     0.0.0.0         UG     eth0


Help me, Obi-Wan Xenobi; you''re my only hope.

Peter

_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

On Sun, Apr 02, 2006 at 09:48:18AM +0200, Peter Fokkinga
wrote:
> I have a really strange routing problem that no amount of
> googling and experimenting has been able to solve. Then
> again, I''m new to Xen and "advanced" networking, so I
could
> be missing something very basic.
My ''very basic'' on fairly similar symptoms was some default
rules that the
Xen networking scripts were inserting -- in my case, the FORWARD chain had
it''s policy set to DROP.  Open that up and all was good.

I''d check your dom0 and your routing domU iptables listing (both the
filter
and nat tables) for *any* rules and policies which could possibly be causing
hassles.  It''s the simplest, most obvious thing to overlook -- and I
was
stumped for waaaaay too long by it.

- Matt

_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

Quoting Matthew Palmer <mpalmer@hezmatt.org>:
> My ''very basic'' on fairly similar symptoms was some
default rules
> that the Xen networking scripts were inserting -- in my case, the 
> FORWARD chain had it''s policy set to DROP.  Open that up and all
was
> good.
>
All chains (checked with `ipchains -L` and `ipchains -L -t nat`)
on the domU have policy ACCEPT. There are also no rules defined.
Removing the netfilter modules (just in case) has no effect; I
can ping the dmz, but not get the daytime of the dmz from the
external system (just like before).

The dom0 doesn''t have netfilter support (not compiled in, nor
as modules). Is it needed in dom0?

Regards, Peter

_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

Hi,
> I have a really strange routing problem
from my experience, most of the "ICMP works but TCP seems connected and no
data gets through" problems in routed xen3 setups are due to the
cheksum-offloading featue. See my post on the devel list:
http://lists.xensource.com/archives/html/xen-devel/2006-03/msg01445.html
for a patch for the Dom0 kernel which disables this. Search for offloading
for more information :) The basic problem: the intention was to save
CPU power by not calculating checksums on internal interfaces (as there 
won''t be any transmission errors there anyway :) and let the real NIC
add
this on the way out (modern NICs have HW accelleration for that).
Unfortunately this doesn''t work out in all cases currently.
> In all these cases I get connected, but no output;
To see if you''ve really got the checksum problem use tcpdump -vv in the
Domain not receiving data. For example:
tcpdump -vv -n -i eth0 
and then transfer some tcp or udp data to it. If you see something like
 [bad udp cksum f566!]
and it looks like the host didn''t receive the packet (no answer, gets
in again
and again), you might well have hit the checksum offloading problem.
> Help me, Obi-Wan Xenobi; you''re my only hope.
Tried my best :)

(:ul8er, r@y

_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

Quoting Florian Kirstein <xenlist@custom.ray.net>:
>> I have a really strange routing problem
> from my experience, most of the "ICMP works but TCP seems connected 
> and no data gets through" problems in routed xen3 setups are due to 
> thecheksum-offloading featue. [snip]
>
> To see if you''ve really got the checksum problem use tcpdump -vv
in
> the > Domain not receiving data. For example:
> tcpdump -vv -n -i eth0 and then transfer some tcp or udp data to it. 
> If you see something like  [bad udp cksum f566!]
>
I get "cksum 0x7fe9 (incorrect (-> 0x3470)", so it looks like
the checksum offloading problem you described.

Applied the patch, installed the new dom0 kernel (and new domU
for good measure) and now routing works fine!

May I suggest The Powers That Be to include this patch in the
upcoming 3.0.2 release? First make it work, _then_ make it fast ;-)

Many thanks, Peter

_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users