Xen users - Jun 2005 - xen, fc4, bridging, iptables and conntrack problem

Hi,

I''m testing out Xen on FC4. I''m using bridging for networking,
as
well as iptables to firewall, configured with the standard Fedora 
''system-config-security-level'' tool. However I have really
strange
problem with conntrack not seeming to catch outbound connections. 
This prevents outbound connections working from dom0. Connections 
from domU''s however /do/ work.

The problem appears to boil down to the following:

Chain INPUT (policy ACCEPT 210K packets, 18M bytes)
  pkts bytes target     prot opt in     out     source               destination
  111K 8778K RH-Firewall-1-INPUT  all  --  xen-br+ any     anywhere            
anywhere
     0     0 RH-Firewall-1-INPUT  all  --  vif+   any     anywhere            
anywhere
     1    73 RH-Firewall-1-INPUT  all  --  eth0   any     anywhere            
anywhere

Chain FORWARD (policy ACCEPT 2812K packets, 311M bytes)
  pkts bytes target     prot opt in     out     source               destination
<empty>

Chain RH-Firewall-1-INPUT (3 references)
  pkts bytes target     prot opt in     out     source               destination
    33  2485 ACCEPT     all  --  lo     any     anywhere             anywhere
   253 16338 ACCEPT     icmp --  any    any     anywhere             anywhere   
icmp any
     0     0 ACCEPT     ipv6-crypt--  any    any     anywhere   anywhere
     0     0 ACCEPT     ipv6-auth--  any    any     anywhere    anywhere
68483 6004K ACCEPT     all  --  any    any     anywhere             anywhere    
state RELATED,ESTABLISHED
<snip remaining standard RH-Firewall rules to allow in certain ports>


The FORWARD chain is empty and policy ACCEPT, which maybe explains 
why domU''s work.

The INPUT side of stuff though seems to not work because the 
RELATED,ESTABLISHED conntrack rule doesn''t match. And this would 
appear to be because the original /outgoing/ packets are never caught 
by connection track and entered into its state.

If I tcpdump xen-br0, I can see packets leave, and I can even see the 
remote SYN|ACK come in, which is very strange (and not inline with my 
only hypothesis so far, a conntrack problem):

# tcpdump -i xen-br0 port 25
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on xen-br0, link-type EN10MB (Ethernet), capture size 96 bytes
18:48:54.138909 IP domain0.38261 > remote.smtp: 
S 710403207:710403207(0) win 5840 <mss 1460,sackOK,timestamp 181127121
0,nop,wscale 2>
18:48:54.271062 IP remote.smtp > domain0.38261: 
S 746149051:746149051(0) ack 710403208 win 5792 <mss 1460,sackOK,timestamp
1332954470 181127121,nop,wscale 0>
18:48:57.138797 IP domain0.38261 > remote.smtp: 
S 710403207:710403207(0) win 5840 <mss 1460,sackOK,timestamp 181127421
0,nop,wscale 2>
18:48:57.270302 IP remote.smtp > domain0.38261: 
S 749148214:749148214(0) ack 710403208 win 5792 <mss 1460,sackOK,timestamp
1332954770 181127421,nop,wscale 0>

Has anyone seen this problem before?

Is it specific to bridging (but it affects local packets though), to 
Xen somehow, to FC4?

regards,
-- 
Paul Jakma	paul@clubi.ie	paul@jakma.org	Key ID: 64A2FF6A
Fortune:
That''s always the way when you discover something new; everyone thinks
you''re crazy.
 		-- Evelyn E. Smith

_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

Hi Paul, I have Fedora Core 4 and I am having exactly the same problem
as you. I will provide some detail below.
Out of two installs this happened both times.
You are right, this is a conntrack failure but I don''t know if
it''s on
the iptables or xen side, although everything works fine until xend
starts-creates the bridge and bingo! conntrack stops working. Bit of a
showstopper really.

Here is some of my info:-
Problem:-
New install of fedora core 4 with xen kernel running. Iptables rules that under
the regular kernel work fine stop working when in bridge mode under xen in dom0.
This stops the conntrack system working on the xen host machine and i
can''t then
log in via ssh.
It seems that the conntrack system is failing to match already accepted
connections. The initial packet seems to get accepted by the INPUT rule, then
the reply packet slips past the ESTABLISHED,RELATED rule and gets logged then
dropped by the default policy.

This is the packet that gets logged:-
xen kernel: OUTPUT IN= OUT=xen-br0 PHYSOUT=eth0 SRC=192.168.0.45
DST=192.168.0.39 LEN=48 TOS=0x00 PREC=0x00 TTL=64 ID=0 DF PROTO=TCP SPT=22
DPT=1152 WINDOW=5840 RES=0x00 ACK SYN URGP=0

This happens whether i start a guest os up or not.
This was reproduced on another machine at work with a Fedora Core 4 install.

xen host machine address:192.168.0.45
ssh client address:192.168.0.39

rules:-
Chain FORWARD (policy DROP 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source               destination
    0     0 ACCEPT     all  --  *      *       0.0.0.0/0            0.0.0.0/0  
        state RELATED,ESTABLISHED
    0     0 LOG        all  --  *      *       0.0.0.0/0            0.0.0.0/0  
        LOG flags 0 level 4 prefix `FORWARD ''

Chain INPUT (policy DROP 54 packets, 7483 bytes)
 pkts bytes target     prot opt in     out     source               destination
    0     0 ACCEPT     all  --  lo     *       0.0.0.0/0            0.0.0.0/0
  304 21532 ACCEPT     all  --  *      *       0.0.0.0/0            0.0.0.0/0  
        state RELATED,ESTABLISHED
    1    48 ACCEPT     tcp  --  *      *       192.168.0.39         192.168.0.45
      tcp spts:1024:65535 dpt:22 state NEW
   54  7483 LOG        all  --  *      *       0.0.0.0/0            0.0.0.0/0  
        LOG flags 0 level 4 prefix `INPUT ''

Chain OUTPUT (policy DROP 8 packets, 384 bytes)
 pkts bytes target     prot opt in     out     source               destination
    0     0 ACCEPT     all  --  *      lo      0.0.0.0/0            0.0.0.0/0
    0     0 ACCEPT     all  --  *      *       0.0.0.0/0            0.0.0.0/0  
        state RELATED,ESTABLISHED
    0     0 ACCEPT     udp  --  *      *       192.168.0.45         192.168.0.19
      udp spts:1024:65535 dpt:53
    0     0 ACCEPT     tcp  --  *      *       192.168.0.45         0.0.0.0/0  
        tcp spts:1024:65535 dpt:80
    0     0 ACCEPT     icmp --  *      *       192.168.0.45         0.0.0.0/0
    0     0 LOG        all  --  *      *       0.0.0.0/0            0.0.0.0/0  
        LOG flags 0 level 4 prefix `OUTPUT ''

interfaces:-
eth0      Link encap:Ethernet  HWaddr 00:08:43:EE:50:CE
          inet addr:192.168.0.45  Bcast:192.168.0.255  Mask:255.255.255.0
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:24684 errors:0 dropped:0 overruns:0 frame:0
          TX packets:4406 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:1992235 (1.8 MiB)  TX bytes:631910 (617.0 KiB)
          Base address:0xecc0 Memory:ff8e0000-ff900000

lo        Link encap:Local Loopback
          inet addr:127.0.0.1  Mask:255.0.0.0
          UP LOOPBACK RUNNING  MTU:16436  Metric:1
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0
          TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:0 (0.0 b)  TX bytes:0 (0.0 b)

xen-br0   Link encap:Ethernet  HWaddr 00:08:43:EE:50:CE
          inet addr:192.168.0.45  Bcast:192.168.0.255  Mask:255.255.255.255
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:24682 errors:0 dropped:0 overruns:0 frame:0
          TX packets:4451 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:1538495 (1.4 MiB)  TX bytes:618890 (604.3 KiB)

routes:-
192.168.0.0     0.0.0.0         255.255.255.0   U     0      0        0 xen-br0
169.254.0.0     0.0.0.0         255.255.0.0     U     0      0        0 xen-br0
0.0.0.0         192.168.0.250   0.0.0.0         UG    0      0        0 xen-br0

operating system:-
Fedora Core 4

kernel version:-
2.6.11-1.1369_FC4xen0

iptables version:-
iptables v1.3.0

xen version:-
xen-2-20050522

network driver:-
e1000

Had everything working under fedora core 3 before with iptables and 5 virtual
machines conntracking beautifully

There''s nothing obvious, all the iptables modules are loaded and work
fine until the bridge goes up. No error messages associated with the
bridge creation either.
Will try to dig further.
Hope somebody has some ideas as I am running out of them!


_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

Hi,

On Sat, 25 Jun 2005, Jon Howse wrote:
> Hi Paul,
> I have Fedora Core 4 and I am having exactly the same problem as 
> you.
Aha, so it''s not just me. Time to raise a bug with fedora.
> I will provide some detail below. Out of two installs this happened 
> both times. You are right, this is a conntrack failure
Seems to be.
> but I don''t know if it''s on the iptables or xen side,
although
> everything works fine until xend starts-creates the bridge and 
> bingo! conntrack stops working.
Yep.
> Bit of a showstopper really.
Definitely.
> machine and i can''t then log in via ssh. It seems that the 
> conntrack system is failing to match already accepted connections.
See above. For me, all dom0 initiated connections fail to appear in 
conntrack state (but strangely the remote replies still get seen by 
tcpdump on xen-br0). domU''s work fine though, as FORWARD is 
unrestricted.
> The initial packet seems to get accepted by the INPUT rule, then 
> the reply packet slips past the ESTABLISHED,RELATED rule and gets 
> logged then dropped by the default policy.
Ah.
> This happens whether i start a guest os up or not. This was 
> reproduced on another machine at work with a Fedora Core 4 install.
> There''s nothing obvious, all the iptables modules are loaded and 
> work fine until the bridge goes up. No error messages associated 
> with the bridge creation either. Will try to dig further.
I created a bug for Fedora. See 
https://bugzilla.redhat.com/bugzilla/show_bug.cgi?id=161792 and 
please add your comments to it.

regards,
-- 
Paul Jakma	paul@clubi.ie	paul@jakma.org	Key ID: 64A2FF6A
Fortune:
Ask not what''s inside your head, but what your head''s inside
of.
 		-- J.J. Gibson

_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

Hi Paul, I too posted to bugzilla at redhat since we last spoke. Should
I delete my bug and add to yours?

I also posted to the xen bugzilla system...bug number 82.

Will check back soon...

Jonny



_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

Paul Jakma wrote:

> On Sat, 25 Jun 2005, Jon Howse wrote:
>
>> Hi Paul,
>
>> I have Fedora Core 4 and I am having exactly the same problem as you.
>
> Aha, so it''s not just me. Time to raise a bug with fedora.
I can confirm the problem here.

[snip]
>> machine and i can''t then log in via ssh. It seems that the
conntrack
>> system is failing to match already accepted connections.
>
> See above. For me, all dom0 initiated connections fail to appear in 
> conntrack state (but strangely the remote replies still get seen by 
> tcpdump on xen-br0). domU''s work fine though, as FORWARD is
unrestricted.
>
>> The initial packet seems to get accepted by the INPUT rule, then the 
>> reply packet slips past the ESTABLISHED,RELATED rule and gets logged
then
>> dropped by the default policy.
[snip]
> https://bugzilla.redhat.com/bugzilla/show_bug.cgi?id=161792 and please add 
> your comments to it.

The snapshot for -unstable used for the latest FC4 package is quite old: * 
Tue Apr 26 2005 Rik van Riel <...> 2-20050424
- upgrade to last night''s snapshot

So perhaps this is already fixed in xen-unstable. Or it was just an artefact 
of code changes, similar to the problem that xm restore does not work 
correctly in that snapshot.

Rik said he would upgrade to a new snapshot for rawhide rather soon. Not 
sure when that will be, though.

Can anyone not using FC4 confirm problems with iptables and conntrack in the 
latest -unstable?

Best Regards,
Michael Paesold 


_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

Hi Paul, I subscribe to the netfilter mailing list and I saw this in a
posting by someone:-

<POSTING>
Has anyone else seen this?  A working bridge running Fedora Core 3 fails
after an upgrade to the one of  the latest Fedora kernels.  What I''ve
found is:

   kernel-2.6.11-1.14_FC3 and earlier work fine

   kernel-2.6.11-1.27_FC3 and kernel-2.6.11-1.35_FC3 fail!

The problem is with netfilter not with bridging.  With iptables
shutdown the bridge works fine but even with very simple iptables
rules any network connections to/from or through the bridge fail.

(Fedora kernels are patched so it''s difficult to say which standard
kernel version they correspond to but it looks like some new kernel
patch has caused or will cause problems with bridge-nf).
<POSTING>

I know it''s fedora core 3...but it may be connected.
It''s not great detail but indicates a trend that seems to fit the
evidence that we have. No network connectivity...ESTABLISHED or RELATED
reply packets not getting out/in?

These kernels are relatively recent for 3 as I have a fc3 workstation at
work and those revisions ring a bell. I''ll check back with you after I
find out at work. The last fc3 working xen environment I had working was
before those revisions...hmmmm.

Any help?

Jonny



_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users