Xen users - Dec 2006 - iptables in dom0 with bridge: no more outbound connections

After successfully consolidating my servers at home with Xen I
wanted to do the same at work. Unfortunately, I ran into a
networking anomaly that baffles my mind...

What I''m looking for is a real simple setup: a dom0 and 4
domU''s
that are all directly accessible; in other words, no NAT required,
each dom has a "real" (public) ip address in our 129.125. range
(that''s University of Groningen, the Netherlands in case
you''re
wondering).

Even though I have no need for iptables to do NAT, I _do_ want
to protect each dom, including dom0, with its own firewall. And
here the problems start.

When I boot into dom0 (Xen 3.0.4 patched to kernel 2.6.16.36), but
without starting xend, things are fine (iptable rules are active
at this point). Yet, after I have started xend (and xenbr0 appears
in my ifconfig output) I am unable to make connections to remote
hosts (dns lookups fail, ping to ip addresses fail, etc). Strange!

Now for the real spooky part:
   1. I booted into dom0 (no xend)
   2. executed `telnet 129.125.14.12 daytime`, it works
   3. started xend
   4. executed `telnet 129.125.14.12 daytime`, it still works (surprise!)
   5. executed `telnet 129.125.14.13 daytime`, it does not work
Wierd, so I rebooted the machine and tried again except for step 2
and the result was the same. Two days later I tried another time
(again leaving out step 2) and now step 4 gives no response...

When I disable the firewall (iptables -F) everything is fine.

My minimal firewall script:
iptables -F
iptables -A INPUT -p tcp --dport ssh   -j ACCEPT
iptables -A INPUT -m state --state ESTABLISHED,RELATED -j ACCEPT
iptables -A INPUT -j DROP

Another observation: with the above firewall in place I can ssh into
dom0, but it takes about 30 seconds to connect; without firewall it
is almost instantaneous.

Now I''m a programmer, not a network engineer. And I don''t have
a
clue how to go from here (i.o.w. I can run tcpdump, but don''t know
what to look for). So suggestions are greatly appreciated!

Cheers, Peter

_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

Peter Fokkinga wrote:
> After successfully consolidating my servers at home with Xen I
> wanted to do the same at work. Unfortunately, I ran into a
> networking anomaly that baffles my mind...
>
> What I''m looking for is a real simple setup: a dom0 and 4
domU''s
> that are all directly accessible; in other words, no NAT required,
> each dom has a "real" (public) ip address in our 129.125. range
> (that''s University of Groningen, the Netherlands in case
you''re
> wondering).
>
> Even though I have no need for iptables to do NAT, I _do_ want
> to protect each dom, including dom0, with its own firewall. And
> here the problems start.
>
> When I boot into dom0 (Xen 3.0.4 patched to kernel 2.6.16.36), but
> without starting xend, things are fine (iptable rules are active
> at this point). Yet, after I have started xend (and xenbr0 appears
> in my ifconfig output) I am unable to make connections to remote
> hosts (dns lookups fail, ping to ip addresses fail, etc). Strange!
>
> Now for the real spooky part:
>   1. I booted into dom0 (no xend)
>   2. executed `telnet 129.125.14.12 daytime`, it works
>   3. started xend
>   4. executed `telnet 129.125.14.12 daytime`, it still works (surprise!)
>   5. executed `telnet 129.125.14.13 daytime`, it does not work
DNS cache, I think.
> Wierd, so I rebooted the machine and tried again except for step 2
> and the result was the same. Two days later I tried another time
> (again leaving out step 2) and now step 4 gives no response...
>
> When I disable the firewall (iptables -F) everything is fine.
>
> My minimal firewall script:
> iptables -F
> iptables -A INPUT -p tcp --dport ssh   -j ACCEPT
> iptables -A INPUT -m state --state ESTABLISHED,RELATED -j ACCEPT
> iptables -A INPUT -j DROP
>
> Another observation: with the above firewall in place I can ssh into
> dom0, but it takes about 30 seconds to connect; without firewall it
> is almost instantaneous.
Reverse DNS being attempted by Dom0 as part of SSHD logging, and 
failing.  Try restarting the SSHD with the "-u0" option to turn this
off.
> Now I''m a programmer, not a network engineer. And I don''t
have a
> clue how to go from here (i.o.w. I can run tcpdump, but don''t know
> what to look for). So suggestions are greatly appreciated!
>
> Cheers, Peter
It''s been discussed before: I haven''t had a chance to pursue
it, myself.
Basically, after you start Xend, traffic going *out* from Dom0 goes 
through peth0, as near as I can tell, not eth0.

_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

Quoting Nico Kadel-Garcia <nkadel@gmail.com>:
> Peter Fokkinga wrote:
>> [...]
>> Now for the real spooky part:
>>  1. I booted into dom0 (no xend)
>>  2. executed `telnet 129.125.14.12 daytime`, it works
>>  3. started xend
>>  4. executed `telnet 129.125.14.12 daytime`, it still works (surprise!)
>>  5. executed `telnet 129.125.14.13 daytime`, it does not work
> DNS cache, I think.
But I''m using ip adresses, not names? I don''t see how DNS fits
in
this picture.
> It''s been discussed before: I haven''t had a chance to
pursue it,
> myself. Basically, after you start Xend, traffic going *out* from Dom0
> goes through peth0, as near as I can tell, not eth0.
Ok, but why is iptables interfering? I''m not refering to eth0 in
my rules. If I flush iptables after starting Xend everything is fine,
troubles start the moment I re-activate the rules.

I get the feeling iptables does not remember its state, so my rule
   iptables -A INPUT -m state --state ESTABLISHED,RELATED -j ACCEPT
has no effect. Kernel modules xt_state and ip_conntrack are loaded.

Peter


_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

Peter Fokkinga wrote:
> Quoting Nico Kadel-Garcia <nkadel@gmail.com>:
>> Peter Fokkinga wrote:
>>> [...]
>>> Now for the real spooky part:
>>>  1. I booted into dom0 (no xend)
>>>  2. executed `telnet 129.125.14.12 daytime`, it works
>>>  3. started xend
>>>  4. executed `telnet 129.125.14.12 daytime`, it still works
(surprise!)
>>>  5. executed `telnet 129.125.14.13 daytime`, it does not work
>> DNS cache, I think.
>
> But I''m using ip adresses, not names? I don''t see how DNS
fits in
> this picture.
I can''t swear to this, but when you use anything to reach out to the 
net, it assumes first that the word or name is a hostname, and tries to 
look that up. It resolves IP addresses as IP addresses, and DNS names as 
IP addresses, and then has to turn that into appropriate local or 
gateway MAC addresses based on ARP data, etc., etc., etc. DNS caches 
store the information locally, so no additional lookups happen. If it''s
not stored locally in your DNS cache, then it tries to do a DNS lookup, 
and in your case fails as it tries to look up 129.154.14.13 from your 
DNS system.

I don''t think a numerical hostname is first resolved as a number, for a
whole bunch of historical and procedural reasons. It still does DNS the 
first time.
>> It''s been discussed before: I haven''t had a chance to
pursue it,
>> myself. Basically, after you start Xend, traffic going *out* from Dom0
>> goes through peth0, as near as I can tell, not eth0.
>
> Ok, but why is iptables interfering? I''m not refering to eth0 in
> my rules. If I flush iptables after starting Xend everything is fine,
> troubles start the moment I re-activate the rules.
I think because when Xen is running, it''s not going through eth0.
It''s
going through peth0.
> I get the feeling iptables does not remember its state, so my rule
>   iptables -A INPUT -m state --state ESTABLISHED,RELATED -j ACCEPT
> has no effect. Kernel modules xt_state and ip_conntrack are loaded.
>
> Peter

_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

On 12/30/06, Nico Kadel-Garcia <nkadel@gmail.com>
wrote:
> Peter Fokkinga wrote:
> > Quoting Nico Kadel-Garcia <nkadel@gmail.com>:
> >> Peter Fokkinga wrote:
> >>> [...]
> >>> Now for the real spooky part:
> >>>  1. I booted into dom0 (no xend)
> >>>  2. executed `telnet 129.125.14.12 daytime`, it works
> >>>  3. started xend
> >>>  4. executed `telnet 129.125.14.12 daytime`, it still works
(surprise!)
> >>>  5. executed `telnet 129.125.14.13 daytime`, it does not work
> >> DNS cache, I think.
> >
> > But I''m using ip adresses, not names? I don''t see
how DNS fits in
> > this picture.
> I can''t swear to this, but when you use anything to reach out to
the
> net, it assumes first that the word or name is a hostname, and tries to
> look that up. It resolves IP addresses as IP addresses, and DNS names as
> IP addresses, and then has to turn that into appropriate local or
> gateway MAC addresses based on ARP data, etc., etc., etc. DNS caches
> store the information locally, so no additional lookups happen. If
it''s
> not stored locally in your DNS cache, then it tries to do a DNS lookup,
> and in your case fails as it tries to look up 129.154.14.13 from your
> DNS system.
>
> I don''t think a numerical hostname is first resolved as a number,
for a
> whole bunch of historical and procedural reasons. It still does DNS the
> first time.
No, DNS was not invloved in the above case. A quad-octet string would
only be treated by telnet as an IP address.
> > Ok, but why is iptables interfering? I''m not refering to eth0
in
> > my rules. If I flush iptables after starting Xend everything is fine,
> > troubles start the moment I re-activate the rules.
> I think because when Xen is running, it''s not going through eth0.
It''s
> going through peth0.
ISTR seeing a recent thread explaining that both are involved at times...??
> > I get the feeling iptables does not remember its state, so my rule
> >   iptables -A INPUT -m state --state ESTABLISHED,RELATED -j ACCEPT
> > has no effect. Kernel modules xt_state and ip_conntrack are loaded.
Depends on your distro. Redhat for example,
"service iptables save" (overwriting /etc/sysconfig/iptables).

jerry

_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

Quoting Jerry Amundson <jamundso@gmail.com>:
>> Peter Fokkinga wrote:
>>> [iptables drops outgoing traffic when xend is running]
>>> I get the feeling iptables does not remember its state, so my rule
>>>   iptables -A INPUT -m state --state ESTABLISHED,RELATED -j ACCEPT
>>> has no effect. Kernel modules xt_state and ip_conntrack are loaded.
>
> Depends on your distro. Redhat for example,
> "service iptables save" (overwriting /etc/sysconfig/iptables).
I did not mean "remember" in the sense of "between reboots",
but more
like that iptables does not register the outgoing packet. So when the
first "response" packet comes back and enters the INPUT rule it is
seen as a NEW packet instead of ESTABLISHED or RELATED and therefore
dropped.

Distro I''m using is Ubuntu 6.06 btw.

Peter


_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

Nico Kadel-Garcia wrote:
> Peter Fokkinga wrote:
>> Quoting Nico Kadel-Garcia <nkadel@gmail.com>:
>>> Peter Fokkinga wrote:
>>>> [...]
>>>> Now for the real spooky part:
>>>>  1. I booted into dom0 (no xend)
>>>>  2. executed `telnet 129.125.14.12 daytime`, it works
>>>>  3. started xend
>>>>  4. executed `telnet 129.125.14.12 daytime`, it still works
(surprise!)
>>>>  5. executed `telnet 129.125.14.13 daytime`, it does not work
I don''t get this part.  Why do you think 5 would work just because 4
worked?  They are different IP addresses.  You never proved that 5 would
have worked before 3 was executed.
>> But I''m using ip adresses, not names? I don''t see how
DNS fits in
>> this picture.
> I can''t swear to this, but when you use anything to reach out to
the
> net, it assumes first that the word or name is a hostname, and tries to
> look that up. It resolves IP addresses as IP addresses, and DNS names as
> IP addresses, and then has to turn that into appropriate local or
> gateway MAC addresses based on ARP data, etc., etc., etc. DNS caches
> store the information locally, so no additional lookups happen. If
it''s
> not stored locally in your DNS cache, then it tries to do a DNS lookup,
> and in your case fails as it tries to look up 129.154.14.13 from your
> DNS system.
> 
> I don''t think a numerical hostname is first resolved as a number,
for a
> whole bunch of historical and procedural reasons. It still does DNS the
> first time.
The resolver shouldn''t try to look up a dotted quad.  The problem here
is possibly the remote host verifying that the forward and reverse
mappings of the client host match in order to avoid hostname spoofing.

-- 
Christopher G. Stach II


_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users