Shorewall users - Mar 2008 - FTP DNAT not working - "Server sent passive reply with unroutable address"

Hi all!

I am a long time lurker, but have not posted until now.

My old trusted firewall machine broke a couple of weeks ago and I replaced it 
with a XEN domU that is using DNAT and has two interfaces. The firewall domU and
the FTP server domU are both guests on the same dom0. All three machines are 
running Debian/etch (stable) and Shorewall has version 3.2.6.

I can''t get FTP to work and Filezilla says:

      Response: 227 Entering Passive Mode (192,168,221,239,19)
      Status: Server sent passive reply with unroutable address. Using
      server address instead.

I cannot understand why the FTP servers private address is leaked since the 
modules ip_nat_ftp and ip_conntrack_ftp are loaded. The FTP rule is
"FTP/DNAT
net loc:192.168.221.3".

In the attached status file I have connected from "213.115.101.134" to
"87.96.134.74". Can any of you see what is wrong?

Thank you in advance.

/Martin Leben

Ps/ DNAT:ting http, imap and other "simple" traffic works. /Ds
Ps2/ My apologies if this mail hits the list twice. I sent the first one before 
subscribing. (Reading through http://gmane.org) /Ds2


-------------------------------------------------------------------------
Check out the new SourceForge.net Marketplace.
It''s the best place to buy or sell services for
just about anything Open Source.
http://ad.doubleclick.net/clk;164216239;13503038;w?http://sf.net/marketplace

On Sun, Mar 30, 2008 at 11:57:41PM +0200, Martin Leben
wrote:
>      Response: 227 Entering Passive Mode (192,168,221,239,19)
>      Status: Server sent passive reply with unroutable address. Using
>      server address instead.
>
> I cannot understand why the FTP servers private address is leaked since 
> the modules ip_nat_ftp and ip_conntrack_ftp are loaded. The FTP rule is 
> "FTP/DNAT net loc:192.168.221.3".
Because the address sent by the FTP server is 192.168.221.239, not
192.168.221.3

I don''t think you''re looking in the right place.

-------------------------------------------------------------------------
Check out the new SourceForge.net Marketplace.
It''s the best place to buy or sell services for
just about anything Open Source.
http://ad.doubleclick.net/clk;164216239;13503038;w?http://sf.net/marketplace

Andrew Suffield wrote:
> On Sun, Mar 30, 2008 at 11:57:41PM +0200, Martin Leben wrote:
>>      Response: 227 Entering Passive Mode (192,168,221,239,19)
>>      Status: Server sent passive reply with unroutable address. Using
>>      server address instead.
>>
>> I cannot understand why the FTP servers private address is leaked since
>> the modules ip_nat_ftp and ip_conntrack_ftp are loaded. The FTP rule is
>> "FTP/DNAT net loc:192.168.221.3".
> 
> Because the address sent by the FTP server is 192.168.221.239, not
> 192.168.221.3
> 
> I don''t think you''re looking in the right place.
Hmm... My fingers must have slipped when typing (No, I didn''t
copy/paste that
one...) the Filezilla response. Just tested again and it says "Response:
227
Entering Passive Mode (192,168,221,3,76,244)" this time. Yes, that is six
blocks
of comma separated numbers.

So, my question still stands.

/Martin Leben


-------------------------------------------------------------------------
Check out the new SourceForge.net Marketplace.
It''s the best place to buy or sell services for
just about anything Open Source.
http://ad.doubleclick.net/clk;164216239;13503038;w?http://sf.net/marketplace

Martin Leben wrote:
> 
> So, my question still stands.
> 
I''ve just spoken on #shorewall with another user who is running the
same
setup as you are (including identical kernel version), and who is having
exactly same problem!

So I think we have a smoking gun...

-Tom
-- 
Tom Eastep    \ Nothing is foolproof to a sufficiently talented fool
Shoreline,     \ http://shorewall.net
Washington USA  \ teastep@shorewall.net
PGP Public Key   \ https://lists.shorewall.net/teastep.pgp.key

-------------------------------------------------------------------------
Check out the new SourceForge.net Marketplace.
It''s the best place to buy or sell services for
just about anything Open Source.
http://ad.doubleclick.net/clk;164216239;13503038;w?http://sf.net/marketplace

Hi Martin,

this reminds me of a FTP conntrack problem I had with Xen a while ago.
It's related to the common Xen checksum offload problem.
First FTP port packet has incorrect checksum and is dropped. Then the resent 
packet is ignored by the ftp_conntrack module and doesn't get masq'ed. E
voila, you have your internal address in the port command.

Diagnosis: run tcpdump -vv (or even better wireshark) on the involved 
interfaces and you'll see a lot of invalid checksums

Solution: disable tx-checksumming on ALL interfaces (ethtool -K <device>
tx
off). I disable it on all Interfaces, because I saw so many different 
problems with that in all kind of Xen setups that I gave up to find out a 
logic behind that error.

Let me know if it helps.

Alex




On Sonntag 30 März 2008, Martin Leben wrote:
> Hi all!
>
> I am a long time lurker, but have not posted until now.
>
> My old trusted firewall machine broke a couple of weeks ago and I replaced
> it with a XEN domU that is using DNAT and has two interfaces. The firewall
> domU and the FTP server domU are both guests on the same dom0. All three
> machines are running Debian/etch (stable) and Shorewall has version 3.2.6.
>
> I can't get FTP to work and Filezilla says:
>
>       Response: 227 Entering Passive Mode (192,168,221,239,19)
>       Status: Server sent passive reply with unroutable address. Using
>       server address instead.
>
> I cannot understand why the FTP servers private address is leaked since the
> modules ip_nat_ftp and ip_conntrack_ftp are loaded. The FTP rule is
> "FTP/DNAT net loc:192.168.221.3".
>
> In the attached status file I have connected from
"213.115.101.134" to
> "87.96.134.74". Can any of you see what is wrong?
>
> Thank you in advance.
>
> /Martin Leben
>
> Ps/ DNAT:ting http, imap and other "simple" traffic works. /Ds
> Ps2/ My apologies if this mail hits the list twice. I sent the first one
> before subscribing. (Reading through http://gmane.org) /Ds2


-------------------------------------------------------------------------
Check out the new SourceForge.net Marketplace.
It's the best place to buy or sell services for
just about anything Open Source.
http://ad.doubleclick.net/clk;164216239;13503038;w?http://sf.net/marketplace
_______________________________________________
Shorewall-users mailing list
Shorewall-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/shorewall-users

Alexander Wilms wrote:
> Hi Martin,
> 
> this reminds me of a FTP conntrack problem I had with Xen a while ago.
> It''s related to the common Xen checksum offload problem.
> First FTP port packet has incorrect checksum and is dropped. Then the
resent
> packet is ignored by the ftp_conntrack module and doesn''t get
masq''ed. E
> voila, you have your internal address in the port command.
> 
> Diagnosis: run tcpdump -vv (or even better wireshark) on the involved 
> interfaces and you''ll see a lot of invalid checksums
> 
> Solution: disable tx-checksumming on ALL interfaces (ethtool -K
<device> tx
> off).
Hi Alexander,

Voila! That did it. Now it works. I ran
# ethtool -K <device> tx off
... on the firewalls both network interfaces as well as on the FTP server 
network interface.

Big thanks to you, Andrew and Tom for your time and input.

Tom, I think this is something for the documentation.

/Martin Leben


-------------------------------------------------------------------------
Check out the new SourceForge.net Marketplace.
It''s the best place to buy or sell services for
just about anything Open Source.
http://ad.doubleclick.net/clk;164216239;13503038;w?http://sf.net/marketplace

Martin, this workaround is documented at
http://www.shorewall.net/XenMyWay-Routed.html
which is an interesting Xen setup.

you can put the line "ethtool -K <device> tx off" in your
interfaces
file (e.g. directly beneath each interface configuration stanza) to
ensure this persists across reboots.

Werner


On Mon, 2008-03-31 at 13:19 +0200, Martin Leben wrote:
> Alexander Wilms wrote:
> > Hi Martin,
> > 
> > this reminds me of a FTP conntrack problem I had with Xen a while ago.
> > It''s related to the common Xen checksum offload problem.
> > First FTP port packet has incorrect checksum and is dropped. Then the
resent
> > packet is ignored by the ftp_conntrack module and doesn''t get
masq''ed. E
> > voila, you have your internal address in the port command.
> > 
> > Diagnosis: run tcpdump -vv (or even better wireshark) on the involved 
> > interfaces and you''ll see a lot of invalid checksums
> > 
> > Solution: disable tx-checksumming on ALL interfaces (ethtool -K
<device> tx
> > off).
> 
> Hi Alexander,
> 
> Voila! That did it. Now it works. I ran
> # ethtool -K <device> tx off
> ... on the firewalls both network interfaces as well as on the FTP server 
> network interface.
> 
> Big thanks to you, Andrew and Tom for your time and input.
> 
> Tom, I think this is something for the documentation.
> 
> /Martin Leben
> 
> 
> -------------------------------------------------------------------------
> Check out the new SourceForge.net Marketplace.
> It''s the best place to buy or sell services for
> just about anything Open Source.
>
http://ad.doubleclick.net/clk;164216239;13503038;w?http://sf.net/marketplace
> _______________________________________________
> Shorewall-users mailing list
> Shorewall-users@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/shorewall-users

-------------------------------------------------------------------------
Check out the new SourceForge.net Marketplace.
It''s the best place to buy or sell services for
just about anything Open Source.
http://ad.doubleclick.net/clk;164216239;13503038;w?http://sf.net/marketplace

Werner van Staden wrote:
> Martin, this workaround is documented at
> http://www.shorewall.net/XenMyWay-Routed.html
> which is an interesting Xen setup.
> 
> you can put the line "ethtool -K <device> tx off" in your
interfaces
> file (e.g. directly beneath each interface configuration stanza) to
> ensure this persists across reboots.
> 
> Werner

Yes, I see that now. And to make things even worse it was documented in 
/etc/network/interfaces on the machines themselves as well, just three lines 
below the "iface eth0 inet dhcp" stanza. Humbling experience.

# The commented out line above will disable TCP checksumming which
# might resolve problems for some users.  It is disabled by default

Might have been xen-create-image [1] that placed it there. (xen-create-image is 
part of xen-tools <http://www.xen-tools.org/software/xen-tools/>.)

Once again: Thank you all. :-)

/Martin


-------------------------------------------------------------------------
Check out the new SourceForge.net Marketplace.
It''s the best place to buy or sell services for
just about anything Open Source.
http://ad.doubleclick.net/clk;164216239;13503038;w?http://sf.net/marketplace