George Mamalakis
2012-Aug-08 11:34 UTC
pf nat fails on msk0 from packets deriving from a jail interface
Hi all,
Suddenly I am facing a problem on a new PC, using a configuration that I
have been using on more than 10 servers for the last few years. The only
thing that I find that differs from my other configuratinos is the NIC
of the PC. If not, I must be missing something very trivial.
I have built a jail on this PC, following the handbook's guidelines
(section: application of jails). The PC has one NIC, msk0, where I run
pf on (built on my kernel; I have already tried using the module). My
pf.conf is as simple as possible:
# cat /etc/pf.conf
nat on msk0 from any to any -> 10.0.3.6
pass quick all
when I jexec inside the jail, and pf is running, I am unable to reach
any machine except my jail (not even the host). If pf is off, the
network works just fine (of course my router knows where to find my
jail's subnet).
What is strange is that if I tcpdump on msk0, then after a few seconds
that I request something from within the jail, I see the packets going
and coming on msk0 using the correct IP (the NAT IP), but it seems that
the machine fails to route them back inside the jail.
My configuration is as follows:
#uname -a
FreeBSD filesrv.svr.noca 9.0-STABLE FreeBSD 9.0-STABLE #1: Fri Jul 27
15:40:48 EEST 2012
root@filesrv.svr.noca:/usr/obj/usr/src/sys/MAMALOPYRINO amd64
#ifconfig -a
msk0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
options=c011b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,TSO4,VLAN_HWTSO,LINKSTATE>
ether 80:ee:73:10:a3:58
inet 10.0.3.6 netmask 0xffffff00 broadcast 10.0.3.255
inet6 fe80::82ee:73ff:fe10:a358%msk0 prefixlen 64 scopeid 0x1
nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
media: Ethernet autoselect (1000baseT
<full-duplex,flowcontrol,rxpause,txpause>)
status: active
pflog0: flags=0<> metric 0 mtu 33152
nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
pfsync0: flags=0<> metric 0 mtu 1500
nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
syncpeer: 0.0.0.0 maxupd: 128
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
options=3<RXCSUM,TXCSUM>
inet6 ::1 prefixlen 128
inet6 fe80::1%lo0 prefixlen 64 scopeid 0x9
inet 127.0.0.1 netmask 0xff000000
nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
lo1: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
options=3<RXCSUM,TXCSUM>
inet 10.3.2.1 netmask 0xff000000
nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
tap1: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
options=80000<LINKSTATE>
ether 00:bd:7b:c3:0c:01
inet6 fe80::2bd:7bff:fec3:c01%tap1 prefixlen 64 scopeid 0xb
inet 10.3.2.2 netmask 0xffffff00 broadcast 10.3.2.255
nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
tap2: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
options=80000<LINKSTATE>
ether 00:bd:7f:c3:0c:02
inet6 fe80::2bd:7fff:fec3:c02%tap2 prefixlen 64 scopeid 0xc
nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
lo3: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
options=3<RXCSUM,TXCSUM>
inet 10.3.2.3 netmask 0xffffff00
nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL
lo3 is used as my jail interface, msk0 is my lan interface.
# pciconf -v
mskc0@pci0:3:0:0: class=0x020000 card=0x40011297 chip=0x438011ab
rev=0x10 hdr=0x00
vendor = 'Marvell Technology Group Ltd.'
device = '88E8057 PCI-E Gigabit Ethernet Controller'
class = network
subclass = ethernet
excerpt of /etc/rc.conf:
jail_test_hostname="test.svr.noca"
jail_test_rootdir="/jails/j/test"
jail_test_devfs_enable="YES"
jail_test_ip="10.3.2.3/24"
jail_test_interface="lo3"
I have even enabled forwarding and fast forwarding (just in case that
this had been the case) with non results.
# netstat -rn
Routing tables
Internet:
Destination Gateway Flags Refs Use Netif Expire
default 10.0.3.1 UGS 0 290 msk0
10.0.3.0/24 link#1 U 0 18825 msk0
10.0.3.6 link#1 UHS 0 1 lo0
10.3.2.0/24 link#11 U 0 0 tap1
10.3.2.1 link#10 UH 0 0 lo1
10.3.2.2 link#11 UHS 0 61 lo0
10.3.2.3 link#13 UH 0 0 lo3
127.0.0.1 link#9 UH 0 64 lo0
Since I don't need NAT on my configuration, I will use simple routing
instead, so there won't be a problem for me. I am just sending this info
in case this is a bug with pf-msk driver (for the specific card?) and
before I send a bug report, I'd like a second opinion in case I am
missing something fundamental.
Thanx all in advance.
--
George Mamalakis
IT and Security Officer
Electrical and Computer Engineer (Aristotle Un. of Thessaloniki),
MSc (Imperial College of London)
Department of Electrical and Computer Engineering
Faculty of Engineering
Aristotle University of Thessaloniki
phone number : +30 (2310) 994379
YongHyeon PYUN
2012-Aug-09 01:00 UTC
pf nat fails on msk0 from packets deriving from a jail interface
On Wed, Aug 08, 2012 at 02:33:25PM +0300, George Mamalakis wrote:> Hi all, > > Suddenly I am facing a problem on a new PC, using a configuration that I > have been using on more than 10 servers for the last few years. The only > thing that I find that differs from my other configuratinos is the NIC > of the PC. If not, I must be missing something very trivial. > > I have built a jail on this PC, following the handbook's guidelines > (section: application of jails). The PC has one NIC, msk0, where I run > pf on (built on my kernel; I have already tried using the module). My > pf.conf is as simple as possible: > > # cat /etc/pf.conf > > nat on msk0 from any to any -> 10.0.3.6 > pass quick all > > when I jexec inside the jail, and pf is running, I am unable to reach > any machine except my jail (not even the host). If pf is off, the > network works just fine (of course my router knows where to find my > jail's subnet). > > What is strange is that if I tcpdump on msk0, then after a few seconds > that I request something from within the jail, I see the packets going > and coming on msk0 using the correct IP (the NAT IP), but it seems that > the machine fails to route them back inside the jail.I guess this is the same issue reported in kern/170081. Some msk(4) controllers lack full hardware checksum offloading capability such that pseudo checksum should be computed by upper layer. It seems pf(4) NAT was broken for controllers that lack pseudo checksumming. This indicates the following ethernet controller do not work with pf(4) NAT. sk(4), msk(4), fxp(4), hme(4) and gem(4) Try disabling RX checksum offloading as a work-around. #ifconfig msk0 -rxcsum