Hi All,
I've a bit of an odd query which I hope somebody may be able to
assist with.
I'm looking to set up several OpenVPN tunnels on a single machine
(each residing in its own jail) and route data to different
destinations over different tunnels by selectively routing the traffic
via a particular jail.
I have three jails set up with OpenVPN tunnels terminated in each,
they all work as expected from the "local" machine.
I can't do a straight forward route over the VPN tunnel as I don't
control the other end of the tunnel, I need to treat it as a
point-to-point connection as a result, hence I need to use NAT.
I've tested this setup with a single tunnel running off a "real"
machine with natd providing NAT, it works like a charm, however, when
I move the config into a jail I run into issues, natd doesn't seem to
be able to see the incoming traffic, nothing shows up in the logs at
all.
I'm not even sure if this is actually possible, I'm starting to
suspect that natd can't hook in low enough from the jails to access
the incoming traffic.
Traffic gets into the jail by way of an epair interface between the
host and the jail, bridged to the ethernet adapter by way of a bridge
device, I can see the traffic attempting to route over the tun
interface in the jail (but obviously it's not being NATted so nothing
comes back) so the traffic is making it in and through the routing
engine, just not via natd.
Any suggestions here?
The host is FreeBSD-8.3.
Thanks,
Morgan
On 22 November 2012 04:00, Morgan Reed <morgan.s.reed at gmail.com> wrote:> Hi All, > > I've a bit of an odd query which I hope somebody may be able to > assist with. > > I'm looking to set up several OpenVPN tunnels on a single machine > (each residing in its own jail) and route data to different > destinations over different tunnels by selectively routing the traffic > via a particular jail. > > I have three jails set up with OpenVPN tunnels terminated in each, > they all work as expected from the "local" machine. > > I can't do a straight forward route over the VPN tunnel as I don't > control the other end of the tunnel, I need to treat it as a > point-to-point connection as a result, hence I need to use NAT. > > I've tested this setup with a single tunnel running off a "real" > machine with natd providing NAT, it works like a charm, however, when > I move the config into a jail I run into issues, natd doesn't seem to > be able to see the incoming traffic, nothing shows up in the logs at > all. > > I'm not even sure if this is actually possible, I'm starting to > suspect that natd can't hook in low enough from the jails to access > the incoming traffic. > > Traffic gets into the jail by way of an epair interface between the > host and the jail, bridged to the ethernet adapter by way of a bridge > device, I can see the traffic attempting to route over the tun > interface in the jail (but obviously it's not being NATted so nothing > comes back) so the traffic is making it in and through the routing > engine, just not via natd. > > Any suggestions here? > > The host is FreeBSD-8.3. >I've not used it myself, but this sound like something VIMAGE may be good for, basically it's a virtual tcp stack per jail, there's some docs at http://wiki.freebsd.org/Image but I seem to remember a more up to date one elsewhere but can't find it at the moment!
Hmm, list was missing from reply-to on this one. ---------- Forwarded message ---------- From: Morgan Reed <morgan.s.reed at gmail.com> Date: Thu, Nov 22, 2012 at 10:36 PM Subject: Re: natd in a jail To: Dewayne Geraghty <dewayne.geraghty at heuristicsystems.com.au> On Thu, Nov 22, 2012 at 9:33 PM, Dewayne Geraghty <dewayne.geraghty at heuristicsystems.com.au> wrote:> We run a lot of jails with kernel nat and ipfw (& ipsec but that's not what > you need here). Some of the hosts haven't migrated from natd to kernel nat, > so we're probably similar to your setup.Sounds very similar, just substituting OpenVPN for IPSec.> 90% of our jails have an 192.168/16 that nat via an external interface with > a routable address, and an internal non-routeable address (ie non-RFC1918); > which is probably what you're doing for your VPN stuff. > > Our openvpn's all use tun, I would suggest that your natd isn't doing > exactly like you'd wish - on a good day it can be tricky to get right and > tcpdump is your friend, which should be monitored in both your host > environment and within the jail. You'll need to enable allow.raw_sockets > and you'll probably want to enable bpf to be available in your jail, if you > haven't already done so.BPF is enabled for the jails, and the traffic is getting to where it needs to (but not via natd). I'll try enabling raw_sockets in the jails, it is entirely conceivable that natd requires that functionality. Thanks for your assistance, I'll see how I go and report back. Best Regards, Morgan Reed -- "They that can give up essential liberty to obtain a little temporary safety deserve neither liberty nor safety." -- Benjamin Franklin, 1759
On 11/22/2012 6:00 AM, Morgan Reed wrote:> Hi All,Hi, [snipped content]> Any suggestions here?A quick one. Could you make a try using "ipfw nat" instead of natd? I am not sure about divert socket and natd per jail, but NATing using ipfw and libalias(which natd uses as well) works. HTH, Nikos
On Thu, Nov 22, 2012 at 10:36 PM, Morgan Reed <morgan.s.reed at gmail.com> wrote:> BPF is enabled for the jails, and the traffic is getting to where it > needs to (but not via natd). I'll try enabling raw_sockets in the > jails, it is entirely conceivable that natd requires that > functionality.So it turns out I'd not bought bpf into the jails, however even with that and raw_sockets enabled I'm still having no joy with natd. I've been looking at ipfw a bit today but I've run into an issue, loading ipfw_nat causes my kernel to instantly panic, I need to recompile with KDB and DDB turned on so I can actually catch the trace though... Might look at netgraph before going too far down that path.
SOLVED: Thanks all for your assistance.
SUMMARY:
- Kernel rebuilt with option IPFIREWALL and friends turned on (not
necessary if your ipfw modules work you should just be able to load
them, mine didn't for reasons I don't really have the time or
inclination to track down)
- OpenVPN configurations modified to use a specific tun device
('device tun' directive replaced with 'device tun0')
- OpenVPN configurations modified to run the following script prior
to dropping privs (via the 'up' directive);
/usr/local/etc/openvpn/up.sh
---
ipfw -q flush
pfw nat 1 config if tun0 reset same_ports deny_in
ipfw add 500 nat 1 ip from any to any via tun0
---
This script assumes that option IPFILTER_DEFAULT_TO_ACCEPT or the
equivalent sysctl frob is set, this is most probably *not* what you
want to do in the "real world". Modify as needed, and be sure to set
the permissions on the file appropriately as the script will be
executed by root.
A warning though; this is a total hack, the ipfw stuff should be moved
to /etc/ipfw.rules or similar and processed by ipfw at boot but I'm
not sure how it'll react if you try to do this config before the tun
device is created, I expect it'll be unhappy so you'll need to create
a static tun device for openvpn, this is the "right" way to do things
but I'm being exceedingly lazy. The script above is a filthy hack, and
potentially dangerous.
- Normal requirements for gateway operation also apply here (which
essentially means set gateway_enable=YES in rc.conf on the host and
all router jails).