Hey All,
I've been working on sharing a 6in4 IPv6 tunnel (via a gif device) I have
with Hurricane Electric (tunnelbroker.net) to my jails via epair devices.
My setup is a bit unique in that the IPv6 tunnel is behind an OpenVPN
connection. I've had varying degrees of success. I might have a bug to
report, but I thought I'd post here to get input from people who know
better than I do about these kinds of things.
I have a bridge device (we'll call it bridge0) with a /64 IPv6 address
(2001:470:8142:1::1). Each jail's epair[n]b device will get an IPv6 address
in that same prefix. For example, one of my jails is 2001:470:8142:1::3.
The default IPv6 gateway is the IPv6 address of bridge0.
Giving one jail an IP address works fine. For each jail after that, the
IPv6 address stays in tentative mode. FreeBSD gets stuck trying to use DAD
to figure out if there's an address conflict. It never leaves tentative
mode. This is the bug I'm working out.
Here's bridge0's config:
# ifconfig bridge0
bridge0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu
1500
ether 02:fe:21:34:d3:00
inet6 2001:470:8142:1::1 prefixlen 64
nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15
maxage 20 holdcnt 6 proto rstp maxaddr 2000 timeout 1200
root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0
member: epair0a flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
ifmaxaddr 0 port 19 priority 128 path cost 2000
member: epair1a flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
ifmaxaddr 0 port 21 priority 128 path cost 2000
member: bge0 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
ifmaxaddr 0 port 5 priority 128 path cost 200000
Here's the relevant epair device for the jail whose IPv6 stack is working:
# jexec "ClamAV_Dev" ifconfig epair1b
epair1b: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu
1500
options=8<VLAN_MTU>
ether 02:fb:c0:00:16:0b
inet6 2001:470:8142:1::3 prefixlen 64
inet6 fe80::fb:c0ff:fe00:160b%epair1b prefixlen 64 scopeid 0x2
inet 10.7.1.172 netmask 0xfffffe00 broadcast 10.7.1.255
nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
media: Ethernet 10Gbase-T (10Gbase-T <full-duplex>)
status: active
Here's the relevant epair device for the jail whose IPv6 stack isn't
working:
# jexec "Dev Template" ifconfig epair0b
epair0b: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu
1500
options=8<VLAN_MTU>
ether 02:80:03:00:14:0b
inet6 2001:470:8142:1::5 prefixlen 64 tentative
inet6 fe80::80:3ff:fe00:140b%epair0b prefixlen 64 tentative scopeid 0x2
inet 10.7.1.92 netmask 0xfffffe00 broadcast 10.7.1.255
nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
media: Ethernet 10Gbase-T (10Gbase-T <full-duplex>)
status: active
I brought up the "Dev Template" jail after bringing up the ClamAV_Dev
jail.
If there's any other output you'd like to see, let me know. If
you're
confused about my setup, visit my blog post about the subject here:
http://0xfeedface.org/blog/lattera/2013-01-12/tunneled-ipv6-freebsd-jails
I'm curious to know if I've got a legit bug or if it's something
I'm doing
wrong. The one thing I haven't tried is setting up rtadvd on the bridge.
That'd be kindof interesting, since my physical NIC is a member on the
bridge. I'd rather not dish out IPv6 addresses for all devices on the
network (a network with lots of devices I don't own or control).
Thanks,
Shawn
Quoth Shawn Webb <lattera at gmail.com>:> > I've been working on sharing a 6in4 IPv6 tunnel (via a gif device) I have > with Hurricane Electric (tunnelbroker.net) to my jails via epair devices. > My setup is a bit unique in that the IPv6 tunnel is behind an OpenVPN > connection. I've had varying degrees of success. I might have a bug to > report, but I thought I'd post here to get input from people who know > better than I do about these kinds of things. > > I have a bridge device (we'll call it bridge0) with a /64 IPv6 address > (2001:470:8142:1::1). Each jail's epair[n]b device will get an IPv6 address > in that same prefix. For example, one of my jails is 2001:470:8142:1::3. > The default IPv6 gateway is the IPv6 address of bridge0. > > Giving one jail an IP address works fine. For each jail after that, the > IPv6 address stays in tentative mode. FreeBSD gets stuck trying to use DAD > to figure out if there's an address conflict. It never leaves tentative > mode. This is the bug I'm working out. > > Here's bridge0's config: > > # ifconfig bridge0 > bridge0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu > 1500 > ether 02:fe:21:34:d3:00 > inet6 2001:470:8142:1::1 prefixlen 64 > nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL> > id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15 > maxage 20 holdcnt 6 proto rstp maxaddr 2000 timeout 1200 > root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0 > member: epair0a flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP> > ifmaxaddr 0 port 19 priority 128 path cost 2000 > member: epair1a flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP> > ifmaxaddr 0 port 21 priority 128 path cost 2000 > member: bge0 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP> > ifmaxaddr 0 port 5 priority 128 path cost 200000Why have you added the physical interface to the bridge? AFAICT you don't need to: a bridge will bridge epairs just fine, and as you explained in that blog post you have to route rather than bridge into the tunnel, since the tunnel isn't an Ethernet device.> Here's the relevant epair device for the jail whose IPv6 stack is working: > > # jexec "ClamAV_Dev" ifconfig epair1b > epair1b: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu > 1500 > options=8<VLAN_MTU> > ether 02:fb:c0:00:16:0b > inet6 2001:470:8142:1::3 prefixlen 64 > inet6 fe80::fb:c0ff:fe00:160b%epair1b prefixlen 64 scopeid 0x2 > inet 10.7.1.172 netmask 0xfffffe00 broadcast 10.7.1.255 > nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL> > media: Ethernet 10Gbase-T (10Gbase-T <full-duplex>) > status: active > > Here's the relevant epair device for the jail whose IPv6 stack isn't > working: > > # jexec "Dev Template" ifconfig epair0b > epair0b: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu > 1500 > options=8<VLAN_MTU> > ether 02:80:03:00:14:0b > inet6 2001:470:8142:1::5 prefixlen 64 tentative > inet6 fe80::80:3ff:fe00:140b%epair0b prefixlen 64 tentative scopeid 0x2 > inet 10.7.1.92 netmask 0xfffffe00 broadcast 10.7.1.255 > nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>I suspect the addresses are only marked tentative because the interface has been marked IFDISABLED. This causes all current addresses to be marked tentative, because the kernel isn't allowed to send or receive IPv6 packets and so can't defend the addresses any more. Is it possible something in the jail's startup scripts is causing the interface to be marked IFDISABLED after the inet6 address has been assigned? Some of the functions in network.subr mark interfaces IFDISABLED automatically if they don't think they have IPv6 addresses.> media: Ethernet 10Gbase-T (10Gbase-T <full-duplex>) > status: active > > I brought up the "Dev Template" jail after bringing up the ClamAV_Dev jail. > If there's any other output you'd like to see, let me know. If you're > confused about my setup, visit my blog post about the subject here: > http://0xfeedface.org/blog/lattera/2013-01-12/tunneled-ipv6-freebsd-jails > > I'm curious to know if I've got a legit bug or if it's something I'm doing > wrong. The one thing I haven't tried is setting up rtadvd on the bridge. > That'd be kindof interesting, since my physical NIC is a member on the > bridge. I'd rather not dish out IPv6 addresses for all devices on the > network (a network with lots of devices I don't own or control).As I said, I don't believe you need the physical interface on the bridge, unless you have to for IPv4 (and you can't route or proxyarp instead). However, before you can run rtadvd you will need to give the bridge its proper link-local address, which probably also means locking down its hardware address in rc.conf. Bridges don't get auto link-local addresses, for reasons I've never entirely understood, and RAs have to use ll addresses. You will need to set up routing so that packets coming in through the tunnel destined for the jails get routed out of the bridge, and packets coming in on the bridge destined for the IPv6 Internet get routed out of the tunnel. Probably that will have happened already, just by assigning an inet6 address and prefixlen to the bridge and the default inet6 route to the tunnel. Ben
Somehow there ended up a typo in the CC to freebsd-stable at freebsd.org. Last email below: On Tue, Jan 15, 2013 at 5:53 PM, Shawn Webb <lattera at gmail.com> wrote:> On Tue, Jan 15, 2013 at 4:52 PM, Ben Morrow <ben at morrow.me.uk> wrote: > >> Quoth Shawn Webb <lattera at gmail.com>: >> > On Tue, Jan 15, 2013 at 2:54 PM, Ben Morrow <ben at morrow.me.uk> wrote: >> > > >> > > ifconfig epair0b inet6 -ifdisabled >> > > >> > > I don't know why you get that error when you miss out the 'inet6'; >> it's >> > > not exactly very clear. >> > > >> > >> > Ah. That works. I'll just have to add that to my scripts. Since the >> device >> > won't come out of tentative mode without manually removing the >> ifdisabled >> > flag, should I go ahead and file a PR? It'd be nice if I could at the >> very >> > least set a timeout for DAD. >> >> DAD already has a timeout: it succeeds iff no packets indicating someone >> else is using the address are received in a given time. The only reason >> for an address remaining tentative indefinitely (without transitioning >> to either valid or duplicated) is if IPv6 on that interface has been >> disable entirely by setting IFDISABLED. If DAD fails for the LL address >> the interface is marked IFDISABLED but the LL address is marked >> duplicated rather than tentative. >> > > I figured it out. In my jail initialization scripts, I'm running '/bin/sh > /bin/rc' after doing initial network setup. The rc script puts the > interface in IFDISABLED mode. So if I run the ifconfig command to remove > the flag, I'm golden. I've committed and pushed the code that fixes the > problem in my scripts. If you're curious, you can look at > https://github.com/lattera/drupal-jailadmin/commit/cbf8509712c3dd237bbc020f49f63b51507b7be4 > > Thanks for the help. I really appreciate it. >
At 5PM -0500 on 15/01/13 you (Shawn Webb) wrote:> > I figured it out. In my jail initialization scripts, I'm running '/bin/sh > /bin/rc' after doing initial network setup. The rc script puts the > interface in IFDISABLED mode. So if I run the ifconfig command to remove > the flag, I'm golden.Yes, that's what I thought. You should be able to avoid this by specifying either ifconfig_epair0b_ipv6="inet6 auto_linklocal" or ipv6_activate_all_interfaces="YES" in the jail's rc.conf. This is cleaner than running ifconfig explicitly outside the jail. Ben