Damien Fleuriot
2015-Aug-20 09:51 UTC
[POSSIBLE BUG] 10-STABLE CARP erroneously becomes master on boot
On 19 August 2015 at 18:20, Damien Fleuriot <ml at my.gd> wrote:> > > On 19 August 2015 at 11:29, Damien Fleuriot <ml at my.gd> wrote: > >> Hello list, Freddie, >> >> >> I've been able to run extensive tests, the results of which I'm pasting >> below. >> >> >> First of all, I've been unable to replicate the problem in our >> preproduction and QA environments. >> >> The differences between our production and pre/QA envs are as follows : >> - we use link aggregation and VLAN tagging in production , we cannot >> replicate this in pre/QA because of limitations with KVM guests. >> - we use multiple CARP addresses with the same VHID on our public VLAN in >> production, we COULD replicate that in pre/QA if required. >> >> >> The context remains the following : >> - host A is supposed to be CARP MASTER, has advskew 20 and preempt >> - host B is supposed to be CARP BACKUP, has advskew 150 and preempt >> - host B assumes mastership if the CARPs are configured from rc.conf , >> doesn't if they're set up manually after boot >> >> Note that these 2 boxes were upgraded from 8-STABLE to 10-STABLE. >> Host A runs 10.2-BETA1 >> Host B runs 10.2-PRERELEASE >> >> I used the exact same versions in our pre/QA environments (one BETA1, one >> PRERELEASE from the same build) and couldn't replicate the issue. >> >> >> >> Now, on to the tests themselves. >> >> >> A/ Create a new CARP address with a new VHID, configure it in rc.conf and >> see if we get double MASTERS >> - on Host A : CARP created manually >> - on Host B : CARP created manually >> Host A is MASTER and Host B is BACKUP >> >> - on Host B : setup the new CARP in rc.conf , reboot >> Host A is MASTER and Host B is BACKUP , problem not replicated >> >> >> B/ Try with only one of our production addresses >> - on Host B : uncomment the production CARP address from rc.conf , reboot >> Host A is MASTER and Host B is MASTER >> Host A shows net.inet.carp.demotion=0 >> Host B shows net.inet.carp.demotion=240 >> >> >> C/ Try with the new CARP address + one of our production addresses , >> different VHIDs >> - on Host B : uncomment the new CARP address from rc.conf , reboot >> Host A is MASTER and Host B is MASTER >> Host A shows net.inet.carp.demotion=0 >> Host B shows net.inet.carp.demotion=240 >> >> >> D/ Try the new syntax in rc.conf , as per Freddie's suggestion >> - on Host B : change the rc.conf syntax , reboot >> Host A is MASTER and Host B is MASTER >> Host A shows net.inet.carp.demotion=0 >> Host B shows net.inet.carp.demotion=240 >> >> >> E/ Try, just for the sake of it, to remove old files and libs on host B >> - on Host B : cd /usr/src ; yes | make delete-old ; yes | make >> delete-old-libs ; reboot >> Host A is MASTER and Host B is MASTER >> Host A shows net.inet.carp.demotion=0 >> Host B shows net.inet.carp.demotion=240 >> >> >> F/ Edit sysctls to disable CARP demotion on advertisement send errors >> - on Host A : sysctl net.inet.carp.senderr_demotion_factor=0 >> - on Host B : set "net.inet.carp.senderr_demotion_factor=0" in >> sysctl.conf , reboot >> Host A is MASTER and Host B is MASTER >> Host A shows net.inet.carp.demotion=0 >> Host B shows net.inet.carp.demotion=240 >> >> >> >> Now after this F/ test, I'm thinking there's some voodoo going on here >> and it sure shows up : >> - on Host A, 'pfctl -si' shows 3k states >> - on Host B, 'pfctl -si' shows 800 states >> >> Now that would explain why my CARP gets demoted on Host B (as per man 4 >> carp, pfsync failures result in a -240 demotion). >> It doesn't, however, explain why the demoted CARP chooses to remain in a >> MASTER state, or assumed MASTERship in the first place. >> >> Surely enough, I can find some CARP errors in-between my reboots : >> messages:2420:2015-08-19T03:51:38.273600+00:00 pf1-gs kernel: carp: >> demoted by -240 to 0 (pfsync bulk fail) >> messages:2429:2015-08-19T03:56:37.178575+00:00 pf1-gs kernel: carp: VHID >> 110 at vlan410: INIT -> BACKUP >> messages:2430:2015-08-19T03:56:40.664568+00:00 pf1-gs kernel: carp: VHID >> 110 at vlan410: BACKUP -> MASTER (master down) >> messages:2637:2015-08-19T04:00:22.482071+00:00 pf1-gs kernel: carp: VHID >> 111 at vlan410: BACKUP -> MASTER (master down) >> messages:2857:2015-08-19T04:04:02.330167+00:00 pf1-gs kernel: carp: VHID >> 110 at vlan410: BACKUP -> MASTER (master down) >> messages:2877:2015-08-19T04:05:03.288199+00:00 pf1-gs kernel: carp: >> demoted by -240 to 0 (pfsync bulk fail) >> messages:3088:2015-08-19T04:08:48.961985+00:00 pf1-gs kernel: carp: VHID >> 110 at vlan410: BACKUP -> MASTER (master down) >> >> >> Things I have not tested yet : >> - use /24 CARPs instead of /32s >> - switch my CARPs to all use different VHIDs >> >> Things I CANNOT test : >> - set up a *dedicated* pfsync link between the firewalls , they're in >> different DCs >> - set up a *dedicated* VLAN for pfsync , that would entail huge changes >> in our PCI-DSS environment >> >> >> After all these tests, I find myself in a situation where : >> - manually set up CARPs on Host B work fine , and pfsync works >> - CARPs from rc.conf on Host B result in MASTER-MASTER , and pfsync fails >> >> >> >> >> I must say I'm stuck here. >> The "master down" message is very confusing, when the firewalls *can* see >> each other. >> The "pfsync bulk fail" is rather interesting as well, since it doesn't >> occur when the CARPs are unconfigured, or set up manually without rc.conf. >> tcpdump -nei pflog0 does not show any dropped packet. >> PF is configured to "pass in quick" pfsync and CARP packets. >> >> >> >> I'm afraid a STFW hasn't helped overmuch here, although I'll try some >> more. >> >> >> If anyone's got a pointer, I'll bite. >> >> Cheers >> >> > > FWIW , additional testing shows that on Host A (10.2-BETA) CARP > advertisements are sent from the CARP IP itself. > As in, my Host A has physical IPs X and Y, but sends its CARP announces > from CARP IP Z. > > I have not been able to replicate this behaviour in our preproduction > environment where both -BETA and -PRERELEASE send the advertisements from > their physical IP. > These boxes however, have just the one IP, as opposed to the production > environment which has 2 physical IPs. > > > I will get some additional testing done, by swapping over to make the Host > B running -PRERELEASE master, and see if it also sends its CARP > announcements sourced from its physical IP (would be good) or the CARP > itself (would be bad). >Hello list, We've managed to find the source of the bug, if it is indeed a bug. It all comes down to the order in which the IP addresses are assigned to the interface from /etc/rc.conf. When using the following syntax, the physical IP address is configured AFTER the CARPs on the interface, which results in the CARP advertisements being sourced from the CARP IP, triggering the double MASTER situation : ipv4_addrs_int="1.2.3.4/24" ifconfig_int_alias0="1.2.3.6/32 vhid 1 pass test advskew 20" When using either of the following syntaxes, the physical IP address is configured BEFORE the CARPs, which results in the CARP advertisements being sourced from the physical IP and restoring normal functionality : ifconfig_int="inet 1.2.3.4/24" ifconfig_int_alias0="1.2.3.6/32 vhid 1 pass test advskew 20" OR ifconfig_int_alias0="1.2.3.4/24" ifconfig_int_alias1="1.2.3.6/32 vhid 1 pass test advskew 20" I cannot find any reference to this behaviour in UPDATING, I'm not even sure it's actually wanted. I'm going to update the boxes to the latest 10-STABLE to see if the problem still occurs, no point in submitting a bug for something that's fixed.> > > >> >> On 17 August 2015 at 18:38, Damien Fleuriot <ml at my.gd> wrote: >> >>> >>> On 17 August 2015 at 18:32, Freddie Cash <fjwcash at gmail.com> wrote: >>> >>>> >>>> On Aug 17, 2015 9:22 AM, "Damien Fleuriot" <ml at my.gd> wrote: >>>> > >>>> > Hello list, >>>> > >>>> > >>>> > >>>> > I'm seeing this very peculiar behaviour between 2 10-STABLE boxes. >>>> > >>>> > Host A is CARP Master with advskew 20 and runs 10.2-BETA1 from 10/07 >>>> > Host B is CARP Backup with advskew 150 and runs 10.2-PRERELEASE from >>>> 12/08 >>>> > >>>> > >>>> > When I configure CARP in rc.conf on host B, it becomes Master on >>>> boot, and >>>> > host A remains Master as well. >>>> > When I force a state change on host B (ifconfig vlanx vhid y state >>>> backup), >>>> > it transitions to Backup then again to Master. >>>> > >>>> > When I comment out the CARP configuration in rc.conf , and configure >>>> CARP >>>> > manually on host B's interfaces after it boots, it correctly becomes >>>> and >>>> > remains Backup. >>>> > >>>> > >>>> > >>>> > Below is the excerpt from rc.conf pertaining to CARP configuration, >>>> the >>>> > only difference between the 2 hosts being their advskew. >>>> > >>>> > Host A >>>> > == BEGIN >>>> > >>>> > ifconfig_vlan410_alias0="vhid 110 pass passhere advskew 20 alias >>>> > 10.104.10.251/32" >>>> > >>>> > == END >>>> > >>>> > Host B >>>> > == BEGIN >>>> > >>>> > ifconfig_vlan410_alias0="vhid 110 pass passhere advskew 150 alias >>>> > 10.104.10.251/32" >>>> > >>>> > == END >>>> >>>> Put the IP first, and the vhid stuff last in rc.conf for things to work >>>> the most reliably. And drop the extra alias. >>>> >>>> ifconfig_vlan410_alias0="inet 10.104.10.251/32 vhid 110 pass passhere >>>> advskew 150" >>>> >>>> CARP requires that all IPs on an interface that are part of the same >>>> vhid to be listed (added) in the exact same order for the vhid to be >>>> considered "the same". That one trips me up all the time when manually >>>> adding an IP to a CARP pair, and then later rebooting one box as they both >>>> think they're master for that interface, while being a mix of master/backup >>>> for the other interfaces. >>>> >>>> Cheers, >>>> Freddie >>>> (running CARP on 2 10-CURRENT boxes and 2 10.1-p13 boxes) >>>> >>> >>> Cheers Freddie, will try and keep the thread up to date on the results. >>> >>> >> >
Eugene M. Zheganin
2015-Aug-21 07:06 UTC
[POSSIBLE BUG] 10-STABLE CARP erroneously becomes master on boot
Hi. On 20.08.2015 14:51, Damien Fleuriot wrote:> > Hello list, > > > > We've managed to find the source of the bug, if it is indeed a bug. > > It all comes down to the order in which the IP addresses are assigned to > the interface from /etc/rc.conf. > > > When using the following syntax, the physical IP address is configured > AFTER the CARPs on the interface, which results in the CARP advertisements > being sourced from the CARP IP, triggering the double MASTER situation : > ipv4_addrs_int="1.2.3.4/24" > ifconfig_int_alias0="1.2.3.6/32 vhid 1 pass test advskew 20" > > When using either of the following syntaxes, the physical IP address is > configured BEFORE the CARPs, which results in the CARP advertisements being > sourced from the physical IP and restoring normal functionality : > ifconfig_int="inet 1.2.3.4/24" > ifconfig_int_alias0="1.2.3.6/32 vhid 1 pass test advskew 20" > OR > ifconfig_int_alias0="1.2.3.4/24" > ifconfig_int_alias1="1.2.3.6/32 vhid 1 pass test advskew 20" >It has been there since carp-ng was commited to the 10-CURRENT 2 years ago. The thing is, carp-ng doesn't need a non-carp address on an interface anymore, both nodes can work fine using only shared address. This isn't comfortable in lots of cases, but still. Thus, kernel sends carp advertisements from a primary address on the interface (which is normal behavior for any known network stack) and for FreeBSD that primary address has always been the first address on an interface for a given AF. Thus, your split-brain carp situation cause lies definitely somewhere else. I'm running carp on FreeBSD for years, including legacy one; if there is a bug - the situation you are describing probably isn't one. Eugene.