Hello, I'm Running 12.0-REL in a VM under W2016S with CARP enabled and paired to a baremetal FreeBSD server. All of a sudden I realized that thjis machine is unable to become a CARP MASTER - because it sees it's own ACRP announces, but instead of seeing them from a CARP synthetic MAC address only, it sees additional extra packets with several MACs derived from the original one (I'm well awared about the -MacAddressSpoof on SetVmNetworkAdapterVlan switch, and it's running with this thingg on, but still). These packets always almost (but not 100%) accompany each valid CARP advertisement. Say, we have a CARP-enabled interface: vlan2: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500 ??????? description: AS WAN ??????? options=80000<LINKSTATE> ??????? ether 00:15:5d:0a:79:12 ??????? inet 91.206.242.9/28 broadcast 91.206.242.15 ??????? inet 91.206.242.12/28 broadcast 91.206.242.15 vhid 3 ??????? groups: vlan ??????? carp: BACKUP vhid 3 advbase 1 advskew 250 ??????? vlan: 2 vlanpcp: 0 parent interface: hn1 ??????? media: Ethernet autoselect (10Gbase-T <full-duplex>) ??????? status: active ??????? nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL> Notice the MAC and now look at this: ===Cut== [root at gw1:~]# tcpdump -T carp -nepi vlan2 carp tcpdump: verbose output suppressed, use -v or -vv for full protocol decode listening on vlan2, link-type EN10MB (Ethernet), capture size 262144 bytes 20:45:54.152619 00:00:5e:00:01:03 > 01:00:5e:00:00:12, ethertype IPv4 (0x0800), length 70: 91.206.242.9 > 224.0.0.18: CARPv2-advertise 36: vhid=3 advbase=1 advskew=100 authlen=7 counter=13769798250643227035 ^^^ this is the ordinary and valid CARP advertisement, notice the synthetic MAC which is requiring setting mac address spoofing. 20:45:54.152880 9c:8e:99:0f:79:42 > 01:00:5e:00:00:12, ethertype IPv4 (0x0800), length 70: 91.206.242.9 > 224.0.0.18: CARPv2-advertise 36: vhid=3 advbase=1 advskew=100 authlen=7 counter=13769798250643227035 ^^^ this is some insanity happening 20:45:54.153234 9c:8e:99:0f:79:42 > 01:00:5e:00:00:12, ethertype IPv4 (0x0800), length 70: 91.206.242.9 > 224.0.0.18: CARPv2-advertise 36: vhid=3 advbase=1 advskew=100 authlen=7 counter=13769798250643227035 ^^^ and again 20:45:54.153401 9c:8e:99:0f:79:42 > 01:00:5e:00:00:12, ethertype IPv4 (0x0800), length 70: 91.206.242.9 > 224.0.0.18: CARPv2-advertise 36: vhid=3 advbase=1 advskew=100 authlen=7 counter=13769798250643227035 ^^^ and again 20:45:57.562470 00:00:5e:00:01:03 > 01:00:5e:00:00:12, ethertype IPv4 (0x0800), length 70: 91.206.242.9 > 224.0.0.18: CARPv2-advertise 36: vhid=3 advbase=1 advskew=100 authlen=7 counter=13769798250643227036 ^^^ valid CARP advertisement, next one-second advbase cycle 20:45:57.562874 9c:8e:99:0f:79:3c > 01:00:5e:00:00:12, ethertype IPv4 (0x0800), length 70: 91.206.242.9 > 224.0.0.18: CARPv2-advertise 36: vhid=3 advbase=1 advskew=100 authlen=7 counter=13769798250643227036 ^^^ more insane stuff, notice the NEW (sic !) MAC-address 20:45:57.562955 9c:8e:99:0f:79:3c > 01:00:5e:00:00:12, ethertype IPv4 (0x0800), length 70: 91.206.242.9 > 224.0.0.18: CARPv2-advertise 36: vhid=3 advbase=1 advskew=100 authlen=7 counter=13769798250643227036 20:45:57.562989 9c:8e:99:0f:79:3c > 01:00:5e:00:00:12, ethertype IPv4 (0x0800), length 70: 91.206.242.9 > 224.0.0.18: CARPv2-advertise 36: vhid=3 advbase=1 advskew=100 authlen=7 counter=13769798250643227036 ^C 8 packets captured 3195 packets received by filter ===Cut== Does anyone has, by any chance, some idea about what's happening ? As soon as I stop CARP stack on this VM these "mad" MACs aren't received anymore, so I'm pretty confident these are somehow procuced on the Hyper-V side. Another weird this is that vlan1? is refusing to work (seems like packets are never received on the VM side) unless its configured on another adapter in the -Untagged (once again powershell term for SetVmNetworkAdapterVlan). Thanks. Eugene.
Stefan Bethke
2020-Jun-01 09:18 UTC
Using VLAN ID 1 (was: CARP under Hyper-V: weird things happen)
Only tangential to your main issues, but:> Am 31.05.2020 um 18:07 schrieb Eugene M. Zheganin <emz at norma.perm.ru>: > > Another weird this is that vlan1 is refusing to work (seems like packets are never received on the VM side) unless its configured on another adapter in the -Untagged (once again powershell term for SetVmNetworkAdapterVlan).I believe it is best practice to not use VLAN ID 1 in your network design because many vendors assign it a special role, and it can be hard to reconfigure that on the devices. The virtual switch might have the same issue. Stefan -- Stefan Bethke <stb at lassitu.de> Fon +49 151 14070811
Hi Eugene, Might it be the Hyper-V doesn?t properly implement multicast? Or there is perhaps some setting in there to let it work. From memory CARP is not trivial on vmware as well, unless you make special settings. Some ideas here: https://docs.netgate.com/pfsense/en/latest/highavailability/troubleshooting-high-availability-clusters.html#hypervisor-users-especially-vmware-esx-esxi <https://docs.netgate.com/pfsense/en/latest/highavailability/troubleshooting-high-availability-clusters.html#hypervisor-users-especially-vmware-esx-esxi> Daniel> On 31 May 2020, at 19:07, Eugene M. Zheganin <emz at norma.perm.ru> wrote: > > Hello, > > I'm Running 12.0-REL in a VM under W2016S with CARP enabled and paired to a baremetal FreeBSD server. > > All of a sudden I realized that thjis machine is unable to become a CARP MASTER - because it sees it's own ACRP announces, but instead of seeing them from a CARP synthetic MAC address only, it sees additional extra packets with several MACs derived from the original one (I'm well awared about the -MacAddressSpoof on SetVmNetworkAdapterVlan switch, and it's running with this thingg on, but still). These packets always almost (but not 100%) accompany each valid CARP advertisement. > > Say, we have a CARP-enabled interface: > > vlan2: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500 > description: AS WAN > options=80000<LINKSTATE> > ether 00:15:5d:0a:79:12 > inet 91.206.242.9/28 broadcast 91.206.242.15 > inet 91.206.242.12/28 broadcast 91.206.242.15 vhid 3 > groups: vlan > carp: BACKUP vhid 3 advbase 1 advskew 250 > vlan: 2 vlanpcp: 0 parent interface: hn1 > media: Ethernet autoselect (10Gbase-T <full-duplex>) > status: active > nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL> > > Notice the MAC and now look at this: > > ===Cut==> > [root at gw1:~]# tcpdump -T carp -nepi vlan2 carp > tcpdump: verbose output suppressed, use -v or -vv for full protocol decode > listening on vlan2, link-type EN10MB (Ethernet), capture size 262144 bytes > 20:45:54.152619 00:00:5e:00:01:03 > 01:00:5e:00:00:12, ethertype IPv4 (0x0800), length 70: 91.206.242.9 > 224.0.0.18: CARPv2-advertise 36: vhid=3 advbase=1 advskew=100 authlen=7 counter=13769798250643227035 > > ^^^ this is the ordinary and valid CARP advertisement, notice the synthetic MAC which is requiring setting mac address spoofing. > > 20:45:54.152880 9c:8e:99:0f:79:42 > 01:00:5e:00:00:12, ethertype IPv4 (0x0800), length 70: 91.206.242.9 > 224.0.0.18: CARPv2-advertise 36: vhid=3 advbase=1 advskew=100 authlen=7 counter=13769798250643227035 > > ^^^ this is some insanity happening > > 20:45:54.153234 9c:8e:99:0f:79:42 > 01:00:5e:00:00:12, ethertype IPv4 (0x0800), length 70: 91.206.242.9 > 224.0.0.18: CARPv2-advertise 36: vhid=3 advbase=1 advskew=100 authlen=7 counter=13769798250643227035 > > ^^^ and again > > 20:45:54.153401 9c:8e:99:0f:79:42 > 01:00:5e:00:00:12, ethertype IPv4 (0x0800), length 70: 91.206.242.9 > 224.0.0.18: CARPv2-advertise 36: vhid=3 advbase=1 advskew=100 authlen=7 counter=13769798250643227035 > > ^^^ and again > > 20:45:57.562470 00:00:5e:00:01:03 > 01:00:5e:00:00:12, ethertype IPv4 (0x0800), length 70: 91.206.242.9 > 224.0.0.18: CARPv2-advertise 36: vhid=3 advbase=1 advskew=100 authlen=7 counter=13769798250643227036 > > ^^^ valid CARP advertisement, next one-second advbase cycle > > 20:45:57.562874 9c:8e:99:0f:79:3c > 01:00:5e:00:00:12, ethertype IPv4 (0x0800), length 70: 91.206.242.9 > 224.0.0.18: CARPv2-advertise 36: vhid=3 advbase=1 advskew=100 authlen=7 counter=13769798250643227036 > > ^^^ more insane stuff, notice the NEW (sic !) MAC-address > > 20:45:57.562955 9c:8e:99:0f:79:3c > 01:00:5e:00:00:12, ethertype IPv4 (0x0800), length 70: 91.206.242.9 > 224.0.0.18: CARPv2-advertise 36: vhid=3 advbase=1 advskew=100 authlen=7 counter=13769798250643227036 > 20:45:57.562989 9c:8e:99:0f:79:3c > 01:00:5e:00:00:12, ethertype IPv4 (0x0800), length 70: 91.206.242.9 > 224.0.0.18: CARPv2-advertise 36: vhid=3 advbase=1 advskew=100 authlen=7 counter=13769798250643227036 > ^C > 8 packets captured > 3195 packets received by filter > > ===Cut==> > > Does anyone has, by any chance, some idea about what's happening ? As soon as I stop CARP stack on this VM these "mad" MACs aren't received anymore, so I'm pretty confident these are somehow procuced on the Hyper-V side. > > Another weird this is that vlan1 is refusing to work (seems like packets are never received on the VM side) unless its configured on another adapter in the -Untagged (once again powershell term for SetVmNetworkAdapterVlan). > > > Thanks. > > Eugene. > > _______________________________________________ > freebsd-stable at freebsd.org mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-stable > To unsubscribe, send any mail to "freebsd-stable-unsubscribe at freebsd.org"