Some CARP interfaces send out unusual mcast packets when set for CARP unicast

[nzkiwi68]

Important notices

Before you add a new report, we ask you kindly to acknowledge the following:

• I have read the contributing guide lines at https://github.com/opnsense/src/blob/master/CONTRIBUTING.md
• I am convinced that my issue is new after having checked both open and closed issues at https://github.com/opnsense/src/issues?q=is%3Aissue

Describe the bug

Despite setting a peer ipv4 unicast address on a CARP virtual IP address, and the recent 25.1 OPNsense release with the FreeBSD upstream bug fix incorporated, some interfaces still send out an unusual multicast packet that sort of looks like a unicast packet but Wireshark sees this as a multicast packet.

This is also causing our ISP upstream to repeat back the CARP mcast on WAN2 from our fw1, coming from the ISP Juniper router with their src mac address and src IP address of our fw1 WAN2 addresses, destined to fw2 WAN2 correct dst mac and correct IP address. This is causing mac flapping on fw2 and CARP instability issues. We suspect because the CARP packet outbound from fw1 is malformed and the Juniper router doesn't quite know what to do with it.

To Reproduce

1. Setup HA clustered OPNsense with CARP addresses set with an ipv4 peer IP address (to use unicast CARP)
2. setup multiple interfaces, LAN, WAN1, WAN2, DMZ, DMZ2 (we have 7 in total)
3. Run a packet capture and observe on some interfaces, malformed "multicast" packets are being sent out, others send a normal unicast packet

Expected behavior

All CARP addresses set to use unicast should send out only unicast packets.

Screenshots

fw1 - interface hn0 - sending unusual mcast CARP packet
Frame 1: 70 bytes on wire (560 bits), 70 bytes captured (560 bits)
Encapsulation type: Ethernet (1)
Arrival Time: Feb 12, 2025 14:11:43.214572000 New Zealand Daylight Time
UTC Arrival Time: Feb 12, 2025 01:11:43.214572000 UTC
Epoch Arrival Time: 1739322703.214572000
[Time shift for this packet: 0.000000000 seconds]
[Time delta from previous captured frame: 0.000000000 seconds]
[Time delta from previous displayed frame: 0.000000000 seconds]
[Time since reference or first frame: 0.000000000 seconds]
Frame Number: 1
Frame Length: 70 bytes (560 bits)
Capture Length: 70 bytes (560 bits)
[Frame is marked: False]
[Frame is ignored: False]
[Protocols in frame: eth:ethertype:ip:carp]
[Coloring Rule Name: Routing]
[Coloring Rule String: hsrp || eigrp || ospf || bgp || cdp || vrrp || carp || gvrp || igmp || ismp]
Ethernet II, Src: Microsoft_de:ed:00 (00:15:5d:de:ed:00), Dst: IPv4mcast_28:c9:ef (01:00:5e:28:c9:ef)
Destination: IPv4mcast_28:c9:ef (01:00:5e:28:c9:ef)
Source: Microsoft_de:ed:00 (00:15:5d:de:ed:00)
Type: IPv4 (0x0800)
[Stream index: 0]
Internet Protocol Version 4, Src: 192.168.201.238, Dst: 192.168.201.239
0100 .... = Version: 4
.... 0101 = Header Length: 20 bytes (5)
Differentiated Services Field: 0xe0 (DSCP: CS7, ECN: Not-ECT)
Total Length: 56
Identification: 0x0000 (0)
010. .... = Flags: 0x2, Don't fragment
...0 0000 0000 0000 = Fragment Offset: 0
Time to Live: 255
Protocol: VRRP (112)
Header Checksum: 0x6546 [validation disabled]
[Header checksum status: Unverified]
Source Address: 192.168.201.238
Destination Address: 192.168.201.239
[Stream index: 0]
Common Address Redundancy Protocol
Version 2, Packet type 1 (Advertisement)
Virtual Host ID: 10
Advertisement Skew: 0
Auth Len: 7
Demotion indicator: 0
Adver Int: 2
Checksum: 0x3649 [correct]
[Checksum Status: Good]
Counter: 15684383851564536600
HMAC: 325b9b5fb73a149f2defcd59ccf7722055c878be

** same fw1 - hn4 interface - sending proper unicast CARP packet **
Frame 1: 70 bytes on wire (560 bits), 70 bytes captured (560 bits)
Encapsulation type: Ethernet (1)
Arrival Time: Feb 12, 2025 14:11:43.827888000 New Zealand Daylight Time
UTC Arrival Time: Feb 12, 2025 01:11:43.827888000 UTC
Epoch Arrival Time: 1739322703.827888000
[Time shift for this packet: 0.000000000 seconds]
[Time delta from previous captured frame: 0.000000000 seconds]
[Time delta from previous displayed frame: 0.000000000 seconds]
[Time since reference or first frame: 0.000000000 seconds]
Frame Number: 1
Frame Length: 70 bytes (560 bits)
Capture Length: 70 bytes (560 bits)
[Frame is marked: False]
[Frame is ignored: False]
[Protocols in frame: eth:ethertype:ip:carp]
[Coloring Rule Name: Routing]
[Coloring Rule String: hsrp || eigrp || ospf || bgp || cdp || vrrp || carp || gvrp || igmp || ismp]
Ethernet II, Src: Microsoft_de:ed:04 (00:15:5d:de:ed:04), Dst: Microsoft_e9:0a:05 (00:15:5d:e9:0a:05)
Destination: Microsoft_e9:0a:05 (00:15:5d:e9:0a:05)
.... ..0. .... .... .... .... = LG bit: Globally unique address (factory default)
.... ...0 .... .... .... .... = IG bit: Individual address (unicast)
Source: Microsoft_de:ed:04 (00:15:5d:de:ed:04)
.... ..0. .... .... .... .... = LG bit: Globally unique address (factory default)
.... ...0 .... .... .... .... = IG bit: Individual address (unicast)
Type: IPv4 (0x0800)
[Stream index: 0]
Internet Protocol Version 4, Src: 192.168.202.251, Dst: 192.168.202.252
0100 .... = Version: 4
.... 0101 = Header Length: 20 bytes (5)
Differentiated Services Field: 0xe0 (DSCP: CS7, ECN: Not-ECT)
Total Length: 56
Identification: 0x0000 (0)
010. .... = Flags: 0x2, Don't fragment
...0 0000 0000 0000 = Fragment Offset: 0
Time to Live: 255
Protocol: VRRP (112)
Header Checksum: 0x632c [validation disabled]
[Header checksum status: Unverified]
Source Address: 192.168.202.251
Destination Address: 192.168.202.252
[Stream index: 0]
Common Address Redundancy Protocol
Version 2, Packet type 1 (Advertisement)
Virtual Host ID: 14
Advertisement Skew: 0
Auth Len: 7
Demotion indicator: 0
Adver Int: 2
Checksum: 0x2820 [correct]
[Checksum Status: Good]
Counter: 316573603036265274
HMAC: 013599daa4ab77fa22176fc00978bb8bf78f3cc7

Additional context

We suspect this is an upstream FreeBSD issue with CARP handling for unicast packets and somewhere in the code it's not getting it quite right on CARP startup under certain circumstances.

We have noted that once an interface starts sending out mcast packets for CARP, it never stops, even if you disable and then re-enable CARP, it still carries on sending out mcast packets.

We also tried under "Interfaces > Neighbors" setting static MAC addresses on fw1 for fw2 and the reverse for fw2, setting static MAC addresses for fw1. This didn't help.

We have checked FreeBSD and haven't seen a bug report for this issue on FreeBSD.

Environment

OPNsense 25.1 (amd64)
Under Hyper-V as a VM

Acknowledgements

Brett Merrick contributed considerably to diagnosing this issue.

[nzkiwi68]

Quick note:

Brett has identified the code in FreeBSD causing the issue:
sys/netinet/ip_carp.c - line 1248:
if (IN_MULTICAST(sc->sc_carpaddr.s_addr)) m->m_flags |= M_MCAST;

Should read:
if (IN_MULTICAST(ntohl(sc->sc_carpaddr.s_addr))) m->m_flags |= M_MCAST;

A present, any announcement where the peer IP address ends in .244-.239 rather than begins with 244.-239. get sent to an invalid multicast destination MAC address, which just happens to be the case with WAN HA implementation we have in production.

Brett is obtaining a login etc with FreeBSD to properly raise this there and will link back to here with the proper fix when we get there. The reason for this post is save others time from looking into this bug when we are certain on the underlying code issue and the fix required.

Steven.

[BrettMerrick]

Thanks Steven,

Bug submitted upstream as follows: https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=284872

[fichtner]

@nzkiwi68 @BrettMerrick really nice work guys, I stashed a test patch 8f86d0fdd3 and will provide a test kernel in a few minutes.

Looks like ccff207 was committed to address this partially before.

Cheers,
Franco

[fichtner]

Took a bit longer but here it is now:

# opnsense-update -zkr 25.1.1-carp

Make sure uname -a says "8f86d0fdd37" after a reboot as I had to redo the kernel two times because it wasn't building the right revision.

Cheers,
Franco

[nzkiwi68]

Thanks for the patch. Implemented last night (we're UTC +13) and got mixed results.

The good.
CARP unicast mode definitely now sends properly formed unicast packets.

The strange
fw2 on wan1 and wan2 sends out properly formed unicast CARP packets, but with a dst MAC of the wan gateway and not of the dst MAC of the configured peer IP.

• fw1 can ping fw2 wan1 & wan2 ip
• fw2 can ping fw2 wan1 & wan2 ip

Detail

1. fw1 is the CARP master
2. switch fw1 to "enter persistent CARP maintenance mode"
3. on fw1 - 5 out 7 interfaces transition to backup
4. on fw2 - 7 out of 7 interfaces transition to master
5. the two interfaces on fw1 that persist in master mode are wan1 and wan2
6. Packet captures on fw2 show that fw2 on wan1 is sending a CARP unicast packet destined to the MAC dst of wan1's gateway
7. Packet captures on fw2 show that fw2 on wan2 is sending a CARP unicast packet destined to the MAC dst of wan2's gateway
8. Packet captures on fw1 show no CARP packet ever arrives on wan1 or wan2, hence fw1 remains the CARP master too

Hypothesis
Somehow on the backup firewall, on an interface that has gateway set, when CARP unicast creates the unicast packet for the peer destination, it does not correctly lookup the arp IP of the peer as set, but instead uses the interface gateway as the destination.

We thought the ISP was doing silly things replying but I'm convinced we are actually generating that packet. The ISP at our request has put a silent drop all protocol 112 packets on the wan1 and wan2 interfaces. If that's true then OPNsense/FreeBSD must be generating the packet.

[fichtner]

Thanks for the feedback. Would you mind sharing this in the FreeBSD bug tracker as well as long as there is attention there?

I’ll take a look tomorrow morning and move that patch to the next stable.

Cheers,
Franco

[gmshake]

Thanks for the patch. Implemented last night (we're UTC +13) and got mixed results.

The good. CARP unicast mode definitely now sends properly formed unicast packets.

The strange fw2 on wan1 and wan2 sends out properly formed unicast CARP packets, but with a dst MAC of the wan gateway and not of the dst MAC of the configured peer IP.

For unicast CARP setup, if the two interfaces (configured with same vhid) are not in the same network, i.e. directly reachable, then it is expected that you see the CARP packet with dst MAC been the gateway's ether address, otherwise the CARP packets are not able to reach the peer.

• fw1 can ping fw2 wan1 & wan2 ip
• fw2 can ping fw2 wan1 & wan2 ip

Detail

1. fw1 is the CARP master
2. switch fw1 to "enter persistent CARP maintenance mode"
3. on fw1 - 5 out 7 interfaces transition to backup
4. on fw2 - 7 out of 7 interfaces transition to master
5. the two interfaces on fw1 that persist in master mode are wan1 and wan2
6. Packet captures on fw2 show that fw2 on wan1 is sending a CARP unicast packet destined to the MAC dst of wan1's gateway
7. Packet captures on fw2 show that fw2 on wan2 is sending a CARP unicast packet destined to the MAC dst of wan2's gateway
8. Packet captures on fw1 show no CARP packet ever arrives on wan1 or wan2, hence fw1 remains the CARP master too

Hypothesis Somehow on the backup firewall, on an interface that has gateway set, when CARP unicast creates the unicast packet for the peer destination, it does not correctly lookup the arp IP of the peer as set, but instead uses the interface gateway as the destination.

I'm confused. It appears that you add arp entry of the CARP peer, so is the peer directly reachable ? If yes, then that ( adding arp entry of the CARP peer ) is not necessary. Can you share your setup ?

We thought the ISP was doing silly things replying but I'm convinced we are actually generating that packet. The ISP at our request has put a silent drop all protocol 112 packets on the wan1 and wan2 interfaces. If that's true then OPNsense/FreeBSD must be generating the packet.

[nzkiwi68]

For unicast CARP setup, if the two interfaces (configured with same vhid) are not in the same network, i.e. directly reachable, then it is expected that you see the CARP packet with destination MAC been the gateway's ether address, otherwise the CARP packets are not able to reach the peer.

Sure, but in our case it's the same L2 subnet inside a VLAN, all layer 2. Normal networking and routing should apply. If the IP address is directly layer 2 reachable, then the packet should be generated with a destination MAC of the IP address of the peer, the MAC coming from the local ARP table.

I'm confused. It appears that you add ARP entry of the CARP peer, so is the peer directly reachable ? If yes, then that ( adding ARP entry of the CARP peer ) is not necessary. Can you share your setup ?

Adding peer (ipv4) address is how you switch from sending multicast CARP to using unicast. To switch to unicast CARP it is necessary to enter a Peer (ipv4) destination address. If you do not enter a destination address, then it defaults to multicast 224.0.0.18 and we would like to use unicast, not multicast.

For setup, just assume a simple network, say 192.168.100.0/24, fw1 192.168.100.1, fw1 192.168.100.2 and an ISP gateway of 192.168.100.254

I guess you might ask "why" would we want to switch to unicast on the same L2 subnet, why not just use multicast?

We have some good reasons:

• In this case, the pair of core switches are misbehaving and if any sort of broadcast packet enters switch1, once it crosses to switch2, switch2 duplicates the packet. A packet capture show duplicate broadcast packets on switch2 for mcast and any sort of broadcast packet. This also happens the other way around too, switch2 broadcast packet first > packet over LACP to switch1 > duplicate broadcasts now on switch1. We have a case running with the switch manufacturing but it's going to a long process we suspect.
• In other cases I have seen naughty stacked switches that misbehave with multicast frames and just don't work reliably with CARP and HA
• It does reduce the broadcast packets on each interface by not generating any multicast packets
• With some ISPs we have met problems with protocol 112 on the WAN interface. Unicast CARP I expect will help overcome those cases too.

Note
We have 7 interfaces on this HA firewall pair. On 5 interfaces, everything works as expected. Only on the two WAN1 and WAN2 interfaces is this issue occurring. It does seem very much to me that when the interface has a gateway set, CARP unicast generates all frames destined to the MAC address of the interface gateway.

[gmshake]

For unicast CARP setup, if the two interfaces (configured with same vhid) are not in the same network, i.e. directly reachable, then it is expected that you see the CARP packet with destination MAC been the gateway's ether address, otherwise the CARP packets are not able to reach the peer.

Sure, but in our case it's the same L2 subnet inside a VLAN, all layer 2. Normal networking and routing should apply. If the IP address is directly layer 2 reachable, then the packet should be generated with a destination MAC of the IP address of the peer, the MAC coming from the local ARP table.

Yes, exactly.

I'm confused. It appears that you add ARP entry of the CARP peer, so is the peer directly reachable ? If yes, then that ( adding ARP entry of the CARP peer ) is not necessary. Can you share your setup ?

Adding peer (ipv4) address is how you switch from sending multicast CARP to using unicast. To switch to unicast CARP it is necessary to enter a Peer (ipv4) destination address. If you do not enter a destination address, then it defaults to multicast 224.0.0.18 and we would like to use unicast, not multicast.

For setup, just assume a simple network, say 192.168.100.0/24, fw1 192.168.100.1, fw1 192.168.100.2 and an ISP gateway of 192.168.100.254

I guess you might ask "why" would we want to switch to unicast on the same L2 subnet, why not just use multicast?

We have some good reasons:

• In this case, the pair of core switches are misbehaving and if any sort of broadcast packet enters switch1, once it crosses to switch2, switch2 duplicates the packet. A packet capture show duplicate broadcast packets on switch2 for mcast and any sort of broadcast packet. This also happens the other way around too, switch2 broadcast packet first > packet over LACP to switch1 > duplicate broadcasts now on switch1. We have a case running with the switch manufacturing but it's going to a long process we suspect.
• In other cases I have seen naughty stacked switches that misbehave with multicast frames and just don't work reliably with CARP and HA
• It does reduce the broadcast packets on each interface by not generating any multicast packets
• With some ISPs we have met problems with protocol 112 on the WAN interface. Unicast CARP I expect will help overcome those cases too.

Perfect valid cases.

Note We have 7 interfaces on this HA firewall pair. On 5 interfaces, everything works as expected. Only on the two WAN1 and WAN2 interfaces is this issue occurring. It does seem very much to me that when the interface has a gateway set, CARP unicast generates all frames destined to the MAC address of the interface gateway.

I tried to setup unicast CARP hosts within the same network ( same L2 broadcast domain ), but failed to repeat your issue, with / without interface gateway .

The topology, both fw1 and fw2 are FreeBSD 14.2,

fw1, IP 192.168.100.1/24, carp peer 192.168.100.2, default route 192.168.100.254
fw2, IP 192.168.100.2/24, carp peer 192.168.100.1, default route 192.168.100.254

I guess that is an edge case that a simple topology such as above can not repeat. You may want to post the issue to FreeBSD bugzilla. Best to have minimal steps to repeat.