problem

I'm running 4.22.0.0 which supports specifying static routes using nexthops instead of just referencing an existing gateway id.

For this testing I have 3 VPCs connected back to back using private gateways on the same vlan (999). I'm using 172.17.0.0/24 as the subnet for the private gateways with addresses of .2, .3, and .4. There is no "gateway" for this subnet, so I use a dummy ".1".

I can see when adding a static route of 10.252.0.0/16, it adds it to the main table, but not the policy-based-routing table:

root@r-441701-VM:~# ip route show table all | grep eth2
default via 172.17.0.1 dev eth2 table Table_eth2 proto static
throw 10.251.4.0/22 table Table_eth2 proto static
throw 172.17.0.0/24 table Table_eth2 proto static
10.252.0.0/16 via 172.17.0.2 dev eth2
172.17.0.0/24 dev eth2 proto kernel scope link src 172.17.0.3
local 172.17.0.3 dev eth2 table local proto kernel scope host src 172.17.0.3
broadcast 172.17.0.255 dev eth2 table local proto kernel scope link src 172.17.0.3

And we can see in the ip rules that traffic coming in from eth2 uses the PBR table:

root@r-441701-VM:~# ip rule | grep Table_eth2
32760:	from 172.17.0.0/24 lookup Table_eth2
32761:	from all fwmark 0x66 lookup Table_eth2

root@r-441701-VM:~# iptables-save | grep 0x66
-A PREROUTING -s 10.251.4.0/22 -d 172.17.0.0/24 -m state --state NEW -j MARK --set-xmark 0x66/0xffffffff
-A PREROUTING -i eth2 -m state --state NEW -j CONNMARK --set-xmark 0x66/0xffffffff

I haven't fully investigated the underlying cause, but I think probably the rp_filter is rejecting the packets as the return packet would be coming back from some address other than the default gateway configured for Table_eth2. So I think it thinks they are spoofed packets returning.

Of course if you configure the gateways on either end of 2 of the private gateways to point to eachother everything works.

versions

4.22.0.0

The steps to reproduce the bug

See description above

What to do about it?

Either don't set a firewall mark so it uses the default/main routing table for private gateways, or when static routes are created with a nexthop in the same subnet as the private gateway interface, add to the PBR table.