Enable BBR TCP Congestion Control Algorithm out of the box

[sidewinder12s]

Is there a reason this is not enabled out of the box?

I saw another issue that appeared to indicate it's enabled but its still not on the latest AMI.

v20241011:

$ sysctl net.ipv4.tcp_available_congestion_control
net.ipv4.tcp_available_congestion_control = reno cubic

[ndbaker1]

were you referring to #432? In it, the author already mentions how they enabled BBR, but i think its not going to show up until the kmod is explicitly loaded, like -

[root@ip-192-168-143-4 bin]# sysctl net.ipv4.tcp_available_congestion_control
net.ipv4.tcp_available_congestion_control = reno cubic
[root@ip-192-168-143-4 bin]# modinfo tcp_bbr
filename:       /lib/modules/6.1.102-108.177.amzn2023.x86_64/kernel/net/ipv4/tcp_bbr.ko
description:    TCP BBR (Bottleneck Bandwidth and RTT)
license:        Dual BSD/GPL
author:         Soheil Hassas Yeganeh <soheil@google.com>
author:         Yuchung Cheng <ycheng@google.com>
author:         Neal Cardwell <ncardwell@google.com>
author:         Van Jacobson <vanj@google.com>
srcversion:     52FE9DC9CC96C6188A84576
depends:
retpoline:      Y
intree:         Y
name:           tcp_bbr
vermagic:       6.1.102-108.177.amzn2023.x86_64 SMP preempt mod_unload modversions
sig_id:         PKCS#7
signer:         Amazon Linux Kernel Signing Key
sig_key:        68:CE:0B:E9:C8:08:61:0D:28:E6:EE:2D:B9:7C:51:F7:0B:2F:41:5D
sig_hashalgo:   sha256
signature:      22:AF:C1:0F:FF:A5:C8:68:AE:A0:4B:14:E9:F8:10:BF:25:CE:C4:4A:
                4A:0C:EC:BE:90:5F:29:CB:AF:F9:68:FC:4C:C0:53:24:CB:28:A9:52:
                6D:11:FD:19:40:DC:73:D9:8C:DA:FA:D4:0F:48:30:4A:38:B7:6E:41:
                00:90:57:92:52:28:03:1E:31:27:6A:9A:94:21:AD:E5:03:52:41:19:
                F3:B5:81:DA:B6:FE:E6:02:B3:4E:FA:3C:D3:51:56:B5:4A:1A:6A:74:
                F2:EB:68:2C:9F:5B:95:28:A2:7C:B5:B1:1A:64:88:A6:7C:C5:3A:52:
                6C:80:5A:32:32:65:77:CA:DD:CA:B8:87:50:5F:D6:AA:EB:84:A7:1B:
                9C:35:6C:B1:88:70:3F:AC:D8:0D:E1:A5:09:41:39:BE:D5:F9:50:8E:
                A6:93:F3:F3:A5:41:C1:4F:F7:04:95:52:C5:2F:E9:53:85:D4:BE:4D:
                59:6E:3B:D5:1F:E2:C8:35:40:EF:16:97:F0:56:47:DD:44:3D:97:80:
                AF:CF:0B:7C:0E:EA:6A:D1:DB:15:B6:7B:15:0E:87:C9:24:8C:F9:AB:
                EF:49:CA:B9:EE:33:83:8F:0B:71:86:55:2B:B1:B3:28:25:CC:9E:4D:
                64:0C:6F:A2:EE:F3:FC:28:BC:A7:1A:71:5F:21:BF:8D:20:19:69:11:
                6B:7F:74:64:D9:89:0F:CA:C1:21:0A:F0:A9:9B:FF:D6:78:5A:E2:17:
                80:08:64:2A:23:FE:07:10:B2:33:31:E1:D7:DE:37:86:2D:5B:A0:A7:
                65:53:EE:32:E6:CC:22:0B:62:93:76:8C:63:23:81:C1:96:0F:5C:64:
                53:06:7A:D6:D9:DD:51:6F:94:0D:17:04:3F:9F:2D:C9:80:69:7C:1E:
                5F:10:9B:1D:E4:F5:BC:66:59:EA:AE:0B:AB:D8:4E:B5:D5:C2:F5:26:
                FC:87:EB:3B:91:17:9F:D6:C0:B3:C9:57:31:8A:38:35:C6:AC:AF:94:
                AC:97:68:D5:7F:01:83:10:28:F6:4F:DF:8B:FF:60:89:78:EC:FC:BB:
                9A:BD:43:FB:1D:8B:7A:AE:FE:D2:8A:EF:28:E8:34:D0:09:D3:05:A2:
                BF:3B:D6:4D:05:B3:00:BF:A7:3C:71:05:60:11:CE:38:E2:CF:90:E9:
                02:D1:44:15:E2:A1:F7:B0:B5:00:53:2A:11:70:CC:DB:D9:02:EB:86:
                DB:9B:EA:70:1B:5E:D1:81:1E:C2:A1:17:4E:A7:51:6E:FA:B3:79:A1:
                3C:76:7D:68:16:C4:C5:02:7E:CD:AE:69:BB:7F:C8:17:22:8A:09:86:
                0F:EA:49:0E:F4:81:6E:58:18:DE:60:CE
[root@ip-192-168-143-4 bin]# modprobe tcp_bbr
[root@ip-192-168-143-4 bin]# sysctl net.ipv4.tcp_available_congestion_control
net.ipv4.tcp_available_congestion_control = reno cubic bbr

[sidewinder12s]

I mean, I had read that ticket like BBR was already enabled out of the box for the AMI. It is not from what I was checking. So this ticket is linked/asking can we just make it the default.

[ndbaker1]

I think that's a conscious decision the end-user should make based on observed network metrics. BBR might conflict with fairness guarantees from the other congestion control algorithms, so deviating from the kernel's default seems likely to confuse IMO

[cartermckinnon]

Changing the default on AL2 at this point is not something we're likely to do, but I'm not opposed to doing this on AL2023.

[sidewinder12s]

I think that's a conscious decision the end-user should make based on observed network metrics. BBR might conflict with fairness guarantees from the other congestion control algorithms, so deviating from the kernel's default seems likely to confuse IMO

I see (and looking up some benchmarking of BBR)

[sidewinder12s]

So what prompted this ask was that we were seeing microbursting on the ENA was leading to dropped traffic so we were looking for potential mitigations in the EKS AMI, VPC CNI or elsewhere that might reduce the impact of these microbursts.

[ndbaker1]

@sidewinder12s Were the symptoms here something the CNI bandwidth plugin could help with? (ref: https://github.com/aws/amazon-vpc-cni-k8s/tree/master?tab=readme-ov-file#enable_bandwidth_plugin-v1100)

[sidewinder12s]

I was not sure. I think I had generally found just as many warnings about the native bandwidth plugin potentially introducing its own performance issues due to how it works, especially when the entire node is under pressure (mostly referring to some of the benchmarking Cilium did a few years ago). Our issues were intermittent enough that it was difficult to find the time or use case to repro or benchmark.

[github-actions]

This issue is stale because it has been open for 60 days with no activity. Remove the stale label or comment to avoid closure in 14 days