[EKS] [eks-node-monitoring-agent] [request]: Support `kube-proxy` with IPVS mode

[z0rc]

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

Tell us about your request
Properly support kube-proxy networking setup when it uses IPVS.

Which service(s) is this request for?
EKS, kube-proxy addon installed with configuration {"mode":"ipvs"}

Tell us about the problem you're trying to solve. What are you trying to do, and why is it hard?
eks-node-monitoring-agent produces following logs and kubernetes events:

{"level":"info","ts":"2025-02-06T10:45:52Z","msg":"handling export request","source":"networking","condition":{"Reason":"UnexpectedRejectRule","Message":"Found an unexpected iptables reject rule: \"-A KUBE-SOURCE-RANGES-FIREWALL -j DROP\"","Severity":"Warning","MinOccurrences":0}}
{"level":"info","ts":"2025-02-06T10:45:52Z","msg":"handling export request","source":"networking","condition":{"Reason":"UnexpectedRejectRule","Message":"Found an unexpected iptables reject rule: \"-A KUBE-IPVS-FILTER -m conntrack --ctstate NEW -m set --match-set KUBE-6-IPVS-IPS dst -j REJECT --reject-with icmp6-port-unreachable\"","Severity":"Warning","MinOccurrences":0}}

Are you currently working around this issue?
By uninstalling eks-node-monitoring-agent.

Additional context
N/A

Attachments
N/A

[ndbaker1]

yep that makes sense, we'll work on a change to pardon the rules related to IPVS 👍

I'm also curious, if these lower severity detections (the Events and not NodeConditions) aren't important for you would it benefit to have a configurable severity threshold/verbosity? some API that closely resembles the logr implementaton

[z0rc]

I'm also curious, if these lower severity detections (the Events and not NodeConditions) aren't important for you would it benefit to have a configurable severity threshold/verbosity? some API that closely resembles the logr implementaton

Configurable severity would be nice. In my case I would use it (lower the severity), if I understand "warning" and its implications for my use case. Though in general I'd use the default levels.

[mohag]

Calico added rules results in similar issues on 1.0.2.

{"level":"info","ts":"2025-03-13T09:32:18Z","msg":"handling export request","source":"networking","condition":{"Reason":"UnexpectedRejectRule","Message":"Found an unexpected iptables reject rule: \"-A cali-tw-enif60d019c205 -m comment --comment \\\"cali:qhfZUMLL7QonXLhF\\\" -m conntrack --ctstate INVALID -j DROP\"","Severity":"Warning","MinOccurrences":0}}
{"level":"info","ts":"2025-03-13T09:32:18Z","msg":"sending condition to exporter"}
{"level":"info","ts":"2025-03-13T09:32:18Z","msg":"handling export request","source":"networking","condition":{"Reason":"UnexpectedRejectRule","Message":"Found an unexpected iptables reject rule: \"-A cali-tw-enif60d019c205 -m comment --comment \\\"cali:on3uZRyrGlllUJ7w\\\" -m comment --comment \\\"Drop if no profiles matched\\\" -j DROP\"","Severity":"Warning","MinOccurrences":0}}

It results in events like this: (mainly for people Googling those events)

NetworkingReady | Node ip-172-28-226-214.eu-central-1.compute.internal | (combined from similar events): UnexpectedRejectRule: Found an unexpected iptables reject rule: "-A cali-PREROUTING -m comment --comment \"cali:B2t7yHkp_oKgWOGM\" -m mark --mark 0x40000/0x40000 -m rpfilter --validmark --invert -j DROP"

[mohag]

Upgrading to 1.1.0 did not fix it for Calico

[woehrl01]

This also happens for just having an IPv6 cluster:

UnexpectedRejectRule: Found an unexpected iptables reject rule: "-A FORWARD -d 169.254.172.0/22 -m conntrack --ctstate NEW -m comment --comment \"Block Node Local Pod access via IPv4\" -j REJECT --reject-with icmp-port-unreachable"

[elcaos]

Calico added rules results in similar issues on 1.0.2.

{"level":"info","ts":"2025-03-13T09:32:18Z","msg":"handling export request","source":"networking","condition":{"Reason":"UnexpectedRejectRule","Message":"Found an unexpected iptables reject rule: \"-A cali-tw-enif60d019c205 -m comment --comment \\\"cali:qhfZUMLL7QonXLhF\\\" -m conntrack --ctstate INVALID -j DROP\"","Severity":"Warning","MinOccurrences":0}}
{"level":"info","ts":"2025-03-13T09:32:18Z","msg":"sending condition to exporter"}
{"level":"info","ts":"2025-03-13T09:32:18Z","msg":"handling export request","source":"networking","condition":{"Reason":"UnexpectedRejectRule","Message":"Found an unexpected iptables reject rule: \"-A cali-tw-enif60d019c205 -m comment --comment \\\"cali:on3uZRyrGlllUJ7w\\\" -m comment --comment \\\"Drop if no profiles matched\\\" -j DROP\"","Severity":"Warning","MinOccurrences":0}}

It results in events like this: (mainly for people Googling those events)

NetworkingReady | Node ip-172-28-226-214.eu-central-1.compute.internal | (combined from similar events): UnexpectedRejectRule: Found an unexpected iptables reject rule: "-A cali-PREROUTING -m comment --comment \"cali:B2t7yHkp_oKgWOGM\" -m mark --mark 0x40000/0x40000 -m rpfilter --validmark --invert -j DROP"

We do have the very same issue at our company, using Calico. It would be a considerable improvement if we could completely silence or disable UnexpectedRejectRule

[ariel-rivo]

+1

[mohag]

The Calico issue is still there, but Kubernetes is dropping support for IPVS mode in kube-proxy....

[dcoppa]

+1

Our logs are full of messages like this:

{"level":"info","ts":"2025-12-30T18:47:12Z","msg":"sending condition to exporter","hostname":"ip-10-212-26-96.eu-central-1.compute.internal","condition":{"Reason":"UnexpectedRejectRule","Message":"Found an unexpected iptables reject rule: \"-A cali-tw-eni59dda123e85 -m comment --comment \\\"cali:WZhY19FJlqRWkTTm\\\" -m comment --comment \\\"Drop if no profiles matched\\\" -j DROP\"","Severity":"Warning","MinOccurrences":0},"conditionType":"NetworkingReady"}

[csplinter]

Thanks @dcoppa for the PR. This was fixed in EKS node monitoring agent v1.6.1 (release notes).