Whitelisted rule still being triggered

[matias-at-skribble]

Initially posted on StackOverflow. Downgrading Falco fixed the issue so it seems like a recently introduced bug.

Describe the bug
Added custom conditions to the user_known_contact_k8s_api_server_activities macro to prevent the Contact K8S API Server From Container rule from being triggered. This works on 0.38.1 but not on 0.39.2

How to reproduce it

# Only alert when the metadata fields are populated
- macro: k8s_fields
  condition: >
    (
      k8smeta.ns.name exists and 
      k8smeta.pod.name exists
    )
# Don't alert for specific namespaces
- macro: privileged_namespace
  condition: >
    (
      (k8smeta.ns.name startswith calico)
      or (k8smeta.ns.name startswith cattle)
      or (k8smeta.ns.name startswith syn)
    )

- macro: user_known_contact_k8s_api_server_activities
  condition: >
    (
      not (k8s_fields)
      or (privileged_namespace)
      or (...other macros...)
    )

Expected behaviour
Rule should not trigger for the syn-infra-monitoring namespace, but it does

{
   "hostname":"worker-a7f3",
   "output":"10:26:00.800107028: Notice Unexpected connection to K8s API Server (...))",
   "output_fields":{
      (...)
      "k8smeta.deployment.name":null,
      "k8smeta.ns.name":"syn-infra-monitoring",
      "k8smeta.pod.name":"nodeexporter-infra-l9csj",
      "proc.cmdline":"kube-rbac-proxy --secure-listen-address=[172.18.200.237]:9100 (...)",
      "proc.exepath":"/usr/local/bin/kube-rbac-proxy",
      (...)
   },
   "priority":"Notice",
   "rule":"Contact K8S API Server From Container",
   "source":"syscall",
   "tags":[
      "T1565",
      "container",
      "k8s",
      "maturity_stable",
      "mitre_discovery",
      "network"
   ],
   "time":"2025-01-21T10:26:00.800107028Z"
}

Environment

• Falco version: 0.39.2
• Image: docker.io/falcosecurity/falco-no-driver:0.39.2
• Installation method: Helm chart (falco-4.17.2)

[leogr]

Hey @matias-at-skribble

Thank you for reporting this. I have a few questions that can help us troubleshoot the problem.

• Can you try to use k8s.ns.name and k8s.pod.name instead?
• Can you share the k8smeta plugin version used with 0.38.1 and the one used with 0.39.2?

Also, it would be very significant for us if you could try 0.40-rc. But I'd understand if you can't.

[matias-at-skribble]

Can you try to use k8s.ns.name and k8s.pod.name instead?

That worked. I had changed that to k8smeta after the 0.37 release, as the k8s fields were not being populated, but it seems to work now

Can you share the k8smeta plugin version used with 0.38.1 and the one used with 0.39.2?

Chart 4.6.1 includes Falco 0.38.1 with k8smeta:0.1.0
Chart 4.17.2 includes Falco 0.39.2 with k8smeta:0.2.1
I also tried Chart 4.7.0, with Falco 0.38.1 and k8smeta:0.2.0, which seems to work fine

Also, it would be very significant for us if you could try 0.40-rc

I tried, but that tag is not available on the falco-no-driver image used by the Chart.
I tried changing the image to falco, but the pods fail to start, and I don't really know why

[leogr]

Can you try to use k8s.ns.name and k8s.pod.name instead?

That worked. I had changed that to k8smeta after the 0.37 release, as the k8s fields were not being populated, but it seems to work now

Can you share the k8smeta plugin version used with 0.38.1 and the one used with 0.39.2?

Chart 4.6.1 includes Falco 0.38.1 with k8smeta:0.1.0 Chart 4.17.2 includes Falco 0.39.2 with k8smeta:0.2.1 I also tried Chart 4.7.0, with Falco 0.38.1 and k8smeta:0.2.0, which seems to work fine

This indicates potential problems in plugins' extraction mechanisms. We will investigate soon.

cc for visibility @alacuku @jasondellaluce @LucaGuerra

Also, it would be very significant for us if you could try 0.40-rc

I tried, but that tag is not available on the falco-no-driver image used by the Chart. I tried changing the image to falco, but the pods fail to start, and I don't really know why

No problem.

Thank you so much for your support. I will keep you posted once we find anything relevant.

[matias-at-skribble]

Update: I tested several charts, my rules work until chart 4.13.0, and break with chart 4.14.0
The changelog shows one change for this version:

Bump k8smeta plugin to 0.2.1

I then tried the latest 4.17.2 chart, with k8smeta:0.2.0 and everything works fine, so it seems that is the issue
I will keep the k8smeta version pinned for now, until a fix is found.

Thank you for your help!

[leogr]

Update: I tested several charts, my rules work until chart 4.13.0, and break with chart 4.14.0 The changelog shows one change for this version:

Bump k8smeta plugin to 0.2.1

I then tried the latest 4.17.2 chart, with k8smeta:0.2.0 and everything works fine, so it seems that is the issue I will keep the k8smeta version pinned for now, until a fix is found.

So, do you confirm there is no issue with Falco 0.39.2 and k8smeta:0.2.0 ?

[matias-at-skribble]

Yes, the issue seems to start with k8smeta:0.2.1

[matias-at-skribble]

It seems like there is still one single case in which our whitelist is not working. This generates an alert every 5-10min, instead of ~15sec as before, so most events are getting whitelisted.

[leogr]

It seems like there is still one single case in which our whitelist is not working. This generates an alert every 5-10min, instead of ~15sec as before, so most events are getting whitelisted.

Okay. This suggests that there can be a race condition, the frequency of which varies depending on the setup.
May you try a long run with 0.38.2? I'd like to know if, with older Falco, the issue disappears definitively.

[leogr]

Also, we suspect #3460 can share the same root cause.
@matias-at-skribble, have you noticed the same issue with the other rules, too?

[leogr]

Furthermore, if you still have a chance to test this with 0.40.0-rc, here are the instructions for deploying with Helm:

# download WIP chart for 0.40 https://github.com/falcosecurity/charts/pull/818 
git clone --single-branch --branch klc/falco-0.40.0 https://github.com/alacuku/charts.git

# change directory to Falco chart.
cd charts/charts/falco

# build dependencies.
helm dependency build

helm install falco $CHART \
    --namespace falco \
    --replace \
    --create-namespace \
    --set tty=true \
    --values=custom-values

[matias-at-skribble]

have you noticed the same issue with the other rules, too?

The other whitelisted rules don't get triggered so often, so I cannot be sure, but I haven't seen any whitelisted alert other than the K8s API connection

if you still have a chance to test this with 0.40.0-rc, here are the instructions for deploying with Helm

I have deployed it now, the behaviour seems to be the same as with k8smeta:0.2.0: I get alerts which should be whitelisted, but not as many as I did before.

This suggests that there can be a race condition, the frequency of which varies depending on the setup.

Indeed. To get a sense of the difference between versions, here is a chart showing the alerts we get for each case:

• Red: Falco 0.39.2 + k8smeta:0.2.1 -> Lots of false positives
• Blue: Falco 0.38.1 + k8smeta:0.1.0 -> Normal activity
• Yellow: Falco 0.39.2 + k8smeta:0.2.0 / Falco 0.40.0 + k8smeta:0.2.1 -> Some false positives

May you try a long run with 0.38.2?

I'll deploy that now, will leave it running for the weekend and post an update on Monday

[matias-at-skribble]

The issue is gone with falco 0.38.2 + k8smeta 0.2.0 (chart 4.8.0)
This has been running since Friday and we have received no alerts since then.