Condition for "Modify Shell Configuration File" rule does not work

[Jmorkcho]

Describe the bug

I have a condition for not proc.pcmdline="systemd --switched-root --system --deserialize 21" yet when I redeploy my falco pods, the event will still be in the log output:

I even tried to specify the condition to even isolate the container.id like this:
and not (proc.pcmdline contains "systemd --switched-root --system --deserialize 21" and container.id="host")
But it still shows up.

How to reproduce it

- rule: Modify Shell Configuration File
  desc: Detect attempt to modify shell configuration files
  condition: >
    open_write and
    (fd.filename in (shell_config_filenames) or
     fd.name in (shell_config_files) or
     fd.directory in (shell_config_directories))
    and not proc.name in (shell_binaries)
    and not exe_running_docker_save
    and not proc.pcmdline="systemd --switched-root --system --deserialize 21"
  output: >
    a shell configuration file has been modified (user=%user.name user_loginuid=%user.loginuid command=%proc.cmdline pcmdline=%proc.pcmdline file=%fd.name container_id=%container.id image=%container.image.repository)
  priority:
    WARNING
  tags: [file, mitre_persistence]

Expected behaviour

The event should not be sent to Slack due to the condition of:
and not proc.pcmdline="systemd --switched-root --system --deserialize 21"
Screenshots

Environment

• Falco version:
  Tue Jan 21 15:11:16 2025: Falco version: 0.39.2 (x86_64)

• System info:

root@falco-cc2jt:/# falco --support | jq .system_info
Tue Jan 21 15:10:20 2025: Falco version: 0.39.2 (x86_64)
Tue Jan 21 15:10:20 2025: Falco initialized with configuration files:
Tue Jan 21 15:10:20 2025:    /etc/falco/falco.yaml | schema validation: ok
Tue Jan 21 15:10:20 2025: System info: Linux version 5.10.230-223.885.amzn2.x86_64 (mockbuild@ip-10-0-38-89) (gcc10-gcc (GCC) 10.5.0 20230707 (Red Hat 10.5.0-1), GNU ld version 2.35.2-9.amzn2.0.1) #1 SMP Tue Dec 3 14:36:00 UTC 2024
Tue Jan 21 15:10:20 2025: Loading rules from:
Tue Jan 21 15:10:20 2025:    /etc/falco/falco_rules.yaml | schema validation: ok
Tue Jan 21 15:10:20 2025:    /etc/falco/falco_rules.local.yaml | schema validation: ok
{
  "machine": "x86_64",
  "nodename": "falco-cc2jt",
  "release": "5.10.230-223.885.amzn2.x86_64",
  "sysname": "Linux",
  "version": "#1 SMP Tue Dec 3 14:36:00 UTC 2024"
}

• Cloud provider or hardware configuration:
  Running on AWS EKS 1.29.

• OS:

root@falco-cc2jt:/# cat /etc/os-release 
PRETTY_NAME="Debian GNU/Linux 12 (bookworm)"
NAME="Debian GNU/Linux"
VERSION_ID="12"
VERSION="12 (bookworm)"
VERSION_CODENAME=bookworm
ID=debian
HOME_URL="https://www.debian.org/"
SUPPORT_URL="https://www.debian.org/support"
BUG_REPORT_URL="https://bugs.debian.org/"

• Kernel:

root@falco-cc2jt:/# uname -a
Linux falco-cc2jt 5.10.230-223.885.amzn2.x86_64 #1 SMP Tue Dec 3 14:36:00 UTC 2024 x86_64 GNU/Linux

• Installation method:
  Kubernetes daemonset

[LucaGuerra]

Could you try using proc.acmdline[1] instead of proc.pcmdline? We recently fixed a bug with proc.p* fields, the fix will be released in Falco 0.40.0 . Another possibility is that the whitespace that you see is not actually a space but maybe another character (tab or anything else). To confirm this you can look at /proc/<PID>/cmdline | xxd of the process that you are trying to exclude.

[Jmorkcho]

I will try to use proc.acmdline[1] after BitBucket gets fixed, 'cause right now I cannot push.
I also extracted this output from xxd and you may be right on this one:

[root@REDACTED bin]# xxd /proc/1/cmdline
00000000: 2f75 7372 2f6c 6962 2f73 7973 7465 6d64  /usr/lib/systemd
00000010: 2f73 7973 7465 6d64 002d 2d73 7769 7463  /systemd.--switc
00000020: 6865 642d 726f 6f74 002d 2d73 7973 7465  hed-root.--syste
00000030: 6d00 2d2d 6465 7365 7269 616c 697a 6500  m.--deserialize.
00000040: 3231 00                                  21.

What would be the correct way to include this text in the rule?
As /usr/lib/systemd/systemd--switched-root--system--deserialize21 ?

[Jmorkcho]

Using proc.acmdline[1] with the following rule:

- rule: Run not allowed shell command
  desc: an attempt to run a command that is not allowed
  condition: >
    evt.type in (execve)
    and not proc.aname[2] in (supervisord, systemd)
    and proc.name in (monitoring_shell_runs_cmdlines)
    ...more conditions...
    and not (proc.acmdline[1] contains "systemd--switched-root--system--deserialize21" and container.id="host")
  output: >
    Not allowed command run (user=%user.name shell=%proc.name cmdline=%proc.cmdline pcmdline=%proc.pcmdline
    parent=%proc.pname gparent=%proc.aname[2] ggparent=%proc.aname[3]
    aname[4]=%proc.aname[4] aname[5]=%proc.aname[5] aname[6]=%proc.aname[6] aname[7]=%proc.aname[7])
  priority: ERROR
  tags: [shell]

returned again the following event:
19:00:26.653263243: Error Not allowed command run (user=root shell=ps cmdline=ps ax pcmdline=systemd --switched-root --system --deserialize 21 parent=systemd gparent=<NA> ggparent=<NA> aname[4]=<NA> aname[5]=<NA> aname[6]=<NA> aname[7]=<NA>) container_id=4d129397cb50 container_image=REDACTED container_image_tag=b25ee5392aeca215f75b578214e8aff365628f79 container_name=REDACTED k8s_ns=REDACTED k8s_pod_name=REDACTED

I will try conditioning it out only with contains deserialize then:
and not (proc.acmdline[1] contains "deserialize")

[Jmorkcho]

The condition for deserialize might have worked since there were no such events during the night but I don't want it to be so widely-scoped, so I need to find how exactly to specify the string.

Meanwhile, if you have any ideas why the following does not work, I will be really glad:

This macro is inside the Run not allowed shell command rule specified in the following way:
and not run_by_qualys

- macro: run_by_qualys
  condition: >
    (proc.pname contains "qualys-cloud-ag" or
     proc.aname[2]="qualys-cloud-ag" or
     proc.aname[3]="qualys-cloud-ag" or
     proc.aname[4]="qualys-cloud-ag" or
     proc.aname[5]="qualys-cloud-ag" or
     proc.aname[6]="qualys-cloud-ag" or
     proc.aname[7]="qualys-cloud-ag" or
     proc.aname contains "qualys" or
     proc.aname[2]="qualys-scan-uti" or
     proc.aname[3]="qualys-scan-uti" or
     proc.aname[4]="qualys-scan-uti" or
     proc.aname[5]="qualys-scan-uti" or
     proc.aname[6]="qualys-scan-uti" or
     proc.aname[7]="qualys-scan-uti" or
     proc.aname[2] contains "qualys" or
     proc.aname[3] contains "qualys" or
     proc.aname[4] contains "qualys" or
     proc.aname[5] contains "qualys" or
     proc.aname[6] contains "qualys" or
     proc.aname[7] contains "qualys" or
     proc.pname contains "qualys" or
     proc.pcmdline contains "qualys")

Yet we still receive the following event in Slack:
07:19:20.940326223: Error Not allowed command run (user=root shell=ps parent=sh cmdline=ps -ef pcmdline=REDACTED gparent=sh ggparent=timeout aname[4]=sh aname[5]=qualys-scan-uti aname[6]=qualys-cloud-ag aname[7]=systemd) container_id=host container_image=<NA> container_image_tag=<NA> container_name=host k8s_ns=<NA> k8s_pod_name=<NA>

[Jmorkcho]

No success with the following rule... as you can see on the attached screenshot, the events still show up:

root@falco-wr8jw:/# cat /etc/falco/falco_rules.yaml | grep deserialize -C 10 | tail -n15
- rule: Modify Shell Configuration File
  desc: Detect attempt to modify shell configuration files
  condition: >
    open_write and
    (fd.filename in (shell_config_filenames) or
     fd.name in (shell_config_files) or
     fd.directory in (shell_config_directories))
    and not proc.name in (shell_binaries)
    and not exe_running_docker_save
    and not (proc.acmdline[1] contains "deserialize")
  output: >
    a shell configuration file has been modified (user=%user.name user_loginuid=%user.loginuid command=%proc.cmdline pcmdline=%proc.pcmdline file=%fd.name container_id=%container.id image=%container.image.repository)
  priority:
    WARNING
  tags: [file, mitre_persistence]

Should I try with proc.acmdline without an argument or do you have another suggestion?

[LucaGuerra]

Thanks for the additional details! Looks odd, will try to reproduce and investigate.

[LucaGuerra]

With such a /proc/pid/cmdline (showing zeroes) using spaces is the right way to go. I have tried to reproduce this behavior, but no luck yet. I created a single rule file

- list: bash_config_filenames
  items: [.bashrc, .bash_profile, .bash_history, .bash_login, .bash_logout, .inputrc, .profile]

- list: bash_config_files
  items: [/etc/profile, /etc/bashrc]

# Covers both csh and tcsh
- list: csh_config_filenames
  items: [.cshrc, .login, .logout, .history, .tcshrc, .cshdirs]

- list: csh_config_files
  items: [/etc/csh.cshrc, /etc/csh.login]

- list: zsh_config_filenames
  items: [.zshenv, .zprofile, .zshrc, .zlogin, .zlogout]

- list: shell_config_filenames
  items: [bash_config_filenames, csh_config_filenames, zsh_config_filenames]

- list: shell_config_files
  items: [bash_config_files, csh_config_files]

- list: shell_config_directories
  items: [/etc/zsh]

- macro: open_write
  condition: (evt.type in (open,openat,openat2) and evt.is_open_write=true and fd.typechar='f' and fd.num>=0)

- macro: open_read
  condition: (evt.type in (open,openat,openat2) and evt.is_open_read=true and fd.typechar='f' and fd.num>=0)

- list: shell_binaries
  items: [ash, bash, csh, ksh, sh, tcsh, zsh, dash]

- rule: Modify Shell Configuration File
  desc: Detect attempt to modify shell configuration files
  condition: >
    (open_read or open_write) and
    (fd.filename in (shell_config_filenames) or
     fd.name in (shell_config_files) or
     fd.directory in (shell_config_directories))
    and not proc.name in (shell_binaries)
    and not proc.pcmdline="bash -c echo $$; sleep 1; cat /home/ubuntu/.bashrc > /dev/null"
  output: >
    a shell configuration file has been opened (user=%user.name user_loginuid=%user.loginuid command=%proc.cmdline pcmdline=%proc.pcmdline ppid=%proc.ppid file=%fd.name container_id=%container.id image=%container.image.repository)
  priority:
    WARNING
  tags: [file, mitre_persistence]

I then only loaded this file in Falco by running

sudo docker run -v $(pwd)/test_rule.yaml:/etc/falco/falco_rules.local.yaml --name falco --rm -i -t --privileged -v /var/run/docker.sock:/host/var/run/docker.sock -v /dev:/host/dev -v /proc:/host/proc:ro -v /boot:/host/boot:ro -v /lib/modules:/host/lib/modules:ro -v /usr:/host/usr:ro -v /etc:/host/etc:ro falcosecurity/falco:0.39.2

And then on another shell:

bash -c 'echo $$; sleep 1; cat /home/ubuntu/.bashrc > /dev/null'

The event is excluded as it should be. If I change the bash line (e.g. sleep 2) the event is detected. If you can try, does the smaller example I showed above also fail?

[LucaGuerra]

Another attempt would be trying, if possible, an older Falco version (0.39.0, 0.38.2) to see if we introduced anything in these versions

[Jmorkcho]

The proposed configuration/rule works as expected.
If I initiate the following line, I will not receive an event in Slack:
bash -c 'echo $$; sleep 1; cat /home/ubuntu/.bashrc > /dev/null'

However, if I change anything in this command, an event will be sent.

My next config will be as I suggested in my previous comment:
and not (proc.acmdline contains "systemd --switched-root --system --deserialize 21")

[LucaGuerra]

Definitely odd, as some rule appear to work and some don't in your environment. proc.acmdline (no arg) should work as well as you suggested, just note that you cannot use %proc.acmdline (without argument) in the output

[LucaGuerra]

This also looks very similar to #3463 . I'm setting up a cluster to stress test with a large amount of events to see if this is easier to reproduce.

[Jmorkcho]

Update on the situation:
and not proc.acmdline contains "systemd --switched-root --system --deserialize 21" continues to not work;

I can suppress the event with the following line:
and not (proc.pcmdline startswith "systemd" and container.id="host"), which is still not ideal, but I will use it until there is another solution.

One more thing to mention - I've tried to include the %proc.acmdline[1] in the output with an argument, but it does not show at all. It's as if the field itself does not exist.

  output: >
    Not allowed command run (user=%user.name shell=%proc.name cmdline=%proc.cmdline pcmdline=%proc.pcmdline
    parent=%proc.pname gparent=%proc.aname[2] ggparent=%proc.aname[3]
    aname[4]=%proc.aname[4] aname[5]=%proc.aname[5] aname[6]=%proc.aname[6] aname[7]=%proc.aname[7]
    proc.acmdline[1]=%proc.acmdline[1])

[leogr]

Have you tried just printing %proc.acmdline (with no argument)?
Sorry, you can't do that

When used without any arguments, proc.acmdline is applicable only in filters and matches any of the process ancestors

[LucaGuerra]

Thank you for checking this further!

It's as if the field itself does not exist.

Does it always happen (i.e. %proc.acmdline[1] seems to be empty for every event) or only when you observe the problem? As I understand it doesn't show at all means that you read

Not allowed command run [... snip ...] aname[7]=systemd proc.acmdline[1]=)

It shows like an empty string and nothing else (not even <NA> or similar)
I went as far as trying to replicate the exact command line for my filter by having a custom binary called systemd and passing the same arguments but I couldn't reproduce the issue yet.

[Jmorkcho]

I've only configured it for this rule, but I will add output for proc.acmdline today for our 2 local rules as well and check whether it happens always.

Update on your question:

It shows like an empty string and nothing else (not even or similar)

No, it doesn't even show the "proc.acmdline[1]" before the equal sign. Also, even with this line:
and not (proc.pcmdline startswith "systemd" and container.id="host")
the process event cannot be suppressed; check screenshot below:

[Jmorkcho]

Quick update, proc.acmdline[1] exists in the output now(at least on one of my local rules), but is value is "N/A".

[LucaGuerra]

I've only configured it for this rule, but I will add output for proc.acmdline today for our 2 local rules as well and check whether it happens always.

Update on your question:

It shows like an empty string and nothing else (not even or similar)

No, it doesn't even show the "proc.acmdline[1]" before the equal sign. Also, even with this line: and not (proc.pcmdline startswith "systemd" and container.id="host") the process event cannot be suppressed; check screenshot below:

In this screenshot here the output format configured in the rule does not match the output in the alert, e.g. the configured output says image but the alert shows container_image. Could it be that there are multiple rules active with similar output/descriptions?

[Jmorkcho]

No, the rule is only one. Verification below:

root@falco-ztfbg:/# cat /etc/falco/falco.yaml | grep rules_files -A 6
rules_files:
  - /etc/falco/falco_rules.yaml
  - /etc/falco/falco_rules.local.yaml
  - /etc/falco/k8s_audit_rules.yaml
  - /etc/falco/rules-php-fpm.yaml
  - /etc/falco/rules.d
stdout_output:

root@falco-ztfbg:/# ls /etc/falco/
falco.yaml  falco_rules.local.yaml  falco_rules.yaml

root@falco-ztfbg:/# cat /etc/falco/falco_rules.yaml -n | grep "a shell configuration file has been modified" -A 2
  1790      a shell configuration file has been modified (user=%user.name user_loginuid=%user.loginuid command=%proc.cmdline pcmdline=%proc.pcmdline
  1791      file=%fd.name container_id=%container.id image=%container.image.repository proc.acmdline[1]=%proc.acmdline[1])
  1792    priority:

root@falco-ztfbg:/# cat /etc/falco/falco_rules.local.yaml -n | grep "a shell configuration file has been modified" -A 2
root@falco-ztfbg:/# 

The output for
container_id, container_image, container_image_tag, container_name, k8s_ns and k8s_pod_name seems to be after every rule's output closing bracket as can be noticed on my latest screenshot.

Also, on the screenshot that you've replied to, there is image=N/A inside of the brackets, the container_image you're referring to is outside of the brackets.