Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Enhanced session recording may skip events under load #48774

Open
Joerger opened this issue Nov 11, 2024 · 0 comments
Open

Enhanced session recording may skip events under load #48774

Joerger opened this issue Nov 11, 2024 · 0 comments
Labels
bpf Used to bugs with bpf and enhanced session recording. bug

Comments

@Joerger
Copy link
Contributor

Joerger commented Nov 11, 2024

Expected behavior:

When enhanced session recording is enabled, all session commands and other BPF events should be recorded.

Current behavior:

The BPF service can be overloaded and miss events. It was surprisingly easy to overload on a virtual Centos7 box with Kernel 5.15.171 manually installed, 16gb ram and 16 cores.

Running while sleep .5; do ls; done, I would get the following error every several seconds, and it would affect all events for a few seconds, like the service is temporarily overloaded and down. These missing events would occur in the session, but would not be recorded in the audit log.

> tsh ssh centos7
(centos7) > while sleep .5; do ls; done

### Teleport logs
2024-11-11T23:48:37Z INFO  emitting audit event event_type:session.command fields:map[argv:[.5] cgroup_id:496 code:T4000I ei:0 event:session.command login:vagrant namespace:default path:/bin/sleep pid:9599 ppid:9450 program:sleep return_code:0 server_hostname:centos7 server_id:ee77d106-9076-4788-b146-f1b37e45ebdb server_version:17.0.0-alpha.5 sid:23d58260-dbe2-4301-8f4f-d63088733fb4 time:2024-11-11T23:48:37.26Z trace.component:audit uid:05c42a70-3cbc-4989-8698-9e9b58d6a96c user:dev] events/emitter.go:287
2024-11-11T23:48:37Z INFO  emitting audit event event_type:session.command fields:map[argv:[--color=auto] cgroup_id:496 code:T4000I ei:0 event:session.command login:vagrant namespace:default path:/bin/ls pid:9600 ppid:9450 program:ls return_code:0 server_hostname:centos7 server_id:ee77d106-9076-4788-b146-f1b37e45ebdb server_version:17.0.0-alpha.5 sid:23d58260-dbe2-4301-8f4f-d63088733fb4 time:2024-11-11T23:48:37.757Z trace.component:audit uid:c20e5b93-078b-4a8d-9824-c6d27731aaad user:dev] events/emitter.go:287
2024-11-11T23:48:37Z DEBU [BPF]       Got event with missing args: skipping. bpf/bpf.go:390
2024-11-11T23:48:38Z DEBU [BPF]       Got event with missing args: skipping. bpf/bpf.go:390
2024-11-11T23:48:38Z DEBU [BPF]       Got event with missing args: skipping. bpf/bpf.go:390
2024-11-11T23:48:38Z DEBU [BPF]       Got event with missing args: skipping. bpf/bpf.go:390
2024-11-11T23:48:38Z DEBU [BPF]       Got event with missing args: skipping. bpf/bpf.go:390
2024-11-11T23:48:39Z DEBU [BPF]       Got event with missing args: skipping. bpf/bpf.go:390
2024-11-11T23:48:39Z DEBU [BPF]       Got event with missing args: skipping. bpf/bpf.go:390
2024-11-11T23:48:39Z DEBU [BPF]       Got event with missing args: skipping. bpf/bpf.go:390
2024-11-11T23:48:39Z DEBU [BPF]       Got event with missing args: skipping. bpf/bpf.go:390
2024-11-11T23:48:40Z DEBU [BPF]       Got event with missing args: skipping. bpf/bpf.go:390
2024-11-11T23:48:40Z DEBU [BPF]       Got event with missing args: skipping. bpf/bpf.go:390
2024-11-11T23:48:40Z DEBU [BPF]       Got event with missing args: skipping. bpf/bpf.go:390
2024-11-11T23:48:40Z DEBU [BPF]       Got event with missing args: skipping. bpf/bpf.go:390
2024-11-11T23:48:41Z DEBU [BPF]       Got event with missing args: skipping. bpf/bpf.go:390
2024-11-11T23:48:41Z DEBU [BPF]       Got event with missing args: skipping. bpf/bpf.go:390
2024-11-11T23:48:41Z DEBU [BPF]       Got event with missing args: skipping. bpf/bpf.go:390
2024-11-11T23:48:41Z DEBU [BPF]       Got event with missing args: skipping. bpf/bpf.go:390
2024-11-11T23:48:42Z DEBU [BPF]       Got event with missing args: skipping. bpf/bpf.go:390
2024-11-11T23:48:42Z DEBU [BPF]       Got event with missing args: skipping. bpf/bpf.go:390
2024-11-11T23:48:42Z DEBU [BPF]       Got event with missing args: skipping. bpf/bpf.go:390
2024-11-11T23:48:42Z DEBU [BPF]       Got event with missing args: skipping. bpf/bpf.go:390
2024-11-11T23:48:43Z DEBU [BPF]       Got event with missing args: skipping. bpf/bpf.go:390
2024-11-11T23:48:43Z DEBU [BPF]       Got event with missing args: skipping. bpf/bpf.go:390
2024-11-11T23:48:43Z DEBU [BPF]       Got event with missing args: skipping. bpf/bpf.go:390
2024-11-11T23:48:43Z DEBU [BPF]       Got event with missing args: skipping. bpf/bpf.go:390
2024-11-11T23:48:44Z DEBU [BPF]       Got event with missing args: skipping. bpf/bpf.go:390
2024-11-11T23:48:44Z DEBU [BPF]       Got event with missing args: skipping. bpf/bpf.go:390
2024-11-11T23:48:44Z DEBU [BPF]       Got event with missing args: skipping. bpf/bpf.go:390
2024-11-11T23:48:44Z INFO  emitting audit event event_type:session.command fields:map[argv:[.5] cgroup_id:496 code:T4000I ei:0 event:session.command login:vagrant namespace:default path:/bin/sleep pid:9629 ppid:9450 program:sleep return_code:0 server_hostname:centos7 server_id:ee77d106-9076-4788-b146-f1b37e45ebdb server_version:17.0.0-alpha.5 sid:23d58260-dbe2-4301-8f4f-d63088733fb4 time:2024-11-11T23:48:44.84Z trace.component:audit uid:5245bd68-662a-4712-ae2c-7cb8832deea6 user:dev] events/emitter.go:287

Bug details:

  • Teleport version: v17.0.0-alpha.2, also tested back to v15.0.0
@Joerger Joerger added the bug label Nov 11, 2024
@zmb3 zmb3 added the bpf Used to bugs with bpf and enhanced session recording. label Nov 12, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bpf Used to bugs with bpf and enhanced session recording. bug
Projects
None yet
Development

No branches or pull requests

2 participants