Kafka consumes 100% disk space causing the whole setup to crash

[MeikoSch]

Environment

self-hosted (https://develop.sentry.dev/self-hosted/)

What are you trying to accomplish?

I'm tyring to run sentry on my local environment. It all works fine for a while but after a few hours the 220Gig harddisk got full.

How are you getting stuck?

I nailed it down to the kafka storrage of docker compose. After docker volume rm sentry-kafka and docker volume create sentry-kafka and docker compose up again the system was functional again but the kafka storrage was again increasing constantly untill the disc was full again.
I did a little deep dive and saw that the top 3 directories where constantly "growing"

[appuser@ed0fe2c012eb data]$ du -h /var/lib/kafka/data | sort -hr | head -20 15G /var/lib/kafka/data 5.3G /var/lib/kafka/data/snuba-spans-0 4.4G /var/lib/kafka/data/transactions-0 4.2G /var/lib/kafka/data/ingest-transactions-0 259M /var/lib/kafka/data/snuba-generic-metrics-0 193M /var/lib/kafka/data/ingest-performance-metrics-0 1.8M /var/lib/kafka/data/outcomes-billing-0 1.5M /var/lib/kafka/data/events-0 1.2M /var/lib/kafka/data/ingest-events-0 1.1M /var/lib/kafka/data/generic-events-0 436K /var/lib/kafka/data/__consumer_offsets-0 124K /var/lib/kafka/data/snuba-transactions-commit-log-0 116K /var/lib/kafka/data/__cluster_metadata-0 84K /var/lib/kafka/data/ingest-occurrences-0 68K /var/lib/kafka/data/snuba-metrics-0 52K /var/lib/kafka/data/snuba-generic-metrics-distributions-commit-log-0 48K /var/lib/kafka/data/snuba-generic-metrics-sets-commit-log-0 48K /var/lib/kafka/data/snuba-generic-metrics-gauges-commit-log-0 48K /var/lib/kafka/data/snuba-generic-metrics-counters-commit-log-0 48K /var/lib/kafka/data/ingest-metrics-0

im not sure where to go from here? I fist installed 25.3.0 and then tired upgrading to lated master (25.5.0.dev0) but the problem persists.

Where in the product are you?

Other

Link

No response

DSN

sentry.mls2.de

Version

25.3.0 and 25.5.0.dev0

[getsantry]

Assigning to @getsentry/support for routing ⏲️

[aldy505]

A few questions:

1. What is your machine (or server) specification?
2. How big is your traffic to Sentry?
3. If you do docker compose ps, do you see any containers that keeps restarting or have upnormal uptimes?

[MeikoSch]

The machine is a managed Server:
16 vCPUs, 32 GiB of memory and 360 GB of storage
I cant realy tell how bit the treffic is but i guess it is not much, since our application has a small audience.
When running docker compose ps no container was restarting, as long as the disk was not full. All containers where up for the same time since i started docker compose.

Note: Independend of the traffic the Kafka storage was constantly growing. I observed it also during a weekend where usually none of our customers is using our app. I did shot down the machine now since we cannot use it as it is, but i can spin it up quickly in case.