snuba-errors-consumer dying on requests, errors are not shown

[max-wittig]

Self-Hosted Version

25.02.0

CPU Architecture

x86_64

Docker Version

25.0.8

Docker Compose Version

2.34.0

Machine Specification

• My system meets the minimum system requirements of Sentry

Steps to Reproduce

1. Try to upgrade to Sentry 25.03.0
2. Notice that no events are received
3. Check logs for snuba-errors-consumer
4. Notice the crash and constant restart, when events are sent

Expected Result

Everything works as expected and no crashes are seen

Actual Result

snuba-errors-consumer-1  | 2025-04-30 08:38:40,541 Initializing Snuba...
snuba-errors-consumer-1  | 2025-04-30 08:38:46,786 Snuba initialization took 6.2461343029281124s
snuba-errors-consumer-1  | 2025-04-30 08:38:47,635 Initializing Snuba...
snuba-errors-consumer-1  | 2025-04-30 08:38:53,318 Snuba initialization took 5.684111953014508s
snuba-errors-consumer-1  | {"timestamp":"2025-04-30T08:38:53.334648Z","level":"INFO","fields":{"message":"Storage: errors, ClickHouse Table Name: errors_local, Message Processor: MessageProcessorConfig { python_class_name: \"ErrorsProcessor\", python_module: \"snuba.datasets.processors.errors_processor\" }, ClickHouse host: clickhouse, ClickHouse port: 9000, ClickHouse HTTP port: 8123, ClickHouse database: default"},"target":"rust_snuba::consumer"}
snuba-errors-consumer-1  | {"timestamp":"2025-04-30T08:38:53.334784Z","level":"INFO","fields":{"message":"Starting consumer for \"errors\"","storage":"errors"},"target":"rust_snuba::consumer"}
snuba-errors-consumer-1  | {"timestamp":"2025-04-30T08:38:55.372293Z","level":"INFO","fields":{"message":"Kafka consumer member id: \"rdkafka-8cf661db-1d78-4574-aac8-62349fbfa2d7\""},"target":"sentry_arroyo::backends::kafka"}
snuba-errors-consumer-1  | {"timestamp":"2025-04-30T08:38:55.373050Z","level":"INFO","fields":{"message":"New partitions assigned: {Partition { topic: Topic(\"events\"), index: 0 }: 7}"},"target":"sentry_arroyo::processing"}
snuba-errors-consumer-1  | {"timestamp":"2025-04-30T08:39:15.436072Z","level":"INFO","fields":{"message":"Caught error, terminating strategy"},"target":"sentry_arroyo::processing"}
snuba-errors-consumer-1  | {"timestamp":"2025-04-30T08:39:15.436255Z","level":"INFO","fields":{"message":"Partitions to revoke: [Partition { topic: Topic(\"events\"), index: 0 }]"},"target":"sentry_arroyo::processing"}
snuba-errors-consumer-1  | {"timestamp":"2025-04-30T08:39:15.436300Z","level":"INFO","fields":{"message":"Joining strategy"},"target":"sentry_arroyo::processing"}
snuba-errors-consumer-1  | {"timestamp":"2025-04-30T08:39:15.436309Z","level":"WARN","fields":{"message":"Timeout Some(0ns) reached while waiting for tasks to finish"},"target":"sentry_arroyo::processing::strategies::reduce"}
snuba-errors-consumer-1  | {"timestamp":"2025-04-30T08:39:15.527813Z","level":"INFO","fields":{"message":"Partition revocation complete."},"target":"sentry_arroyo::processing"}
snuba-errors-consumer-1  | {"timestamp":"2025-04-30T08:39:15.660616Z","level":"ERROR","fields":{"message":"Strategy(bad response while inserting rows, status: 500 Internal Server Error, response body: Ok(\"Code: 36. DB::Exception: The pattern argument is not a valid re2 pattern: missing ): (=|\\\\): while executing 'FUNCTION replaceRegexpAll(k :: 0, '(=|\\\\\\\\)' :: 2, '\\\\\\\\\\\\\\\\1' :: 3) -> replaceRegexpAll(k, '(=|\\\\\\\\)', '\\\\\\\\\\\\\\\\1') String : 5'. (BAD_ARGUMENTS) (version 23.8.11.29.altinitystable (altinity build))\\n\")\n\nStack backtrace:\n   0: anyhow::error::<impl anyhow::Error>::msg\n             at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/anyhow-1.0.82/src/backtrace.rs:27:14\n   1: rust_snuba::strategies::clickhouse::batch::BatchFactory::new_batch::{{closure}}\n             at ./rust_snuba/src/strategies/clickhouse/batch.rs:144:21\n   2: tokio::runtime::task::core::Core<T,S>::poll::{{closure}}\n             at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.37.0/src/runtime/task/core.rs:328:17\n   3: tokio::loom::std::unsafe_cell::UnsafeCell<T>::with_mut\n             at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.37.0/src/loom/std/unsafe_cell.rs:16:9\n   4: tokio::runtime::task::core::Core<T,S>::poll\n             at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.37.0/src/runtime/task/core.rs:317:30\n   5: tokio::runtime::task::harness::poll_future::{{closure}}\n             at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.37.0/src/runtime/task/harness.rs:485:19\n   6: <core::panic::unwind_safe::AssertUnwindSafe<F> as core::ops::function::FnOnce<()>>::call_once\n             at /rustc/90b35a6239c3d8bdabc530a6a0816f7ff89a0aaf/library/core/src/panic/unwind_safe.rs:272:9\n   7: std::panicking::try::do_call\n             at /rustc/90b35a6239c3d8bdabc530a6a0816f7ff89a0aaf/library/std/src/panicking.rs:557:40\n   8: std::panicking::try\n             at /rustc/90b35a6239c3d8bdabc530a6a0816f7ff89a0aaf/library/std/src/panicking.rs:520:19\n   9: std::panic::catch_unwind\n             at /rustc/90b35a6239c3d8bdabc530a6a0816f7ff89a0aaf/library/std/src/panic.rs:358:14\n  10: tokio::runtime::task::harness::poll_future\n             at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.37.0/src/runtime/task/harness.rs:473:18\n  11: tokio::runtime::task::harness::Harness<T,S>::poll_inner\n             at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.37.0/src/runtime/task/harness.rs:208:27\n  12: tokio::runtime::task::harness::Harness<T,S>::poll\n             at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.37.0/src/runtime/task/harness.rs:153:15\n  13: tokio::runtime::task::raw::poll\n             at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.37.0/src/runtime/task/raw.rs:271:5\n  14: tokio::runtime::task::raw::RawTask::poll\n             at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.37.0/src/runtime/task/raw.rs:201:18\n  15: tokio::runtime::task::LocalNotified<S>::run\n             at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.37.0/src/runtime/task/mod.rs:427:9\n  16: tokio::runtime::scheduler::multi_thread::worker::Context::run_task::{{closure}}\n             at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.37.0/src/runtime/scheduler/multi_thread/worker.rs:576:18\n  17: tokio::runtime::coop::with_budget\n             at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.37.0/src/runtime/coop.rs:107:5\n  18: tokio::runtime::coop::budget\n             at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.37.0/src/runtime/coop.rs:73:5\n  19: tokio::runtime::scheduler::multi_thread::worker::Context::run_task\n             at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.37.0/src/runtime/scheduler/multi_thread/worker.rs:575:9\n  20: tokio::runtime::scheduler::multi_thread::worker::Context::run\n  21: tokio::runtime::scheduler::multi_thread::worker::run::{{closure}}::{{closure}}\n             at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.37.0/src/runtime/scheduler/multi_thread/worker.rs:491:21\n  22: tokio::runtime::context::scoped::Scoped<T>::set\n             at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.37.0/src/runtime/context/scoped.rs:40:9\n  23: tokio::runtime::context::set_scheduler::{{closure}}\n             at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.37.0/src/runtime/context.rs:176:26\n  24: std::thread::local::LocalKey<T>::try_with\n             at /rustc/90b35a6239c3d8bdabc530a6a0816f7ff89a0aaf/library/std/src/thread/local.rs:283:12\n  25: std::thread::local::LocalKey<T>::with\n             at /rustc/90b35a6239c3d8bdabc530a6a0816f7ff89a0aaf/library/std/src/thread/local.rs:260:9\n  26: tokio::runtime::context::set_scheduler\n             at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.37.0/src/runtime/context.rs:176:17\n  27: tokio::runtime::scheduler::multi_thread::worker::run::{{closure}}\n             at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.37.0/src/runtime/scheduler/multi_thread/worker.rs:486:9\n  28: tokio::runtime::context::runtime::enter_runtime\n             at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.37.0/src/runtime/context/runtime.rs:65:16\n  29: tokio::runtime::scheduler::multi_thread::worker::run\n             at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.37.0/src/runtime/scheduler/multi_thread/worker.rs:478:5\n  30: tokio::runtime::scheduler::multi_thread::worker::Launch::launch::{{closure}}\n             at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.37.0/src/runtime/scheduler/multi_thread/worker.rs:447:45\n  31: <tokio::runtime::blocking::task::BlockingTask<T> as core::future::future::Future>::poll\n             at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.37.0/src/runtime/blocking/task.rs:42:21\n  32: tokio::runtime::task::core::Core<T,S>::poll::{{closure}}\n             at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.37.0/src/runtime/task/core.rs:328:17\n  33: tokio::loom::std::unsafe_cell::UnsafeCell<T>::with_mut\n             at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.37.0/src/loom/std/unsafe_cell.rs:16:9\n  34: tokio::runtime::task::core::Core<T,S>::poll\n             at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.37.0/src/runtime/task/core.rs:317:30\n  35: tokio::runtime::task::harness::poll_future::{{closure}}\n             at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.37.0/src/runtime/task/harness.rs:485:19\n  36: <core::panic::unwind_safe::AssertUnwindSafe<F> as core::ops::function::FnOnce<()>>::call_once\n             at /rustc/90b35a6239c3d8bdabc530a6a0816f7ff89a0aaf/library/core/src/panic/unwind_safe.rs:272:9\n  37: std::panicking::try::do_call\n             at /rustc/90b35a6239c3d8bdabc530a6a0816f7ff89a0aaf/library/std/src/panicking.rs:557:40\n  38: std::panicking::try\n             at /rustc/90b35a6239c3d8bdabc530a6a0816f7ff89a0aaf/library/std/src/panicking.rs:520:19\n  39: std::panic::catch_unwind\n             at /rustc/90b35a6239c3d8bdabc530a6a0816f7ff89a0aaf/library/std/src/panic.rs:358:14\n  40: tokio::runtime::task::harness::poll_future\n             at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.37.0/src/runtime/task/harness.rs:473:18\n  41: tokio::runtime::task::harness::Harness<T,S>::poll_inner\n             at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.37.0/src/runtime/task/harness.rs:208:27\n  42: tokio::runtime::task::harness::Harness<T,S>::poll\n             at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.37.0/src/runtime/task/harness.rs:153:15\n  43: tokio::runtime::task::raw::RawTask::poll\n             at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.37.0/src/runtime/task/raw.rs:201:18\n  44: tokio::runtime::task::UnownedTask<S>::run\n             at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.37.0/src/runtime/task/mod.rs:464:9\n  45: tokio::runtime::blocking::pool::Task::run\n             at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.37.0/src/runtime/blocking/pool.rs:159:9\n  46: tokio::runtime::blocking::pool::Inner::run\n             at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.37.0/src/runtime/blocking/pool.rs:513:17\n  47: tokio::runtime::blocking::pool::Spawner::spawn_thread::{{closure}}\n             at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.37.0/src/runtime/blocking/pool.rs:471:13\n  48: std::sys::backtrace::__rust_begin_short_backtrace\n             at /rustc/90b35a6239c3d8bdabc530a6a0816f7ff89a0aaf/library/std/src/sys/backtrace.rs:154:18\n  49: std::thread::Builder::spawn_unchecked_::{{closure}}::{{closure}}\n             at /rustc/90b35a6239c3d8bdabc530a6a0816f7ff89a0aaf/library/std/src/thread/mod.rs:538:17\n  50: <core::panic::unwind_safe::AssertUnwindSafe<F> as core::ops::function::FnOnce<()>>::call_once\n             at /rustc/90b35a6239c3d8bdabc530a6a0816f7ff89a0aaf/library/core/src/panic/unwind_safe.rs:272:9\n  51: std::panicking::try::do_call\n             at /rustc/90b35a6239c3d8bdabc530a6a0816f7ff89a0aaf/library/std/src/panicking.rs:557:40\n  52: std::panicking::try\n             at /rustc/90b35a6239c3d8bdabc530a6a0816f7ff89a0aaf/library/std/src/panicking.rs:520:19\n  53: std::panic::catch_unwind\n             at /rustc/90b35a6239c3d8bdabc530a6a0816f7ff89a0aaf/library/std/src/panic.rs:358:14\n  54: std::thread::Builder::spawn_unchecked_::{{closure}}\n             at /rustc/90b35a6239c3d8bdabc530a6a0816f7ff89a0aaf/library/std/src/thread/mod.rs:537:30\n  55: core::ops::function::FnOnce::call_once{{vtable.shim}}\n             at /rustc/90b35a6239c3d8bdabc530a6a0816f7ff89a0aaf/library/core/src/ops/function.rs:250:5\n  56: <alloc::boxed::Box<F,A> as core::ops::function::FnOnce<Args>>::call_once\n             at /rustc/90b35a6239c3d8bdabc530a6a0816f7ff89a0aaf/library/alloc/src/boxed.rs:2454:9\n  57: <alloc::boxed::Box<F,A> as core::ops::function::FnOnce<Args>>::call_once\n             at /rustc/90b35a6239c3d8bdabc530a6a0816f7ff89a0aaf/library/alloc/src/boxed.rs:2454:9\n  58: std::sys::pal::unix::thread::Thread::new::thread_start\n             at /rustc/90b35a6239c3d8bdabc530a6a0816f7ff89a0aaf/library/std/src/sys/pal/unix/thread.rs:105:17\n  59: <unknown>\n  60: __clone)"},"target":"rust_snuba::consumer"}
snuba-errors-consumer-1  | 2025-04-30 08:39:18,014 Initializing Snuba...
snuba-errors-consumer-1  | 2025-04-30 08:39:23,815 Snuba initialization took 5.801053549046628s
snuba-errors-consumer-1  | 2025-04-30 08:39:24,658 Initializing Snuba...
snuba-errors-consumer-1  | 2025-04-30 08:39:30,457 Snuba initialization took 5.80023129703477s
snuba-errors-consumer-1  | {"timestamp":"2025-04-30T08:39:30.467322Z","level":"INFO","fields":{"message":"Storage: errors, ClickHouse Table Name: errors_local, Message Processor: MessageProcessorConfig { python_class_name: \"ErrorsProcessor\", python_module: \"snuba.datasets.processors.errors_processor\" }, ClickHouse host: clickhouse, ClickHouse port: 9000, ClickHouse HTTP port: 8123, ClickHouse database: default"},"target":"rust_snuba::consumer"}
snuba-errors-consumer-1  | {"timestamp":"2025-04-30T08:39:30.467423Z","level":"INFO","fields":{"message":"Starting consumer for \"errors\"","storage":"errors"},"target":"rust_snuba::consumer"}
snuba-errors-consumer-1  | {"timestamp":"2025-04-30T08:39:33.071293Z","level":"INFO","fields":{"message":"Kafka consumer member id: \"rdkafka-efbe457e-f61c-42b1-9da5-4bf95d26be01\""},"target":"sentry_arroyo::backends::kafka"}
snuba-errors-consumer-1  | {"timestamp":"2025-04-30T08:39:33.072401Z","level":"INFO","fields":{"message":"New partitions assigned: {Partition { topic: Topic(\"events\"), index: 0 }: 8}"},"target":"sentry_arroyo::processing"}

Seems like that all snuba migrations were successful:

snuba@edeb2336c3d3:/usr/src/snuba$ snuba migrations list
2025-04-30 08:49:31,129 Initializing Snuba...
2025-04-30 08:49:36,884 Snuba initialization took 5.756623110966757s
system (readiness_state: complete)
[X]  0001_migrations

events (readiness_state: complete)
[X]  0001_events_initial
[X]  0002_events_onpremise_compatibility
[X]  0003_errors
[X]  0004_errors_onpremise_compatibility
[X]  0005_events_tags_hash_map
[X]  0006_errors_tags_hash_map
[X]  0007_groupedmessages
[X]  0008_groupassignees
[X]  0009_errors_add_http_fields
[X]  0010_groupedmessages_onpremise_compatibility
[X]  0011_rebuild_errors
[X]  0012_errors_make_level_nullable
[X]  0013_errors_add_hierarchical_hashes
[X]  0014_backfill_errors
[X]  0015_truncate_events
[X]  0016_drop_legacy_events
[X]  0017_errors_add_indexes
[X]  0018_errors_ro_add_tags_hash_map
[X]  0019_add_replay_id_column
[X]  0020_add_main_thread_column
[X]  0021_add_replay_id_errors_ro
[X]  0022_add_main_thread_column_errors_ro
[X]  0023_add_trace_sampled_num_processing_errors_columns
[X]  0024_add_trace_sampled_num_processing_errors_columns_ro
[X]  0025_add_flags_column

transactions (readiness_state: complete)
[X]  0001_transactions
[X]  0002_transactions_onpremise_fix_orderby_and_partitionby
[X]  0003_transactions_onpremise_fix_columns
[X]  0004_transactions_add_tags_hash_map
[X]  0005_transactions_add_measurements
[X]  0006_transactions_add_http_fields
[X]  0007_transactions_add_discover_cols
[X]  0008_transactions_add_timestamp_index
[X]  0009_transactions_fix_title_and_message
[X]  0010_transactions_nullable_trace_id
[X]  0011_transactions_add_span_op_breakdowns
[X]  0012_transactions_add_spans
[X]  0013_transactions_reduce_spans_exclusive_time
[X]  0014_transactions_remove_flattened_columns
[X]  0015_transactions_add_source_column
[X]  0016_transactions_add_group_ids_column
[X]  0017_transactions_add_app_start_type_column
[X]  0018_transactions_add_profile_id
[X]  0019_transactions_add_indexes_and_context_hash
[X]  0020_transactions_add_codecs
[X]  0021_transactions_add_replay_id
[X]  0022_transactions_add_index_on_trace_id
[X]  0023_add_profiler_id_column

discover (readiness_state: complete)
[X]  0001_discover_merge_table
[X]  0002_discover_add_deleted_tags_hash_map
[X]  0003_discover_fix_user_column
[X]  0004_discover_fix_title_and_message
[X]  0005_discover_fix_transaction_name
[X]  0006_discover_add_trace_id
[X]  0007_discover_add_span_id
[X]  0008_discover_fix_add_local_table
[X]  0009_discover_add_replay_id

outcomes (readiness_state: complete)
[X]  0001_outcomes
[X]  0002_outcomes_remove_size_and_bytes
[X]  0003_outcomes_add_category_and_quantity
[X]  0004_outcomes_matview_additions
[X]  0005_outcomes_ttl
[X]  0006_outcomes_add_size_col
[X]  0007_outcomes_add_event_id_ttl_codec
[X]  0008_outcomes_add_indexes

metrics (readiness_state: complete)
[X]  0001_metrics_buckets
[X]  0002_metrics_sets
[X]  0003_counters_to_buckets
[X]  0004_metrics_counters
[X]  0005_metrics_distributions_buckets
[X]  0006_metrics_distributions
[X]  0007_metrics_sets_granularity_10
[X]  0008_metrics_counters_granularity_10
[X]  0009_metrics_distributions_granularity_10
[X]  0010_metrics_sets_granularity_1h
[X]  0011_metrics_counters_granularity_1h
[X]  0012_metrics_distributions_granularity_1h
[X]  0013_metrics_sets_granularity_1d
[X]  0014_metrics_counters_granularity_1d
[X]  0015_metrics_distributions_granularity_1d
[X]  0016_metrics_sets_consolidated_granularity
[X]  0017_metrics_counters_consolidated_granularity
[X]  0018_metrics_distributions_consolidated_granularity
[X]  0019_aggregate_tables_add_ttl
[X]  0020_polymorphic_buckets_table
[X]  0021_polymorphic_bucket_materialized_views
[X]  0022_repartition_polymorphic_table
[X]  0023_polymorphic_repartitioned_bucket_matview
[X]  0024_metrics_distributions_add_histogram
[X]  0025_metrics_counters_aggregate_v2
[X]  0026_metrics_counters_v2_writing_matview
[X]  0027_fix_migration_0026
[X]  0028_metrics_sets_aggregate_v2
[X]  0029_metrics_distributions_aggregate_v2
[X]  0030_metrics_distributions_v2_writing_mv
[X]  0031_metrics_sets_v2_writing_mv
[X]  0032_redo_0030_and_0031_without_timestamps
[X]  0033_metrics_cleanup_old_views
[X]  0034_metrics_cleanup_old_tables
[X]  0035_metrics_raw_timeseries_id

sessions (readiness_state: deprecate)
[X]  0001_sessions
[X]  0002_sessions_aggregates
[X]  0003_sessions_matview
[X]  0004_sessions_ttl
[X]  0005_drop_sessions_tables

profiles (readiness_state: complete)
[X]  0001_profiles
[X]  0002_disable_vertical_merge_algorithm
[X]  0003_add_device_architecture
[X]  0004_drop_profile_column

functions (readiness_state: complete)
[X]  0001_functions
[X]  0002_add_new_columns_to_raw_functions
[X]  0003_add_new_columns_to_raw_functions
[X]  0004_functions_v2

replays (readiness_state: complete)
[X]  0001_replays
[X]  0002_add_url
[X]  0003_alter_url_allow_null
[X]  0004_add_error_ids_column
[X]  0005_add_urls_user_agent_replay_start_timestamp
[X]  0006_add_is_archived_column
[X]  0007_add_replay_type_column
[X]  0008_add_sample_rate
[X]  0009_add_dom_index_columns
[X]  0010_add_nullable_columns
[X]  0011_add_is_dead_rage
[X]  0012_materialize_counts
[X]  0013_add_low_cardinality_codecs
[X]  0014_add_id_event_columns
[X]  0015_index_frequently_accessed_columns
[X]  0016_materialize_new_event_counts
[X]  0017_add_component_name_column
[X]  0018_add_viewed_by_id_column
[X]  0019_add_materialization
[X]  0020_add_dist_migration_for_materialization
[X]  0021_index_tags

generic_metrics (readiness_state: complete)
[X]  0001_sets_aggregate_table
[X]  0002_sets_raw_table
[X]  0003_sets_mv
[X]  0004_sets_raw_add_granularities
[X]  0005_sets_replace_mv
[X]  0006_sets_raw_add_granularities_dist_table
[X]  0007_distributions_aggregate_table
[X]  0008_distributions_raw_table
[X]  0009_distributions_mv
[X]  0010_counters_aggregate_table
[X]  0011_counters_raw_table
[X]  0012_counters_mv
[X]  0013_distributions_dist_tags_hash
[X]  0014_distribution_add_options
[X]  0015_sets_add_options
[X]  0016_counters_add_options
[X]  0017_distributions_mv2
[X]  0018_sets_update_opt_default
[X]  0019_counters_update_opt_default
[X]  0020_sets_mv2
[X]  0021_counters_mv2
[X]  0022_gauges_aggregate_table
[X]  0023_gauges_raw_table
[X]  0024_gauges_mv
[X]  0025_counters_add_raw_tags_hash_column
[X]  0026_gauges_add_raw_tags_hash_column
[X]  0027_sets_add_raw_tags_column
[X]  0028_distributions_add_indexed_tags_column
[X]  0029_add_use_case_id_index
[X]  0030_add_record_meta_column
[X]  0031_counters_meta_table
[X]  0032_counters_meta_table_mv
[X]  0033_counters_meta_tag_values_table
[X]  0034_counters_meta_tag_values_table_mv
[X]  0035_recreate_counters_meta_tag_value_table_mv
[X]  0036_counters_meta_tables_final
[X]  0037_add_record_meta_column_sets
[X]  0038_add_record_meta_column_distributions
[X]  0039_add_record_meta_column_gauges
[X]  0040_remove_counters_meta_tables
[X]  0041_adjust_partitioning_meta_tables
[X]  0042_rename_counters_meta_tables
[X]  0043_sets_meta_tables
[X]  0044_gauges_meta_tables
[X]  0045_distributions_meta_tables
[X]  0046_distributions_add_disable_percentiles
[X]  0047_distributions_mv3
[X]  0048_counters_meta_tables_support_empty_tags
[X]  0049_sets_meta_tables_support_empty_tags
[X]  0050_distributions_meta_tables_support_empty_tags
[X]  0051_gauges_meta_tables_support_empty_tags
[X]  0052_counters_raw_add_sampling_weight
[X]  0053_counters_aggregated_add_sampling_weight
[X]  0054_counters_mv3
[X]  0055_gauges_raw_add_sampling_weight
[X]  0056_gauges_aggregated_add_weighted_columns
[X]  0057_gauges_mv3
[X]  0058_distributions_raw_add_sampling_weight
[X]  0059_distributions_aggregated_add_weighted_columns
[X]  0060_distributions_mv4
[X]  0061_remove_distribution_meta_tag_values_mv

search_issues (readiness_state: complete)
[X]  0001_search_issues
[X]  0002_search_issues_add_tags_hash_map
[X]  0003_search_issues_modify_occurrence_type_id_size
[X]  0004_rebuild_search_issues_with_version
[X]  0005_search_issues_v2
[X]  0006_add_subtitle_culprit_level_resource_id
[X]  0007_add_transaction_duration
[X]  0008_add_profile_id_replay_id
[X]  0009_add_message

spans (readiness_state: complete)
[X]  0001_spans_v1
[X]  0002_spans_add_tags_hashmap
[X]  0003_spans_add_ms_columns
[X]  0004_spans_group_raw_col
[X]  0005_spans_add_sentry_tags
[X]  0006_spans_add_profile_id
[X]  0007_spans_add_metrics_summary
[X]  0008_spans_add_index_on_span_id
[X]  0009_spans_add_measure_hashmap
[X]  0010_spans_add_compression
[X]  0011_spans_add_index_on_trace_id
[X]  0012_spans_add_index_on_transaction_name
[X]  0013_spans_add_indexes_for_tag_columns
[X]  0014_spans_add_microsecond_precision_timestamps

events_analytics_platform (readiness_state: complete)
[X]  0001_spans
[X]  0002_spans_attributes_mv
[X]  0003_eap_spans_project_id_index
[X]  0004_modify_sampling_weight
[X]  0005_remove_attribute_mv
[X]  0006_drop_attribute_key_project_id_indexes
[X]  0007_drop_project_id_index
[X]  0008_drop_index_attribute_key_bucket_0
[X]  0009_drop_index_attribute_key_buckets_1_19
[X]  0010_drop_indexes_on_attribute_values
[X]  0011_span_attribute_table
[X]  0012_span_attribute_table_numeric
[X]  0013_span_attribute_table_shard_keys
[X]  0014_span_attribute_table_smaller
[X]  0015_span_attribute_table_namespaced
[X]  0016_spans_v2
[X]  0017_span_attribute_table_v3
[X]  0018_drop_unused_span_tables
[X]  0019_uptime_monitors_init
[X]  0020_ourlogs_init
[X]  0021_ourlogs_attrs
[X]  0022_uptime_monitors_init_v2
[X]  0023_smart_autocomplete_mv
[X]  0024_items
[X]  0025_smart_autocomplete_index
[X]  0026_items_add_attributes_hash_map
[X]  0027_uptime_checks_add_column_in_incident
[X]  0028_ourlogs_v3
[X]  0029_remove_smart_autocomplete_experimental
[X]  0030_smart_autocomplete_items
[X]  0032_sampled_storage_views
[X]  0033_items_attribute_table_v1
[X]  0034_materialize_sampled_storage_views
[X]  0035_drop_item_attrs
[X]  0036_items_attribute_table_v1
[X]  0037_remove_items_attribute_mv_v1

group_attributes (readiness_state: complete)
[X]  0001_group_attributes
[X]  0002_add_priority_to_group_attributes
[X]  0003_add_first_release_id_to_group_attributes
[X]  0004_add_new_first_release_column_to_group_attributes

cdc (readiness_state: deprecate)

/cc @fh1ch @dlouzan

[aldy505]

How did you upgrade Sentry? What commands did you execute?

[max-wittig]

Turns out, the fault here is at clickhouse-backup. We created an issue in their tracker: Altinity/clickhouse-backup#1151