Sentry: replica.go:1957: log.Fatal: pebble: backing file 000028 error: open ×: operation not permitted (1) attached stack trace -- stack trace: | github.com/cockroachdb/cockroach/pkg/kv/kvserver.... #156749
Description
This issue was auto filed by Sentry. It represents a crash or reported error on a live cluster with telemetry enabled.
Sentry Link: https://cockroach-labs.sentry.io/issues/6992659748/?referrer=webhooks_plugin
Panic Message:
replica.go:1957: log.Fatal: pebble: backing file 000028 error: open ×: operation not permitted
(1) attached stack trace
-- stack trace:
| github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).assertStateRaftMuLockedReplicaMuRLocked
| pkg/kv/kvserver/replica.go:1957
| github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).handleRaftReadyRaftMuLocked
| pkg/kv/kvserver/replica_raft.go:1283
| github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Store).processRaftSnapshotRequest.func1
| pkg/kv/kvserver/store_raft.go:558
| github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Store).withReplicaForRequest
| pkg/kv/kvserver/store_raft.go:435
| github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Store).processRaftSnapshotRequest
| pkg/kv/kvserver/store_raft.go:499
| github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Store).receiveSnapshot
| pkg/kv/kvserver/store_snapshot.go:600
| github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Store).HandleSnapshot.func1
| pkg/kv/kvserver/store_raft.go:282
| github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunTaskWithErr
| pkg/util/stop/stopper.go:349
| github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Store).HandleSnapshot
| pkg/kv/kvserver/store_raft.go:279
| github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*RaftTransport).raftSnapshot
| pkg/kv/kvserver/raft_transport.go:566
| github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*RaftTransport).RaftSnapshot
| pkg/kv/kvserver/raft_transport.go:544
| github.com/cockroachdb/cockroach/pkg/kv/kvserver._MultiRaft_RaftSnapshot_Handler
| bazel-out/k8-opt/bin/pkg/kv/kvserver/kvserver_go_proto_/github.com/cockroachdb/cockroach/pkg/kv/kvserver/storage_services.pb.go:267
| github.com/cockroachdb/cockroach/pkg/rpc.NewServerEx.StreamServerInterceptor.func13
| pkg/util/tracing/grpcinterceptor/grpc_interceptor.go:159
| google.golang.org/grpc.getChainStreamHandler.func1
| external/org_golang_google_grpc/server.go:1504
| github.com/cockroachdb/cockroach/pkg/rpc.NewServerEx.func4
| pkg/rpc/context.go:187
| google.golang.org/grpc.getChainStreamHandler.func1
| external/org_golang_google_grpc/server.go:1504
| github.com/cockroachdb/cockroach/pkg/rpc.NewServerEx.func2.1
| pkg/rpc/context.go:145
| github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunTaskWithErr
| pkg/util/stop/stopper.go:349
| github.com/cockroachdb/cockroach/pkg/rpc.NewServerEx.func2
| pkg/rpc/context.go:144
| google.golang.org/grpc.NewServer.chainStreamServerInterceptors.chainStreamInterceptors.func2
| external/org_golang_google_grpc/server.go:1495
| google.golang.org/grpc.(*Server).processStreamingRPC
| external/org_golang_google_grpc/server.go:1659
| google.golang.org/grpc.(*Server).handleStream
| external/org_golang_google_grpc/server.go:1739
| google.golang.org/grpc.(*Server).serveStreams.func1.1
| external/org_golang_google_grpc/server.go:970
| runtime.goexit
| src/runtime/asm_arm64.s:1223
Wraps: (2) secondary error attachment
| pebble: backing file 000028 error: open ×: operation not permitted
| (1) attached stack trace
| -- stack trace:
| | github.com/cockroachdb/pebble.NewFileCache.func1
| | external/com_github_cockroachdb_pebble/file_cache.go:468
| | github.com/cockroachdb/pebble/internal/genericcache.(*shard[...]).findOrCreateValue
| | external/com_github_cockroachdb_pebble/internal/genericcache/shard.go:191
| | github.com/cockroachdb/pebble/internal/genericcache.(*Cache[...]).FindOrCreate
| | external/com_github_cockroachdb_pebble/internal/genericcache/cache.go:84
| | github.com/cockroachdb/pebble.(*fileCacheHandle).findOrCreateTable
| | external/com_github_cockroachdb_pebble/file_cache.go:260
| | github.com/cockroachdb/pebble.(*fileCacheHandle).newIters
| | external/com_github_cockroachdb_pebble/file_cache.go:553
| | github.com/cockroachdb/pebble.(*ingestedFlushable).constructRangeDelIter
| | external/com_github_cockroachdb_pebble/flushable.go:238
| | github.com/cockroachdb/pebble/internal/keyspan/keyspanimpl.(*LevelIter).setPosAtFile
| | external/com_github_cockroachdb_pebble/internal/keyspan/keyspanimpl/level_iter.go:460
| | github.com/cockroachdb/pebble/internal/keyspan/keyspanimpl.(*LevelIter).SeekLT
| | external/com_github_cockroachdb_pebble/internal/keyspan/keyspanimpl/level_iter.go:247
| | github.com/cockroachdb/pebble/internal/keyspan/keyspanimpl.(*MergingIter).SeekGE
| | external/com_github_cockroachdb_pebble/internal/keyspan/keyspanimpl/merging_iter.go:390
| | github.com/cockroachdb/pebble.(*mergingIter).seekGE
| | external/com_github_cockroachdb_pebble/merging_iter.go:997
| | github.com/cockroachdb/pebble.(*mergingIter).SeekPrefixGEStrict
| | external/com_github_cockroachdb_pebble/merging_iter.go:1044
| | github.com/cockroachdb/pebble.(*mergingIter).SeekPrefixGE
| | external/com_github_cockroachdb_pebble/merging_iter.go:1035
| | github.com/cockroachdb/pebble/internal/keyspan.(*InterleavingIter).SeekPrefixGE
| | external/com_github_cockroachdb_pebble/internal/keyspan/interleaving_iter.go:323
| | github.com/cockroachdb/pebble.(*Iterator).SeekPrefixGE
| | external/com_github_cockroachdb_pebble/iterator.go:1502
| | github.com/cockroachdb/cockroach/pkg/storage.(*pebbleIterator).SeekGE
| | pkg/storage/pebble_iterator.go:343
| | github.com/cockroachdb/cockroach/pkg/storage.(*pebbleMVCCScanner).get
| | pkg/storage/pebble_mvcc_scanner.go:584
| | github.com/cockroachdb/cockroach/pkg/storage.mvccGet
| | pkg/storage/mvcc.go:1583
| | github.com/cockroachdb/cockroach/pkg/storage.MVCCGetWithValueHeader
| | pkg/storage/mvcc.go:1508
| | github.com/cockroachdb/cockroach/pkg/storage.MVCCGet
| | pkg/storage/mvcc.go:1408
| | github.com/cockroachdb/cockroach/pkg/storage.MVCCGetProto
| | pkg/storage/mvcc.go:1104
| | github.com/cockroachdb/cockroach/pkg/kv/kvserver/abortspan.(*AbortSpan).Get
| | pkg/kv/kvserver/abortspan/abortspan.go:90
| | github.com/cockroachdb/cockroach/pkg/kv/kvserver.checkIfTxnAborted
| | pkg/kv/kvserver/replica.go:2705
| | github.com/cockroachdb/cockroach/pkg/kv/kvserver.evaluateBatch
| | pkg/kv/kvserver/replica_evaluate.go:265
| | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).executeReadOnlyBatchWithServersideRefreshes
| | pkg/kv/kvserver/replica_read.go:479
| | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).executeReadOnlyBatch
| | pkg/kv/kvserver/replica_read.go:131
| | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).executeBatchWithConcurrencyRetries
| | pkg/kv/kvserver/replica_send.go:510
| | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).SendWithWriteBytes
| | pkg/kv/kvserver/replica_send.go:185
| | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Store).SendWithWriteBytes
| | pkg/kv/kvserver/store_send.go:188
| | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Stores).SendWithWriteBytes
| | pkg/kv/kvserver/stores.go:199
| | github.com/cockroachdb/cockroach/pkg/server.(*Node).batchInternal
| | node.go:1654
| | github.com/cockroachdb/cockroach/pkg/server.(*Node).Batch
| | node.go:1847
| Wraps: (2) pebble: backing file 000028 error
| Wraps: (3) attached stack trace
| -- stack trace:
| | github.com/cockroachdb/pebble/vfs.defaultFS.Open
| | external/com_github_cockroachdb_pebble/vfs/vfs.go:275
| | github.com/cockroachdb/pebble/vfs.(*diskHealthCheckingFS).Open
| | external/com_github_cockroachdb_pebble/vfs/disk_health.go:805
| | github.com/cockroachdb/pebble/vfs.(*enospcFS).Open
| | external/com_github_cockroachdb_pebble/vfs/disk_full.go:221
| | github.com/cockroachdb/cockroach/pkg/storage/fs.(*Env).Open
| | pkg/storage/fs/fs.go:363
| | github.com/cockroachdb/pebble/objstorage/objstorageprovider.(*provider).vfsOpenForReading
| | external/com_github_cockroachdb_pebble/objstorage/objstorageprovider/vfs.go:27
| | github.com/cockroachdb/pebble/objstorage/ob...
Stacktrace (expand for inline code snippets):
src/runtime/asm_arm64.s#L1222-L1224
external/org_golang_google_grpc/server.go#L969-L971
external/org_golang_google_grpc/server.go#L1738-L1740
external/org_golang_google_grpc/server.go#L1658-L1660
external/org_golang_google_grpc/server.go#L1494-L1496
pkg/rpc/context.go#L143-L145
pkg/util/stop/stopper.go#L348-L350
pkg/rpc/context.go#L144-L146
external/org_golang_google_grpc/server.go#L1503-L1505
pkg/rpc/context.go#L186-L188
external/org_golang_google_grpc/server.go#L1503-L1505
pkg/util/tracing/grpcinterceptor/grpc_interceptor.go#L158-L160
bazel-out/k8-opt/bin/pkg/kv/kvserver/kvserver_go_proto_/https://github.com/cockroachdb/cockroach/blob/a5e060ee11ad11e1649e7569486b579e9da8c8d7/pkg/kv/kvserver/storage_services.pb.go#L266-L268
pkg/kv/kvserver/raft_transport.go#L543-L545
pkg/kv/kvserver/raft_transport.go#L565-L567
pkg/kv/kvserver/store_raft.go#L278-L280
pkg/util/stop/stopper.go#L348-L350
pkg/kv/kvserver/store_raft.go#L281-L283
pkg/kv/kvserver/store_snapshot.go#L599-L601
pkg/kv/kvserver/store_raft.go#L498-L500
pkg/kv/kvserver/store_raft.go#L434-L436
pkg/kv/kvserver/store_raft.go#L557-L559
pkg/kv/kvserver/replica_raft.go#L1282-L1284
pkg/kv/kvserver/replica.go#L1956-L1958
src/runtime/asm_arm64.s in runtime.goexit at line 1223
external/org_golang_google_grpc/server.go in google.golang.org/grpc.(*Server).serveStreams.func1.1 at line 970
external/org_golang_google_grpc/server.go in google.golang.org/grpc.(*Server).handleStream at line 1739
external/org_golang_google_grpc/server.go in google.golang.org/grpc.(*Server).processStreamingRPC at line 1659
external/org_golang_google_grpc/server.go in google.golang.org/grpc.NewServer.chainStreamServerInterceptors.chainStreamInterceptors.func2 at line 1495
pkg/rpc/context.go in pkg/rpc.NewServerEx.func2 at line 144
pkg/util/stop/stopper.go in pkg/util/stop.(*Stopper).RunTaskWithErr at line 349
pkg/rpc/context.go in pkg/rpc.NewServerEx.func2.1 at line 145
external/org_golang_google_grpc/server.go in google.golang.org/grpc.getChainStreamHandler.func1 at line 1504
pkg/rpc/context.go in pkg/rpc.NewServerEx.func4 at line 187
external/org_golang_google_grpc/server.go in google.golang.org/grpc.getChainStreamHandler.func1 at line 1504
pkg/util/tracing/grpcinterceptor/grpc_interceptor.go in pkg/rpc.NewServerEx.StreamServerInterceptor.func13 at line 159
bazel-out/k8-opt/bin/pkg/kv/kvserver/kvserver_go_proto_/github.com/cockroachdb/cockroach/pkg/kv/kvserver/storage_services.pb.go in pkg/kv/kvserver._MultiRaft_RaftSnapshot_Handler at line 267
pkg/kv/kvserver/raft_transport.go in pkg/kv/kvserver.(*RaftTransport).RaftSnapshot at line 544
pkg/kv/kvserver/raft_transport.go in pkg/kv/kvserver.(*RaftTransport).raftSnapshot at line 566
pkg/kv/kvserver/store_raft.go in pkg/kv/kvserver.(*Store).HandleSnapshot at line 279
pkg/util/stop/stopper.go in pkg/util/stop.(*Stopper).RunTaskWithErr at line 349
pkg/kv/kvserver/store_raft.go in pkg/kv/kvserver.(*Store).HandleSnapshot.func1 at line 282
pkg/kv/kvserver/store_snapshot.go in pkg/kv/kvserver.(*Store).receiveSnapshot at line 600
pkg/kv/kvserver/store_raft.go in pkg/kv/kvserver.(*Store).processRaftSnapshotRequest at line 499
pkg/kv/kvserver/store_raft.go in pkg/kv/kvserver.(*Store).withReplicaForRequest at line 435
pkg/kv/kvserver/store_raft.go in pkg/kv/kvserver.(*Store).processRaftSnapshotRequest.func1 at line 558
pkg/kv/kvserver/replica_raft.go in pkg/kv/kvserver.(*Replica).handleRaftReadyRaftMuLocked at line 1283
pkg/kv/kvserver/replica.go in pkg/kv/kvserver.(*Replica).assertStateRaftMuLockedReplicaMuRLocked at line 1957
Tags
| Tag | Value |
|---|---|
| Command | server |
| Environment | v25.3.2 |
| Go Version | go1.23.12 X:nocoverageredesign |
| Platform | darwin arm64 |
| Distribution | CCL |
| Cockroach Release | v25.3.2 |
| Cockroach SHA | a5e060e |
| # of CPUs | 10 |
| # of Goroutines | 990 |
Jira issue: CRDB-56168