Sentry: pebble.go:1066: log.Fatal: local corruption detected: pebble: backing file 000761 error: open ×: The system cannot find the file specified. (1) attached stack trace -- stack trace: | gith... #156752
Description
This issue was auto filed by Sentry. It represents a crash or reported error on a live cluster with telemetry enabled.
Sentry Link: https://cockroach-labs.sentry.io/issues/6992883922/?referrer=webhooks_plugin
Panic Message:
pebble.go:1066: log.Fatal: local corruption detected: pebble: backing file 000761 error: open ×: The system cannot find the file specified.
(1) attached stack trace
-- stack trace:
| github.com/cockroachdb/cockroach/pkg/storage.(*Pebble).makeMetricEtcEventListener.func2
| pkg/storage/pebble.go:1066
| github.com/cockroachdb/pebble.TeeEventListener.func2
| external/com_github_cockroachdb_pebble/event.go:994
| github.com/cockroachdb/pebble.(*DB).reportFileCorruption
| external/com_github_cockroachdb_pebble/event.go:1194
| github.com/cockroachdb/pebble.(*DB).reportCorruption
| external/com_github_cockroachdb_pebble/event.go:1153
| github.com/cockroachdb/pebble.(*fileCacheHandle).findOrCreateTable
| external/com_github_cockroachdb_pebble/file_cache.go:245
| github.com/cockroachdb/pebble.(*fileCacheHandle).withReader
| external/com_github_cockroachdb_pebble/file_cache.go:350
| github.com/cockroachdb/pebble.(*DB).estimateSizesBeneath-range1
| external/com_github_cockroachdb_pebble/table_stats.go:518
| github.com/cockroachdb/pebble.(*DB).estimateSizesBeneath.LevelSlice.All.func3
| external/com_github_cockroachdb_pebble/internal/manifest/level_metadata.go:277
| github.com/cockroachdb/pebble.(*DB).estimateSizesBeneath
| external/com_github_cockroachdb_pebble/table_stats.go:513
| github.com/cockroachdb/pebble.(*DB).loadTablePointKeyStats
| external/com_github_cockroachdb_pebble/table_stats.go:358
| github.com/cockroachdb/pebble.(*DB).loadTableStats.func1
| external/com_github_cockroachdb_pebble/table_stats.go:324
| github.com/cockroachdb/pebble.(*fileCacheHandle).withCommonReader
| external/com_github_cockroachdb_pebble/file_cache.go:341
| github.com/cockroachdb/pebble.(*DB).loadTableStats
| external/com_github_cockroachdb_pebble/table_stats.go:310
| github.com/cockroachdb/pebble.(*DB).loadNewFileStats
| external/com_github_cockroachdb_pebble/table_stats.go:198
| github.com/cockroachdb/pebble.(*DB).collectTableStats
| external/com_github_cockroachdb_pebble/table_stats.go:113
| runtime.goexit
| src/runtime/asm_amd64.s:1700
Wraps: (2) secondary error attachment
| pebble: backing file 000761 error: open ×: The system cannot find the file specified.
| (1) ×
| Wraps: (2) attached stack trace
| -- stack trace:
| | github.com/cockroachdb/pebble.NewFileCache.func1
| | external/com_github_cockroachdb_pebble/file_cache.go:461
| | [...repeated from below...]
| Wraps: (3) pebble: backing file 000761 error
| Wraps: (4) forced error mark
| | ×
| | github.com/cockroachdb/errors/withstack/*withstack.withStack::
| Wraps: (5) ×
| Wraps: (6) attached stack trace
| -- stack trace:
| | github.com/cockroachdb/pebble/vfs.defaultFS.Open
| | external/com_github_cockroachdb_pebble/vfs/vfs.go:275
| | github.com/cockroachdb/pebble/vfs.(*diskHealthCheckingFS).Open
| | external/com_github_cockroachdb_pebble/vfs/disk_health.go:805
| | github.com/cockroachdb/pebble/vfs.(*enospcFS).Open
| | external/com_github_cockroachdb_pebble/vfs/disk_full.go:221
| | github.com/cockroachdb/cockroach/pkg/storage/fs.(*Env).Open
| | pkg/storage/fs/fs.go:363
| | github.com/cockroachdb/pebble/objstorage/objstorageprovider.(*provider).vfsOpenForReading
| | external/com_github_cockroachdb_pebble/objstorage/objstorageprovider/vfs.go:27
| | github.com/cockroachdb/pebble/objstorage/objstorageprovider.(*provider).OpenForReading
| | external/com_github_cockroachdb_pebble/objstorage/objstorageprovider/provider.go:304
| | github.com/cockroachdb/pebble.(*fileCacheHandle).openFile
| | external/com_github_cockroachdb_pebble/file_cache.go:196
| | github.com/cockroachdb/pebble.NewFileCache.func1
| | external/com_github_cockroachdb_pebble/file_cache.go:459
| | github.com/cockroachdb/pebble/internal/genericcache.(*shard[...]).findOrCreateValue
| | external/com_github_cockroachdb_pebble/internal/genericcache/shard.go:191
| | github.com/cockroachdb/pebble/internal/genericcache.(*Cache[...]).FindOrCreate
| | external/com_github_cockroachdb_pebble/internal/genericcache/cache.go:84
| | github.com/cockroachdb/pebble.(*fileCacheHandle).findOrCreateTable
| | external/com_github_cockroachdb_pebble/file_cache.go:243
| | github.com/cockroachdb/pebble.(*fileCacheHandle).withReader
| | external/com_github_cockroachdb_pebble/file_cache.go:350
| | github.com/cockroachdb/pebble.(*DB).estimateSizesBeneath-range1
| | external/com_github_cockroachdb_pebble/table_stats.go:518
| | github.com/cockroachdb/pebble.(*DB).estimateSizesBeneath.LevelSlice.All.func3
| | external/com_github_cockroachdb_pebble/internal/manifest/level_metadata.go:277
| | github.com/cockroachdb/pebble.(*DB).estimateSizesBeneath
| | external/com_github_cockroachdb_pebble/table_stats.go:513
| | github.com/cockroachdb/pebble.(*DB).loadTablePointKeyStats
| | external/com_github_cockroachdb_pebble/table_stats.go:358
| | github.com/cockroachdb/pebble.(*DB).loadTableStats.func1
| | external/com_github_cockroachdb_pebble/table_stats.go:324
| | github.com/cockroachdb/pebble.(*fileCacheHandle).withCommonReader
| | external/com_github_cockroachdb_pebble/file_cache.go:341
| | github.com/cockroachdb/pebble.(*DB).loadTableStats
| | external/com_github_cockroachdb_pebble/table_stats.go:310
| | github.com/cockroachdb/pebble.(*DB).loadNewFileStats
| | external/com_github_cockroachdb_pebble/table_stats.go:198
| | github.com/cockroachdb/pebble.(*DB).collectTableStats
| | external/com_github_cockroachdb_pebble/table_stats.go:113
| | runtime.goexit
| | src/runtime/asm_amd64.s:1700
| Wraps: (7) open ×
| Wraps: (8) The system cannot find the file specified.
| Error types: (1) *hintdetail.withHint (2) *withstack.withStack (3) *errutil.withPrefix (4) *markers.withMark (5) *hintdetail.withDetail (6) *withstack.withStack (7) *fs.PathError (8) syscall.Errno
Wraps: (3) log.Fatal: local corruption detected: pebble: backing file 000761 error: open ×: The system cannot find the file specified.
| (1) ×
| Wraps: (2) attached stack trace
| -- stack trace:
| | github.com/cockroachdb/pebble.NewFileCache.func1
| | external/com_github_cockroachdb_pebble/file_cache.go:461
| | [...repeated from below...]
| Wraps: (3) pebble: backing file 000761 error
| Wraps: (4) forced error mark
| | ×
| | github.com/cockroachdb/errors/withstack/*withstack.withStack::
| Wraps: (5) ×
| Wraps: (6) attached stack trace
| -- stack trace:
| | github.com/cockroachdb/pebble/vfs.defaultFS.Open
| | external/com_github_cockroachdb_pebble/vfs/vfs.go:275
| | github.com/cockroachdb/pebble/vfs.(*diskHealthCheckingFS).Open
| | external/com_github_cockroachdb_pebble/vfs/disk_health.go:805
| | github.com/cockroachdb/pebble/vfs.(*enospcFS).Open
| | external/com_github_cockroachdb_pebble/vfs/disk_full.go:221
| | github.com/cockroachdb/cockroach/pkg/storage/fs.(*Env).Open
| | pkg/storage/fs/fs.go:363
| | github.com/cockroachdb/pebble/objstorage/objstorageprovider.(*provider).vfsOpenForReading
| | external/com_github_cockroachdb_pebble/objstorage/objstorageprovider/vfs.go:27
| | github.com/cockroachdb/pebble/objstorage/objstorageprovider.(*provider).OpenForReading
| | external/com_github_cockroachdb_pebble/objstorage/objstorageprovider/provider.go:304
| | github.com/cockroachdb/pebble.(*fileCacheHandle).openFile
| | external/com_github_cockroachdb_pebble/file_cache.go:196
| | github.com/cockroachdb/pebble.NewFileCache.func1
| | external/com_github_cockroachdb_pebble/file_cache.go:459
| | github.com/cockroachdb/pebble/internal/genericcache.(*shard[...]).findOrCreateValue
| | external/com_github_cockroachdb_pebble/internal/genericcache/shard.go:191
| | github.com/cockroachdb/pebble/internal/genericcache.(*Cache[...]).FindOrCreate
| | external/com_github_cockroachdb_pebble/internal/genericcache/cache.go:84
| | github.com/cockroachdb/pebble.(*fileCacheHandle).findOrCreateTable
| | ...
Stacktrace (expand for inline code snippets):
src/runtime/asm_amd64.s#L1699-L1701
external/com_github_cockroachdb_pebble/table_stats.go#L112-L114
external/com_github_cockroachdb_pebble/table_stats.go#L197-L199
external/com_github_cockroachdb_pebble/table_stats.go#L309-L311
external/com_github_cockroachdb_pebble/file_cache.go#L340-L342
external/com_github_cockroachdb_pebble/table_stats.go#L323-L325
external/com_github_cockroachdb_pebble/table_stats.go#L357-L359
external/com_github_cockroachdb_pebble/table_stats.go#L512-L514
external/com_github_cockroachdb_pebble/internal/manifest/level_metadata.go#L276-L278
external/com_github_cockroachdb_pebble/table_stats.go#L517-L519
external/com_github_cockroachdb_pebble/file_cache.go#L349-L351
external/com_github_cockroachdb_pebble/file_cache.go#L244-L246
external/com_github_cockroachdb_pebble/event.go#L1152-L1154
external/com_github_cockroachdb_pebble/event.go#L1193-L1195
external/com_github_cockroachdb_pebble/event.go#L993-L995
pkg/storage/pebble.go#L1065-L1067
src/runtime/asm_amd64.s in runtime.goexit at line 1700
external/com_github_cockroachdb_pebble/table_stats.go in github.com/cockroachdb/pebble.(*DB).collectTableStats at line 113
external/com_github_cockroachdb_pebble/table_stats.go in github.com/cockroachdb/pebble.(*DB).loadNewFileStats at line 198
external/com_github_cockroachdb_pebble/table_stats.go in github.com/cockroachdb/pebble.(*DB).loadTableStats at line 310
external/com_github_cockroachdb_pebble/file_cache.go in github.com/cockroachdb/pebble.(*fileCacheHandle).withCommonReader at line 341
external/com_github_cockroachdb_pebble/table_stats.go in github.com/cockroachdb/pebble.(*DB).loadTableStats.func1 at line 324
external/com_github_cockroachdb_pebble/table_stats.go in github.com/cockroachdb/pebble.(*DB).loadTablePointKeyStats at line 358
external/com_github_cockroachdb_pebble/table_stats.go in github.com/cockroachdb/pebble.(*DB).estimateSizesBeneath at line 513
external/com_github_cockroachdb_pebble/internal/manifest/level_metadata.go in github.com/cockroachdb/pebble.(*DB).estimateSizesBeneath.LevelSlice.All.func3 at line 277
external/com_github_cockroachdb_pebble/table_stats.go in github.com/cockroachdb/pebble.(*DB).estimateSizesBeneath-range1 at line 518
external/com_github_cockroachdb_pebble/file_cache.go in github.com/cockroachdb/pebble.(*fileCacheHandle).withReader at line 350
external/com_github_cockroachdb_pebble/file_cache.go in github.com/cockroachdb/pebble.(*fileCacheHandle).findOrCreateTable at line 245
external/com_github_cockroachdb_pebble/event.go in github.com/cockroachdb/pebble.(*DB).reportCorruption at line 1153
external/com_github_cockroachdb_pebble/event.go in github.com/cockroachdb/pebble.(*DB).reportFileCorruption at line 1194
external/com_github_cockroachdb_pebble/event.go in github.com/cockroachdb/pebble.TeeEventListener.func2 at line 994
pkg/storage/pebble.go in pkg/storage.(*Pebble).makeMetricEtcEventListener.func2 at line 1066
Tags
| Tag | Value |
|---|---|
| Command | server |
| Environment | v25.2.6 |
| Go Version | go1.23.12 X:nocoverageredesign |
| Platform | windows amd64 |
| Distribution | CCL |
| Cockroach Release | v25.2.6 |
| Cockroach SHA | e9b11cf |
| # of CPUs | 16 |
| # of Goroutines | 766 |
Jira issue: CRDB-56171