cachedict inflight registry: sqlite 'file is not a database' under concurrent job-array access

[sdascoli]

Summary

When a grid of experiments is submitted as a single slurm job array (via TaskInfra.job_array()), and each job independently accesses a shared cached step (same folder) that computes a large set of items, exca's cachedict inflight registry sqlite database gets corrupted:

sqlite3.DatabaseError: file is not a database

The corruption then cascades into FileNotFoundError on full-uid.yaml and submitit.core.utils.UncompletedJobError for the sub-tasks, failing the whole run.

Environment

• exca 0.5.27
• Python 3.12, Linux, slurm cluster (submitit backend)

Scenario

• ~18 experiments submitted together via TaskInfra.job_array().
• Each job calls into the same shared cached pipeline (a Cached step backend + a map that computes ~432 items) pointing at the same cache folder.
• All ~18 jobs start ~simultaneously, so many concurrently query/populate the same inflight registry.
• Pre-warming the cache once (single builder) before submitting the array does not prevent it: the concurrent jobs still re-open/validate the shared registry and corrupt it.

Stacktrace

WARNING - exca.cachedict.inflight:349 - Waiting for 432 in-flight items (of 432 requested) held by: <jobid> [slurm:RUNNING] x432
WARNING - exca.cachedict.registry:234 - Inflight registry query: file is not a database
Traceback (most recent call last):
  File ".../exca/cachedict/registry.py", line 222, in _safe_execute
    return self._retry_on_lock(run)
  File ".../exca/cachedict/registry.py", line 130, in _retry_on_lock
    return fn()
  File ".../exca/cachedict/registry.py", line 211, in run
    return fn(conn)
  File ".../exca/cachedict/inflight.py", line 316, in _do
    rows = registry.select_in_chunks(
  File ".../exca/cachedict/registry.py", line 49, in select_in_chunks
    out.extend(conn.execute(sql, batch).fetchall())
sqlite3.DatabaseError: file is not a database

Also observed alongside it:

FileNotFoundError: [Errno 2] No such file or directory: '.../<cache-uid>/full-uid.yaml'
submitit.core.utils.UncompletedJobError: Job <jobid> (task: 0) with path .../logs/<jobid>/<jobid>_0_result.pkl

Expected

Concurrent readers/writers of a shared cachedict (typical when a job array shares one cache folder) should not corrupt the sqlite inflight registry. Either serialize/guard registry writes robustly across processes, tolerate concurrent access, or fail loudly with a clear message instead of silently corrupting the DB.

Notes

Happy to help repro. Workaround we used: disable the Slurm timelines sub-infra so items are computed in-process (removing the concurrent writers to that registry).

[jrapin]

For the record, neuralset config

study = {
                "name": "Lebel2023Natural",
                "path": DATADIR,
                "timelines": {
                    "infra": {
                        "backend": "Slurm",
                        "min_items_per_job": 20,  # ~22 jobs for 432 timelines
                        "max_jobs": 2048,
                        "folder": CACHEDIR,
                        "partition": SLURM_PARTITION,
                        "cpus_per_task": 10,
                        "timeout_min": 60 * 4,  # Lebel TextGrid/audio loading is slow
                    },
                },
                "infra": {
                    "backend": "Cached",
                    "folder": CACHEDIR,
                },
            }

and coment

Pre-warm worked — the study built once cleanly and the 18-job sweep launched ("study cache ready; submitting 18 jobs"). But the run still failed because each of the 18 concurrent jobs re-runs study.run(), which re-accesses the study's sqlite Cached backend + Slurm timelines builder concurrently → sqlite3.DatabaseError: file is not a database. Pre-warm alone isn't enough: exca's sqlite study cache isn't safe under 18-way concurrent access (the workers re-validate/rebuild it, not just read).

[jrapin]

#220 — replace the flaky JobChecker with an SQLite advisory registry (journal_mode=DELETE, BEGIN IMMEDIATE claims).
#224 — general robustness hardening of the inflight path.
#226 — batch the worker-info updates (_QUERY_BATCH_SIZE, single BEGIN) to fix slow DB updates under many items.
#235 — add a second advisory DB (errors.db) for Step error caching, same pattern.
#263 — factor the shared SQLite plumbing into a reusable AdvisoryRegistry base.
#273 — best-effort chmod on transient DB files (avoid crashing on permission races).
#276 — better logging around inflight DB operations.
#290 — fix an infinite poll in the wait loop.
#298 — fix the update_worker_info freeze (per-row autocommit → NFS fcntl/-journal/fsync storm parking in D-state): batch it into one BEGIN…COMMIT and switch DELETE→WAL to kill the per-transaction journal-sidecar churn. The WAL half ignored the "unsafe on NFS" caveat, which is #304.