Register bwm_lfp as an official, opt-in agent dataset

CHANGELOG.md

@@ -12,6 +12,13 @@ This project adheres to [Semantic Versioning](https://semver.org/).
 ### Added
 - `bwm_behavior` dataset version 2.0.0 with wheel recovery for all 459 sessions
   and Lightning Pose preferred over DeepLabCut per camera.
+- `bwm_lfp` dataset version 1.0.0 (standard-compression tier only): opt-in
+  download/registration via `scripts/download_datasets.py --lfp`, plugged
+  into the standard `data_locations`/`schema.yaml` dataset registry, plus
+  `ibl-neuropixel`/`ibl-load` skill guidance for `lfpack.LFPackReader`.
+  `lfpack` is an optional `lfp` extra, not a core dependency.
+- `scripts/validate_bwm_lfp_release.py`: release validator for `bwm_lfp`
+  archives, mirroring `validate_bwm_ephys_release.py`'s role.
 See `CHANGELOG_DATA.md` for further details.
 
 ### Changed

CHANGELOG_DATA.md

@@ -11,6 +11,44 @@ Dataset versions follow [Semantic Versioning](https://semver.org/).
 
 ---
 
+## [bwm_lfp 1.0.0] - 2026-07-25
+
+### Added
+- First registered release of `bwm_lfp`, a compressed LFP companion to
+  `bwm_ephys`/`bwm_behavior`: all `699` BWM probe recordings in a single
+  `lf_compressed_all_bwm.h5` file, `384` channels for 695 recordings and `96`
+  for four NP2.4 recordings, `250 Hz` (decimated from `2500 Hz`), built with
+  `lfpack`'s SVD + wavelet-packet codec.
+- Only the **standard** compression tier (~119x) is distributed to the
+  agent; `lfpack` also produces an aggressive tier for its own uses, which
+  is intentionally not shipped here.
+- Each recording embeds per-channel brain-region annotations, a time axis
+  (`sr.times`) that is session-clock synchronized when `sr.t0` is finite, and
+  a saturation (ADC-clipping) QC table.
+- Opt-in download via `scripts/download_datasets.py --lfp`; unlike
+  `bwm_ephys`/`bwm_behavior`, not part of the default download flow because
+  of its size (~14 GB).
+- Unlike the other two datasets, the archive is produced upstream by the
+  `lfpack` package (in `ephys-atlas`), not built by this repo; its
+  `schema.yaml`/`provenance.yaml`/`manifest.json` are authored by
+  `scripts/download_datasets.py` itself at download time to plug it into
+  the same schema-based dataset registry.
+
+### Known limitations
+- Sync could not be computed for 7 probes across 4 sessions; `sr.t0` is
+  `NaN` and `sr.times` is recording-relative for these
+  ([lfpack#8](https://github.com/int-brain-lab/lfpack/issues/8)).
+- Saturation affects more than 1% of source samples in 121 recordings and more
+  than 10% in 16 recordings (maximum 23.9%); use `sr.saturation_mask` rather
+  than testing decompressed values for exact zeros.
+- The compression codec attenuates power above roughly 20-30 Hz.
+- Per-channel impedance/analog-filter differences show up as amplitude
+  outliers below ~1.5 Hz.
+
+See [docs/bwm/lfp.md](docs/bwm/lfp.md) for the full dataset spec.
+
+---
+
 ## [bwm_behavior 2.0.0] - 2026-07-12
 
 ### Added

README.md

@@ -117,6 +117,9 @@ Yes. It will download the main compressed BWM data, and may offer to download mo
 **How much disk space do the public BWM derived datasets need?**
 
 Less than 10 GB for all spikes from good units and behavior data from the BWM.
+A compressed LFP dataset (`bwm_lfp`, ~14 GB) is also available but opt-in —
+it is not downloaded by default; fetch it with
+`scripts/download_datasets.py --lfp`.
 
 See [docs/bwm/README.md](docs/bwm/README.md) for more details. You
 will also need working space for generated artifacts.

data_locations.example.yaml

@@ -14,5 +14,12 @@ datasets:
     root:
     preferred_version: latest
 
+  bwm_lfp:
+    # Opt-in, ~14 GB. Directory containing version folders such as 1.0.0/,
+    # or a specific dataset directory that contains schema.yaml.
+    # Download with: uv run python scripts/download_datasets.py --lfp
+    root:
+    preferred_version: latest
+
 one_cache:
   root:

docs/CONTRIBUTING.md

@@ -6,6 +6,9 @@
 UV_CACHE_DIR=.uv-cache uv sync --extra ibl --extra notebook
 ```
 
+Add `--extra lfp` if you're working on `bwm_lfp` (needed to import `lfpack`,
+e.g. for `scripts/validate_bwm_lfp_release.py`'s deeper check).
+
 ## Quality gates
 
 ```bash
@@ -45,6 +48,27 @@ Checklist before uploading a new `<dataset>-<version>.tar`:
    ```
    The script checks version strings, manifest completeness, row/column counts, and array shapes.
 
+## Releasing a new single-file dataset (e.g. `bwm_lfp`)
+
+`bwm_lfp` isn't a tar built by this repo's own pipeline — it's a single `.h5`
+file produced upstream by `lfpack`. `scripts/download_datasets.py` authors
+its `schema.yaml`/`provenance.yaml`/`manifest.json` itself at download time
+(there is nothing to unpack). To pick up a new upstream release:
+
+1. Bump `BWM_LFP_VERSION` in `scripts/download_datasets.py`.
+2. Update `LFP_STANDARD`'s `url` and `sha1` (`shasum -a 1
+   lf_compressed_all_bwm.h5`) to the new upstream file. Only the standard
+   compression tier is exposed; do not add an aggressive-tier spec.
+3. Add a `CHANGELOG_DATA.md` entry under a new `[bwm_lfp <version>]` heading.
+4. Run the release validator against the freshly downloaded version dir:
+   ```bash
+   UV_CACHE_DIR=.uv-cache uv run python scripts/validate_bwm_lfp_release.py \
+       reports/datasets/bwm_lfp/<version>
+   ```
+   With `lfpack` installed (`uv sync --extra lfp`), it also opens the file
+   and checks the recording count; without it, that deeper check is skipped
+   with a note.
+
 ## Docs policy
 
 Authoritative docs are:

docs/bwm/README.md

@@ -106,6 +106,39 @@ cell-level waveform/ACG surfaces for analyses that need them.
 Best detailed references:
 - [Dataset spec](./ephys.md)
 
+## `bwm_lfp`
+
+Local compressed LFP dataset: all 699 BWM probe recordings in a single HDF5
+file, read via `lfpack.LFPackReader` (a drop-in for `spikeglx.Reader`).
+Opt-in — not downloaded by the default `download_datasets.py` run because of
+its size.
+
+- Config key: `datasets.bwm_lfp.root`
+- Version: `1.0.0`
+- Compression tier: `standard` only (the `lfpack` package also produces an
+  `aggressive` tier for its own uses; it is intentionally not distributed to
+  the agent)
+- Approx. size on disk: `14G`
+- Recordings: `699` (one per probe insertion, keyed by `pid`)
+- Channels per recording: `384` for 695 recordings; `96` for four NP2.4
+  recordings
+- Sample rate: `250 Hz` (decimated from `2500 Hz`)
+- Main contents:
+  - `lf_compressed_all_bwm.h5` — all 699 recordings, per-channel brain-region
+    annotations, saturation QC, and session-clock sync where available
+  - `schema.yaml` / `provenance.yaml` / `manifest.json` — authored by
+    `scripts/download_datasets.py` at download time (not shipped by `lfpack`)
+
+Requires the `lfp` extra to read (`uv sync --extra lfp`); download with:
+
+```bash
+UV_CACHE_DIR=.uv-cache uv run python scripts/download_datasets.py --lfp
+```
+
+Best detailed references:
+- [Dataset spec](./lfp.md)
+- `lfpack`'s own how-to: <https://int-brain-lab.github.io/lfpack/how-to/bwm-dataset.html>
+
 ## Proposed Compact Query Layer
 
 Many broad questions about where task, movement, pose, or behavioral
@@ -119,5 +152,5 @@ Design references:
 
 ## Supporting Docs
 
-See the detailed behavior and ephys pages in this directory for schema notes and
-loading guidance.
+See the detailed behavior, ephys, and LFP pages in this directory for schema
+notes and loading guidance.

docs/bwm/lfp.md

@@ -0,0 +1,101 @@
+# BWM LFP Dataset Specification
+
+## Status
+
+`bwm_lfp` is a local, compressed companion to `bwm_ephys`/`bwm_behavior`
+covering the raw LFP band. Unlike those two datasets, it is not built by this
+repo: the archive is produced upstream by the `lfpack` package (in the
+`ephys-atlas` repo) and this repo only downloads, verifies, and registers it
+so it can be discovered through the same `data_locations.local.yaml` /
+`ibl_ai_agent.data_locations.resolve_dataset_dir` path as every other BWM
+dataset.
+
+It is opt-in: `scripts/download_datasets.py` does not fetch it by default
+(unlike `bwm_ephys`/`bwm_behavior`) because of its size (~14 GB). Fetch and
+register it with:
+
+```bash
+UV_CACHE_DIR=.uv-cache uv run python scripts/download_datasets.py --lfp
+```
+
+Reading the file needs the `lfp` extra, not installed by default:
+
+```bash
+UV_CACHE_DIR=.uv-cache uv sync --extra lfp
+```
+
+Dataset identity:
+- Name: `bwm_lfp`
+- Current version: `1.0.0`
+- Config key: `datasets.bwm_lfp.root`
+- Compression tier: `standard` only. `lfpack` also produces an `aggressive`
+  tier (higher compression, more high-frequency roll-off); it is not
+  distributed to the agent.
+
+## Layout
+
+```text
+bwm_lfp/
+  1.0.0/
+    lf_compressed_all_bwm.h5
+    schema.yaml
+    provenance.yaml
+    manifest.json
+```
+
+`schema.yaml`, `provenance.yaml`, and `manifest.json` are authored by
+`scripts/download_datasets.py` at download time (see
+`_write_lfp_sidecars`) — `lfpack` ships only the `.h5` file itself. This
+keeps `bwm_lfp` schema-discoverable the same way as the other BWM datasets,
+even though nothing here is built by this repo's own pipeline.
+
+## Contents
+
+One HDF5 file holding all `699` BWM probe recordings, keyed by `pid`:
+- `384` channels for `695` recordings and `96` channels for four NP2.4
+  recordings from subject `NR_0029`
+- `250 Hz` sample rate (decimated from `2500 Hz`)
+- per-channel brain-region annotations (`acronym`, `atlas_id`, MNI coordinates)
+- per-recording saturation (ADC-clipping) QC table
+- a sync-corrected session-clock time axis (`sr.times`) when `sr.t0` is finite;
+  seven recordings without sync instead expose recording-relative times
+
+## Reading
+
+Use `lfpack.LFPackReader` — a drop-in for `spikeglx.Reader` that decompresses
+chunks on demand and never loads the whole file into memory. Do not restate
+the API here; see:
+- `skills/ibl-neuropixel/references/neuropixel_routing.md` and
+  `neuropixel_function_signatures.md` for agent-facing routing
+- `lfpack`'s own how-to doc:
+  <https://int-brain-lab.github.io/lfpack/how-to/bwm-dataset.html>
+
+## Known Limitations
+
+Carried over from `lfpack`'s own release notes — check these before trusting
+a result:
+
+- **Saturation events**: ADC-clipped stretches are detected and muted before
+  lossy compression. Decompressed values in those stretches are not guaranteed
+  to be exactly zero, so use `sr.saturation_mask`/`sr.saturation_times()` rather
+  than testing sample values. In this release, 121 of 699 recordings have more
+  than 1% saturated source samples and 16 exceed 10% (maximum 23.9%).
+- **Missing sync for 7 probes**: `t0_sync`/`fs_sync` could not be computed
+  for 7 probes across 4 sessions due to an upstream session-level sync
+  data-quality issue; `sr.t0` returns `NaN` and `sr.times` is
+  recording-relative for these
+  ([lfpack#8](https://github.com/int-brain-lab/lfpack/issues/8)).
+- **Saturation interval boundaries**: interval stop times can extend up to one
+  decimated sample beyond `sr.times[-1]` because intervals are rounded
+  outwards; `sr.saturation_mask` clips them safely.
+- **High-frequency roll-off**: the SVD + wavelet-packet codec trades off
+  power above roughly 20-30 Hz for compression ratio, more so at the
+  (undistributed) aggressive tier.
+- **Low-frequency channel-to-channel mismatch**: per-channel
+  impedance/analog-filter differences show up as amplitude outliers below
+  ~1.5 Hz and are not corrected by the current pre-processing pipeline.
+
+## Related
+
+- [Dataset layering decision](../decisions/bwm_dataset_layering.md)
+- [BWM dataset overview](./README.md)

docs/data_locations.md

@@ -16,6 +16,18 @@ The script downloads the public `bwm_ephys` and `bwm_behavior` archives into
 `reports/datasets/` and writes `data_locations.local.yaml`. Approximate download
 sizes are listed in `docs/bwm/README.md`.
 
+The compressed LFP dataset (`bwm_lfp`, ~14 GB, standard-compression tier) is
+opt-in and not part of this default flow, since most sessions never need raw
+LFP access. Fetch and register it separately with:
+
+```bash
+UV_CACHE_DIR=.uv-cache uv run python scripts/download_datasets.py --lfp
+```
+
+Reading it also needs the `lfp` extra (`uv sync --extra lfp`) for the
+`lfpack` package — see `docs/bwm/README.md` and
+`skills/ibl-neuropixel/references/neuropixel_routing.md`.
+
 If the datasets are already present under `reports/datasets/`, the script skips
 the download and refreshes the local config. Runtime loading also auto-detects
 valid BWM datasets in the default `reports/datasets/<dataset_name>/` location
@@ -64,6 +76,9 @@ datasets:
   bwm_behavior:
     root: reports/datasets/bwm_behavior
     preferred_version: latest
+  bwm_lfp:
+    root: reports/datasets/bwm_lfp
+    preferred_version: latest
 one_cache:
   root: "C:/Users/<user>/Downloads/ONE"
 ```
@@ -97,6 +112,7 @@ from ibl_ai_agent.data_locations import resolve_dataset_dir
 
 bwm_ephys_dir = resolve_dataset_dir("bwm_ephys")
 bwm_behavior_dir = resolve_dataset_dir("bwm_behavior")
+bwm_lfp_dir = resolve_dataset_dir("bwm_lfp")
 ```
 
 If no local data location is configured, BWM agents may offer to run the public

ibl_ai_agent/data_locations.py

@@ -18,11 +18,19 @@
         "root": REPO_ROOT / "reports" / "datasets" / "bwm_ephys",
         "size": "about 5 GB",
         "why": "local spike shards, unit/session metadata, and passive ephys tables avoid slow per-session ONE loading",
+        "download_hint": "UV_CACHE_DIR=.uv-cache uv run python scripts/download_datasets.py",
     },
     "bwm_behavior": {
         "root": REPO_ROOT / "reports" / "datasets" / "bwm_behavior",
         "size": "about 3.5 GB",
         "why": "local trial, wheel, movement-state, pose, and behavior feature tables avoid slow per-session ONE loading",
+        "download_hint": "UV_CACHE_DIR=.uv-cache uv run python scripts/download_datasets.py",
+    },
+    "bwm_lfp": {
+        "root": REPO_ROOT / "reports" / "datasets" / "bwm_lfp",
+        "size": "about 14 GB (opt-in)",
+        "why": "the compressed LFP store lets the agent read any of the 699 BWM recordings without a raw ONE/SpikeGLX download",
+        "download_hint": "UV_CACHE_DIR=.uv-cache uv run python scripts/download_datasets.py --lfp",
     },
 }
 
@@ -202,6 +210,7 @@ def _missing_bwm_dataset_message(name: str, *, configured_root: Path | None) ->
         f"For BWM analyses, stop before falling back to ONE/session loaders. "
         f"Ask the user whether to download/configure {name}: it is {info['size']} and is expected by default under "
         f"{_display_path(expected)}. This is needed because {info['why']}. "
+        f"Download with: {info['download_hint']}. "
         "Alternatives are: point data_locations.local.yaml at an existing dataset root, run the public downloader, "
         "or explicitly use ONE/session loaders only after explaining which required field is missing from the local dataset."
     )

pyproject.toml

@@ -29,6 +29,9 @@ ibl = [
   "ibllib>=3.3",
   "brainbox<=0.0.9",
 ]
+lfp = [
+  "lfpack>=0.1.0",
+]
 notebook = [
   "jupyter>=1.1",
   "jupyter-mcp-server>=0.1",
@@ -38,7 +41,9 @@ notebook = [
 dev = [
   "pytest>=8.3",
   "pytest-cov>=5.0",
-  "ruff>=0.6",
+  # ruff 0.16.0 changed its default rule set and surfaces ~368 pre-existing
+  # violations across unrelated files; cap until those are cleaned up separately.
+  "ruff>=0.6,<0.16",
   "mypy>=1.11",
 ]
 
@@ -66,6 +71,11 @@ target-version = "py310"
 constraint-dependencies = [
   "PyQt5-Qt5==5.15.2; sys_platform == 'win32'",
 ]
+required-environments = [
+  "sys_platform == 'darwin' and platform_machine == 'arm64'",
+  "sys_platform == 'linux' and platform_machine == 'x86_64'",
+  "sys_platform == 'win32'",
+]
 
 [tool.uv.sources]
 brainwidemap = { git = "https://github.com/int-brain-lab/paper-brain-wide-map.git", rev = "de37454426bc4107047b5a18b854fc1ef7a5959c" }