Merge pull request #13 from int-brain-lab/release/v0.2.0

Release v0.2.0: bwm_ephys 1.2.0 with waveforms and ACGs

Author: rossant

CHANGELOG.md

@@ -0,0 +1,39 @@
+# Changelog
+
+All notable changes to the IBL AI Agent are documented here.
+
+The format follows [Keep a Changelog](https://keepachangelog.com/en/1.1.0/).
+This project adheres to [Semantic Versioning](https://semver.org/).
+
+---
+
+## [Unreleased]
+
+---
+
+## [0.2.0] - 2026-06-08
+
+### Added
+- `bwm_ephys` dataset version 1.2.0 with waveforms, additional waveform features for each cluster and autocorrelograms.
+See `CHANGELOG_DATA.md` for further details.
+
+### Changed
+- `scripts/download_datasets.py`: `bwm_ephys` archive updated to version 1.2.0
+  (new filename, SHA1, and URL).
+- Skill references updated to document the new cell-level files and loading
+  guidance for `bwm_ephys ≥ 1.2.0`.
+
+---
+
+## [0.1.0] - 2026-02 *(initial release)*
+
+### Added
+- Initial agent scaffold with `bwm_ephys 1.1.0` and `bwm_behavior 1.1.0` datasets.
+- Skill system for IBL data loading, analysis, anatomy, and Neuropixels access.
+- `scripts/download_datasets.py` for bootstrapping public BWM archives.
+
+---
+
+[Unreleased]: https://github.com/int-brain-lab/ibl-ai-agent/compare/v0.2.0...HEAD
+[0.2.0]: https://github.com/int-brain-lab/ibl-ai-agent/compare/v0.1.0...v0.2.0
+[0.1.0]: https://github.com/int-brain-lab/ibl-ai-agent/releases/tag/v0.1.0

CHANGELOG_DATA.md

@@ -0,0 +1,52 @@
+# Data Changelog
+
+All notable changes to the IBL AI Agent datasets are documented here.
+
+The format follows [Keep a Changelog](https://keepachangelog.com/en/1.1.0/).
+Dataset versions follow [Semantic Versioning](https://semver.org/).
+
+---
+
+## [Unreleased]
+
+---
+
+## [bwm_ephys 1.2.0] - 2026-06-08
+
+### Added
+- `clusters.pqt` (621 733 × 59): full-release BWM unit table covering all 699 probe
+  insertions. Replaces the `2024_Q2_IBL_et_al_BWM` aggregate `clusters.pqt`.
+  Includes `eid` and 25 new columns relative to the 2024_Q2 aggregate:
+  - Firing statistics: `burstiness`, `memory`
+  - QC / annotation: `labels`, `rawInd`, `peak_channel`, `invert_sign_peak`
+  - Waveform shape: `peak_time_idx`, `peak_val`, `trough_time_idx`, `trough_val`,
+    `tip_time_idx`, `tip_val`
+  - Waveform timing: `peak_to_trough_duration`, `half_peak_duration`,
+    `half_peak_post_time_idx`, `half_peak_pre_time_idx`,
+    `half_peak_post_val`, `half_peak_pre_val`,
+    `recovery_time_idx`, `recovery_val`
+  - Waveform slopes / ratios: `peak_to_trough_ratio`, `peak_to_trough_ratio_log`,
+    `depolarisation_slope`, `repolarisation_slope`, `recovery_slope`
+- `clusters.waveforms_peak.npy` (621 733 × 128, float16): peak-channel waveform
+  per unit at 30 kHz. Row-aligned with `clusters.pqt`.
+- `clusters.acgs_log.npy` (621 733 × 128, float16): log-binned autocorrelogram
+  per unit. Row-aligned with `clusters.pqt`.
+- `acgs_log.times.npy` (128, float64): shared ACG lag-time axis in seconds.
+- generation code on IBL cluster is available [here](https://github.com/int-brain-lab/sdsc-slurms/blob/main/2026-03_EA_Cells/cells.py), leveraging the [eatools](https://github.com/int-brain-lab/eatools) library.
+
+
+
+---
+
+## [bwm_ephys 1.1.0] - 2026-02 *(baseline)*
+
+Initial public release of the `bwm_ephys` archive containing:
+- `metadata/`: sessions, insertions, units, channels, trials, events, passive sessions/events.
+- `features/`: unit features, event-response features, passive response features.
+- `spikes/`: per-probe blosc-compressed spike shards (good units only).
+
+---
+
+[Unreleased]: https://github.com/int-brain-lab/ibl-ai-agent/compare/v0.2.0...HEAD
+[bwm_ephys 1.2.0]: https://github.com/int-brain-lab/ibl-ai-agent/releases/tag/v0.2.0
+[bwm_ephys 1.1.0]: https://github.com/int-brain-lab/ibl-ai-agent/releases/tag/v0.1.0

docs/CONTRIBUTING.md

@@ -20,6 +20,31 @@ UV_CACHE_DIR=.uv-cache uv run pytest -q
 - Prefer typed contracts over untyped dict plumbing in runtime internals.
 - Keep CLI commands thin; shared behavior should live in `commands/common.py` or `commands/kernel.py`.
 
+## Changelog and versioning
+
+This project follows documents evolutions in two changelogs.
+
+- **`CHANGELOG.md`** — agent/code changes (version bumps, new features, skill updates).
+- **`CHANGELOG_DATA.md`** — dataset changes (new archive versions, schema additions, column changes).
+
+Dataset versions are independent of the agent version; both use semver. A dataset minor bump (e.g. 1.1.0 → 1.2.0) adds columns or new files without breaking existing queries.
+
+## Releasing a new dataset archive
+
+When bumping a dataset version (e.g. `bwm_ephys`), the archive's own metadata files must match the new version **before** the tar is uploaded to S3. Forgetting this is the most common release mistake.
+
+Checklist before uploading a new `<dataset>-<version>.tar`:
+
+1. **`provenance.yaml`** inside the archive — set `dataset_version: <new_version>`.
+2. **`manifest.json`** inside the archive — set `dataset_version: <new_version>` and ensure the `files` list includes every new file added in this release.
+3. **`schema.yaml`** — set `dataset_version: <new_version>` and add entries for any new tables or stores.
+4. Repack the tar, compute its SHA1 (`shasum -a 1 <archive>.tar`), and update the `sha1` field in `scripts/download_datasets.py`.
+5. Run the release validator against the extracted archive before pushing:
+   ```bash
+   uv run python scripts/validate_bwm_ephys_release.py <path/to/extracted/version>
+   ```
+   The script checks version strings, manifest completeness, row/column counts, and array shapes.
+
 ## Docs policy
 
 Authoritative docs are:

docs/bwm/README.md

@@ -58,13 +58,15 @@ Local ephys-centered BWM dataset with spikes, units, insertions, channels,
 trials, events, and passive-response context.
 
 - Config key: `datasets.bwm_ephys.root`
-- Version: `1.1.0`
-- Approx. size on disk: `5.3G`
+- Version: `1.2.0`
+- Public archive size: `6.03G`
+- Previous `1.1.0` local-table/spike footprint: `5.3G`
 - Approx. file count: `4,215`
 - Mice: `139`
 - Sessions: `459`
 - Insertions: `699`
 - Units: `75,395`
+- Full cluster rows in root `clusters.pqt`: `621,733`
 - Channels: `267,264`
 - Trials: `295,920`
 - Events: `2,066,041`
@@ -82,6 +84,15 @@ trials, events, and passive-response context.
   - `features/event_response_features.parquet`
   - `features/passive_response_features.parquet`
   - `spikes/<pid>/...` blosc shard directories
+  - `clusters.pqt` full cluster table
+  - `clusters.waveforms_peak.npy` peak-channel unit waveforms
+  - `clusters.acgs_log.npy` log-binned autocorrelograms
+  - `acgs_log.times.npy` shared ACG lag-time axis
+
+Compatibility note: the `metadata/units.parquet`, `features/*`, and `spikes/*`
+surfaces keep the good-unit local analysis contract from `1.1.0`. The new
+root-level `clusters.pqt` and companion arrays expose the full cluster table and
+cell-level waveform/ACG surfaces for analyses that need them.
 
 Best detailed references:
 - [Dataset spec](./ephys.md)

docs/bwm/ephys.md

@@ -31,7 +31,9 @@ located through `data_locations.local.yaml`, so this page should be read as a mi
 Reference local artifact snapshot:
 
 - config key: `datasets.bwm_ephys.root`
-- final total size on disk: `5.3G`
+- current dataset version: `1.2.0`
+- public archive size: `6.03G`
+- previous `1.1.0` local-table/spike footprint: `5.3G`
 - build timestamp: `2026-04-22T18:14:47.760768+00:00`
 - upgraded from dataset version: `1.0.0`
 - source freeze: `2023_12_bwm_release`
@@ -40,6 +42,7 @@ Reference local artifact snapshot:
 - sessions: `459`
 - insertions: `699`
 - units: `75,395`
+- full cluster rows in root `clusters.pqt`: `621,733`
 - channels: `267,264`
 - trials: `295,920`
 - events: `2,066,041`
@@ -62,6 +65,13 @@ Important implementation note:
   - `100 us`
   - adaptive `uint16` / `uint32`
   - container format: `blosc_file_shards`
+- `bwm_ephys 1.2.0` additionally ships root-level full-cluster files:
+  - `clusters.pqt`
+  - `clusters.waveforms_peak.npy`
+  - `clusters.acgs_log.npy`
+  - `acgs_log.times.npy`
+- `metadata/units.parquet` remains the good-unit table used with local spike
+  shards; root `clusters.pqt` is the full cluster table.
 
 Where this document differs from the configured local `bwm_ephys` artifact,
 treat the built artifact as the source of truth.
@@ -83,6 +93,7 @@ features. See `docs/decisions/bwm_dataset_layering.md`.
 
 Dataset identity:
 - Name: `bwm_ephys`
+- Current public version: `1.2.0`
 - Initial target version: `1.0.0`
 
 Pinned source scope (initial intent):
@@ -98,7 +109,7 @@ Core product definition:
 - Does not claim to include all possible BWM modalities such as raw video
 
 Recommended distribution artifact:
-- `reports/releases/bwm_ephys/1.1.0/bwm_ephys-1.1.0.tar`
+- `reports/releases/bwm_ephys/1.2.0/bwm_ephys-1.2.0.tar`
 
 The latest `bwm_ephys` archive is generated by the passive upgrade path and
 written to that release folder automatically. To write or refresh only the
@@ -173,6 +184,11 @@ bwm_ephys/
     build_report.yaml
     SUMMARY.md
 
+    clusters.pqt
+    clusters.waveforms_peak.npy
+    clusters.acgs_log.npy
+    acgs_log.times.npy
+
     metadata/
       sessions.parquet
       insertions.parquet
@@ -199,6 +215,10 @@ bwm_ephys/
 
 Notes:
 - The current local artifact populates:
+  - `clusters.pqt`
+  - `clusters.waveforms_peak.npy`
+  - `clusters.acgs_log.npy`
+  - `acgs_log.times.npy`
   - `metadata/sessions.parquet`
   - `metadata/insertions.parquet`
   - `metadata/units.parquet`

docs/data_locations.md

@@ -72,6 +72,12 @@ one_cache:
 - a directory containing version folders such as `1.1.0/schema.yaml`; or
 - one exact dataset directory containing `schema.yaml`.
 
+For automatic dataset updates, prefer a version-parent root such as
+`reports/datasets/bwm_ephys` with `preferred_version: latest`. If `root` points
+directly at an exact version such as `reports/datasets/bwm_ephys/1.1.0`, the
+downloader treats that as a manual pin and will not install a newer version into
+that path.
+
 Use a project-level override only when a project has a special data location:
 
 ```text

pyproject.toml

@@ -4,7 +4,7 @@ build-backend = "setuptools.build_meta"
 
 [project]
 name = "ibl-ai-agent"
-version = "0.1.0"
+version = "0.2.0"
 description = "Skill-first Codex agent for IBL neural data analysis"
 readme = "README.md"
 requires-python = ">=3.10"