Enveda-180: generation of a large open multimodal MS/MS and ion mobility spectral library for drug-like small molecules

This repository contains the code and notebooks for the Enveda-180 project, which generates a large open multimodal MS/MS + ion mobility spectral library for drug-like small molecules from the Enamine REAL space. The project is a collaboration between Enveda and the scientific community, and we welcome contributions and feedback.

Citation

Krettler, C., Kind, T., DeBloois, E., Killian, M., Prince, J., Taylor, J., Gadiya, Y., Simpson, P., Domingo-Fernández, D., Allen, A., Colluru, V., Healey, D. (2026). Enveda-180: generation of a large open multimodal MS/MS and ion mobility spectral library for drug-like small molecules. ChemRxiv. https://doi.org/10.26434/chemrxiv.15004319/v1

Repository layout

enveda-180/
  data_processing/        # Python package: ms_prepper + extraction + curation
  scripts/                # CLI runners (build manifest, run ms_prepper, run pipeline)
  notebooks/              # process.ipynb — papermill-driven pipeline entry point
  tests/                  # Unit tests
  docs/                   # Design / agent notes
  pyproject.toml          # Hatchling build; the package installs as `data_processing`
  README.md, LICENSE

The package lives at data_processing/ and is installed under the import name data_processing.

Install

uv sync
# or, with pip:
pip install -e .

Python 3.11–3.13 is supported.

Pipeline overview

The data_processing package has two halves:

1. data_processing.ms_prepper — read a Bruker .d folder and write four per-msrun parquets (ms1, ms2, ms2_event, frame).
2. data_processing.{extraction, curation, pipeline} — targeted extraction of MS/MS spectra for each compound-pool member across all common adducts, plus quality-curation helpers.

Adducts targeted by the extraction pipeline

The default adduct list is surfaced at the package root:

from data_processing import COMMON_ADDUCTS, CATION_ADDUCTS

Positive mode: [M+H]+, [M+Na]+, [M+NH4]+, [M+K]+, [2M+Na]+, [2M+H]+, [M-H2O+H]+, [M-H4O2+H]+

Negative mode: [M-H]-, [M+Cl]-, [M+FA-H]-, [M+Br]-, [M+Hac-H]-, [2M-H]-, [2M+FA-H]-, [2M+Hac-H]-, [M-H2O-H]-

Cation (natively charged) mode: [M]+

All exact monoisotopic offsets are computed from the adduct name strings via molmass in data_processing.extraction.load.compute_adduct_metadata. To add an adduct, append its name to COMMON_ADDUCTS (or CATION_ADDUCTS) in data_processing/extraction/load.py.

Quick start

1. Convert a Bruker .d to parquets

create-signal-tables --input /path/to/sample.d --output-dir /data/enamine_prep

or from Python:

from data_processing.ms_prepper import pipeline
pipeline.run(input_path="/path/to/sample.d", output_dir="/data/enamine_prep")

This writes four parquets named {msrun_id}.{ms1,ms2,ms2_event,frame}.parquet.

2. Configure where the inputs and outputs live

Set environment variables (a .env file at the repo root is auto-loaded):

ENAMINE_PROCESSING_DATA_PATH=/data/data_processing
ENAMINE_PREP_DATA_PATH=/data/data_processing/input/enamine_prep
ENAMINE_PROCESSING_LOG_LEVEL=INFO

Expected input layout:

${ENAMINE_PROCESSING_DATA_PATH}/
  input/
    enamine_prep/
      {msrun_id}.ms1.parquet
      {msrun_id}.ms2.parquet
      {msrun_id}.ms2_event.parquet
      {msrun_id}.frame.parquet
    preprocessed/
      <my-batch>/
        compound_metadata.parquet
        compound_pool_id_to_msrun_ids_df.parquet
  output/
    <my-batch>/
      <version>/
        extracted/...
        annotated/...
        annotated_matched/...
        final/...

3. Run the extraction + curation pipeline

import asyncio
from data_processing import pipeline

asyncio.run(pipeline.run(
    name="my-batch",
    version="v0.1",
    log_path="/tmp/enamine-processing-run",
))

Or call the headline extraction function directly:

from pathlib import Path
from data_processing.pipeline import extract_spectra

spectra = extract_spectra(
    compound_pool_id_to_msrun_ids_df_path=Path(".../compound_pool_id_to_msrun_ids_df.parquet"),
    compound_metadata_df_path=Path(".../compound_metadata.parquet"),
    export_folder=Path(".../output/my-batch/v0.1"),
    version="v0.1",
)

Running the 14 enamine batches end-to-end

scripts/ contains the end-to-end runners used to produce the Enveda-180 library:

# 1. Build a manifest mapping MSB ids -> raw .d filenames for every batch.
#    (Enveda-internal: loads data from Databricks.)
python scripts/build_run_manifest.py --all

# 2. Download the .d folders listed in the manifest to a local directory.
#    (Use whatever tool your environment provides — azcopy, gsutil, rclone, ...)

# 3. Run ms_prepper on every .d folder for one batch.
python scripts/run_ms_prepper.py \
    --batch enamine-part1 \
    --raw-dir /scratch/enamine/raw \
    --output-dir $ENAMINE_PREP_DATA_PATH \
    --skip-existing

# 4. Run the extraction + curation pipeline.
python scripts/run_pipeline.py --batch enamine-part1 --version v0.1

External users without Enveda Databricks access should skip step 1 and hand-author raw_file_manifest.parquet with at least the columns msrun_id, raw_file.

Notebooks

notebooks/process.ipynb is the canonical papermill-driven entry point for running the extraction + curation pipeline against one preprocessed batch. It is the notebook-flavoured equivalent of scripts/run_pipeline.py — pick whichever you prefer.

name="my-batch"
version=$(date +%b-%d-%Y)
folder_path="$ENAMINE_PROCESSING_DATA_PATH/logs/${name}"
log_path="${folder_path}/${version}"
mkdir -p "$folder_path"

papermill notebooks/process.ipynb "${log_path}.ipynb" \
    -p name "$name" \
    -p version "$version" \
    -p log_path "$log_path"

Public configuration surface

Every tunable knob is exposed in data_processing.config and re-exported from the package root:

Name	Source	Description
COMMON_ADDUCTS	extraction.load	Adducts searched per compound.
CATION_ADDUCTS	extraction.load	Adducts for natively charged species.
DATA_PATH	extraction.constants	Root data directory ($ENAMINE_PROCESSING_DATA_PATH).
ENAMINE_PREP_DATA_PATH	extraction.constants	ms_prepper parquet directory.
ENAMINE_PREP_VERSION	extraction.constants	Stamped on every extracted spectrum.
INDEX_COLUMNS	curation.constants	["msrun_id", "msms_aggregate_id"].
HASH_PRECISION	curation.constants	Spectrum-hash mz rounding (default 1).
TOP_N, TOLERANCE_DA	curation.constants	Library-match knobs.
EXPORT_KEYS, DS8_EXPORT_KEYS	curation.constants	Columns written to disk.

SIRIUS annotation

SIRIUS annotation is not invoked automatically. Run SIRIUS yourself (https://bio.informatik.uni-jena.de/software/sirius/) on the extracted jsonl output and merge the resulting columns into the dataframe before continuing. All downstream curation code reads SIRIUS columns defensively, so the pipeline still completes without them.

License

MIT. See LICENSE.