feat: Add docs explaining engineering vs research

Author: JavRedstone

README.md

@@ -114,6 +114,13 @@ python src/train.py +experiment=long_run   # apply a preset
 
 Hydra saves the full resolved config alongside every run for reproducibility.
 
+## Docs
+
+- [docs/research.md](docs/research.md) — research workflow: MLflow, multirun sweeps, reproducibility checklist
+- [docs/engineering.md](docs/engineering.md) — engineering workflow: locking configs, CI, inference
+- [docs/experiment-tracking.md](docs/experiment-tracking.md) — TensorBoard vs MLflow vs W&B
+- [docs/data.md](docs/data.md) — data formats, Parquet guide, adding custom datasets
+
 ## License
 
 MIT

data/README.md

@@ -1,74 +1,5 @@
 # data/
 
-Raw and processed datasets. This directory is gitignored — do not commit data files.
+Raw and processed datasets. Gitignored — do not commit data files.
 
-## Convention
-
-```
-data/
-├── cifar-10-batches-py/   # auto-downloaded by torchvision
-├── custom/                # your custom dataset
-│   ├── train/
-│   ├── val/
-│   └── test/
-└── raw/                   # unprocessed source data
-```
-
-## Recommended data format: Parquet (via PyArrow)
-
-For tabular or structured data, use **Parquet** instead of CSV. It's faster to read, smaller on disk, and preserves column types.
-
-### Saving data as Parquet
-
-```python
-import pandas as pd
-
-df = pd.DataFrame({"feature1": [1, 2, 3], "label": ["cat", "dog", "cat"]})
-
-# Save — uses pyarrow backend automatically
-df.to_parquet("data/my_dataset.parquet")
-```
-
-### Loading Parquet data
-
-```python
-import pandas as pd
-
-# Single file
-df = pd.read_parquet("data/my_dataset.parquet")
-
-# Specific columns only (fast — Parquet is columnar)
-df = pd.read_parquet("data/my_dataset.parquet", columns=["feature1", "label"])
-
-# Multiple files / partitioned dataset
-df = pd.read_parquet("data/my_dataset/")
-```
-
-### Converting CSV to Parquet
-
-```python
-df = pd.read_csv("data/raw/data.csv")
-df.to_parquet("data/processed/data.parquet")
-```
-
-### Why Parquet over CSV
-
-| | CSV | Parquet |
-|---|---|---|
-| Read speed | Slow (parsed line-by-line) | Fast (columnar, zero-copy) |
-| File size | Large (text) | Small (compressed binary) |
-| Column types | Lost (everything is strings) | Preserved (int, float, datetime) |
-| Column selection | Must read entire file | Reads only requested columns |
-
-### Other supported formats
-
-- **Excel** (`.xlsx`): `pd.read_excel("file.xlsx")` — requires `openpyxl`
-- **HDF5** (`.h5`): `pd.read_hdf("file.h5")` — requires `h5py`, common in research
-- **CSV**: `pd.read_csv("file.csv")` — use for small files or one-off imports
-
-## Where to store data
-
-- **Small datasets** (CIFAR-10, MNIST): auto-downloaded by torchvision into this folder
-- **Medium datasets**: download manually, place here, document the source in your experiment config
-- **Large datasets** (ImageNet, COCO): store on a shared drive or cluster filesystem, symlink into this folder
-- **Google Drive** (Colab): mount and symlink, or set `dataset.path` in your config to the Drive path
+See [docs/data.md](../docs/data.md) for format recommendations, Parquet usage, and how to add a custom dataset to the pipeline.

docs/data.md

@@ -0,0 +1,108 @@
+# Data Guide
+
+## Directory layout
+
+```
+data/
+├── cifar-10-batches-py/   # auto-downloaded by torchvision
+├── custom/                # your custom dataset
+│   ├── train/
+│   ├── val/
+│   └── test/
+└── raw/                   # unprocessed source data
+```
+
+`data/` is gitignored. Never commit data files.
+
+## Where to get data
+
+- **Small standard datasets** (CIFAR-10, MNIST, ImageNet subsets) — auto-downloaded by torchvision on first run
+- **Custom datasets** — download manually, place in `data/`, document the source in your dataset config as a comment
+- **Large datasets** (ImageNet, COCO) — store on a shared drive or cluster filesystem, set `dataset.path` in your config to that location
+
+## Recommended format: Parquet
+
+For tabular or structured data, use Parquet instead of CSV. It's faster, smaller, and preserves column types.
+
+### Save
+
+```python
+import pandas as pd
+
+df = pd.DataFrame({"feature1": [1, 2, 3], "label": ["cat", "dog", "cat"]})
+df.to_parquet("data/my_dataset.parquet")
+```
+
+### Load
+
+```python
+import pandas as pd
+
+# Single file
+df = pd.read_parquet("data/my_dataset.parquet")
+
+# Specific columns only (fast — Parquet is columnar)
+df = pd.read_parquet("data/my_dataset.parquet", columns=["feature1", "label"])
+
+# Partitioned dataset (multiple files in a folder)
+df = pd.read_parquet("data/my_dataset/")
+```
+
+### Convert from CSV
+
+```python
+df = pd.read_csv("data/raw/data.csv")
+df.to_parquet("data/processed/data.parquet")
+```
+
+### Why Parquet over CSV
+
+| | CSV | Parquet |
+|---|---|---|
+| Read speed | Slow (text parsing) | Fast (columnar, binary) |
+| File size | Large | Small (compressed) |
+| Column types | Lost (all strings) | Preserved |
+| Column selection | Reads entire file | Reads only what you ask for |
+
+## Other formats
+
+| Format | Read | When to use |
+|---|---|---|
+| Excel `.xlsx` | `pd.read_excel("file.xlsx")` | Data from non-technical collaborators |
+| HDF5 `.h5` | `pd.read_hdf("file.h5")` | Large arrays, research datasets |
+| NumPy `.npy` / `.npz` | `np.load("file.npy")` | Pre-processed tensor data |
+| CSV | `pd.read_csv("file.csv")` | Small files, one-off imports only |
+
+## Adding a custom dataset to the training pipeline
+
+1. Create `configs/dataset/my_dataset.yaml`:
+
+```yaml
+name: my_dataset
+path: ./data/my_dataset
+num_classes: 5
+augment: true
+```
+
+2. Add a builder in `src/data.py`:
+
+```python
+def _build_my_dataset(cfg):
+    # Load your data, return (train_dataset, val_dataset)
+    df = pd.read_parquet(cfg.path)
+    # ... wrap in a torch Dataset
+    return train_ds, val_ds
+```
+
+3. Register it in `build_dataset()`:
+
+```python
+if cfg.name == "my_dataset":
+    return _build_my_dataset(cfg)
+```
+
+4. Train:
+
+```bash
+python src/train.py dataset=my_dataset
+```

docs/engineering.md

@@ -0,0 +1,94 @@
+# Engineering Projects
+
+Engineering projects are pipeline-driven. You have a specific model to build, a dataset to handle, and something to deliver — a working classifier, a detection system, an inference API. This guide covers how to use the seed for that workflow.
+
+## What makes it engineering
+
+- You have a deliverable: "a model that classifies X with >90% accuracy"
+- You run a handful of experiments to tune, then lock the config
+- Reliability and reproducibility of the pipeline matter more than exploring many approaches
+- Output is a deployed model, an API, or an integration into a larger system
+
+## Recommended setup
+
+The defaults are already engineering-appropriate. TensorBoard is on, no experiment tracking overhead:
+
+```bash
+python src/train.py
+```
+
+Once you've settled on a config, lock it in an experiment file so anyone can reproduce the exact run:
+
+```yaml
+# configs/experiment/production.yaml
+# @package _global_
+training:
+  epochs: 30
+  lr: 0.001
+  batch_size: 64
+  seed: 42
+
+logging:
+  tensorboard: true
+  mlflow: false
+```
+
+```bash
+python src/train.py +experiment=production
+```
+
+## Typical workflow
+
+1. **Explore** — run a few experiments with different LRs/architectures to find what works
+2. **Lock** — commit your best experiment config to `configs/experiment/`
+3. **Train final model** — run with the locked config, save the checkpoint
+4. **Share weights** — push to HuggingFace Hub (see [../checkpoints/README.md](../checkpoints/README.md))
+5. **Evaluate** — run `src/evaluate.py` and log the final metrics
+
+## Adding your data pipeline
+
+Replace CIFAR-10 with your dataset (see README "Adding a dataset"). For engineering projects, data preprocessing is often the most important part — invest time here:
+
+- Clean and validate inputs before training
+- Use Parquet for tabular data (see [data.md](data.md))
+- Add a data validation step before your `build_dataset()` function
+
+## CI / automated testing
+
+The seed ships with a basic GitHub Actions CI that lints and runs smoke tests. For engineering projects, extend it to include:
+
+```yaml
+# .github/workflows/ci.yml — add after existing jobs
+  integration-test:
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v4
+      - uses: actions/setup-python@v5
+        with:
+          python-version: "3.12"
+      - run: pip install torch torchvision --index-url https://download.pytorch.org/whl/cpu
+      - run: pip install hydra-core omegaconf pytest numpy
+      # Run a real 1-epoch training to catch pipeline regressions
+      - run: python src/train.py training.epochs=1 training.device=cpu
+```
+
+## Inference
+
+After training, evaluate the model on a held-out test set:
+
+```bash
+python src/evaluate.py checkpoint=outputs/<date>/<time>/best_model.pt
+```
+
+For serving predictions in a larger system, load the checkpoint directly:
+
+```python
+from src.models import build_model
+from src.utils import load_checkpoint
+from omegaconf import OmegaConf
+
+cfg = OmegaConf.load("outputs/<date>/<time>/config.yaml")
+model = build_model(cfg.model)
+load_checkpoint("checkpoints/best_model.pt", model)
+model.eval()
+```