Llm experiments by bigabig · Pull Request #687 · uhh-lt/dats

bigabig · 2026-07-01T13:32:23Z

this adds llm assistant experiments / benchmarks

for more information, see https://pre-commit.ci

Copilot

Pull request overview

This PR introduces a new benchmarks/ subproject for running LLM benchmark experiments (Hydra configs, prompt templates, typed Pydantic schemas, evaluation/metrics, MLflow logging, and dataset prep scripts), and includes small formatting updates to generated frontend OpenAPI model types.

Changes:

Added a standalone benchmarking framework under benchmarks/ (schemas, runner, prompts, evaluation utilities, configs, dockerized MLflow, dataset prep).
Added multiple benchmark datasets’ documentation and preprocessing scripts/notebooks (with .gitignore rules to keep large artifacts out of git).
Minor formatting changes in generated frontend OpenAPI TypeScript models/enums.

Reviewed changes

Copilot reviewed 175 out of 177 changed files in this pull request and generated 13 comments.

Show a summary per file

File	Description
frontend/src/api/openapi/models/SDocStatus.ts	Adjust enum member naming for status values.
frontend/src/api/openapi/models/LLMJobOutput.ts	Reformat union type for `specific_task_result`.
frontend/src/api/openapi/models/FilterExpression_WordFrequencyColumns_.ts	Reformat union type for `operator`.
frontend/src/api/openapi/models/FilterExpression_SpanColumns_.ts	Reformat union type for `operator`.
frontend/src/api/openapi/models/FilterExpression_SentAnnoColumns_.ts	Reformat union type for `operator`.
frontend/src/api/openapi/models/FilterExpression_SdocColumns_.ts	Reformat union type for `operator`.
frontend/src/api/openapi/models/FilterExpression_MemoColumns_.ts	Reformat union type for `operator`.
frontend/src/api/openapi/models/FilterExpression_BBoxColumns_.ts	Reformat union type for `operator`.
benchmarks/src/schemas/reference/reference_schema.py	Add typed “gold/reference” schemas with validation helpers.
benchmarks/src/schemas/prediction/template_filling_schema.py	Add template-filling answer schema.
benchmarks/src/schemas/prediction/span_classification_schema.py	Add span-classification answer schemas (FewNERD/GermanLER/etc.).
benchmarks/src/schemas/prediction/sequential_sentence_classification_schema.py	Add sequential sentence classification answer schemas.
benchmarks/src/schemas/prediction/schema_resolver.py	Add dynamic resolver for configured answer schema classes.
benchmarks/src/schemas/prediction/prediction_schema.py	Add base answer schema types and common prediction structures.
benchmarks/src/schemas/prediction/extractive_qa_schema.py	Add extractive QA answer schema.
benchmarks/src/schemas/prediction/document_classification_schema.py	Add document classification answer schemas (Newsgroups/BBC/IMDB/Tagesschau).
benchmarks/src/schemas/dataset/template_filling_dataset_schema.py	Add dataset config + reference building for template filling.
benchmarks/src/schemas/dataset/sequential_sentence_classification_dataset_schema.py	Add dataset config + prompt/references for sequential sentence classification.
benchmarks/src/schemas/dataset/extractive_qa_dataset_schema.py	Add dataset config + references for extractive QA.
benchmarks/src/schemas/dataset/document_classification_dataset_schema.py	Add dataset config + references for single/multi-label classification.
benchmarks/src/schemas/dataset/dataset_schema.py	Add dataset base config + column validation helpers.
benchmarks/src/schemas/config/run_schema.py	Add top-level run config schema (output dir, MLflow URI, backend/experiment).
benchmarks/src/schemas/config/README.md	Add short schema-doc README.
benchmarks/src/schemas/config/model_schema.py	Add model config schema for vLLM container runs.
benchmarks/src/schemas/config/experiment_schema.py	Add experiment config schema (templates, schema, metrics/artifacts).
benchmarks/src/schemas/config/dataset_schema.py	Add discriminated union for dataset config types.
benchmarks/src/schemas/config/backend_schema.py	Add vLLM backend config schema + env var validation.
benchmarks/src/run_experiment.py	Add Hydra entrypoint for running benchmark experiments.
benchmarks/src/prompts/template_filling_system_en.j2	Add system prompt for template filling.
benchmarks/src/prompts/template_filling_muc4_v1_en.j2	Add MUC4 template-filling user prompt template.
benchmarks/src/prompts/span_classification_v1_en.j2	Add span classification user prompt (EN).
benchmarks/src/prompts/span_classification_v1_de.j2	Add span classification user prompt (DE).
benchmarks/src/prompts/span_classification_system_en.j2	Add span classification system prompt (EN).
benchmarks/src/prompts/span_classification_system_de.j2	Add span classification system prompt (DE).
benchmarks/src/prompts/sequential_sentence_classification_v1_en.j2	Add sequential sentence classification user prompt.
benchmarks/src/prompts/sequential_sentence_classification_system_v1_en.j2	Add sequential sentence classification system prompt.
benchmarks/src/prompts/extractive_qa_v1_de.j2	Add extractive QA user prompt (DE).
benchmarks/src/prompts/extractive_qa_system_en.j2	Add extractive QA system prompt (EN).
benchmarks/src/prompts/extractive_qa_system_de.j2	Add extractive QA system prompt (DE).
benchmarks/src/prompts/extractive_qa_squad2_v1_en.j2	Add extractive QA SQuAD2 user prompt (EN).
benchmarks/src/prompts/extractive_qa_squad1_v1_en.j2	Add extractive QA SQuAD1 user prompt (EN).
benchmarks/src/prompts/document_classification_system_en.j2	Add document classification system prompt (EN).
benchmarks/src/prompts/document_classification_system_de.j2	Add document classification system prompt (DE).
benchmarks/src/prompts/document_classification_single_label_v1_en.j2	Add single-label doc classification user prompt (EN).
benchmarks/src/prompts/document_classification_single_label_v1_de.j2	Add single-label doc classification user prompt (DE).
benchmarks/src/prompts/document_classification_multi_label_v1_en.j2	Add multi-label doc classification user prompt (EN).
benchmarks/src/evaluation/template_filling_metrics.py	Add template filling metric wrapper (slot-based scoring).
benchmarks/src/evaluation/span_classification_utils.py	Add span classification parsing/normalization helpers.
benchmarks/src/evaluation/span_classification_metrics.py	Add span classification metrics computation (seqeval).
benchmarks/src/evaluation/span_classification_artifacts.py	Add span classification report artifact generation.
benchmarks/src/evaluation/sequential_sentence_classification_utils.py	Add sequential sentence classification label-sequence builders.
benchmarks/src/evaluation/sequential_sentence_classification_metrics.py	Add sequential sentence classification metrics computation (seqeval).
benchmarks/src/evaluation/sequential_sentence_classification_artifacts.py	Add sequential sentence classification report artifact generation.
benchmarks/src/evaluation/metric_registry.py	Register available metrics by name.
benchmarks/src/evaluation/metric_base.py	Add shared metric wrapper base (schema enforcement, filtering).
benchmarks/src/evaluation/extractive_qa_metrics.py	Add extractive QA SQuAD2 metrics wrapper.
benchmarks/src/evaluation/eval_utils.py	Add label normalization/extraction utilities.
benchmarks/src/evaluation/artifact_registry.py	Register available artifacts by name.
benchmarks/src/evaluation/artifact_base.py	Add shared artifact builder base (schema enforcement, filtering).
benchmarks/src/core/llm_client.py	Add async OpenAI-compatible batch inference client for vLLM.
benchmarks/src/core/docker_manager.py	Add managed vLLM docker container lifecycle helper.
benchmarks/src/.env.example	Add example HF token env var file.
benchmarks/README.md	Add benchmark framework documentation.
benchmarks/pyproject.toml	Add Python project configuration and dependencies for benchmarks.
benchmarks/docker/README.md	Add docker README pointer.
benchmarks/docker/docker-compose.yml	Add MLflow + Postgres + S3 gateway compose stack.
benchmarks/docker/.env.example	Add compose environment template.
benchmarks/datasets/tagesschau/README.md	Add dataset documentation for Tagesschau benchmark.
benchmarks/datasets/tagesschau/.gitignore	Ignore dataset artifacts except docs/notebook.
benchmarks/datasets/squad2/squad2_dataset_creation.py	Add SQuAD v2 preprocessing script.
benchmarks/datasets/squad2/README.md	Add SQuAD v2 dataset documentation.
benchmarks/datasets/squad2/.gitignore	Ignore dataset artifacts except docs/script.
benchmarks/datasets/squad/squad_dataset_creation.py	Add SQuAD preprocessing script.
benchmarks/datasets/squad/README.md	Add SQuAD dataset documentation.
benchmarks/datasets/squad/.gitignore	Ignore dataset artifacts except docs/script.
benchmarks/datasets/pubmed200k/README.md	Add PubMed200k dataset documentation.
benchmarks/datasets/pubmed200k/.gitignore	Ignore generated artifacts.
benchmarks/datasets/muc/README.md	Add MUC dataset documentation.
benchmarks/datasets/muc/.gitignore	Ignore dataset artifacts except docs/notebook.
benchmarks/datasets/imdb/README.md	Add IMDB dataset documentation.
benchmarks/datasets/imdb/.gitignore	Ignore dataset artifacts except docs/notebook.
benchmarks/datasets/germanquad/README.md	Add GermanQuAD dataset documentation.
benchmarks/datasets/germanquad/germanquad_dataset_creation.py	Add GermanQuAD preprocessing script.
benchmarks/datasets/germanquad/.gitignore	Ignore dataset artifacts except docs/script.
benchmarks/datasets/german-quotations/README.md	Add German quotations dataset documentation.
benchmarks/datasets/german-quotations/.gitignore	Ignore dataset artifacts except docs/notebook.
benchmarks/datasets/german-ler/README.md	Add German-LER dataset documentation.
benchmarks/datasets/german-ler/.gitignore	Ignore dataset artifacts except docs/notebook.
benchmarks/datasets/fewnerd/README.md	Add Few-NERD dataset documentation.
benchmarks/datasets/fewnerd/fewnerd_dataset_creation.py	Add Few-NERD download script.
benchmarks/datasets/fewnerd/.gitignore	Ignore dataset artifacts except docs/script.
benchmarks/datasets/emotion_lines/README.md	Add EmotionLines dataset documentation.
benchmarks/datasets/emotion_lines/emotion_lines.ipynb	Add EmotionLines preprocessing notebook.
benchmarks/datasets/emotion_lines/.gitignore	Ignore generated artifacts.
benchmarks/datasets/disfl-qa/README.md	Add Disfl-QA dataset documentation.
benchmarks/datasets/disfl-qa/.gitignore	Ignore dataset artifacts except docs/notebook.
benchmarks/datasets/disco/README.md	Add DISCO dataset documentation.
benchmarks/datasets/disco/.gitignore	Ignore dataset artifacts except docs/notebook.
benchmarks/datasets/daily_dialog/README.md	Add DailyDialog dataset documentation.
benchmarks/datasets/daily_dialog/.gitignore	Ignore generated artifacts.
benchmarks/datasets/csabstruct/README.md	Add CSAbstruct dataset documentation.
benchmarks/datasets/csabstruct/csabstruct.ipynb	Add CSAbstruct preprocessing notebook.
benchmarks/datasets/csabstruct/.gitignore	Ignore generated artifacts.
benchmarks/datasets/coarsediscourse/README.md	Add CoarseDiscourse dataset documentation.
benchmarks/datasets/coarsediscourse/coarsediscourse.ipynb	Add CoarseDiscourse preprocessing notebook.
benchmarks/datasets/coarsediscourse/.gitignore	Ignore generated artifacts.
benchmarks/datasets/bbc/README.md	Add BBC dataset documentation.
benchmarks/datasets/bbc/.gitignore	Ignore dataset artifacts except docs/notebook.
benchmarks/datasets/20newsgroups/README.md	Add 20 Newsgroups dataset documentation.
benchmarks/datasets/20newsgroups/20newsgroups_dataset_creation.py	Add 20 Newsgroups preprocessing script.
benchmarks/datasets/20newsgroups/.gitignore	Ignore dataset artifacts except docs/script.
benchmarks/configs/model/llama32_3b.yaml	Add model config for Llama 3.2 3B instruct.
benchmarks/configs/experiment/template_filling_muc4_v1.yaml	Add template filling experiment config.
benchmarks/configs/experiment/span_classification_germanler_fine_v1.yaml	Add German-LER fine span classification experiment config.
benchmarks/configs/experiment/span_classification_germanler_coarse_v1.yaml	Add German-LER coarse span classification experiment config.
benchmarks/configs/experiment/span_classification_fewnerd_fine_v1.yaml	Add Few-NERD fine span classification experiment config.
benchmarks/configs/experiment/span_classification_fewnerd_coarse_v1.yaml	Add Few-NERD coarse span classification experiment config.
benchmarks/configs/experiment/span_classification_direct_quotation_v1.yaml	Add direct quotation span classification experiment config.
benchmarks/configs/experiment/sequential_sentence_classification_pubmed200k_v1.yaml	Add PubMed200k sequential sentence classification experiment config.
benchmarks/configs/experiment/sequential_sentence_classification_emotion_lines_v1.yaml	Add EmotionLines sequential sentence classification experiment config.
benchmarks/configs/experiment/sequential_sentence_classification_daily_dialog_v1.yaml	Add DailyDialog sequential sentence classification experiment config.
benchmarks/configs/experiment/sequential_sentence_classification_csabstruct_v1.yaml	Add CSAbstruct sequential sentence classification experiment config.
benchmarks/configs/experiment/sequential_sentence_classification_coarsediscourse_v1.yaml	Add CoarseDiscourse sequential sentence classification experiment config.
benchmarks/configs/experiment/extractive_qa_squad2_v1.yaml	Add SQuAD2 extractive QA experiment config.
benchmarks/configs/experiment/extractive_qa_squad1_v1.yaml	Add SQuAD1 extractive QA experiment config.
benchmarks/configs/experiment/extractive_qa_germanquad_v1.yaml	Add GermanQuAD extractive QA experiment config.
benchmarks/configs/experiment/document_classification_tagesschau_fine_v1.yaml	Add Tagesschau fine doc classification experiment config.
benchmarks/configs/experiment/document_classification_tagesschau_coarse_v1.yaml	Add Tagesschau coarse doc classification experiment config.
benchmarks/configs/experiment/document_classification_imdb_multi_label_v1.yaml	Add IMDB multi-label doc classification experiment config.
benchmarks/configs/experiment/document_classification_imdb_coarse_v1.yaml	Add IMDB coarse doc classification experiment config.
benchmarks/configs/experiment/document_classification_bbc_fine_v1.yaml	Add BBC fine doc classification experiment config.
benchmarks/configs/experiment/document_classification_bbc_coarse_v1.yaml	Add BBC coarse doc classification experiment config.
benchmarks/configs/experiment/document_classification_20newsgroups_coarse_v1.yaml	Add 20 Newsgroups doc classification experiment config.
benchmarks/configs/dataset/tagesschau-fine.yaml	Add Tagesschau fine dataset config.
benchmarks/configs/dataset/tagesschau-coarse.yaml	Add Tagesschau coarse dataset config.
benchmarks/configs/dataset/squad2.yaml	Add SQuAD2 dataset config.
benchmarks/configs/dataset/squad.yaml	Add SQuAD dataset config.
benchmarks/configs/dataset/pubmed200k.yaml	Add PubMed200k dataset config.
benchmarks/configs/dataset/muc4.yaml	Add MUC4 dataset config.
benchmarks/configs/dataset/imdb-multi-label.yaml	Add IMDB multi-label dataset config.
benchmarks/configs/dataset/imdb-coarse.yaml	Add IMDB coarse dataset config.
benchmarks/configs/dataset/germanquad.yaml	Add GermanQuAD dataset config.
benchmarks/configs/dataset/german-quotations-direct.yaml	Add German quotations dataset config.
benchmarks/configs/dataset/german-ler-fine.yaml	Add German-LER fine dataset config.
benchmarks/configs/dataset/german-ler-coarse.yaml	Add German-LER coarse dataset config.
benchmarks/configs/dataset/fewnerd-fine.yaml	Add Few-NERD fine dataset config.
benchmarks/configs/dataset/fewnerd-coarse.yaml	Add Few-NERD coarse dataset config.
benchmarks/configs/dataset/emotion-lines.yaml	Add EmotionLines dataset config.
benchmarks/configs/dataset/daily-dialog.yaml	Add DailyDialog dataset config.
benchmarks/configs/dataset/csabstruct.yaml	Add CSAbstruct dataset config.
benchmarks/configs/dataset/coarsediscourse.yaml	Add CoarseDiscourse dataset config.
benchmarks/configs/dataset/bbc-fine.yaml	Add BBC fine dataset config.
benchmarks/configs/dataset/bbc-coarse.yaml	Add BBC coarse dataset config.
benchmarks/configs/dataset/20newsgroups.yaml	Add 20 Newsgroups dataset config.
benchmarks/configs/config.yaml	Add top-level Hydra defaults/config.
benchmarks/configs/backend/vllm.yaml	Add vLLM backend config.
benchmarks/.vscode/settings.json	Add benchmark-local VS Code settings (pyright, ruff, prettier).
benchmarks/.vscode/launch.json	Add benchmark-local launch configurations.
benchmarks/.gitignore	Add benchmark-local ignore rules.
.vscode/settings.json	Exclude `benchmarks/` in workspace explorer settings.
.vscode/dats.code-workspace	Add `benchmarks/` folder to multi-root VS Code workspace.

@@ -0,0 +1,3 @@
+# Config Schemas
+
+This diretory contains schemas for all files located in /benchmarks/configs. The schemas validate the Hydra configs.


+- Prompt templates are always loaded from `src/prompts/templates` (not configurable).
+- Schema is configured as a single dotted path (for example `newsgroups20_schema.NewsgroupClassificationSchemaV1`).


+- `configs/`: Runtime config and Hydra groups (`experiment/`, `model/`, `dataset/`, `backend/`)
+- `data/`: Datasets and preprocessing scripts
+- `docker/`: MLflow compose files and environment templates
+- `outputs/`: Local output artifacts (CSV/JSON)
+- `src/`: Core runner, LLM clients, schemas, prompts, evaluation, tracking


+defaults:
+  - experiment: 20newsgroups_v1_zeroshot
+  - backend: vllm
+  - _self_


+  - _self_
+
+output_dir: outputs
+mlflow_uri: http://localhost:19274


+uv run python src/run_experiment.py \
+	experiment=20newsgroups_v1_zeroshot \
+	backend=vllm \
+	backend.gpu_id=1


+        for raw_value in df[self.label_column].tolist():
+            value = raw_value.tolist() if hasattr(raw_value, "tolist") else raw_value
+            references.append(SingleLabelReference(label=str(value).strip()))
+


+        for raw_value in df[self.label_column].tolist():
+            value = raw_value.tolist() if hasattr(raw_value, "tolist") else raw_value
+
+            if isinstance(value, str):
+                labels = [item.strip() for item in value.split(",") if item.strip()]
+            elif isinstance(value, (list, tuple, set)):
+                labels = [str(item).strip() for item in value if str(item).strip()]
+            else:
+                labels = [str(value).strip()] if str(value).strip() else []
+
+            references.append(MultiLabelReference(labels=labels))
+


+    @field_validator("mlflow_uri")
+    @classmethod
+    def validate_mlflow_running(cls, value: str) -> str:
+        try:
+            import requests
+
+            health_url = f"{value.rstrip('/')}/health"
+            response = requests.get(health_url, timeout=5)
+            if response.status_code != 200:
+                raise ValueError(
+                    f"MLflow tracking server at '{value}' is not healthy. /health endpoint returned status code {response.status_code}."
+                )
+        except Exception as exc:
+            raise ValueError(
+                f"Could not connect to MLflow tracking server at '{value}': {exc}"
+            )
+        return value


+def read_dataset_columns(path: Path) -> set[str]:
+    if path.suffix == ".csv":
+        return set(pd.read_csv(path, nrows=0).columns)
+    if path.suffix == ".parquet":
+        return set(pd.read_parquet(path).columns)
+    raise ValueError(f"Unsupported dataset format: {path.suffix}")


bigabig added 8 commits July 1, 2026 13:30

added dats benchmarks framework

e9c5383

added document classification experiments

ee60d7d

added extractive qa experiments

37a4d5a

added template-filling experiments

cbb77e8

added span-classification experiment

902d48c

improved typing

45ceffc

added sequential-sentence-classification experiments

c9ced6a

refactored schemas

dab2455

Copilot AI review requested due to automatic review settings July 1, 2026 13:32

[pre-commit.ci] auto fixes from pre-commit.com hooks

77db3fa

for more information, see https://pre-commit.ci

Copilot started reviewing on behalf of bigabig July 1, 2026 13:32 View session

Copilot AI reviewed Jul 1, 2026

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Llm experiments#687

Llm experiments#687
bigabig wants to merge 9 commits into
mainfrom
llm-experiments

bigabig commented Jul 1, 2026

Uh oh!

Copilot AI left a comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

		@@ -0,0 +1,3 @@
		# Config Schemas

		This diretory contains schemas for all files located in /benchmarks/configs. The schemas validate the Hydra configs.

		- Prompt templates are always loaded from `src/prompts/templates` (not configurable).
		- Schema is configured as a single dotted path (for example `newsgroups20_schema.NewsgroupClassificationSchemaV1`).

Uh oh!

Conversation

bigabig commented Jul 1, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants