Skip to content

Rename DataJoint Works to DataJoint Platform #59

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
dimitri-yatsenko merged 1 commit into from
Dec 18, 2025
Merged

Rename DataJoint Works to DataJoint Platform #59

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
107 changes: 2 additions & 105 deletions book/40-operations/060-orchestration.ipynb
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4,110 +4,7 @@
"cell_type": "markdown",
"id": "cell-0",
"metadata": {},
"source": [
"# Orchestration\n",
"\n",
"While the `populate` operation provides the logic for automated computation, **orchestration** addresses the infrastructure and operational concerns of running these computations at scale:\n",
"\n",
"- **Infrastructure provisioning** — Allocating compute resources (servers, containers, cloud instances)\n",
"- **Dependency management** — Ensuring consistent runtime environments across workers\n",
"- **Automated execution** — Scheduling and triggering `populate` calls\n",
"- **Observability** — Monitoring job progress, failures, and system health\n",
"- **Performance and cost tracking** — Understanding resource utilization and expenses\n",
"\n",
"These concerns are **outside the scope of the core DataJoint library** (`datajoint-python`), which focuses on the data model and workflow logic. Orchestration is solved through complementary infrastructure.\n",
"\n",
"## The Orchestration Challenge\n",
"\n",
"A typical DataJoint workflow requires:\n",
"\n",
"1. **Database server** — MySQL/MariaDB instance with appropriate configuration\n",
"2. **Worker processes** — Python environments with DataJoint and domain-specific packages\n",
"3. **File storage** — For external blob storage (if using `dj.config['stores']`)\n",
"4. **Job coordination** — Managing which workers process which jobs\n",
"5. **Error handling** — Retrying failed jobs, alerting on persistent failures\n",
"6. **Scaling** — Adding workers during high-demand periods\n",
"\n",
"The `populate(reserve_jobs=True)` option handles job coordination at the database level, but provisioning and managing the workers themselves requires additional infrastructure.\n",
"\n",
"## Commercial Solution: DataJoint Works\n",
"\n",
"[DataJoint Works](https://datajoint.com) is a managed platform that provides comprehensive orchestration:\n",
"\n",
"| Feature | Description |\n",
"|---------|-------------|\n",
"| **Managed databases** | Provisioned and configured MySQL instances |\n",
"| **Container registry** | Store and version workflow container images |\n",
"| **Compute clusters** | Auto-scaling worker pools (cloud or on-premise) |\n",
"| **Job scheduler** | Automated triggering of `populate` operations |\n",
"| **Monitoring dashboard** | Real-time visibility into job status and errors |\n",
"| **Cost analytics** | Track compute and storage costs per workflow |\n",
"\n",
"This platform integrates directly with DataJoint schemas, providing a turnkey solution for teams that prefer managed infrastructure.\n",
"\n",
"## DIY Solutions\n",
"\n",
"Many teams build custom orchestration using standard DevOps tools. Common approaches include:\n",
"\n",
"### Containerization\n",
"\n",
"- **Docker** — Package DataJoint workflows with all dependencies\n",
"- **Singularity/Apptainer** — Container runtime for HPC environments\n",
"- **Conda environments** — Dependency management without full containerization\n",
"\n",
"### Container Orchestration\n",
"\n",
"- **Kubernetes** — Production-grade container orchestration\n",
"- **Docker Swarm** — Simpler container clustering\n",
"- **Nomad** — HashiCorp's workload orchestrator\n",
"\n",
"### Job Schedulers\n",
"\n",
"- **SLURM** — Common in academic HPC clusters\n",
"- **PBS/Torque** — Traditional batch scheduling\n",
"- **HTCondor** — High-throughput computing scheduler\n",
"- **Apache Airflow** — DAG-based workflow orchestration\n",
"- **Prefect** — Modern Python-native orchestration\n",
"- **Celery** — Distributed task queue\n",
"\n",
"### Cloud Infrastructure\n",
"\n",
"- **AWS Batch** — Managed batch computing on AWS\n",
"- **Google Cloud Run Jobs** — Serverless container execution\n",
"- **Azure Container Instances** — On-demand container execution\n",
"\n",
"### Monitoring and Observability\n",
"\n",
"- **Prometheus + Grafana** — Metrics collection and visualization\n",
"- **DataDog** — Commercial observability platform\n",
"- **CloudWatch / Stackdriver** — Cloud-native monitoring\n",
"\n",
"### Database Hosting\n",
"\n",
"- **Amazon RDS** — Managed MySQL on AWS\n",
"- **Google Cloud SQL** — Managed MySQL on GCP\n",
"- **Self-hosted MySQL/MariaDB** — On-premise or VM-based\n",
"\n",
"## Choosing an Approach\n",
"\n",
"The right orchestration strategy depends on your team's context:\n",
"\n",
"| Factor | Managed Platform | DIY |\n",
"|--------|-----------------|-----|\n",
"| **Setup time** | Hours | Days to weeks |\n",
"| **Maintenance** | Included | Team responsibility |\n",
"| **Customization** | Platform constraints | Full flexibility |\n",
"| **Cost model** | Subscription | Infrastructure costs |\n",
"| **Existing infrastructure** | May duplicate | Leverages investments |\n",
"| **Compliance requirements** | Check with vendor | Full control |\n",
"\n",
"Many teams start with DIY solutions using familiar tools, then evaluate managed platforms as workflows scale and operational overhead increases.\n",
"\n",
":::{seealso}\n",
"- [DataJoint Works](https://datajoint.com) — Managed orchestration platform\n",
"- [Populate](050-populate.ipynb) — The underlying automation mechanism\n",
":::"
]
"source": "# Orchestration\n\nWhile the `populate` operation provides the logic for automated computation, **orchestration** addresses the infrastructure and operational concerns of running these computations at scale:\n\n- **Infrastructure provisioning** — Allocating compute resources (servers, containers, cloud instances)\n- **Dependency management** — Ensuring consistent runtime environments across workers\n- **Automated execution** — Scheduling and triggering `populate` calls\n- **Observability** — Monitoring job progress, failures, and system health\n- **Performance and cost tracking** — Understanding resource utilization and expenses\n\nThese concerns are **outside the scope of the core DataJoint library** (`datajoint-python`), which focuses on the data model and workflow logic. Orchestration is solved through complementary infrastructure.\n\n## The Orchestration Challenge\n\nA typical DataJoint workflow requires:\n\n1. **Database server** — MySQL/MariaDB instance with appropriate configuration\n2. **Worker processes** — Python environments with DataJoint and domain-specific packages\n3. **File storage** — For external blob storage (if using `dj.config['stores']`)\n4. **Job coordination** — Managing which workers process which jobs\n5. **Error handling** — Retrying failed jobs, alerting on persistent failures\n6. **Scaling** — Adding workers during high-demand periods\n\nThe `populate(reserve_jobs=True)` option handles job coordination at the database level, but provisioning and managing the workers themselves requires additional infrastructure.\n\n## Commercial Solution: DataJoint Platform\n\n[DataJoint Platform](https://datajoint.com) is a managed platform that provides comprehensive orchestration:\n\n| Feature | Description |\n|---------|-------------|\n| **Managed databases** | Provisioned and configured MySQL instances |\n| **Container registry** | Store and version workflow container images |\n| **Compute clusters** | Auto-scaling worker pools (cloud or on-premise) |\n| **Job scheduler** | Automated triggering of `populate` operations |\n| **Monitoring dashboard** | Real-time visibility into job status and errors |\n| **Cost analytics** | Track compute and storage costs per workflow |\n\nThis platform integrates directly with DataJoint schemas, providing a turnkey solution for teams that prefer managed infrastructure.\n\n## DIY Solutions\n\nMany teams build custom orchestration using standard DevOps tools. Common approaches include:\n\n### Containerization\n\n- **Docker** — Package DataJoint workflows with all dependencies\n- **Singularity/Apptainer** — Container runtime for HPC environments\n- **Conda environments** — Dependency management without full containerization\n\n### Container Orchestration\n\n- **Kubernetes** — Production-grade container orchestration\n- **Docker Swarm** — Simpler container clustering\n- **Nomad** — HashiCorp's workload orchestrator\n\n### Job Schedulers\n\n- **SLURM** — Common in academic HPC clusters\n- **PBS/Torque** — Traditional batch scheduling\n- **HTCondor** — High-throughput computing scheduler\n- **Apache Airflow** — DAG-based workflow orchestration\n- **Prefect** — Modern Python-native orchestration\n- **Celery** — Distributed task queue\n\n### Cloud Infrastructure\n\n- **AWS Batch** — Managed batch computing on AWS\n- **Google Cloud Run Jobs** — Serverless container execution\n- **Azure Container Instances** — On-demand container execution\n\n### Monitoring and Observability\n\n- **Prometheus + Grafana** — Metrics collection and visualization\n- **DataDog** — Commercial observability platform\n- **CloudWatch / Stackdriver** — Cloud-native monitoring\n\n### Database Hosting\n\n- **Amazon RDS** — Managed MySQL on AWS\n- **Google Cloud SQL** — Managed MySQL on GCP\n- **Self-hosted MySQL/MariaDB** — On-premise or VM-based\n\n## Choosing an Approach\n\nThe right orchestration strategy depends on your team's context:\n\n| Factor | Managed Platform | DIY |\n|--------|-----------------|-----|\n| **Setup time** | Hours | Days to weeks |\n| **Maintenance** | Included | Team responsibility |\n| **Customization** | Platform constraints | Full flexibility |\n| **Cost model** | Subscription | Infrastructure costs |\n| **Existing infrastructure** | May duplicate | Leverages investments |\n| **Compliance requirements** | Check with vendor | Full control |\n\nMany teams start with DIY solutions using familiar tools, then evaluate managed platforms as workflows scale and operational overhead increases.\n\n:::{seealso}\n- [DataJoint Platform](https://datajoint.com) — Managed orchestration platform\n- [Populate](050-populate.ipynb) — The underlying automation mechanism\n:::"
}
],
"metadata": {
Expand All @@ -123,4 +20,4 @@
},
"nbformat": 4,
"nbformat_minor": 5
}
}