feat: add IVIM workflow DAG with MinIO integration and platform docum…

kaapana_ivim_osipi/README.md

@@ -0,0 +1,98 @@
+# Kaapana Installation Guide
+
+## Overview
+
+The Kaapana installation process consists of three main steps:
+
+1. **Building** all the container images required for Kaapana and pushing them to a remote registry, local registry, or local tarball
+2. **Installation** of Kaapana after building the images
+3. **Deployment** of Kaapana after installation
+
+For my setup, I used the remote registry approach, which is also the recommended format.
+
+## Machine Requirements
+
+These are the machine requirements which I used:
+
+### System Specifications
+- **Operating System**: Ubuntu 24.04
+- **Platform**: Google Cloud Platform (GCP)
+
+### VM Configuration
+- **vCPU**: 8
+- **RAM**: 64 GB
+- **Disk**: 250 GB balanced persistent disk
+
+## Step 1: Building Container Images
+
+### Registry Setup (GitLab)
+
+For the build process, I used GitLab registry. You'll need to configure three key parameters in your build-configuration file:
+
+#### 1. `default_registry`
+- Create a repository on GitLab
+- Navigate to **Deploy** → **Container Registry**
+- Copy the registry link (format: `registry.gitlab.com/your-username/repository-name`)
+- Example: `registry.gitlab.com/unique-usman/kaapana-tutorial`
+
+#### 2. `registry_password`
+- Go to your repository **Settings** → **Access Tokens**
+- Create a new token with read and write registry permissions
+- *Note: I granted all permissions for simplicity*
+
+#### 3. `registry_username`
+- Use the GitLab username under which you created the repository
+- Since I created my repository under my personal GitLab account, I used my GitLab username
+
+### Build Process
+
+Follow the comprehensive build guide available at:
+https://kaapana.readthedocs.io/en/stable/installation_guide/build.html
+
+## Step 2: Server Installation
+
+### Installation Process
+
+Refer to the server installation documentation:
+https://kaapana.readthedocs.io/en/stable/installation_guide/server_installation.html
+
+Since I didn't use any proxy configuration, I simply executed the server installation script as provided in the documentation.
+
+## Step 3: Platform Deployment
+
+### Deployment Process
+
+Follow the deployment guide:
+https://kaapana.readthedocs.io/en/stable/installation_guide/deployment.html
+
+I ran the deployment script directly, and it worked perfectly without any additional configuration.
+
+## Post-Deployment Configuration
+
+### Accessing Your Kaapana Instance
+
+After successful deployment, you'll receive a web link to access your Kaapana instance. 
+
+### DNS Configuration
+
+To simplify access, I mapped the deployment link to the server's IP address in `/etc/hosts`:
+
+```bash
+# Example entry in /etc/hosts
+your-server-ip    your-kaapana-domain
+```
+
+## Architecture Notes
+
+- **Single Machine Setup**: I used only one GCP machine for the entire Kaapana deployment
+- **All-in-One Configuration**: The build, installation, and deployment processes were all executed on the same machine
+
+## Key Resources
+
+- [Build Guide](https://kaapana.readthedocs.io/en/stable/installation_guide/build.html)
+- [Server Installation](https://kaapana.readthedocs.io/en/stable/installation_guide/server_installation.html)
+- [Deployment Guide](https://kaapana.readthedocs.io/en/stable/installation_guide/deployment.html)
+
+---
+
+*This guide reflects a successful single-machine Kaapana deployment using GCP and GitLab registry.*

kaapana_ivim_osipi/Workflow_Extension/README.md

@@ -0,0 +1,145 @@
+# Installing Otsus Template Method in Kaapana
+
+## Overview
+
+This guide documents the process of installing the Otsus template method workflow in Kaapana to understand how to add new workflows to the system. By following this installation, you'll become familiar with the workflow addition process in Kaapana.
+
+## Getting Started
+
+To access the Otsus workflow, clone the Kaapana repository:
+```bash
+git clone https://github.com/kaapana/kaapana
+```
+
+The sample Otsus workflow is located at:
+```
+./kaapana/templates_and_examples/examples/processing-pipelines/otsus-method
+```
+
+## Workflow Components
+
+Before proceeding, let's understand the components of a Kaapana workflow. For detailed information, refer to the [Kaapana documentation](https://kaapana.readthedocs.io/en/stable/development_guide/workflow_dev_guide.html).
+
+### Folder Structure
+
+```
+otsus-method
+├── extension
+│   ├── docker
+│   │   ├── Dockerfile
+│   │   └── files
+│   │       ├── dag_otsus_method.py
+│   │       └── otsus-method
+│   │           ├── OtsusMethodOperator.py
+│   │           └── OtsusNotebookOperator.py
+│   └── otsus-method-workflow
+│       ├── Chart.yaml
+│       ├── README.md
+│       ├── requirements.yaml
+│       └── values.yaml
+└── processing-containers
+    └── otsus-method
+        ├── Dockerfile
+        └── files
+            ├── otsus_method.py
+            ├── otsus_notebooks
+            │   ├── run_otsus_report_notebook.ipynb
+            │   └── run_otsus_report_notebook.sh
+            └── requirements.txt
+```
+
+### Processing Containers
+
+The `processing-containers` folder contains:
+- **Algorithm implementations**: All the algorithms you want to run
+- **Dockerfile**: For containerizing the algorithm
+- **otsus_method.py**: The main Otsus algorithm
+- **otsus_notebooks**: Report analysis components including graphs
+
+### Extension Folder
+
+The extension folder contains configuration and files needed for the DAG, divided into two main parts:
+
+#### 1. otsus-method-workflow
+Contains everything related to Helm chart configuration used to deploy the workflow.
+
+#### 2. docker
+Contains all the code for specifying the DAG workflow:
+
+- **Dockerfile**: DAG codes need to be containerized
+- **otsus-method folder**: Contains all operators (nodes) for the DAG
+  - **OtsusMethodOperator.py**: Operator for the Otsus algorithm using the Docker image from processing-containers
+  - **OtsusNotebookOperator.py**: Operator for the Notebook using the image from processing-containers
+  - **Built-in operators**: Such as "DcmConverterOperator" for converting DICOM images to NIfTI
+- **dag_otsus_method.py**: Configures the DAG using the operators
+
+> **Note**: Each operator is essentially a Docker container uploaded to your registry. The processing-containers images will be pulled from the registry.
+
+### Workflow Architecture
+
+Below is a visual representation of a workflow in Kaapana:
+
+![Workflow Extension Architecture](./workflow_extension.png)
+
+## Installation Steps
+
+Follow these four steps to install the workflow:
+
+### Step 1: Build Container Images
+
+Build the container image using:
+```bash
+docker build -t <registry-name>/<image-name>:<version>
+```
+
+Where:
+- `registry-name`: Name of your registry
+- `image-name`: Name of the image (found in Dockerfile)
+- `version`: Version number of Kaapana
+
+Repeat this process for both images in the files.
+
+After building, push the image to your registry:
+```bash
+docker push <registry-name>/<image-name>:<version>
+```
+
+### Step 2: Package Helm Chart
+
+1. Update the `Chart.yaml` version to match your Kaapana version
+
+2. Inside the `otsus-method-workflow` folder, run:
+   ```bash
+   helm dep up
+   ```
+   This installs all requirements.
+
+3. Package the chart:
+   ```bash
+   helm package .
+   ```
+   This command generates a `.tgz` file.
+
+### Step 3: Upload to Kaapana Platform
+
+1. Navigate to the Extension view in your Kaapana platform
+2. Upload the `.tgz` file
+3. Add "Experimental" to the Version field
+4. You'll see the Otsus method added as part of the workflow
+5. Install the Otsus method extension
+
+### Step 4: Run and Test the Workflow
+
+If the Otsus method doesn't appear in the Workflow Execution tab, follow the guidance from the [Kaapana FAQ](https://kaapana.readthedocs.io/en/stable/faq/workflow_not_listed.html#workflow-not-allowed-in-the-selected-project):
+
+> "If you cannot find a workflow in the list of executable workflows in the Workflow Execution view, it might be that the workflow is not allowed in the selected project. A platform administrator can enable the workflow for the project in the Project Management view."
+
+## Conclusion
+
+Following these steps will successfully install the Otsus template method workflow in Kaapana. This process provides a solid foundation for understanding how to add custom workflows to the Kaapana system.
+
+## Additional Resources
+
+- [Kaapana Documentation](https://kaapana.readthedocs.io/en/stable/development_guide/workflow_dev_guide.html)
+- [Workflow FAQ](https://kaapana.readthedocs.io/en/stable/faq/workflow_not_listed.html#workflow-not-allowed-in-the-selected-project)
+- [Kaapana GitHub Repository](https://github.com/kaapana/kaapana)

kaapana_ivim_osipi/ivim-fitting-method/extension/docker/Dockerfile

@@ -0,0 +1,8 @@
+FROM local-only/base-installer:latest
+
+LABEL IMAGE="dag-ivim-fitting-method"
+LABEL VERSION="0.1.0"
+LABEL BUILD_IGNORE="False"
+
+COPY files/dag_ivim-fitting_method.py /kaapana/tmp/dags/
+COPY files/ivim-fitting-method/MyIvimFittingOperator.py /kaapana/tmp/dags/ivimfitting_method/

kaapana_ivim_osipi/ivim-fitting-method/extension/docker/files/dag_ivim-fitting_method.py

@@ -0,0 +1,68 @@
+from airflow.utils.log.logging_mixin import LoggingMixin
+from airflow.utils.dates import days_ago
+from datetime import timedelta
+from airflow.models import DAG
+
+from ivimfitting_method.MyIvimFittingOperator import MyIvimFittingOperator
+from kaapana.operators.MinioOperator import MinioOperator
+from kaapana.operators.LocalWorkflowCleanerOperator import LocalWorkflowCleanerOperator
+
+ui_forms = {
+    "workflow_form": {
+        "type": "object",
+        "properties": {
+            "single_execution": {
+                "title": "single execution",
+                "description": "Should each series be processed separately?",
+                "type": "boolean",
+                "default": False,
+                "readOnly": False,
+            }
+        },
+    }
+}
+
+log = LoggingMixin().log
+
+# Default DAG args
+args = {
+    "owner": "a",
+    "start_date": days_ago(0),
+    "retries": 0,
+    "retry_delay": timedelta(seconds=30),
+    "ui_visible": True,
+    "ui_forms": ui_forms
+}
+
+# Create the DAG
+dag = DAG(
+    dag_id="ivim-fitting-method",
+    default_args=args,
+    schedule_interval=None)
+
+get_data = MinioOperator(
+    dag=dag,
+    name="download-data",
+    minio_prefix="uploads/Data/",
+    action="get",
+    source_files=["brain.bvec", "brain.bval", "brain.nii.gz"]
+)
+
+# Define your operator task
+ivim_fitting_task = MyIvimFittingOperator(
+    dag=dag,
+    input_operator=get_data
+)
+
+put_output_to_minio = MinioOperator(
+    dag=dag,
+    name="upload-out",
+    minio_prefix="uploads",
+    action="put",
+    none_batch_input_operators=[ivim_fitting_task],
+    whitelisted_file_extensions=(".nii.gz",),
+)
+
+clean = LocalWorkflowCleanerOperator(dag=dag, clean_workflow_dir=True)
+
+get_data >> ivim_fitting_task >> put_output_to_minio >> clean

kaapana_ivim_osipi/ivim-fitting-method/extension/docker/files/ivim-fitting-method/MyIvimFittingOperator.py

@@ -0,0 +1,23 @@
+from datetime import timedelta
+from kaapana.operators.KaapanaBaseOperator import KaapanaBaseOperator
+from kaapana.blueprints.kaapana_global_variables import DEFAULT_REGISTRY, KAAPANA_BUILD_VERSION
+
+
+class MyIvimFittingOperator(KaapanaBaseOperator):
+    def __init__(
+        self,
+        dag,
+        name="ivim-fitting-method",
+        execution_timeout=timedelta(seconds=3000),
+        *args,
+        **kwargs,
+    ):
+        super().__init__(
+            dag=dag,
+            name=name,
+            image=f"{DEFAULT_REGISTRY}/ivim-fitting-method:{KAAPANA_BUILD_VERSION}",
+            image_pull_secrets=["registry-secret"],
+            execution_timeout=execution_timeout,
+            *args,
+            **kwargs,
+        )

kaapana_ivim_osipi/ivim-fitting-method/extension/ivim-fitting-method-workflow/Chart.yaml

@@ -0,0 +1,15 @@
+---
+apiVersion: v2
+description: Workflow for IVIM model fitting using standardized pipelines and test data in Kaapana.
+deprecated: False
+appVersion: "0.4.0"
+name: ivim-fitting-method-workflow
+version: "0.5.0"
+type: application
+keywords:
+  - ivimworkflow
+  - ivimkaapana
+maintainers:
+  - name: IVIM Team
+    email: o.j.gurney-champion@amsterdamumc.nl
+    url: https://osipi.ismrm.org/task-forces/task-force-2-4/