Examples

.github/workflows/docs.yml

@@ -30,7 +30,7 @@ jobs:
         run: |
           python -m pip install --upgrade pip setuptools wheel
           pip install --upgrade -r requirements_deepcell.txt
-          pip install --no-deps deepcell==0.12.2
+          pip install --no-deps deepcell==0.12.3
           pip install --upgrade -r requirements.txt
           pip install -e .
           pip install --upgrade -r requirements_docs.txt

CHANGELOG.md

@@ -5,6 +5,19 @@ All notable changes to this project will be documented in this file.
 The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
 and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
 
+## [0.15.0] - 2022-09-15
+
+  - [#136](https://github.com/BodenmillerGroup/steinbock/issues/136) Fix histoCAT mask export
+  - [#146](https://github.com/BodenmillerGroup/steinbock/issues/146) Fix relabeling during stitching
+  - Upgrade deepcell 0.12.2 to 0.12.3
+  - Upgrade CellProfiler 4.2.1 to 4.2.4
+  - Upgrade other dependencies
+  - [#109](https://github.com/BodenmillerGroup/steinbock/issues/109) Support 2-6 channel TIFF files
+  - [#66](https://github.com/BodenmillerGroup/steinbock/issues/66) Include panel/image metadata in anndata exports
+  - [#117](https://github.com/BodenmillerGroup/steinbock/issues/117) Extract and process images from .zip archives file by file
+  - [#104](https://github.com/BodenmillerGroup/steinbock/issues/104) [#121](https://github.com/BodenmillerGroup/steinbock/issues/121) Documentation improvements
+  - [#139](https://github.com/BodenmillerGroup/steinbock/issues/139) [#140](https://github.com/BodenmillerGroup/steinbock/issues/140) [#141](https://github.com/BodenmillerGroup/steinbock/issues/141) API improvements
+
 ## [0.14.2] - 2022-06-29
 
   - Upgrade DeepCell (to 0.12.2) and other dependencies
@@ -327,6 +340,7 @@ Added:
 Initial release for beta testing
 
 
+[0.15.0]: https://github.com/BodenmillerGroup/steinbock/compare/v0.14.2...v0.15.0
 [0.14.2]: https://github.com/BodenmillerGroup/steinbock/compare/v0.14.1...v0.14.2
 [0.14.1]: https://github.com/BodenmillerGroup/steinbock/compare/v0.14.0...v0.14.1
 [0.14.0]: https://github.com/BodenmillerGroup/steinbock/compare/v0.13.5...v0.14.0

CONTRIBUTING.md

@@ -6,15 +6,17 @@ For major changes, please open an issue first to discuss what you would like to
 
 ## Development
 
-*steinbock* is developed using [Visual Studio Code](https://code.visualstudio.com).
+*steinbock* is developed using [Visual Studio Code](https://code.visualstudio.com). For building the *steinbock* Docker image, BuildKit needs to be enabled for Docker.
 
 For convenience, the following [Docker Compose](https://docs.docker.com/compose) services are available:
+
   - `steinbock` for running *steinbock*
   - `steinbock-debug` for debugging *steinbock* using [debugpy](https://github.com/microsoft/debugpy)
   - `pytest` for running unit tests with [pytest](https://pytest.org)
   - `pytest-debug` for debugging unit tests with [pytest](https://pytest.org) and [debugpy](https://github.com/microsoft/debugpy)
 
 Matching Visual Studio Code launch configurations are provided for debugging:
+
   - `Docker: Python General` for debugging *steinbock* using [Docker](https://www.docker.com) directly
   - `Python: Remote Attach (steinbock-debug)` for debugging *steinbock* using [Docker Compose](https://docs.docker.com/compose)
   - `Python: Remote Attach (pytest-debug)` for debugging unit tests with [pytest](https://pytest.org) using [Docker Compose](https://docs.docker.com/compose)

Dockerfile

@@ -8,7 +8,7 @@ ARG TENSORFLOW_VERSION
 ARG TENSORFLOW_SUFFIX
 ARG FIXUID_VERSION=0.5.1
 ARG ILASTIK_BINARY=ilastik-1.3.3post3-Linux.tar.bz2
-ARG CELLPROFILER_VERSION=4.2.1
+ARG CELLPROFILER_VERSION=4.2.4
 ARG CELLPROFILER_PLUGINS_VERSION=4.2.1
 ARG TZ=Europe/Zurich
 
@@ -69,7 +69,7 @@ RUN mkdir /opt/cellprofiler_plugins && \
 COPY requirements.txt /app/steinbock/requirements.txt
 COPY requirements_deepcell${TENSORFLOW_SUFFIX}.txt /app/steinbock/requirements_deepcell${TENSORFLOW_SUFFIX}.txt
 RUN python -m pip install --upgrade -r /app/steinbock/requirements_deepcell${TENSORFLOW_SUFFIX}.txt && \
-    python -m pip install --no-deps deepcell==0.12.2 && \
+    python -m pip install --no-deps deepcell==0.12.3 && \
     python -m pip install --upgrade -r /app/steinbock/requirements.txt
 ENV TF_CPP_MIN_LOG_LEVEL="2" NO_AT_BRIDGE="1"
 

docs/install-docker.md

@@ -38,11 +38,11 @@ In principle, the *steinbock* Docker container can be run on any Docker-enabled
 
 For reproducibility, it is recommended to always pull a specific release, e.g.:
 
-    docker run ghcr.io/bodenmillergroup/steinbock:0.14.2
+    docker run ghcr.io/bodenmillergroup/steinbock:0.15.0
 
 To run the *steinbock* Docker container with NVIDIA GPU support (Linux only):
 
-    docker run --gpus all ghcr.io/bodenmillergroup/steinbock-gpu:0.14.2
+    docker run --gpus all ghcr.io/bodenmillergroup/steinbock-gpu:0.15.0
 
 [Bind mounts](https://docs.docker.com/storage/bind-mounts/) can be used to make data from the host system available to the Docker container (see below). Commands that launch a graphical user interface may require further system configuration and additional arguments to `docker run` as outlined in the following.
 
@@ -51,11 +51,11 @@ To run the *steinbock* Docker container with NVIDIA GPU support (Linux only):
 
 On the command line, use the following command to run the *steinbock* Docker container:
 
-    docker run -v "C:\Data":/data ghcr.io/bodenmillergroup/steinbock:0.14.2
+    docker run -v "C:\Data":/data ghcr.io/bodenmillergroup/steinbock:0.15.0
 
-In the command above, adapt the bind mount path to your data/working directory (`C:\Data`) and the *steinbock* Docker container version (`0.14.2`) as needed. To simplify the use of the *steinbock* command-line interface, it is recommended to set up a `steinbock` command alias:
+In the command above, adapt the bind mount path to your data/working directory (`C:\Data`) and the *steinbock* Docker container version (`0.15.0`) as needed. To simplify the use of the *steinbock* command-line interface, it is recommended to set up a `steinbock` command alias:
 
-    doskey steinbock=docker run -v "C:\Data":/data ghcr.io/bodenmillergroup/steinbock:0.14.2 $*
+    doskey steinbock=docker run -v "C:\Data":/data ghcr.io/bodenmillergroup/steinbock:0.15.0 $*
 
 The created command alias is retained for the current session and enables running `steinbock` from the current command line without typing the full Docker command, for example:
 
@@ -68,19 +68,21 @@ The created command alias is retained for the current session and enables runnin
 
 On the terminal, use the following command to run the *steinbock* Docker container:
 
-    docker run -v /path/to/data:/data -v /tmp/.X11-unix:/tmp/.X11-unix -v ~/.Xauthority:/home/steinbock/.Xauthority:ro -u $(id -u):$(id -g) -e DISPLAY ghcr.io/bodenmillergroup/steinbock:0.14.2
+    docker run -v /path/to/data:/data -v /tmp/.X11-unix:/tmp/.X11-unix -v ~/.Xauthority:/home/steinbock/.Xauthority:ro -u $(id -u):$(id -g) -e DISPLAY ghcr.io/bodenmillergroup/steinbock:0.15.0
 
 To run the *steinbock* Docker container with NVIDIA GPU support, use `steinbock-gpu` Docker image instead:
 
-    docker run -v /path/to/data:/data -v /tmp/.X11-unix:/tmp/.X11-unix -v ~/.Xauthority:/home/steinbock/.Xauthority:ro -u $(id -u):$(id -g) -e DISPLAY --gpus all ghcr.io/bodenmillergroup/steinbock-gpu:0.14.2
+    docker run -v /path/to/data:/data -v /tmp/.X11-unix:/tmp/.X11-unix -v ~/.Xauthority:/home/steinbock/.Xauthority:ro -u $(id -u):$(id -g) -e DISPLAY --gpus all ghcr.io/bodenmillergroup/steinbock-gpu:0.15.0
 
-In the commands above, adapt the bind mount path to your data/working directory (`/path/to/data`) and the *steinbock* Docker container version (`0.14.2`) as needed. To simplify the use of the *steinbock* command-line interface, it is recommended to set up a `steinbock` command alias:
+In the commands above, adapt the bind mount path to your data/working directory (`/path/to/data`) and the *steinbock* Docker container version (`0.15.0`) as needed. The environment variable `DISPLAY` as well as the bind mounts for `/tmp/.X11-unix` and `~/.Xauthority` are required only when running graphical user interfaces.
 
-    alias steinbock="docker run -v /path/to/data:/data -v /tmp/.X11-unix:/tmp/.X11-unix -v ~/.Xauthority:/home/steinbock/.Xauthority:ro -u $(id -u):$(id -g) -e DISPLAY ghcr.io/bodenmillergroup/steinbock:0.14.2"
+To simplify the use of the *steinbock* command-line interface, it is recommended to set up a `steinbock` command alias:
+
+    alias steinbock="docker run -v /path/to/data:/data -v /tmp/.X11-unix:/tmp/.X11-unix -v ~/.Xauthority:/home/steinbock/.Xauthority:ro -u $(id -u):$(id -g) -e DISPLAY ghcr.io/bodenmillergroup/steinbock:0.15.0"
 
 To run the *steinbock* Docker container with NVIDIA GPU support, use `steinbock-gpu` Docker image instead:
 
-    alias steinbock="docker run -v /path/to/data:/data -v /tmp/.X11-unix:/tmp/.X11-unix -v ~/.Xauthority:/home/steinbock/.Xauthority:ro -u $(id -u):$(id -g) -e DISPLAY --gpus all ghcr.io/bodenmillergroup/steinbock-gpu:0.14.2"
+    alias steinbock="docker run -v /path/to/data:/data -v /tmp/.X11-unix:/tmp/.X11-unix -v ~/.Xauthority:/home/steinbock/.Xauthority:ro -u $(id -u):$(id -g) -e DISPLAY --gpus all ghcr.io/bodenmillergroup/steinbock-gpu:0.15.0"
 
 The created command alias is retained for the current session and enables running `steinbock` from the current terminal without typing the full Docker command, for example:
 
@@ -95,11 +97,13 @@ The created command alias is retained for the current session and enables runnin
 
 On the terminal, use the following command to run the *steinbock* Docker container (Docker must be running):
 
-    docker run -v /path/to/data:/data -v /tmp/.X11-unix:/tmp/.X11-unix -v ~/.Xauthority:/home/steinbock/.Xauthority:ro -u $(id -u):$(id -g) -e DISPLAY=$(hostname):0 ghcr.io/bodenmillergroup/steinbock:0.14.2
+    docker run -v /path/to/data:/data -v /tmp/.X11-unix:/tmp/.X11-unix -v ~/.Xauthority:/home/steinbock/.Xauthority:ro -u $(id -u):$(id -g) -e DISPLAY=$(hostname):0 ghcr.io/bodenmillergroup/steinbock:0.15.0
+
+In the command above, adapt the bind mount path to your data/working directory (`/path/to/data`) and the *steinbock* Docker container version (`0.15.0`) as needed. The environment variable `DISPLAY` as well as the bind mounts for `/tmp/.X11-unix` and `~/.Xauthority` are required only when running graphical user interfaces.
 
-In the command above, adapt the bind mount path to your data/working directory (`/path/to/data`) and the *steinbock* Docker container version (`0.14.2`) as needed. To simplify the use of the *steinbock* command-line interface, it is recommended to set up a `steinbock` command alias:
+To simplify the use of the *steinbock* command-line interface, it is recommended to set up a `steinbock` command alias:
 
-    alias steinbock="docker run -v /path/to/data:/data -v /tmp/.X11-unix:/tmp/.X11-unix -v ~/.Xauthority:/home/steinbock/.Xauthority:ro -u $(id -u):$(id -g) -e DISPLAY=$(hostname):0 ghcr.io/bodenmillergroup/steinbock:0.14.2"
+    alias steinbock="docker run -v /path/to/data:/data -v /tmp/.X11-unix:/tmp/.X11-unix -v ~/.Xauthority:/home/steinbock/.Xauthority:ro -u $(id -u):$(id -g) -e DISPLAY=$(hostname):0 ghcr.io/bodenmillergroup/steinbock:0.15.0"
 
 The created command alias is retained for the current session and enables running `steinbock` from the current terminal without typing the full Docker command, for example:
 

docs/install-python.md

@@ -34,7 +34,7 @@ To install all extras, use the `all` extra:
         # choose between one of the following:
         # pip install --upgrade -r requirements_deepcell.txt
         # pip install --upgrade -r requirements_deepcell-gpu.txt
-        pip install --no-deps deepcell==0.12.2
+        pip install --no-deps deepcell==0.12.3
         pip install --upgrade -r requirements.txt
         pip install --no-deps "steinbock[all]"
 

examples/01_preprocessing_if.ipynb

@@ -0,0 +1,239 @@
+{
+ "cells": [
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "4544b34e",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "import numpy as np\n",
+    "import pandas as pd\n",
+    "import tifffile\n",
+    "import xtiff\n",
+    "\n",
+    "from pathlib import Path\n",
+    "\n",
+    "from steinbock import io\n",
+    "from steinbock.preprocessing import external\n",
+    "\n",
+    "import helpers"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "7b752cfb",
+   "metadata": {},
+   "source": [
+    "# Multiplex images preprocessing pipeline\n",
+    "\n",
+    "This pipeline will run image segmentation within the steinbock workframe. It can be applied to any multiplex image stacks. The full pipeline consists of three separate notebooks that have to be run successively: `preprocessing` (current notebook), `segmentation`, and `measurement`.\n",
+    "\n",
+    "Before running your own script please check the [steinbock documentation](https://bodenmillergroup.github.io/steinbock).\n",
+    "\n",
+    "*Installation*  \n",
+    "To install the required python environment, follow the instructions here: https://bodenmillergroup.github.io/steinbock/latest/install-python/"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "83d9d1e0",
+   "metadata": {},
+   "source": [
+    "## Settings\n",
+    "\n",
+    "Example data can be downloaded using the `download_examples.ipynb` script.\n",
+    "\n",
+    "### Input and output directories\n",
+    "Edit the working directory if needed (by default, the `examples` folder).\n",
+    "\n",
+    "Folder structure:"
+   ]
+  },
+  {
+   "cell_type": "raw",
+   "id": "4e2d0ccd",
+   "metadata": {},
+   "source": [
+    "steinbock data/working directory  \n",
+    "├── panel.csv (user-provided)  \n",
+    "├── raw (*.tiff stacks, user-provided)\n",
+    "├── img (*.tiff image stacks, user-provided or generated from stacks in \"raw\")\n",
+    "├── masks (created by this script)  \n",
+    "├── intensities (created by this script)  \n",
+    "├── regionprops (created by this script)  \n",
+    "└── neighbors (created by this script)  "
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "09658e6a",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "working_dir = Path(\".\")\n",
+    "\n",
+    "# Paths to zipped acquisition files\n",
+    "raw_dir = working_dir / \"raw\"\n",
+    "\n",
+    "# Output directories\n",
+    "img_dir = working_dir / \"img\"\n",
+    "\n",
+    "# Create directories (if they do not already exist)\n",
+    "raw_dir.mkdir(exist_ok=True)\n",
+    "img_dir.mkdir(exist_ok=True)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "8e98844b",
+   "metadata": {},
+   "source": [
+    "## Antibody panel\n",
+    "\n",
+    "### Import the panel\n",
+    "The antibody panel should meet the steinbock format: https://bodenmillergroup.github.io/steinbock/latest/file-types/#panel  \n",
+    "\n",
+    "Customized panels should contain the following columns:\n",
+    "+ `channel`: unique channel id, typically metal and isotope mass (e.g. `Ir191`) or fluorophore.\n",
+    "+ `name`: unique channel name.\n",
+    "+ `deepcell`: channels to use for segmentation (1=nuclear, 2=membrane, empty/NaN=ignored).\n",
+    "+ `keep`: *(optional)* 1 for channels to preprocess, 0 for channels to ignore"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "964e4be4",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "panel_file = working_dir / \"panel.csv\""
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "f7fe4ff7",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "panel = io.read_panel(panel_file)\n",
+    "nb_of_channels = len(panel)\n",
+    "panel.head()"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "1f625996",
+   "metadata": {},
+   "source": [
+    "## Re-shape `.tiff` stacks\n",
+    "\n",
+    "The multichannel image stacks provided by the user in the `raw` folder is reshaped as \"CYX\" (channel, height, width). Other dimensions are ignored: for a time series for instance, please provide one stack per time point.\n",
+    "\n",
+    "Alternately, the user can provide single-channel `tiff` files that will be converted to multichannel stacks. For this, the `generate_from_single_tiffs` variable has to be set to `True` and all images belonging to the same stack should start with the same prefix.\n",
+    "\n",
+    "If the `raw` folder is empty, it is assumed that  properly shaped stacks are directly provided by the user in the `img` folder."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "127cabad",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "generate_from_single_tiffs = True\n",
+    "\n",
+    "raw_tiffs = sorted(Path(raw_dir).glob(\"*.tiff\"))"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "d0b7852e",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "if raw_tiffs and generate_from_single_tiffs:\n",
+    "    from itertools import cycle\n",
+    "    from os.path import commonprefix\n",
+    "    \n",
+    "    for i, j in zip(range(len(raw_tiffs)),\n",
+    "                    cycle(range(nb_of_channels))):\n",
+    "        cur_img = tifffile.imread(raw_tiffs[i])\n",
+    "        cur_img = np.expand_dims(cur_img, axis=0)\n",
+    "        \n",
+    "        if (j == 0):\n",
+    "            img = cur_img\n",
+    "            img_names = []\n",
+    "        else:\n",
+    "            img = np.concatenate((img, cur_img), axis = 0)\n",
+    "            \n",
+    "        img_names.append(raw_tiffs[i].name)\n",
+    "        \n",
+    "        if (j == (nb_of_channels-1)):\n",
+    "            img_file = img_dir / (commonprefix(img_names) + \".tiff\")\n",
+    "            tifffile.imwrite(img_file, img,\n",
+    "                             photometric='minisblack',\n",
+    "                             metadata={'axes': 'CYX'}) "
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "1ebc673c",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "if raw_tiffs and not generate_from_single_tiffs:\n",
+    "    for raw_tiff in raw_tiffs:\n",
+    "        with tifffile.TiffFile(raw_tiff) as tif:\n",
+    "            volume = tif.asarray()\n",
+    "            axes = tif.series[0].axes\n",
+    "            imagej_metadata = tif.imagej_metadata\n",
+    "            \n",
+    "            img_file = img_dir / raw_tiff.name\n",
+    "            tifffile.imwrite(img_file, volume,\n",
+    "                             photometric='minisblack',\n",
+    "                             metadata={'axes': 'CYX'}) "
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "7788a1af",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "!conda list"
+   ]
+  }
+ ],
+ "metadata": {
+  "jupytext": {
+   "formats": "ipynb,py:percent"
+  },
+  "kernelspec": {
+   "display_name": "Python 3 (ipykernel)",
+   "language": "python",
+   "name": "python3"
+  },
+  "language_info": {
+   "codemirror_mode": {
+    "name": "ipython",
+    "version": 3
+   },
+   "file_extension": ".py",
+   "mimetype": "text/x-python",
+   "name": "python",
+   "nbconvert_exporter": "python",
+   "pygments_lexer": "ipython3",
+   "version": "3.8.15"
+  }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 5
+}