notebooks/community/model_garden/model_garden_pytorch_biomedclip.ipynb

{
  "cells": [
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "cellView": "form",
        "id": "7d9bbf86da5e"
      },
      "outputs": [],
      "source": [
        "# Copyright 2025 Google LLC\n",
        "#\n",
        "# Licensed under the Apache License, Version 2.0 (the \"License\");\n",
        "# you may not use this file except in compliance with the License.\n",
        "# You may obtain a copy of the License at\n",
        "#\n",
        "#     https://www.apache.org/licenses/LICENSE-2.0\n",
        "#\n",
        "# Unless required by applicable law or agreed to in writing, software\n",
        "# distributed under the License is distributed on an \"AS IS\" BASIS,\n",
        "# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n",
        "# See the License for the specific language governing permissions and\n",
        "# limitations under the License."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "99c1c3fc2ca5"
      },
      "source": [
        "# Vertex AI Model Garden - BiomedCLIP\n",
        "\n",
        "<table><tbody><tr>\n",
        "  <td style=\"text-align: center\">\n",
        "    <a href=\"https://console.cloud.google.com/vertex-ai/workbench/instances\">\n",
        "      <img alt=\"Workbench logo\" src=\"https://lh3.googleusercontent.com/UiNooY4LUgW_oTvpsNhPpQzsstV5W8F7rYgxgGBD85cWJoLmrOzhVs_ksK_vgx40SHs7jCqkTkCk=e14-rj-sc0xffffff-h130-w32\" width=\"32px\"><br> Run in Workbench\n",
        "    </a>\n",
        "  </td>\n",
        "  <td style=\"text-align: center\">\n",
        "    <a href=\"https://console.cloud.google.com/vertex-ai/colab/import/https:%2F%2Fraw.githubusercontent.com%2FGoogleCloudPlatform%2Fvertex-ai-samples%2Fmain%2Fnotebooks%2Fcommunity%2Fmodel_garden%2Fmodel_garden_pytorch_biomedclip.ipynb\">\n",
        "      <img alt=\"Google Cloud Colab Enterprise logo\" src=\"https://lh3.googleusercontent.com/JmcxdQi-qOpctIvWKgPtrzZdJJK-J3sWE1RsfjZNwshCFgE_9fULcNpuXYTilIR2hjwN\" width=\"32px\"><br> Run in Colab Enterprise\n",
        "    </a>\n",
        "  </td>\n",
        "  <td style=\"text-align: center\">\n",
        "    <a href=\"https://github.com/GoogleCloudPlatform/vertex-ai-samples/blob/main/notebooks/community/model_garden/model_garden_pytorch_biomedclip.ipynb\">\n",
        "      <img alt=\"GitHub logo\" src=\"https://cloud.google.com/ml-engine/images/github-logo-32px.png\" width=\"32px\"><br> View on GitHub\n",
        "    </a>\n",
        "  </td>\n",
        "</tr></tbody></table>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "-ThBX_vbxEJD"
      },
      "source": [
        "## Overview\n",
        "\n",
        "This notebook demonstrates how to serve [BiomedCLIP](https://arxiv.org/pdf/2303.00915.pdf), a biomedical vision-language foundation model that is pretrained on PMC-15M, a dataset of 15 million figure-caption pairs extracted from biomedical research articles in PubMed Central, using contrastive learning. BiomedCLIP can perform various tasks including zero-shot image classification and visual question answering.\n",
        "\n",
        "BiomedCLIP [model checkpoints](https://huggingface.co/microsoft/BiomedCLIP-PubMedBERT_256-vit_base_patch16_224) are available on the Hugging Face Hub.\n",
        "\n",
        "In this notebook, we will show you how to:\n",
        "- Serve BiomedCLIP using Vertex AI\n",
        "- Run zero-shot image classification with BiomedCLIP\n",
        "\n",
        "### File a bug\n",
        "\n",
        "File a bug on [GitHub](https://github.com/GoogleCloudPlatform/vertex-ai-samples/issues/new) if you encounter any issue with the notebook.\n",
        "\n",
        "### Costs\n",
        "\n",
        "This tutorial uses billable components of Google Cloud:\n",
        "\n",
        "* Vertex AI\n",
        "* Cloud Storage\n",
        "\n",
        "Learn about [Vertex AI pricing](https://cloud.google.com/vertex-ai/pricing), [Cloud Storage pricing](https://cloud.google.com/storage/pricing), and use the [Pricing Calculator](https://cloud.google.com/products/calculator/) to generate a cost estimate based on your projected usage."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "264c07757582"
      },
      "source": [
        "## Before you begin\n",
        "\n",
        "**NOTE**: Jupyter runs lines prefixed with `!` as shell commands, and it interpolates Python variables prefixed with `$` into these commands."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "cellView": "form",
        "id": "0172207da484"
      },
      "outputs": [],
      "source": [
        "# @title Setup Google Cloud project\n",
        "\n",
        "# @markdown 1. [Make sure that billing is enabled for your project](https://cloud.google.com/billing/docs/how-to/modify-project).\n",
        "\n",
        "# @markdown 2. **[Optional]** Set region. If not set, the region will be set automatically according to Colab Enterprise environment.\n",
        "\n",
        "REGION = \"\"  # @param {type:\"string\"}\n",
        "\n",
        "# @markdown 3. If you want to run predictions with A100 80GB or H100 GPUs, we recommend using the regions listed below. **NOTE:** Make sure you have associated quota in selected regions. Click the links to see your current quota for each GPU type: [Nvidia A100 80GB](https://console.cloud.google.com/iam-admin/quotas?metric=aiplatform.googleapis.com%2Fcustom_model_serving_nvidia_a100_80gb_gpus), [Nvidia H100 80GB](https://console.cloud.google.com/iam-admin/quotas?metric=aiplatform.googleapis.com%2Fcustom_model_serving_nvidia_h100_gpus). You can request for quota following the instructions at [\"Request a higher quota\"](https://cloud.google.com/docs/quota/view-manage#requesting_higher_quota).\n",
        "\n",
        "# @markdown > | Machine Type | Accelerator Type | Recommended Regions |\n",
        "# @markdown | ----------- | ----------- | ----------- |\n",
        "# @markdown | a2-ultragpu-1g | 1 NVIDIA_A100_80GB | us-central1, us-east4, europe-west4, asia-southeast1, us-east4 |\n",
        "# @markdown | a3-highgpu-2g | 2 NVIDIA_H100_80GB | us-west1, asia-southeast1, europe-west4 |\n",
        "# @markdown | a3-highgpu-4g | 4 NVIDIA_H100_80GB | us-west1, asia-southeast1, europe-west4 |\n",
        "# @markdown | a3-highgpu-8g | 8 NVIDIA_H100_80GB | us-central1, europe-west4, us-west1, asia-southeast1 |\n",
        "\n",
        "# Import the necessary packages\n",
        "\n",
        "# Upgrade Vertex AI SDK.\n",
        "! pip3 install --upgrade --quiet 'google-cloud-aiplatform>=1.84.0'\n",
        "\n",
        "import importlib\n",
        "import os\n",
        "from typing import Tuple\n",
        "\n",
        "import numpy as np\n",
        "from google.cloud import aiplatform\n",
        "\n",
        "# Upgrade Vertex AI SDK.\n",
        "if os.environ.get(\"VERTEX_PRODUCT\") != \"COLAB_ENTERPRISE\":\n",
        "    ! pip install --upgrade tensorflow\n",
        "! git clone https://github.com/GoogleCloudPlatform/vertex-ai-samples.git\n",
        "\n",
        "common_util = importlib.import_module(\n",
        "    \"vertex-ai-samples.community-content.vertex_model_garden.model_oss.notebook_util.common_util\"\n",
        ")\n",
        "\n",
        "LABEL = \"biomedclip_serve\"\n",
        "models, endpoints = {}, {}\n",
        "\n",
        "# Get the default cloud project id.\n",
        "PROJECT_ID = os.environ[\"GOOGLE_CLOUD_PROJECT\"]\n",
        "\n",
        "# Get the default region for launching jobs.\n",
        "if not REGION:\n",
        "    REGION = os.environ[\"GOOGLE_CLOUD_REGION\"]\n",
        "\n",
        "# Initialize Vertex AI API.\n",
        "print(\"Initializing Vertex AI API.\")\n",
        "aiplatform.init(project=PROJECT_ID, location=REGION)\n",
        "\n",
        "! gcloud config set project $PROJECT_ID\n",
        "import vertexai\n",
        "\n",
        "vertexai.init(\n",
        "    project=PROJECT_ID,\n",
        "    location=REGION,\n",
        ")"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "cellView": "form",
        "id": "354da31189dc"
      },
      "outputs": [],
      "source": [
        "# @title Set the model parameters\n",
        "\n",
        "base_model_name = \"biomedclip\"\n",
        "PUBLISHER_MODEL_NAME = (\n",
        "    f\"publishers/microsoft/models/microsoft-biomedclip@{base_model_name}\"\n",
        ")\n",
        "model_id = \"hf-hub:microsoft/BiomedCLIP-PubMedBERT_256-vit_base_patch16_224\"\n",
        "accelerator_type = \"NVIDIA_L4\"  # @param [\"NVIDIA_L4\", \"NVIDIA_TESLA_V100\"]\n",
        "TASK = \"zero-shot-image-classification\"\n",
        "PRECISION = \"amp\"\n",
        "SERVE_PORT = 7080\n",
        "\n",
        "# @markdown Set use_dedicated_endpoint to False if you don't want to use [dedicated endpoint](https://cloud.google.com/vertex-ai/docs/general/deployment#create-dedicated-endpoint). Note that [dedicated endpoint does not support VPC Service Controls](https://cloud.google.com/vertex-ai/docs/predictions/choose-endpoint-type), uncheck the box if you are using VPC-SC.\n",
        "use_dedicated_endpoint = True  # @param {type:\"boolean\"}\n",
        "\n",
        "if accelerator_type == \"NVIDIA_L4\":\n",
        "    machine_type = \"g2-standard-8\"\n",
        "    accelerator_count = 1\n",
        "elif accelerator_type == \"NVIDIA_TESLA_V100\":\n",
        "    machine_type = \"n1-standard-8\"\n",
        "    accelerator_count = 4\n",
        "else:\n",
        "    raise ValueError(f\"Recommended GPU setting not found for: {accelerator_type}\")\n",
        "\n",
        "\n",
        "common_util.check_quota(\n",
        "    project_id=PROJECT_ID,\n",
        "    region=REGION,\n",
        "    accelerator_type=accelerator_type,\n",
        "    accelerator_count=1,\n",
        "    is_for_training=False,\n",
        ")"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "cellView": "form",
        "id": "6RXJEDT-dLlL"
      },
      "outputs": [],
      "source": [
        "# @title [Option 1] Deploy with Model Garden SDK\n",
        "\n",
        "# @markdown Deploy with Gen AI model-centric SDK. This section uploads the prebuilt model to Model Registry and deploys it to a Vertex AI Endpoint. It takes 15 minutes to 1 hour to finish depending on the size of the model. See [use open models with Vertex AI](https://cloud.google.com/vertex-ai/generative-ai/docs/open-models/use-open-models) for documentation on other use cases.\n",
        "from vertexai.preview import model_garden\n",
        "\n",
        "model = model_garden.OpenModel(PUBLISHER_MODEL_NAME)\n",
        "endpoints[LABEL] = model.deploy(\n",
        "    machine_type=machine_type,\n",
        "    accelerator_type=accelerator_type,\n",
        "    accelerator_count=accelerator_count,\n",
        "    use_dedicated_endpoint=use_dedicated_endpoint,\n",
        "    accept_eula=True,  # Accept the End User License Agreement (EULA) on the model card before deploy. Otherwise, the deployment will be forbidden.\n",
        ")"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "cellView": "form",
        "id": "OYNaiywCdNYS"
      },
      "outputs": [],
      "source": [
        "# @title [Option 2] Deploy with customized configs\n",
        "\n",
        "# @markdown This section uploads the model to Model Registry and deploys it on the Endpoint.\n",
        "\n",
        "# @markdown The model deployment step will take ~10 minutes to complete.\n",
        "\n",
        "# The pre-built serving docker image.\n",
        "SERVE_DOCKER_URI = \"us-docker.pkg.dev/vertex-ai/vertex-vision-model-garden-dockers/pytorch-open-clip-serve:20250313_0922_RC00\"\n",
        "\n",
        "\n",
        "def deploy_model(\n",
        "    model_name: str,\n",
        "    model_id: str,\n",
        "    task: str,\n",
        "    precision: str,\n",
        "    machine_type: str,\n",
        "    accelerator_type: str,\n",
        "    accelerator_count: int,\n",
        ") -> Tuple[aiplatform.Model, aiplatform.Endpoint]:\n",
        "    \"\"\"Deploys trained models into Vertex AI.\"\"\"\n",
        "    endpoint = aiplatform.Endpoint.create(display_name=f\"{model_name}-endpoint\")\n",
        "    serving_env = {\n",
        "        \"MODEL\": model_id,\n",
        "        \"TASK\": task,\n",
        "        \"PRECISION\": precision,\n",
        "        \"DEPLOY_SOURCE\": \"notebook\",\n",
        "    }\n",
        "    # If the model_id is a GCS path, use artifact_uri to pass it to serving docker.\n",
        "    artifact_uri = model_id if model_id.startswith(\"gs://\") else None\n",
        "    model = aiplatform.Model.upload(\n",
        "        display_name=model_name,\n",
        "        serving_container_image_uri=SERVE_DOCKER_URI,\n",
        "        serving_container_ports=[SERVE_PORT],\n",
        "        serving_container_predict_route=\"/predictions/transformers_serving\",\n",
        "        serving_container_health_route=\"/ping\",\n",
        "        serving_container_environment_variables=serving_env,\n",
        "        artifact_uri=artifact_uri,\n",
        "        model_garden_source_model_name=\"publishers/microsoft/models/microsoft-biomedclip\",\n",
        "    )\n",
        "    model.deploy(\n",
        "        endpoint=endpoint,\n",
        "        machine_type=machine_type,\n",
        "        accelerator_type=accelerator_type,\n",
        "        accelerator_count=accelerator_count,\n",
        "        deploy_request_timeout=1800,\n",
        "        system_labels={\n",
        "            \"NOTEBOOK_NAME\": \"model_garden_pytorch_biomedclip.ipynb\",\n",
        "            \"NOTEBOOK_ENVIRONMENT\": common_util.get_deploy_source(),\n",
        "        },\n",
        "    )\n",
        "    return model, endpoint\n",
        "\n",
        "\n",
        "models[LABEL], endpoints[LABEL] = deploy_model(\n",
        "    model_name=common_util.get_job_name_with_datetime(prefix=\"biomedclip-serve\"),\n",
        "    model_id=model_id,\n",
        "    task=TASK,\n",
        "    precision=PRECISION,\n",
        "    machine_type=machine_type,\n",
        "    accelerator_type=accelerator_type,\n",
        "    accelerator_count=1,\n",
        ")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "80b3fd2ace09"
      },
      "source": [
        "NOTE: The model weights will be downloaded after the deployment succeeds. Thus additional 5 minutes of waiting time is needed **after** the above model deployment step succeeds and before you run the next step below. Otherwise you might see a `ServiceUnavailable: 503 502:Bad Gateway` error when you send requests to the endpoint.\n",
        "\n",
        "Once deployment succeeds, you can send requests to the endpoint with texts and images for zero-shot classification."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "cellView": "form",
        "id": "4ab04da3ec9a"
      },
      "outputs": [],
      "source": [
        "# @title Predict\n",
        "\n",
        "# # Loads an existing endpoint as below.\n",
        "# endpoint_name = endpoints[\"biomedclip_endpoint\"].name\n",
        "# aip_endpoint_name = (\n",
        "#     f\"projects/{PROJECT_ID}/locations/{REGION}/endpoints/{endpoint_name}\"\n",
        "# )\n",
        "# endpoint = aiplatform.Endpoint(aip_endpoint_name)\n",
        "\n",
        "\n",
        "def download_image(url: str) -> str:\n",
        "    \"\"\"Downloads an image from the given URL.\n",
        "\n",
        "    Args:\n",
        "      url: The URL of the image to download.\n",
        "\n",
        "    Returns:\n",
        "      base64 encoded image.\n",
        "    \"\"\"\n",
        "    !wget -O image.jpg $url\n",
        "    !base64 image.jpg > image.txt\n",
        "    return open(\"image.txt\").read()\n",
        "\n",
        "\n",
        "instances = [\n",
        "    {\n",
        "        \"text\": \"This is a photo of adenocarcinoma histopathology\",\n",
        "        \"image\": download_image(\n",
        "            \"https://huggingface.co/microsoft/BiomedCLIP-PubMedBERT_256-vit_base_patch16_224/resolve/main/example_data/biomed_image_classification_example_data/bone_X-ray.jpg\"\n",
        "        ),\n",
        "    },\n",
        "    {\"text\": \"This is a photo of brain MRI\"},\n",
        "    {\"text\": \"This is a photo of covid line chart\"},\n",
        "    {\"text\": \"This is a photo of squamous cell carcinoma histopathology\"},\n",
        "    {\"text\": \"This is a photo of immunohistochemistry histopathology\"},\n",
        "    {\"text\": \"This is a photo of bone X-ray\"},\n",
        "    {\"text\": \"This is a photo of chest X-ray\"},\n",
        "    {\"text\": \"This is a photo of hematoxylin and eosin histopathology\"},\n",
        "    {\"text\": \"This is a photo of pie chart\"},\n",
        "]\n",
        "response = endpoints[LABEL].predict(instances=instances)\n",
        "\n",
        "print(response.predictions)\n",
        "\n",
        "selected_idx = np.argmax(response.predictions[0])\n",
        "print(f\"Selected class: {instances[selected_idx]['text']}\")"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "cellView": "form",
        "id": "911406c1561e"
      },
      "outputs": [],
      "source": [
        "# @title Delete the models and endpoints\n",
        "# @markdown  Delete the experiment models and endpoints to recycle the resources\n",
        "# @markdown  and avoid unnecessary continuous charges that may incur.\n",
        "\n",
        "# Undeploy model and delete endpoint.\n",
        "for endpoint in endpoints.values():\n",
        "    endpoint.delete(force=True)\n",
        "\n",
        "# Delete models.\n",
        "for model in models.values():\n",
        "    model.delete()"
      ]
    }
  ],
  "metadata": {
    "colab": {
      "name": "model_garden_pytorch_biomedclip.ipynb",
      "toc_visible": true
    },
    "kernelspec": {
      "display_name": "Python 3",
      "name": "python3"
    }
  },
  "nbformat": 4,
  "nbformat_minor": 0
}