finetune_colpali.ipynb

{
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "i72MPXVuA7Is"
      },
      "source": [
        "# Fine-tune ColPali 👀\n",
        "\n",
        "[![Colab](https://img.shields.io/badge/Open_in_Colab-F9AB00?logo=googlecolab&logoColor=fff&style=for-the-badge)](https://colab.research.google.com/github/tonywu71/colpali-cookbooks/blob/main/examples/finetune_colpali.ipynb)\n",
        "[![GitHub](https://img.shields.io/badge/ColPali_Cookbooks-100000?style=for-the-badge&logo=github&logoColor=white)](https://github.com/tonywu71/colpali-cookbooks)\n",
        "[![arXiv](https://img.shields.io/badge/arXiv-2407.01449-b31b1b.svg?style=for-the-badge)](https://arxiv.org/abs/2407.01449)\n",
        "[![Hugging Face](https://img.shields.io/badge/Vidore-FFD21E?style=for-the-badge&logo=huggingface&logoColor=000)](https://huggingface.co/vidore)\n",
        "[![X](https://img.shields.io/badge/Thread-%23000000?style=for-the-badge&logo=X&logoColor=white)](https://x.com/tonywu_71/status/1809183824464560138)\n",
        "\n",
        "## Introduction\n",
        "\n",
        "With our new model *ColPali*, we propose to leverage VLMs to construct efficient multi-vector embeddings in the visual space for document retrieval. By feeding the ViT output patches from PaliGemma-3B to a linear projection, we create a multi-vector representation of documents. We train the model to maximize the similarity between these document embeddings and the query embeddings, following the ColBERT method.\n",
        "\n",
        "Using ColPali removes the need for potentially complex and brittle layout recognition and OCR pipelines with a single model that can take into account both the textual and visual content (layout, charts, ...) of a document.\n",
        "\n",
        "![ColPali Architecture](https://github.com/tonywu71/colpali-cookbooks/blob/main/assets/architecture/colpali_architecture.jpeg?raw=true)\n",
        "\n",
        "The following notebook guide you through how to fine-tune ColPali to improve its retrieval performance for the data distribution of your use case. In particular, we will fine-tune ColPali on [VDSID-French](https://huggingface.co/datasets/vidore/vdsid_french), a French-language document retrieval dataset.\n",
        "\n",
        "## What if I want to use my own documents to fine-tune ColPali?\n",
        "\n",
        "If you are a company, you probably want to fine-tune ColPali on your own documents. But they probably lack the queries that are necessary to train a vision retrieval model... But fear not as Daniel van Strien has published an awesome [🤗 blog post](https://danielvanstrien.xyz/posts/post-with-code/colpali/2024-09-23-generate_colpali_dataset.html) on how to use VLMs to generate quality queries for your PDFs and to create a dataset that you can use for fine-tuning.\n",
        "\n",
        "## Hardware Requirements\n",
        "\n",
        "This notebook was tested on GCP VM with an A100-40GB GPU. I recommend seting this VM up with [SkyPilot](https://github.com/skypilot-org/skypilot) using this [config](https://github.com/tonywu71/colpali-cookbooks/blob/main/skypilot/a100/config.yaml). You should also be able to run it on a smaller GPU but you'll need a stronger quantization strategy and a smaller batch size."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 1,
      "metadata": {},
      "outputs": [],
      "source": [
        "# ==========================     USER INPUTS     ==========================\n",
        "\n",
        "# Define the name used for the model you will push to the HuggingFace Hub.\n",
        "# Leave it empty to disable pushing the model.\n",
        "hf_pushed_model_name = \"tonywu71/finetune_colpali_v1_2-vdsid_french-4bit\"\n",
        "\n",
        "# Define the name used for the WandB experiment. Leave it empty to disable WandB logging.\n",
        "# In particular, leave it empty if you don't have a WandB account.\n",
        "wandb_experiment_name = \"finetune_colpali_v1_2-vdsid_french-4bit\"\n",
        "\n",
        "# =========================================================================\n",
        "\n",
        "if not wandb_experiment_name:\n",
        "    print(\"WandB logging is disabled.\")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "## Installation"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "This notebook leverages [`colpali-engine`](https://github.com/illuin-tech/colpali), the official implementation of ColPali. This package also contains the training code (processor, collator, trainer...) for fine-tuning ColPali on your own dataset."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 2,
      "metadata": {},
      "outputs": [],
      "source": [
        "!pip install -q -U \"colpali-engine[train]>=0.3.0,<0.4.0\""
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "VWFA1qPc9Z4J"
      },
      "source": [
        "## Login to a HuggingFace account"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "Because ColPali uses the [PaliGemma3B](https://huggingface.co/google/paligemma-3b-mix-448) checkpoints, you need to accept its terms and conditions before using it. Once accepted, use the following cell to login to your HuggingFace account."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 3,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 145,
          "referenced_widgets": [
            "dd7b36d536434e019a3d84b951c96831",
            "2b8d3565706c4413af5d0c0a30ad132c",
            "b2d9f77ebfb144de9e978e309d640041",
            "5911b53437694805b28573fbeb1594d0",
            "e45351ae2e4f4454a1eea6d56f0398e7",
            "74c83bea924d470d99608f17cb74d338",
            "eb22c372a98e454e997cba3a562c7439",
            "197cf7905f7b493d90a4cf03b66cc0b9",
            "36acb5cf8816463ca36812913f2d163e",
            "730882a26b9741649831beaa86d76db2",
            "fcd47ca1590b43559d08c04f0bb4d164",
            "de744a456e7a49569eaa973837732c7b",
            "bf4518dfbee14600a799f300b00ab41b",
            "bd117e3f5a4240a982ff268c95fb2b4e",
            "55ddc088c8d149d584d5586680123f7a",
            "b4da694310f04f8fab4a9b64a526509e",
            "359825e9c7df4f01b54677b5b27f7916",
            "c8210da876ed4d6ab02d6c1897b10528",
            "631d08c360674a2b9b5aad1a388029fd",
            "8c5818ee930e49fdaeb6249a35bbca34",
            "5791332747c644049b9de417a3369818",
            "d164013a0ad24027a96f98887fd46d6a",
            "edeb62ff425d429cb6cc6de5fe20e68c",
            "b34db738e74a4f3dbc52f42dedc6dd8a",
            "c7b90e0210b44a838cc785acdc4d1298",
            "833923bb415e4d64b3d7d4e8fbcbbd11",
            "bf569df5a1e543f5b2617d2f5778a6a4",
            "a69de83c36cc46289a00ee81f4125cfa",
            "aa5dae53b5534d2abe0467a3f9d52008",
            "511fb228975e48a3b4a31bbf73cc2b42",
            "fffa059ea84f420fbd8940376678b431",
            "0593f018fea244b6907aa06164dd59c5"
          ]
        },
        "id": "TzUccPzV9ZE2",
        "outputId": "f71a86aa-619a-4914-c226-9ed23996e4ca"
      },
      "outputs": [],
      "source": [
        "!pip install -q -U huggingface_hub\n",
        "from huggingface_hub import login\n",
        "\n",
        "login()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "## Login to Weight&Biases (optional)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "You can use Weights&Biases to log the training process. This step is optional."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 4,
      "metadata": {},
      "outputs": [
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "\u001b[34m\u001b[1mwandb\u001b[0m: Using wandb-core as the SDK backend. Please refer to https://wandb.me/wandb-core for more information.\n",
            "\u001b[34m\u001b[1mwandb\u001b[0m: Currently logged in as: \u001b[33mtonywu_71\u001b[0m. Use \u001b[1m`wandb login --relogin`\u001b[0m to force relogin\n"
          ]
        }
      ],
      "source": [
        "if wandb_experiment_name:\n",
        "    !pip install -q -U wandb\n",
        "    import wandb\n",
        "\n",
        "    wandb.login()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "KMj35oGIErey"
      },
      "source": [
        "## Imports"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 5,
      "metadata": {
        "id": "Pr60gcNw7ORF"
      },
      "outputs": [],
      "source": [
        "from pathlib import Path\n",
        "from typing import cast\n",
        "\n",
        "import torch\n",
        "from colpali_engine.collators.visual_retriever_collator import VisualRetrieverCollator\n",
        "from colpali_engine.loss import ColbertPairwiseCELoss\n",
        "from colpali_engine.models import ColPali, ColPaliProcessor\n",
        "from colpali_engine.trainer.contrastive_trainer import ContrastiveTrainer\n",
        "from colpali_engine.utils.torch_utils import get_torch_device, tear_down_torch\n",
        "from datasets import DatasetDict, load_dataset\n",
        "from peft import LoraConfig\n",
        "from torch import nn\n",
        "from transformers import BitsAndBytesConfig, TrainerCallback, TrainingArguments\n",
        "\n",
        "\n",
        "def print_trainable_parameters(model: nn.Module) -> None:\n",
        "    \"\"\"\n",
        "    Print the number of trainable parameters in the model.\n",
        "    \"\"\"\n",
        "    trainable_params = 0\n",
        "    all_param = 0\n",
        "    for _, param in model.named_parameters():\n",
        "        all_param += param.numel()\n",
        "        if param.requires_grad:\n",
        "            trainable_params += param.numel()\n",
        "    print(\n",
        "        f\"trainable params: {trainable_params:,} || all params: {all_param:,} || trainable%: {100 * trainable_params / all_param}\"\n",
        "    )"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "FiUP8tC67ORF"
      },
      "source": [
        "## Choose a quantization strategy"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "ColPali is quite a large model with 3B parameters. While you can load the model and run inference on a L4 GPU or a M1+ Mac (in BF16, you'll need ≈6GB of VRAM), you will need much more VRAM to train it when taking into account the gradients and the AdamW optimizer states (≈48GB).\n",
        "\n",
        "Therefore, we will use LoRA to limit the number of trainable parameters (like how the original ColPali was trained).\n",
        "\n",
        "Even with LoRA, you might struggle to train ColPali on consumer GPUs. This is because of the contrastive loss used in ColPali: the larger the batch size, the more VRAM is used. Thus, we need to quantize the model to further reduce its memory footprint. Hence, we recommend using the 4-bit quantization with LoRA, i.e. QLoRA.\n",
        "\n",
        "See this [🤗 blog post](https://huggingface.co/docs/transformers/main/en/quantization/overview) for more information on quantization and this [🤗 blog post](https://huggingface.co/blog/4bit-transformers-bitsandbytes) for more details on QLoRA."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 6,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 542,
          "referenced_widgets": [
            "8fd70d5c36254bc3a79ac2d89212fe95",
            "dc7853a136234b3baafc99ba4e6376dd",
            "8fa1c9d926a34a5a92332881c487ccdb",
            "c3c8693b555c4145918078d870a2a648",
            "6f071f6dbae84a418e3cc8af8893f921",
            "18490330921d463ba5166f201ad39aca",
            "a6292478607641efa50af2909b4b769a",
            "1b3e5d1b083c47f09561f7537c0a1c6a",
            "93cf11b23d334dcc982c7e3ebd3ba57a",
            "4dc87365768a45678a4d32eac16188c5",
            "a8b6913554c74eb19a319cced459a8f6"
          ]
        },
        "id": "xckrnj6N7ORF",
        "outputId": "44afc9e9-d0c4-4ded-985a-213171736279"
      },
      "outputs": [],
      "source": [
        "# ==========================     USER INPUT     ==========================\n",
        "\n",
        "QUANTIZATION_STRATEGY = \"4bit\"\n",
        "\n",
        "# ========================================================================\n",
        "\n",
        "# Automatically set the device\n",
        "device = get_torch_device(\"auto\")\n",
        "\n",
        "if QUANTIZATION_STRATEGY and device != \"cuda:0\":\n",
        "    raise ValueError(\"This notebook requires a CUDA GPU to use quantization.\")\n",
        "\n",
        "# Prepare quantization config\n",
        "if QUANTIZATION_STRATEGY is None:\n",
        "    bnb_config = None\n",
        "elif QUANTIZATION_STRATEGY == \"8bit\":\n",
        "    bnb_config = BitsAndBytesConfig(\n",
        "        load_in_8bit=True,\n",
        "    )\n",
        "elif QUANTIZATION_STRATEGY == \"4bit\":\n",
        "    bnb_config = BitsAndBytesConfig(\n",
        "        load_in_4bit=True,\n",
        "        bnb_4bit_quant_type=\"nf4\",\n",
        "        bnb_4bit_compute_dtype=torch.bfloat16,\n",
        "    )\n",
        "else:\n",
        "    raise ValueError(f\"Invalid quantization strategy: {QUANTIZATION_STRATEGY}\")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "## Load the pre-trained model"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "For simplicity, we will continue training the LoRA adapter from the original ColPali model."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 7,
      "metadata": {},
      "outputs": [
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "/home/gcpuser/sky_workdir/.venv/lib/python3.10/site-packages/transformers/models/paligemma/configuration_paligemma.py:137: FutureWarning: The `vocab_size` attribute is deprecated and will be removed in v4.44, Please use `text_config.vocab_size` instead.\n",
            "  warnings.warn(\n",
            "`config.hidden_act` is ignored, you should use `config.hidden_activation` instead.\n",
            "Gemma's activation function will be set to `gelu_pytorch_tanh`. Please, use\n",
            "`config.hidden_activation` if you want to override this behaviour.\n",
            "See https://github.com/huggingface/transformers/pull/29402 for more details.\n"
          ]
        },
        {
          "data": {
            "application/vnd.jupyter.widget-view+json": {
              "model_id": "a504f2c269714f5597a60689e7e1a5ef",
              "version_major": 2,
              "version_minor": 0
            },
            "text/plain": [
              "Loading checkpoint shards:   0%|          | 0/2 [00:00<?, ?it/s]"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "trainable params: 39,292,928 || all params: 1,766,287,216 || trainable%: 2.224605808390791\n"
          ]
        }
      ],
      "source": [
        "# Pre-trained model name (with LoRA adapter)\n",
        "model_name = \"vidore/colpali-v1.2\"\n",
        "\n",
        "# Get the LoRA config from the pretrained model\n",
        "lora_config = LoraConfig.from_pretrained(model_name)\n",
        "\n",
        "# Load the model with the loaded pre-trained adapter\n",
        "model = cast(\n",
        "    ColPali,\n",
        "    ColPali.from_pretrained(\n",
        "        model_name,\n",
        "        quantization_config=bnb_config,\n",
        "        torch_dtype=torch.bfloat16,\n",
        "        device_map=device,\n",
        "    ),\n",
        ")\n",
        "\n",
        "if not model.active_adapters():\n",
        "    raise ValueError(\"No adapter found in the model.\")\n",
        "\n",
        "# The LoRA weights are frozen by default. We need to unfreeze them to fine-tune the model.\n",
        "for name, param in model.named_parameters():\n",
        "    if \"lora\" in name:\n",
        "        param.requires_grad = True\n",
        "\n",
        "print_trainable_parameters(model)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "**Additional note:** It is possible to train a new LoRA adapter on top of the existing one. This allows to have extra flexibility in the LoRA config and to faciltate adapter hot-swapping.\n",
        "\n",
        "To do this:\n",
        "\n",
        "1. Load the version of ColPali with the adapter already merged with the base model weights: `model = ColPali.from_pretrained(\"vidore/colpali-v1.2-merged\")`.\n",
        "2. Define your LoRA adapter using `lora_config = LoraConfig(...)`.\n",
        "3. Use `model = get_peft_model(model, lora_config)` to get the model with the new adapter.\n",
        "4. Run fine-tuning as usual."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "## Load the processor and the collator"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 8,
      "metadata": {
        "id": "S0NX3fAx_g1U"
      },
      "outputs": [],
      "source": [
        "if lora_config.base_model_name_or_path is None:\n",
        "    raise ValueError(\"Base model name or path is required in the LoRA config.\")\n",
        "\n",
        "processor = cast(\n",
        "    ColPaliProcessor,\n",
        "    ColPaliProcessor.from_pretrained(model_name),\n",
        ")\n",
        "collator = VisualRetrieverCollator(processor=processor)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "t2iY6U_6_g1U"
      },
      "source": [
        "## Load the dataset"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "VDSID-French is a subset of the [`vidore/vdsid`](https://huggingface.co/datasets/vidore/vdsid). It contains 5000 document-question-answer triplet of French documets, split into a train set of 4700 examples and a test set of 300 examples.\n",
        "\n",
        "This dataset was created and chosen for this fine-tuning because ColPali was mainly trained on English documents. Thus fine-tuning on French documents can help to improve the multilingual capabilities of the model."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 9,
      "metadata": {
        "id": "RY7sVT8q_g1U"
      },
      "outputs": [
        {
          "data": {
            "text/plain": [
              "DatasetDict({\n",
              "    train: Dataset({\n",
              "        features: ['document_filename', 'document_url', 'search_query', 'search_topic', 'search_subtopic', 'search_language', 'search_filetype', 'page_number', 'page_description', 'page_language', 'page_contains_table', 'page_contains_figure', 'page_contains_paragraph', 'image', 'query_type', 'query_answerability', 'query_modality', 'query_language', 'query_reasoning', 'query', 'query_is_self_contained', 'query_is_self_contained_reasoning', 'answer'],\n",
              "        num_rows: 4700\n",
              "    })\n",
              "    test: Dataset({\n",
              "        features: ['document_filename', 'document_url', 'search_query', 'search_topic', 'search_subtopic', 'search_language', 'search_filetype', 'page_number', 'page_description', 'page_language', 'page_contains_table', 'page_contains_figure', 'page_contains_paragraph', 'image', 'query_type', 'query_answerability', 'query_modality', 'query_language', 'query_reasoning', 'query', 'query_is_self_contained', 'query_is_self_contained_reasoning', 'answer'],\n",
              "        num_rows: 300\n",
              "    })\n",
              "})"
            ]
          },
          "execution_count": 9,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "# Load the dataset\n",
        "dataset_name = \"vidore/vdsid_french\"\n",
        "ds = cast(DatasetDict, load_dataset(dataset_name))\n",
        "\n",
        "# Rename the columns to match the trainer's requirements\n",
        "ds = ds.rename_column(\"page_image\", \"image\")\n",
        "ds[\"train\"] = ds[\"train\"].shuffle(seed=42)\n",
        "\n",
        "ds"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "1oiBwyc6_g1V"
      },
      "source": [
        "## Define training args"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "Depending on your hardware, you might need to adjust the train batch size. This parameter is crucial for the training of ColPali as it is trained with a contrastive loss: the larger the batch size, the more representative the negative samples are, and the better the model will perform."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 10,
      "metadata": {
        "id": "Wp0x1ZcG_g1V"
      },
      "outputs": [],
      "source": [
        "checkpoints_dir = Path(\"checkpoints\")\n",
        "checkpoints_dir.mkdir(exist_ok=True, parents=True)\n",
        "\n",
        "training_args = TrainingArguments(\n",
        "    output_dir=str(checkpoints_dir),\n",
        "    hub_model_id=hf_pushed_model_name if hf_pushed_model_name else None,\n",
        "    overwrite_output_dir=True,\n",
        "    num_train_epochs=1.5,\n",
        "    per_device_train_batch_size=4,\n",
        "    per_device_eval_batch_size=4,\n",
        "    gradient_accumulation_steps=4,\n",
        "    gradient_checkpointing=False,\n",
        "    eval_strategy=\"steps\",\n",
        "    save_steps=200,\n",
        "    logging_steps=20,\n",
        "    eval_steps=100,\n",
        "    warmup_steps=100,\n",
        "    learning_rate=5e-5,\n",
        "    save_total_limit=1,\n",
        "    report_to=[\"wandb\"] if wandb_experiment_name else [],\n",
        ")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "## Create the trainer"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "The trainer uses a ColBERT contrastive hard-margin loss. Read the [ColPali paper](https://doi.org/10.48550/arXiv.2407.01449) for more details on this loss function."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 11,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 492
        },
        "id": "bNw9fDvb_g1V",
        "outputId": "6b049ca9-e82b-4df6-f628-7a007ae52aa3"
      },
      "outputs": [],
      "source": [
        "class EvaluateFirstStepCallback(TrainerCallback):\n",
        "    \"\"\"\n",
        "    Run eval after the first training step.\n",
        "    Used to have a more precise evaluation learning curve.\n",
        "    \"\"\"\n",
        "\n",
        "    def on_step_end(self, args, state, control, **kwargs):\n",
        "        if state.global_step == 1:\n",
        "            control.should_evaluate = True\n",
        "\n",
        "\n",
        "trainer = ContrastiveTrainer(\n",
        "    model=model,\n",
        "    train_dataset=ds[\"train\"],\n",
        "    eval_dataset=ds[\"test\"],\n",
        "    args=training_args,\n",
        "    data_collator=collator,\n",
        "    loss_func=ColbertPairwiseCELoss(),\n",
        "    is_vision_model=True,\n",
        ")\n",
        "\n",
        "trainer.args.remove_unused_columns = False\n",
        "trainer.add_callback(EvaluateFirstStepCallback())"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "## Evaluate the model before training"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "Let's see how ColPali performs on the test set prior to fine-tuning."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 12,
      "metadata": {},
      "outputs": [
        {
          "data": {
            "text/html": [
              "\n",
              "    <div>\n",
              "      \n",
              "      <progress value='150' max='75' style='width:300px; height:20px; vertical-align: middle;'></progress>\n",
              "      [75/75 01:26]\n",
              "    </div>\n",
              "    "
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "\u001b[34m\u001b[1mwandb\u001b[0m: \u001b[33mWARNING\u001b[0m The `run_name` is currently set to the same value as `TrainingArguments.output_dir`. If this was not intended, please specify a different run name by setting the `TrainingArguments.run_name` parameter.\n"
          ]
        },
        {
          "data": {
            "text/html": [
              "Tracking run with wandb version 0.18.1"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "data": {
            "text/html": [
              "Run data is saved locally in <code>/home/gcpuser/sky_workdir/examples/wandb/run-20240925_075620-1v8bharg</code>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "data": {
            "text/html": [
              "Syncing run <strong><a href='https://wandb.ai/tonywu_71/huggingface/runs/1v8bharg' target=\"_blank\">checkpoints</a></strong> to <a href='https://wandb.ai/tonywu_71/huggingface' target=\"_blank\">Weights & Biases</a> (<a href='https://wandb.me/run' target=\"_blank\">docs</a>)<br/>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "data": {
            "text/html": [
              " View project at <a href='https://wandb.ai/tonywu_71/huggingface' target=\"_blank\">https://wandb.ai/tonywu_71/huggingface</a>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "data": {
            "text/html": [
              " View run at <a href='https://wandb.ai/tonywu_71/huggingface/runs/1v8bharg' target=\"_blank\">https://wandb.ai/tonywu_71/huggingface/runs/1v8bharg</a>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "data": {
            "text/plain": [
              "{'eval_loss': 0.04893038421869278,\n",
              " 'eval_model_preparation_time': 0.0121,\n",
              " 'eval_runtime': 41.4353,\n",
              " 'eval_samples_per_second': 7.24,\n",
              " 'eval_steps_per_second': 1.81}"
            ]
          },
          "execution_count": 12,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "eval_results = trainer.evaluate()\n",
        "eval_results"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "## Fine-tune the model"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "Finally, time for fine-tuning! Run the following cell then go make yourself a cup of tea or take a walk while the model trains. 🚀 \n",
        "\n",
        "On a A100-40GB GPU and with the default training parameters, fine-tuning should take around 30 minutes. ⏱️"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 13,
      "metadata": {},
      "outputs": [
        {
          "data": {
            "text/html": [
              "Finishing last run (ID:1v8bharg) before initializing another..."
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "data": {
            "application/vnd.jupyter.widget-view+json": {
              "model_id": "8fb8ab33e0664cf392f62ee539c08fb0",
              "version_major": 2,
              "version_minor": 0
            },
            "text/plain": [
              "VBox(children=(Label(value='0.032 MB of 0.032 MB uploaded\\r'), FloatProgress(value=1.0, max=1.0)))"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "data": {
            "text/html": [
              "<style>\n",
              "    table.wandb td:nth-child(1) { padding: 0 10px; text-align: left ; width: auto;} td:nth-child(2) {text-align: left ; width: 100%}\n",
              "    .wandb-row { display: flex; flex-direction: row; flex-wrap: wrap; justify-content: flex-start; width: 100% }\n",
              "    .wandb-col { display: flex; flex-direction: column; flex-basis: 100%; flex: 1; padding: 10px; }\n",
              "    </style>\n",
              "<div class=\"wandb-row\"><div class=\"wandb-col\"><h3>Run history:</h3><br/><table class=\"wandb\"><tr><td>eval/loss</td><td>▁</td></tr><tr><td>eval/model_preparation_time</td><td>▁</td></tr><tr><td>eval/runtime</td><td>▁</td></tr><tr><td>eval/samples_per_second</td><td>▁</td></tr><tr><td>eval/steps_per_second</td><td>▁</td></tr><tr><td>train/global_step</td><td>▁</td></tr></table><br/></div><div class=\"wandb-col\"><h3>Run summary:</h3><br/><table class=\"wandb\"><tr><td>eval/loss</td><td>0.04893</td></tr><tr><td>eval/model_preparation_time</td><td>0.0121</td></tr><tr><td>eval/runtime</td><td>41.4353</td></tr><tr><td>eval/samples_per_second</td><td>7.24</td></tr><tr><td>eval/steps_per_second</td><td>1.81</td></tr><tr><td>train/global_step</td><td>0</td></tr></table><br/></div></div>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "data": {
            "text/html": [
              " View run <strong style=\"color:#cdcd00\">checkpoints</strong> at: <a href='https://wandb.ai/tonywu_71/huggingface/runs/1v8bharg' target=\"_blank\">https://wandb.ai/tonywu_71/huggingface/runs/1v8bharg</a><br/> View project at: <a href='https://wandb.ai/tonywu_71/huggingface' target=\"_blank\">https://wandb.ai/tonywu_71/huggingface</a><br/>Synced 5 W&B file(s), 0 media file(s), 2 artifact file(s) and 0 other file(s)"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "data": {
            "text/html": [
              "Find logs at: <code>./wandb/run-20240925_075620-1v8bharg/logs</code>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "data": {
            "text/html": [
              "Successfully finished last run (ID:1v8bharg). Initializing new run:<br/>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "data": {
            "text/html": [
              "Tracking run with wandb version 0.18.1"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "data": {
            "text/html": [
              "Run data is saved locally in <code>/home/gcpuser/sky_workdir/examples/wandb/run-20240925_075620-luqihcpc</code>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "data": {
            "text/html": [
              "Syncing run <strong><a href='https://wandb.ai/tonywu_71/colpali/runs/luqihcpc' target=\"_blank\">finetune_colpali_v1_2-vdsid_french-4bit</a></strong> to <a href='https://wandb.ai/tonywu_71/colpali' target=\"_blank\">Weights & Biases</a> (<a href='https://wandb.me/run' target=\"_blank\">docs</a>)<br/>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "data": {
            "text/html": [
              " View project at <a href='https://wandb.ai/tonywu_71/colpali' target=\"_blank\">https://wandb.ai/tonywu_71/colpali</a>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "data": {
            "text/html": [
              " View run at <a href='https://wandb.ai/tonywu_71/colpali/runs/luqihcpc' target=\"_blank\">https://wandb.ai/tonywu_71/colpali/runs/luqihcpc</a>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "data": {
            "text/html": [
              "\n",
              "    <div>\n",
              "      \n",
              "      <progress value='440' max='440' style='width:300px; height:20px; vertical-align: middle;'></progress>\n",
              "      [440/440 31:58, Epoch 1/2]\n",
              "    </div>\n",
              "    <table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              " <tr style=\"text-align: left;\">\n",
              "      <th>Step</th>\n",
              "      <th>Training Loss</th>\n",
              "      <th>Validation Loss</th>\n",
              "      <th>Model Preparation Time</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <td>1</td>\n",
              "      <td>No log</td>\n",
              "      <td>0.048930</td>\n",
              "      <td>0.012100</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <td>100</td>\n",
              "      <td>0.017100</td>\n",
              "      <td>0.021548</td>\n",
              "      <td>0.012100</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <td>200</td>\n",
              "      <td>0.010000</td>\n",
              "      <td>0.030120</td>\n",
              "      <td>0.012100</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <td>300</td>\n",
              "      <td>0.009000</td>\n",
              "      <td>0.031809</td>\n",
              "      <td>0.012100</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <td>400</td>\n",
              "      <td>0.012000</td>\n",
              "      <td>0.031864</td>\n",
              "      <td>0.012100</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table><p>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "/home/gcpuser/sky_workdir/.venv/lib/python3.10/site-packages/transformers/integrations/peft.py:397: FutureWarning: The `active_adapter` method is deprecated and will be removed in a future version.\n",
            "  warnings.warn(\n",
            "/home/gcpuser/sky_workdir/.venv/lib/python3.10/site-packages/transformers/models/paligemma/configuration_paligemma.py:137: FutureWarning: The `vocab_size` attribute is deprecated and will be removed in v4.44, Please use `text_config.vocab_size` instead.\n",
            "  warnings.warn(\n",
            "/home/gcpuser/sky_workdir/.venv/lib/python3.10/site-packages/transformers/integrations/peft.py:397: FutureWarning: The `active_adapter` method is deprecated and will be removed in a future version.\n",
            "  warnings.warn(\n",
            "/home/gcpuser/sky_workdir/.venv/lib/python3.10/site-packages/transformers/models/paligemma/configuration_paligemma.py:137: FutureWarning: The `vocab_size` attribute is deprecated and will be removed in v4.44, Please use `text_config.vocab_size` instead.\n",
            "  warnings.warn(\n",
            "/home/gcpuser/sky_workdir/.venv/lib/python3.10/site-packages/transformers/integrations/peft.py:397: FutureWarning: The `active_adapter` method is deprecated and will be removed in a future version.\n",
            "  warnings.warn(\n",
            "/home/gcpuser/sky_workdir/.venv/lib/python3.10/site-packages/transformers/models/paligemma/configuration_paligemma.py:137: FutureWarning: The `vocab_size` attribute is deprecated and will be removed in v4.44, Please use `text_config.vocab_size` instead.\n",
            "  warnings.warn(\n"
          ]
        },
        {
          "data": {
            "text/plain": [
              "TrainOutput(global_step=440, training_loss=0.022981878132982688, metrics={'train_runtime': 1922.3275, 'train_samples_per_second': 3.667, 'train_steps_per_second': 0.229, 'total_flos': 1.0598129422848e+17, 'train_loss': 0.022981878132982688, 'epoch': 1.4978723404255319})"
            ]
          },
          "execution_count": 13,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "# Prepare WandB logging\n",
        "if wandb_experiment_name:\n",
        "    wandb_tags = [\"finetuning\", \"colpali\"]\n",
        "\n",
        "    if bnb_config:\n",
        "        wandb_tags.append(\"quantization\")\n",
        "\n",
        "    run = wandb.init(\n",
        "        project=\"colpali\",\n",
        "        name=wandb_experiment_name,\n",
        "        job_type=\"finetuning\",\n",
        "        tags=wandb_tags,\n",
        "        config={\n",
        "            \"model_name\": model_name,\n",
        "            \"bitsandbytes_config\": bnb_config.to_dict() if bnb_config else None,\n",
        "            \"dataset_name\": dataset_name,\n",
        "        },\n",
        "    )\n",
        "\n",
        "# Train the model\n",
        "train_results = trainer.train()\n",
        "\n",
        "train_results"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "## Evaluate the model after training"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "Now, let's see how the fine-tuned model performs on the test set. You should observe a drop in the evaluation loss. \n",
        "\n",
        "To further evaluate the model, you can use the [`vidore-benchmark`](https://github.com/illuin-tech/vidore-benchmark) to measure the retrieval performance (e.g. NDCG@5)."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 14,
      "metadata": {},
      "outputs": [
        {
          "data": {
            "text/html": [
              "\n",
              "    <div>\n",
              "      \n",
              "      <progress value='75' max='75' style='width:300px; height:20px; vertical-align: middle;'></progress>\n",
              "      [75/75 00:39]\n",
              "    </div>\n",
              "    "
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "data": {
            "text/plain": [
              "{'eval_loss': 0.03164391964673996,\n",
              " 'eval_model_preparation_time': 0.0121,\n",
              " 'eval_runtime': 40.3053,\n",
              " 'eval_samples_per_second': 7.443,\n",
              " 'eval_steps_per_second': 1.861,\n",
              " 'epoch': 1.4978723404255319}"
            ]
          },
          "execution_count": 14,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "eval_results = trainer.evaluate()\n",
        "eval_results"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "During my own experiments, I got the following learning curves:\n",
        "\n",
        "<p align=\"center\"><img width=800 src=\"https://github.com/tonywu71/colpali-cookbooks/blob/main/assets/finetuning/learning_curves.jpeg?raw=true\"/></p>\n",
        "\n",
        "A few observations:\n",
        "\n",
        "- The training loss is globally decreasing, which proves that the model is learning.\n",
        "- The validation loss rapidly decreases for the first 100 steps, goes back up and starts plateauing after 200 steps. This is a sign of overfitting, so we probably should have stopped the training a bit earlier."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "## Conclude the WandB run (optional)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 15,
      "metadata": {},
      "outputs": [
        {
          "data": {
            "application/vnd.jupyter.widget-view+json": {
              "model_id": "6d7de631e6ce44aaa9a5876b16d5a497",
              "version_major": 2,
              "version_minor": 0
            },
            "text/plain": [
              "VBox(children=(Label(value='0.003 MB of 0.003 MB uploaded\\r'), FloatProgress(value=1.0, max=1.0)))"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "data": {
            "text/html": [
              "<style>\n",
              "    table.wandb td:nth-child(1) { padding: 0 10px; text-align: left ; width: auto;} td:nth-child(2) {text-align: left ; width: 100%}\n",
              "    .wandb-row { display: flex; flex-direction: row; flex-wrap: wrap; justify-content: flex-start; width: 100% }\n",
              "    .wandb-col { display: flex; flex-direction: column; flex-basis: 100%; flex: 1; padding: 10px; }\n",
              "    </style>\n",
              "<div class=\"wandb-row\"><div class=\"wandb-col\"><h3>Run history:</h3><br/><table class=\"wandb\"><tr><td>eval/loss</td><td>█▁▃▄▄▄</td></tr><tr><td>eval/model_preparation_time</td><td>▁▁▁▁▁▁</td></tr><tr><td>eval/runtime</td><td>▅▂▁▃▁█</td></tr><tr><td>eval/samples_per_second</td><td>▄▇█▅█▁</td></tr><tr><td>eval/steps_per_second</td><td>▄▇█▅█▁</td></tr><tr><td>train/epoch</td><td>▁▁▂▂▂▃▃▃▃▄▄▄▄▄▅▅▅▆▆▆▆▇▇▇▇████</td></tr><tr><td>train/global_step</td><td>▁▁▂▂▂▃▃▃▃▄▄▄▄▄▅▅▅▆▆▆▆▇▇▇▇████</td></tr><tr><td>train/grad_norm</td><td>▂▁█▂▁▁▂▆▁▁▁▁▁▁▂▁▁▁▁▁▁▁</td></tr><tr><td>train/learning_rate</td><td>▂▄▅▇██▇▇▆▆▆▅▅▄▄▃▃▃▂▂▁▁</td></tr><tr><td>train/loss</td><td>▇█▆▃▃▂▄▇▅▂▃█▂▄▂▂▂▂▁▂▂▁</td></tr></table><br/></div><div class=\"wandb-col\"><h3>Run summary:</h3><br/><table class=\"wandb\"><tr><td>eval/loss</td><td>0.03164</td></tr><tr><td>eval/model_preparation_time</td><td>0.0121</td></tr><tr><td>eval/runtime</td><td>40.3053</td></tr><tr><td>eval/samples_per_second</td><td>7.443</td></tr><tr><td>eval/steps_per_second</td><td>1.861</td></tr><tr><td>total_flos</td><td>1.0598129422848e+17</td></tr><tr><td>train/epoch</td><td>1.49787</td></tr><tr><td>train/global_step</td><td>440</td></tr><tr><td>train/grad_norm</td><td>0.02536</td></tr><tr><td>train/learning_rate</td><td>0</td></tr><tr><td>train/loss</td><td>0.0024</td></tr><tr><td>train_loss</td><td>0.02298</td></tr><tr><td>train_runtime</td><td>1922.3275</td></tr><tr><td>train_samples_per_second</td><td>3.667</td></tr><tr><td>train_steps_per_second</td><td>0.229</td></tr></table><br/></div></div>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "data": {
            "text/html": [
              " View run <strong style=\"color:#cdcd00\">finetune_colpali_v1_2-vdsid_french-4bit</strong> at: <a href='https://wandb.ai/tonywu_71/colpali/runs/luqihcpc' target=\"_blank\">https://wandb.ai/tonywu_71/colpali/runs/luqihcpc</a><br/> View project at: <a href='https://wandb.ai/tonywu_71/colpali' target=\"_blank\">https://wandb.ai/tonywu_71/colpali</a><br/>Synced 5 W&B file(s), 0 media file(s), 2 artifact file(s) and 0 other file(s)"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "data": {
            "text/html": [
              "Find logs at: <code>./wandb/run-20240925_075620-luqihcpc/logs</code>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        }
      ],
      "source": [
        "if wandb_experiment_name:\n",
        "    run.finish()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "## Push the model to the Hub (optional)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "If satisfied with the fine-tuned model, you can push it to the Hub to share it with the community! 😍\n",
        "\n",
        "You can find my fine-tuned ColPali model for reference at [`tonywu71/finetune_colpali_v1_2-vdsid_french-4bit`](https://huggingface.co/tonywu71/finetune_colpali_v1_2-vdsid_french-4bit). For more inspiration, check out the [🤗 Hf Hub](https://huggingface.co/models?other=base_model:finetune:vidore/colpaligemma-3b-pt-448-base) to see the ColPali models that the community has already fine-tuned."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 16,
      "metadata": {},
      "outputs": [
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "/home/gcpuser/sky_workdir/.venv/lib/python3.10/site-packages/transformers/integrations/peft.py:397: FutureWarning: The `active_adapter` method is deprecated and will be removed in a future version.\n",
            "  warnings.warn(\n",
            "/home/gcpuser/sky_workdir/.venv/lib/python3.10/site-packages/transformers/models/paligemma/configuration_paligemma.py:137: FutureWarning: The `vocab_size` attribute is deprecated and will be removed in v4.44, Please use `text_config.vocab_size` instead.\n",
            "  warnings.warn(\n"
          ]
        },
        {
          "data": {
            "application/vnd.jupyter.widget-view+json": {
              "model_id": "d82a70ffb58548e9990a697d25103f10",
              "version_major": 2,
              "version_minor": 0
            },
            "text/plain": [
              "Upload 2 LFS files:   0%|          | 0/2 [00:00<?, ?it/s]"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "data": {
            "application/vnd.jupyter.widget-view+json": {
              "model_id": "0f2bc22225df46078f6a1104a69c7371",
              "version_major": 2,
              "version_minor": 0
            },
            "text/plain": [
              "adapter_model.safetensors:   0%|          | 0.00/157M [00:00<?, ?B/s]"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "data": {
            "application/vnd.jupyter.widget-view+json": {
              "model_id": "17093ae6a9f84c02a7c7d794c22bebdc",
              "version_major": 2,
              "version_minor": 0
            },
            "text/plain": [
              "training_args.bin:   0%|          | 0.00/5.24k [00:00<?, ?B/s]"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        }
      ],
      "source": [
        "if hf_pushed_model_name:\n",
        "    trainer.push_to_hub(tags=[\"colpali\"], dataset=dataset_name)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "Congrats, you have successfully fine-tuned ColPali! 🎉"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "## Load your fine-tuned model (optional)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "To use your freshly fine-tuned model, use the code from the following cell to load your own ColPali in your own project. 🫶🏼"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 17,
      "metadata": {},
      "outputs": [
        {
          "data": {
            "application/vnd.jupyter.widget-view+json": {
              "model_id": "782f6ec31cc04742addc8fc8b47238e3",
              "version_major": 2,
              "version_minor": 0
            },
            "text/plain": [
              "Loading checkpoint shards:   0%|          | 0/2 [00:00<?, ?it/s]"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        }
      ],
      "source": [
        "# Unload the previous model and clean the GPU cache\n",
        "del model\n",
        "tear_down_torch()\n",
        "\n",
        "# Load your fine-tuned ColPali\n",
        "model = ColPali.from_pretrained(\n",
        "    model_name,\n",
        "    torch_dtype=torch.bfloat16,\n",
        "    device_map=device,\n",
        ")"
      ]
    }
  ],
  "metadata": {
    "accelerator": "GPU",
    "colab": {
      "gpuType": "T4",
      "provenance": []
    },
    "kernelspec": {
      "display_name": "Python 3",
      "name": "python3"
    },
    "language_info": {
      "codemirror_mode": {
        "name": "ipython",
        "version": 3
      },
      "file_extension": ".py",
      "mimetype": "text/x-python",
      "name": "python",
      "nbconvert_exporter": "python",
      "pygments_lexer": "ipython3",
      "version": "3.10.12"
    },
    "widgets": {
      "application/vnd.jupyter.widget-state+json": {
        "0593f018fea244b6907aa06164dd59c5": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "DescriptionStyleModel",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "18490330921d463ba5166f201ad39aca": {
          "model_module": "@jupyter-widgets/base",
          "model_module_version": "1.2.0",
          "model_name": "LayoutModel",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "197cf7905f7b493d90a4cf03b66cc0b9": {
          "model_module": "@jupyter-widgets/base",
          "model_module_version": "1.2.0",
          "model_name": "LayoutModel",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "1b3e5d1b083c47f09561f7537c0a1c6a": {
          "model_module": "@jupyter-widgets/base",
          "model_module_version": "1.2.0",
          "model_name": "LayoutModel",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "2b8d3565706c4413af5d0c0a30ad132c": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "HTMLModel",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_197cf7905f7b493d90a4cf03b66cc0b9",
            "placeholder": "​",
            "style": "IPY_MODEL_36acb5cf8816463ca36812913f2d163e",
            "value": "<center> <img\nsrc=https://huggingface.co/front/assets/huggingface_logo-noborder.svg\nalt='Hugging Face'> <br> Copy a token from <a\nhref=\"https://huggingface.co/settings/tokens\" target=\"_blank\">your Hugging Face\ntokens page</a> and paste it below. <br> Immediately click login after copying\nyour token or it might be stored in plain text in this notebook file. </center>"
          }
        },
        "359825e9c7df4f01b54677b5b27f7916": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "DescriptionStyleModel",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "36acb5cf8816463ca36812913f2d163e": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "DescriptionStyleModel",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "4dc87365768a45678a4d32eac16188c5": {
          "model_module": "@jupyter-widgets/base",
          "model_module_version": "1.2.0",
          "model_name": "LayoutModel",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "511fb228975e48a3b4a31bbf73cc2b42": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "DescriptionStyleModel",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "55ddc088c8d149d584d5586680123f7a": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "ButtonStyleModel",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "ButtonStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "button_color": null,
            "font_weight": ""
          }
        },
        "5791332747c644049b9de417a3369818": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "LabelModel",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "LabelModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "LabelView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_c7b90e0210b44a838cc785acdc4d1298",
            "placeholder": "​",
            "style": "IPY_MODEL_833923bb415e4d64b3d7d4e8fbcbbd11",
            "value": "Token is valid (permission: write)."
          }
        },
        "5911b53437694805b28573fbeb1594d0": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "CheckboxModel",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "CheckboxModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "CheckboxView",
            "description": "Add token as git credential?",
            "description_tooltip": null,
            "disabled": false,
            "indent": true,
            "layout": "IPY_MODEL_de744a456e7a49569eaa973837732c7b",
            "style": "IPY_MODEL_bf4518dfbee14600a799f300b00ab41b",
            "value": true
          }
        },
        "631d08c360674a2b9b5aad1a388029fd": {
          "model_module": "@jupyter-widgets/base",
          "model_module_version": "1.2.0",
          "model_name": "LayoutModel",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "6f071f6dbae84a418e3cc8af8893f921": {
          "model_module": "@jupyter-widgets/base",
          "model_module_version": "1.2.0",
          "model_name": "LayoutModel",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "730882a26b9741649831beaa86d76db2": {
          "model_module": "@jupyter-widgets/base",
          "model_module_version": "1.2.0",
          "model_name": "LayoutModel",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "74c83bea924d470d99608f17cb74d338": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "HTMLModel",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_b4da694310f04f8fab4a9b64a526509e",
            "placeholder": "​",
            "style": "IPY_MODEL_359825e9c7df4f01b54677b5b27f7916",
            "value": "\n<b>Pro Tip:</b> If you don't already have one, you can create a dedicated\n'notebooks' token with 'write' access, that you can then easily reuse for all\nnotebooks. </center>"
          }
        },
        "833923bb415e4d64b3d7d4e8fbcbbd11": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "DescriptionStyleModel",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "8c5818ee930e49fdaeb6249a35bbca34": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "DescriptionStyleModel",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "8fa1c9d926a34a5a92332881c487ccdb": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "FloatProgressModel",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "FloatProgressModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "ProgressView",
            "bar_style": "danger",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_1b3e5d1b083c47f09561f7537c0a1c6a",
            "max": 3,
            "min": 0,
            "orientation": "horizontal",
            "style": "IPY_MODEL_93cf11b23d334dcc982c7e3ebd3ba57a",
            "value": 0
          }
        },
        "8fd70d5c36254bc3a79ac2d89212fe95": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "HBoxModel",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HBoxModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HBoxView",
            "box_style": "",
            "children": [
              "IPY_MODEL_dc7853a136234b3baafc99ba4e6376dd",
              "IPY_MODEL_8fa1c9d926a34a5a92332881c487ccdb",
              "IPY_MODEL_c3c8693b555c4145918078d870a2a648"
            ],
            "layout": "IPY_MODEL_6f071f6dbae84a418e3cc8af8893f921"
          }
        },
        "93cf11b23d334dcc982c7e3ebd3ba57a": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "ProgressStyleModel",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "ProgressStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "bar_color": null,
            "description_width": ""
          }
        },
        "a6292478607641efa50af2909b4b769a": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "DescriptionStyleModel",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "a69de83c36cc46289a00ee81f4125cfa": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "DescriptionStyleModel",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "a8b6913554c74eb19a319cced459a8f6": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "DescriptionStyleModel",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "aa5dae53b5534d2abe0467a3f9d52008": {
          "model_module": "@jupyter-widgets/base",
          "model_module_version": "1.2.0",
          "model_name": "LayoutModel",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "b2d9f77ebfb144de9e978e309d640041": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "PasswordModel",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "PasswordModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "PasswordView",
            "continuous_update": true,
            "description": "Token:",
            "description_tooltip": null,
            "disabled": false,
            "layout": "IPY_MODEL_730882a26b9741649831beaa86d76db2",
            "placeholder": "​",
            "style": "IPY_MODEL_fcd47ca1590b43559d08c04f0bb4d164",
            "value": ""
          }
        },
        "b34db738e74a4f3dbc52f42dedc6dd8a": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "LabelModel",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "LabelModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "LabelView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_fffa059ea84f420fbd8940376678b431",
            "placeholder": "​",
            "style": "IPY_MODEL_0593f018fea244b6907aa06164dd59c5",
            "value": "Login successful"
          }
        },
        "b4da694310f04f8fab4a9b64a526509e": {
          "model_module": "@jupyter-widgets/base",
          "model_module_version": "1.2.0",
          "model_name": "LayoutModel",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "bd117e3f5a4240a982ff268c95fb2b4e": {
          "model_module": "@jupyter-widgets/base",
          "model_module_version": "1.2.0",
          "model_name": "LayoutModel",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "bf4518dfbee14600a799f300b00ab41b": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "DescriptionStyleModel",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "bf569df5a1e543f5b2617d2f5778a6a4": {
          "model_module": "@jupyter-widgets/base",
          "model_module_version": "1.2.0",
          "model_name": "LayoutModel",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "c3c8693b555c4145918078d870a2a648": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "HTMLModel",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_4dc87365768a45678a4d32eac16188c5",
            "placeholder": "​",
            "style": "IPY_MODEL_a8b6913554c74eb19a319cced459a8f6",
            "value": " 0/3 [00:09&lt;?, ?it/s]"
          }
        },
        "c7b90e0210b44a838cc785acdc4d1298": {
          "model_module": "@jupyter-widgets/base",
          "model_module_version": "1.2.0",
          "model_name": "LayoutModel",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "c8210da876ed4d6ab02d6c1897b10528": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "LabelModel",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "LabelModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "LabelView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_631d08c360674a2b9b5aad1a388029fd",
            "placeholder": "​",
            "style": "IPY_MODEL_8c5818ee930e49fdaeb6249a35bbca34",
            "value": "Connecting..."
          }
        },
        "d164013a0ad24027a96f98887fd46d6a": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "LabelModel",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "LabelModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "LabelView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_bf569df5a1e543f5b2617d2f5778a6a4",
            "placeholder": "​",
            "style": "IPY_MODEL_a69de83c36cc46289a00ee81f4125cfa",
            "value": "Your token has been saved in your configured git credential helpers (store)."
          }
        },
        "dc7853a136234b3baafc99ba4e6376dd": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "HTMLModel",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_18490330921d463ba5166f201ad39aca",
            "placeholder": "​",
            "style": "IPY_MODEL_a6292478607641efa50af2909b4b769a",
            "value": "Loading checkpoint shards:   0%"
          }
        },
        "dd7b36d536434e019a3d84b951c96831": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "VBoxModel",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "VBoxModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "VBoxView",
            "box_style": "",
            "children": [
              "IPY_MODEL_5791332747c644049b9de417a3369818",
              "IPY_MODEL_d164013a0ad24027a96f98887fd46d6a",
              "IPY_MODEL_edeb62ff425d429cb6cc6de5fe20e68c",
              "IPY_MODEL_b34db738e74a4f3dbc52f42dedc6dd8a"
            ],
            "layout": "IPY_MODEL_eb22c372a98e454e997cba3a562c7439"
          }
        },
        "de744a456e7a49569eaa973837732c7b": {
          "model_module": "@jupyter-widgets/base",
          "model_module_version": "1.2.0",
          "model_name": "LayoutModel",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "e45351ae2e4f4454a1eea6d56f0398e7": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "ButtonModel",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "ButtonModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "ButtonView",
            "button_style": "",
            "description": "Login",
            "disabled": false,
            "icon": "",
            "layout": "IPY_MODEL_bd117e3f5a4240a982ff268c95fb2b4e",
            "style": "IPY_MODEL_55ddc088c8d149d584d5586680123f7a",
            "tooltip": ""
          }
        },
        "eb22c372a98e454e997cba3a562c7439": {
          "model_module": "@jupyter-widgets/base",
          "model_module_version": "1.2.0",
          "model_name": "LayoutModel",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": "center",
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": "flex",
            "flex": null,
            "flex_flow": "column",
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": "50%"
          }
        },
        "edeb62ff425d429cb6cc6de5fe20e68c": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "LabelModel",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "LabelModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "LabelView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_aa5dae53b5534d2abe0467a3f9d52008",
            "placeholder": "​",
            "style": "IPY_MODEL_511fb228975e48a3b4a31bbf73cc2b42",
            "value": "Your token has been saved to /root/.cache/huggingface/token"
          }
        },
        "fcd47ca1590b43559d08c04f0bb4d164": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "DescriptionStyleModel",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "fffa059ea84f420fbd8940376678b431": {
          "model_module": "@jupyter-widgets/base",
          "model_module_version": "1.2.0",
          "model_name": "LayoutModel",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        }
      }
    }
  },
  "nbformat": 4,
  "nbformat_minor": 0
}