label_argilla_datasets_with_llms_annotation_guidelines_and_distilabel.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Label Argilla datasets with LLMs, good annotation guidelines, and distilabel\n",
    "\n",
    "In this notebook, we will use the `distilabel` library to label datasets with LLMs. We will use the `ArgillaLabeller` class to label any dataset that is hosted on Argilla with based on the written dataset definitions, like guidelines, fields, and questions. This is cool because you can use the same labelling interface for any dataset and questions that is hosted on Argilla."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Getting started\n",
    "\n",
    "### Deploy the Argilla server¶\n",
    "\n",
    "If you already have deployed Argilla, you can skip this step. Otherwise, you can quickly deploy Argilla following [this guide](https://docs.argilla.io/latest/getting_started/quickstart/).\n",
    "\n",
    "### Install dependencies"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001b[33mDEPRECATION: git+https://github.com/argilla-io/distilabel.git@develop#egg=distilabel[llama_cpp] contains an egg fragment with a non-PEP 508 name pip 25.0 will enforce this behaviour change. A possible replacement is to use the req @ url syntax, and remove the egg fragment. Discussion can be found at https://github.com/pypa/pip/issues/11617\u001b[0m\u001b[33m\n",
      "\u001b[0mCollecting distilabel (from distilabel[llama_cpp])\n",
      "  Cloning https://github.com/argilla-io/distilabel.git (to revision develop) to /private/var/folders/8z/jnnncfnj7_lfxym0262z4p180000gn/T/pip-install-d1dwfym7/distilabel_27023bb5907f435ca6caae42d5139507\n",
      "  Running command git clone --filter=blob:none --quiet https://github.com/argilla-io/distilabel.git /private/var/folders/8z/jnnncfnj7_lfxym0262z4p180000gn/T/pip-install-d1dwfym7/distilabel_27023bb5907f435ca6caae42d5139507\n",
      "  Running command git checkout -b develop --track origin/develop\n",
      "  Switched to a new branch 'develop'\n",
      "  branch 'develop' set up to track 'origin/develop'.\n",
      "  Resolved https://github.com/argilla-io/distilabel.git to commit 303722c58834fff2cca774686879ac834688c07f\n",
      "  Installing build dependencies ... \u001b[?25ldone\n",
      "\u001b[?25h  Getting requirements to build wheel ... \u001b[?25ldone\n",
      "\u001b[?25h  Preparing metadata (pyproject.toml) ... \u001b[?25ldone\n",
      "\u001b[?25hRequirement already satisfied: datasets>=2.16.0 in ./.venv/lib/python3.11/site-packages (from distilabel->distilabel[llama_cpp]) (3.0.1)\n",
      "Requirement already satisfied: httpx>=0.25.2 in ./.venv/lib/python3.11/site-packages (from distilabel->distilabel[llama_cpp]) (0.27.2)\n",
      "Requirement already satisfied: jinja2>=3.1.2 in ./.venv/lib/python3.11/site-packages (from distilabel->distilabel[llama_cpp]) (3.1.4)\n",
      "Requirement already satisfied: multiprocess>=0.70 in ./.venv/lib/python3.11/site-packages (from distilabel->distilabel[llama_cpp]) (0.70.16)\n",
      "Requirement already satisfied: nest-asyncio>=1.6.0 in ./.venv/lib/python3.11/site-packages (from distilabel->distilabel[llama_cpp]) (1.6.0)\n",
      "Requirement already satisfied: networkx>=3.0 in ./.venv/lib/python3.11/site-packages (from distilabel->distilabel[llama_cpp]) (3.3)\n",
      "Collecting orjson>=3.10.0 (from distilabel->distilabel[llama_cpp])\n",
      "  Using cached orjson-3.10.7-cp311-cp311-macosx_10_15_x86_64.macosx_11_0_arm64.macosx_10_15_universal2.whl.metadata (50 kB)\n",
      "Collecting portalocker>=2.8.2 (from distilabel->distilabel[llama_cpp])\n",
      "  Using cached portalocker-2.10.1-py3-none-any.whl.metadata (8.5 kB)\n",
      "Requirement already satisfied: pydantic>=2.0 in ./.venv/lib/python3.11/site-packages (from distilabel->distilabel[llama_cpp]) (2.9.2)\n",
      "Requirement already satisfied: rich>=13.5.0 in ./.venv/lib/python3.11/site-packages (from distilabel->distilabel[llama_cpp]) (13.9.2)\n",
      "Requirement already satisfied: scipy>=1.10.0 in ./.venv/lib/python3.11/site-packages (from distilabel->distilabel[llama_cpp]) (1.14.1)\n",
      "Collecting tblib>=3.0.0 (from distilabel->distilabel[llama_cpp])\n",
      "  Using cached tblib-3.0.0-py3-none-any.whl.metadata (25 kB)\n",
      "Collecting typer>=0.9.0 (from distilabel->distilabel[llama_cpp])\n",
      "  Using cached typer-0.12.5-py3-none-any.whl.metadata (15 kB)\n",
      "Collecting universal-pathlib>=0.2.2 (from distilabel->distilabel[llama_cpp])\n",
      "  Using cached universal_pathlib-0.2.5-py3-none-any.whl.metadata (25 kB)\n",
      "Requirement already satisfied: llama-cpp-python>=0.2.0 in ./.venv/lib/python3.11/site-packages (from distilabel->distilabel[llama_cpp]) (0.2.85)\n",
      "Requirement already satisfied: filelock in ./.venv/lib/python3.11/site-packages (from datasets>=2.16.0->distilabel->distilabel[llama_cpp]) (3.16.1)\n",
      "Requirement already satisfied: numpy>=1.17 in ./.venv/lib/python3.11/site-packages (from datasets>=2.16.0->distilabel->distilabel[llama_cpp]) (1.26.4)\n",
      "Requirement already satisfied: pyarrow>=15.0.0 in ./.venv/lib/python3.11/site-packages (from datasets>=2.16.0->distilabel->distilabel[llama_cpp]) (17.0.0)\n",
      "Requirement already satisfied: dill<0.3.9,>=0.3.0 in ./.venv/lib/python3.11/site-packages (from datasets>=2.16.0->distilabel->distilabel[llama_cpp]) (0.3.8)\n",
      "Requirement already satisfied: pandas in ./.venv/lib/python3.11/site-packages (from datasets>=2.16.0->distilabel->distilabel[llama_cpp]) (2.2.3)\n",
      "Requirement already satisfied: requests>=2.32.2 in ./.venv/lib/python3.11/site-packages (from datasets>=2.16.0->distilabel->distilabel[llama_cpp]) (2.32.3)\n",
      "Requirement already satisfied: tqdm>=4.66.3 in ./.venv/lib/python3.11/site-packages (from datasets>=2.16.0->distilabel->distilabel[llama_cpp]) (4.66.5)\n",
      "Requirement already satisfied: xxhash in ./.venv/lib/python3.11/site-packages (from datasets>=2.16.0->distilabel->distilabel[llama_cpp]) (3.5.0)\n",
      "Requirement already satisfied: fsspec<=2024.6.1,>=2023.1.0 in ./.venv/lib/python3.11/site-packages (from fsspec[http]<=2024.6.1,>=2023.1.0->datasets>=2.16.0->distilabel->distilabel[llama_cpp]) (2024.6.1)\n",
      "Requirement already satisfied: aiohttp in ./.venv/lib/python3.11/site-packages (from datasets>=2.16.0->distilabel->distilabel[llama_cpp]) (3.10.9)\n",
      "Requirement already satisfied: huggingface-hub>=0.22.0 in ./.venv/lib/python3.11/site-packages (from datasets>=2.16.0->distilabel->distilabel[llama_cpp]) (0.25.2)\n",
      "Requirement already satisfied: packaging in ./.venv/lib/python3.11/site-packages (from datasets>=2.16.0->distilabel->distilabel[llama_cpp]) (24.1)\n",
      "Requirement already satisfied: pyyaml>=5.1 in ./.venv/lib/python3.11/site-packages (from datasets>=2.16.0->distilabel->distilabel[llama_cpp]) (6.0.2)\n",
      "Requirement already satisfied: anyio in ./.venv/lib/python3.11/site-packages (from httpx>=0.25.2->distilabel->distilabel[llama_cpp]) (4.6.0)\n",
      "Requirement already satisfied: certifi in ./.venv/lib/python3.11/site-packages (from httpx>=0.25.2->distilabel->distilabel[llama_cpp]) (2024.8.30)\n",
      "Requirement already satisfied: httpcore==1.* in ./.venv/lib/python3.11/site-packages (from httpx>=0.25.2->distilabel->distilabel[llama_cpp]) (1.0.6)\n",
      "Requirement already satisfied: idna in ./.venv/lib/python3.11/site-packages (from httpx>=0.25.2->distilabel->distilabel[llama_cpp]) (3.10)\n",
      "Requirement already satisfied: sniffio in ./.venv/lib/python3.11/site-packages (from httpx>=0.25.2->distilabel->distilabel[llama_cpp]) (1.3.1)\n",
      "Requirement already satisfied: h11<0.15,>=0.13 in ./.venv/lib/python3.11/site-packages (from httpcore==1.*->httpx>=0.25.2->distilabel->distilabel[llama_cpp]) (0.14.0)\n",
      "Requirement already satisfied: MarkupSafe>=2.0 in ./.venv/lib/python3.11/site-packages (from jinja2>=3.1.2->distilabel->distilabel[llama_cpp]) (3.0.1)\n",
      "Requirement already satisfied: typing-extensions>=4.5.0 in ./.venv/lib/python3.11/site-packages (from llama-cpp-python>=0.2.0->distilabel->distilabel[llama_cpp]) (4.12.2)\n",
      "Requirement already satisfied: diskcache>=5.6.1 in ./.venv/lib/python3.11/site-packages (from llama-cpp-python>=0.2.0->distilabel->distilabel[llama_cpp]) (5.6.3)\n",
      "Requirement already satisfied: annotated-types>=0.6.0 in ./.venv/lib/python3.11/site-packages (from pydantic>=2.0->distilabel->distilabel[llama_cpp]) (0.7.0)\n",
      "Requirement already satisfied: pydantic-core==2.23.4 in ./.venv/lib/python3.11/site-packages (from pydantic>=2.0->distilabel->distilabel[llama_cpp]) (2.23.4)\n",
      "Requirement already satisfied: markdown-it-py>=2.2.0 in ./.venv/lib/python3.11/site-packages (from rich>=13.5.0->distilabel->distilabel[llama_cpp]) (3.0.0)\n",
      "Requirement already satisfied: pygments<3.0.0,>=2.13.0 in ./.venv/lib/python3.11/site-packages (from rich>=13.5.0->distilabel->distilabel[llama_cpp]) (2.18.0)\n",
      "Collecting click>=8.0.0 (from typer>=0.9.0->distilabel->distilabel[llama_cpp])\n",
      "  Using cached click-8.1.7-py3-none-any.whl.metadata (3.0 kB)\n",
      "Collecting shellingham>=1.3.0 (from typer>=0.9.0->distilabel->distilabel[llama_cpp])\n",
      "  Using cached shellingham-1.5.4-py2.py3-none-any.whl.metadata (3.5 kB)\n",
      "Requirement already satisfied: aiohappyeyeballs>=2.3.0 in ./.venv/lib/python3.11/site-packages (from aiohttp->datasets>=2.16.0->distilabel->distilabel[llama_cpp]) (2.4.3)\n",
      "Requirement already satisfied: aiosignal>=1.1.2 in ./.venv/lib/python3.11/site-packages (from aiohttp->datasets>=2.16.0->distilabel->distilabel[llama_cpp]) (1.3.1)\n",
      "Requirement already satisfied: attrs>=17.3.0 in ./.venv/lib/python3.11/site-packages (from aiohttp->datasets>=2.16.0->distilabel->distilabel[llama_cpp]) (24.2.0)\n",
      "Requirement already satisfied: frozenlist>=1.1.1 in ./.venv/lib/python3.11/site-packages (from aiohttp->datasets>=2.16.0->distilabel->distilabel[llama_cpp]) (1.4.1)\n",
      "Requirement already satisfied: multidict<7.0,>=4.5 in ./.venv/lib/python3.11/site-packages (from aiohttp->datasets>=2.16.0->distilabel->distilabel[llama_cpp]) (6.1.0)\n",
      "Requirement already satisfied: yarl<2.0,>=1.12.0 in ./.venv/lib/python3.11/site-packages (from aiohttp->datasets>=2.16.0->distilabel->distilabel[llama_cpp]) (1.14.0)\n",
      "Requirement already satisfied: mdurl~=0.1 in ./.venv/lib/python3.11/site-packages (from markdown-it-py>=2.2.0->rich>=13.5.0->distilabel->distilabel[llama_cpp]) (0.1.2)\n",
      "Requirement already satisfied: charset-normalizer<4,>=2 in ./.venv/lib/python3.11/site-packages (from requests>=2.32.2->datasets>=2.16.0->distilabel->distilabel[llama_cpp]) (3.4.0)\n",
      "Requirement already satisfied: urllib3<3,>=1.21.1 in ./.venv/lib/python3.11/site-packages (from requests>=2.32.2->datasets>=2.16.0->distilabel->distilabel[llama_cpp]) (2.2.3)\n",
      "Requirement already satisfied: python-dateutil>=2.8.2 in ./.venv/lib/python3.11/site-packages (from pandas->datasets>=2.16.0->distilabel->distilabel[llama_cpp]) (2.9.0.post0)\n",
      "Requirement already satisfied: pytz>=2020.1 in ./.venv/lib/python3.11/site-packages (from pandas->datasets>=2.16.0->distilabel->distilabel[llama_cpp]) (2024.2)\n",
      "Requirement already satisfied: tzdata>=2022.7 in ./.venv/lib/python3.11/site-packages (from pandas->datasets>=2.16.0->distilabel->distilabel[llama_cpp]) (2024.2)\n",
      "Requirement already satisfied: six>=1.5 in ./.venv/lib/python3.11/site-packages (from python-dateutil>=2.8.2->pandas->datasets>=2.16.0->distilabel->distilabel[llama_cpp]) (1.16.0)\n",
      "Requirement already satisfied: propcache>=0.2.0 in ./.venv/lib/python3.11/site-packages (from yarl<2.0,>=1.12.0->aiohttp->datasets>=2.16.0->distilabel->distilabel[llama_cpp]) (0.2.0)\n",
      "Using cached orjson-3.10.7-cp311-cp311-macosx_10_15_x86_64.macosx_11_0_arm64.macosx_10_15_universal2.whl (251 kB)\n",
      "Using cached portalocker-2.10.1-py3-none-any.whl (18 kB)\n",
      "Using cached tblib-3.0.0-py3-none-any.whl (12 kB)\n",
      "Using cached typer-0.12.5-py3-none-any.whl (47 kB)\n",
      "Using cached universal_pathlib-0.2.5-py3-none-any.whl (49 kB)\n",
      "Using cached click-8.1.7-py3-none-any.whl (97 kB)\n",
      "Using cached shellingham-1.5.4-py2.py3-none-any.whl (9.8 kB)\n",
      "Building wheels for collected packages: distilabel\n",
      "  Building wheel for distilabel (pyproject.toml) ... \u001b[?25ldone\n",
      "\u001b[?25h  Created wheel for distilabel: filename=distilabel-1.5.0-py3-none-any.whl size=442132 sha256=6314223420ba99abb9c36a055dbeee908eab22547044b11a90de4004e02c767f\n",
      "  Stored in directory: /private/var/folders/8z/jnnncfnj7_lfxym0262z4p180000gn/T/pip-ephem-wheel-cache-5y8vvvoe/wheels/eb/3f/3f/92917b742bac7885b0d125e6c3928d6feb414c0a67b8c12cf4\n",
      "Successfully built distilabel\n",
      "Installing collected packages: universal-pathlib, tblib, shellingham, portalocker, orjson, click, typer, distilabel\n",
      "Successfully installed click-8.1.7 distilabel-1.5.0 orjson-3.10.7 portalocker-2.10.1 shellingham-1.5.4 tblib-3.0.0 typer-0.12.5 universal-pathlib-0.2.5\n",
      "\n",
      "\u001b[1m[\u001b[0m\u001b[34;49mnotice\u001b[0m\u001b[1;39;49m]\u001b[0m\u001b[39;49m A new release of pip is available: \u001b[0m\u001b[31;49m24.1.2\u001b[0m\u001b[39;49m -> \u001b[0m\u001b[32;49m24.2\u001b[0m\n",
      "\u001b[1m[\u001b[0m\u001b[34;49mnotice\u001b[0m\u001b[1;39;49m]\u001b[0m\u001b[39;49m To update, run: \u001b[0m\u001b[32;49mpip install --upgrade pip\u001b[0m\n"
     ]
    }
   ],
   "source": [
    "!pip install \"git+https://github.com/argilla-io/distilabel.git@develop#egg=distilabel[llama_cpp]\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "\n",
    "!pip install -U -qqq \"numpy==1.26.4\" \\\n",
    "                     \"outlines==0.0.36\" \\\n",
    "                     \"llama_cpp_python==0.2.85\" \\\n",
    "                     \"argilla==2.4.1\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Download the model\n",
    "\n",
    "For the example, we will just use a [quantized version of llama 3.2 1B](https://huggingface.co/hugging-quants/Llama-3.2-1B-Instruct-Q8_0-GGUF/tree/main). This model will be good enough for basic labelling tasks."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "llama-3.2-1b-instruct-q8_0.gguf already exists. Skipping download.\n"
     ]
    }
   ],
   "source": [
    "url = \"https://huggingface.co/hugging-quants/Llama-3.2-1B-Instruct-Q8_0-GGUF/resolve/main/llama-3.2-1b-instruct-q8_0.gguf?download=true\"\n",
    "filename = \"llama-3.2-1b-instruct-q8_0.gguf\"\n",
    "\n",
    "import requests\n",
    "import os\n",
    "\n",
    "\n",
    "def download_file(url, filename):\n",
    "    response = requests.get(url, stream=True)\n",
    "    response.raise_for_status()\n",
    "\n",
    "    with open(filename, \"wb\") as file:\n",
    "        for chunk in response.iter_content(chunk_size=8192):\n",
    "            file.write(chunk)\n",
    "\n",
    "\n",
    "if not os.path.exists(filename):\n",
    "    print(f\"Downloading {filename}...\")\n",
    "    download_file(url, filename)\n",
    "    print(f\"Download complete: {filename}\")\n",
    "else:\n",
    "    print(f\"{filename} already exists. Skipping download.\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Upload some a basic datasets to Argilla\n",
    "\n",
    "We will now choose 4 datasets from Hugging Face and upload them to Argilla. A dataset each for the `TextQuestion` and `LabelQuestion`.\n",
    "\n",
    "- https://huggingface.co/datasets/dair-ai/emotion: emotions as classes  \n",
    "- https://huggingface.co/datasets/fka/awesome-chatgpt-prompts: chatgpt prompts as text\n",
    "\n",
    "We will use the `rg.Dataset.from_hub` method to upload example datasets to Argilla. We will also use the `rg.Settings` class to map the fields to the questions."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/Users/ben/code/argilla-cookbook/.venv/lib/python3.11/site-packages/argilla/datasets/_resource.py:264: UserWarning: Workspace not provided. Using default workspace: argilla id: 735cae0d-eb08-45c3-ad79-0a11ad4dd2c2\n",
      "  warnings.warn(f\"Workspace not provided. Using default workspace: {workspace.name} id: {workspace.id}\")\n",
      "/Users/ben/code/argilla-cookbook/.venv/lib/python3.11/site-packages/argilla/records/_mapping/_mapper.py:89: UserWarning: Keys ['label'] in data are not present in the mapping and will be ignored.\n",
      "  warnings.warn(f\"Keys {unknown_keys} in data are not present in the mapping and will be ignored.\")\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\">DatasetRecords: The provided batch size <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">256</span> was normalized. Using value <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">100</span>.\n",
       "</pre>\n"
      ],
      "text/plain": [
       "DatasetRecords: The provided batch size \u001b[1;36m256\u001b[0m was normalized. Using value \u001b[1;36m100\u001b[0m.\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Sending records...: 100%|██████████| 1/1 [00:00<00:00,  6.68batch/s]\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "Dataset(id=UUID('55bb1f50-710a-412b-a411-4dab8349d4c2') inserted_at=datetime.datetime(2024, 10, 9, 13, 42, 27, 624381) updated_at=datetime.datetime(2024, 10, 9, 13, 42, 27, 751505) name='emotion-d5588445-108b-4e73-b1b8-f984fbb6281a' status='ready' guidelines=None allow_extra_metadata=False distribution=OverlapTaskDistributionModel(strategy='overlap', min_submitted=1) workspace_id=UUID('735cae0d-eb08-45c3-ad79-0a11ad4dd2c2') last_activity_at=datetime.datetime(2024, 10, 9, 13, 42, 27, 751505))"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import os\n",
    "from uuid import uuid4\n",
    "\n",
    "import argilla as rg\n",
    "\n",
    "client = rg.Argilla(api_key=\"argilla.apikey\", api_url=\"http://localhost:6900\")\n",
    "\n",
    "settings = rg.Settings(\n",
    "    fields=[\n",
    "        rg.TextField(\n",
    "            name=\"text\",\n",
    "            title=\"Text\",\n",
    "            description=\"Provide a concise response to the prompt\",\n",
    "        )\n",
    "    ],\n",
    "    questions=[\n",
    "        rg.LabelQuestion(\n",
    "            name=\"emotion\",\n",
    "            title=\"Emotion\",\n",
    "            description=\"Provide a single label for the emotion of the text\",\n",
    "            labels=[\"joy\", \"anger\", \"sadness\", \"fear\", \"surprise\", \"love\"],\n",
    "        )\n",
    "    ],\n",
    "    mapping={\"text\": \"text\"},\n",
    ")\n",
    "\n",
    "dataset_name = f\"emotion-{uuid4()}\"\n",
    "\n",
    "rg.Dataset.from_hub(\n",
    "    repo_id=\"dair-ai/emotion\",\n",
    "    name=dataset_name,\n",
    "    split=\"train[:100]\",\n",
    "    client=client,\n",
    "    with_records=True,\n",
    "    settings=settings,\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Label the datasets\n",
    "\n",
    "We will now label the datasets. We will use the `ArgillaLabeller` class to label the datasets. This class will use  will use a `LlamaCppLLM` LLM to label the datasets. These labels will then be converted into `rg.Suggestion` objects and added to the records. For the sake of the example, we will only label 5 records per time using a while loop that continuesly fetches pending records from Argilla for both datasets and labels them with the LLM. After the labelling, we will update the dataset with the new records."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[Record(id=ead1145a-79e5-4170-aa27-d64584adff7e,status=pending,fields={'text': 'i didnt feel humiliated'},metadata={},suggestions={},responses={})]\n"
     ]
    }
   ],
   "source": [
    "from distilabel.llms.llamacpp import LlamaCppLLM\n",
    "from distilabel.steps.tasks import ArgillaLabeller\n",
    "\n",
    "\n",
    "# Initialize the labeller with the model and fields\n",
    "labeller = ArgillaLabeller(\n",
    "    llm=LlamaCppLLM(\n",
    "        model_path=\"llama-3.2-1b-instruct-q8_0.gguf\",\n",
    "        n_ctx=8000,\n",
    "        extra_kwargs={\"max_new_tokens\": 8000, \"temperature\": 0.0},\n",
    "    )\n",
    ")\n",
    "labeller.load()\n",
    "\n",
    "dataset = client.datasets(name=dataset_name, workspace=\"argilla\")\n",
    "pending_records = list(\n",
    "    dataset.records(\n",
    "        query=rg.Query(filter=rg.Filter((\"status\", \"==\", \"pending\"))),\n",
    "        limit=1,\n",
    "    )\n",
    ")\n",
    "\n",
    "print(pending_records)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Distilabel will define a fewshot prompt for you\n",
    "\n",
    "For the sake of understanding the labelling process, we will expose the prompt that is used to label the records. This prompt is generated by the `ArgillaLabeller` class and is based on the questions and fields of the dataset. This prompt is used to label the records with the LLM."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\">You are an expert annotator and labelling assistant that understands complex domains and natural language \n",
       "processing. You are given input fields and a question. You should create a valid JSON object as an answer to the \n",
       "question based on the input fields. <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">1</span>. Understand the input fields and optional guidelines. <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">2</span>. Understand the \n",
       "question type and the question settings. <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">3</span>. Reason through your response step-by-step. <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">4</span>. Provide a valid JSON \n",
       "object as an answer to the question.\n",
       "Please provide an answer to the question based on the input fields.\n",
       "\n",
       "# Input Fields\n",
       "title: Text\n",
       "i didnt feel humiliated\n",
       "# Question\n",
       "title: Emotion\n",
       "description: Provide a single label for the emotion of the text\n",
       "label_instruction: Select the appropriate label from the list of provided labels.\n",
       "labels: <span style=\"font-weight: bold\">[</span><span style=\"color: #008000; text-decoration-color: #008000\">'joy'</span>, <span style=\"color: #008000; text-decoration-color: #008000\">'anger'</span>, <span style=\"color: #008000; text-decoration-color: #008000\">'sadness'</span>, <span style=\"color: #008000; text-decoration-color: #008000\">'fear'</span>, <span style=\"color: #008000; text-decoration-color: #008000\">'surprise'</span>, <span style=\"color: #008000; text-decoration-color: #008000\">'love'</span><span style=\"font-weight: bold\">]</span>\n",
       "\n",
       "</pre>\n"
      ],
      "text/plain": [
       "You are an expert annotator and labelling assistant that understands complex domains and natural language \n",
       "processing. You are given input fields and a question. You should create a valid JSON object as an answer to the \n",
       "question based on the input fields. \u001b[1;36m1\u001b[0m. Understand the input fields and optional guidelines. \u001b[1;36m2\u001b[0m. Understand the \n",
       "question type and the question settings. \u001b[1;36m3\u001b[0m. Reason through your response step-by-step. \u001b[1;36m4\u001b[0m. Provide a valid JSON \n",
       "object as an answer to the question.\n",
       "Please provide an answer to the question based on the input fields.\n",
       "\n",
       "# Input Fields\n",
       "title: Text\n",
       "i didnt feel humiliated\n",
       "# Question\n",
       "title: Emotion\n",
       "description: Provide a single label for the emotion of the text\n",
       "label_instruction: Select the appropriate label from the list of provided labels.\n",
       "labels: \u001b[1m[\u001b[0m\u001b[32m'joy'\u001b[0m, \u001b[32m'anger'\u001b[0m, \u001b[32m'sadness'\u001b[0m, \u001b[32m'fear'\u001b[0m, \u001b[32m'surprise'\u001b[0m, \u001b[32m'love'\u001b[0m\u001b[1m]\u001b[0m\n",
       "\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "from rich import print\n",
    "\n",
    "prompt = [\n",
    "    labeller.format_input(\n",
    "        {\n",
    "            \"record\": record,\n",
    "            \"fields\": dataset.fields,\n",
    "            \"question\": dataset.questions[0],\n",
    "            \"guidelines\": dataset.guidelines,\n",
    "        }\n",
    "    )\n",
    "    for record in pending_records\n",
    "][0]\n",
    "\n",
    "print(*[row[\"content\"] for row in prompt], sep=\"\\n\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Distilabel can label your records\n",
    "\n",
    "We can use the `process` method of the `ArgillaLabeller` class to label the records. This method will label the records with the LLM and update the records with the new labels. We will use this method to label the records of the datasets."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"font-weight: bold\">{</span><span style=\"color: #008000; text-decoration-color: #008000\">'value'</span>: <span style=\"color: #008000; text-decoration-color: #008000\">'sadness'</span>, <span style=\"color: #008000; text-decoration-color: #008000\">'question_name'</span>: <span style=\"color: #008000; text-decoration-color: #008000\">'emotion'</span>, <span style=\"color: #008000; text-decoration-color: #008000\">'type'</span>: <span style=\"color: #008000; text-decoration-color: #008000\">'model'</span>, <span style=\"color: #008000; text-decoration-color: #008000\">'agent'</span>: <span style=\"color: #008000; text-decoration-color: #008000\">'llama-3.2-1b-instruct-q8_0.gguf'</span><span style=\"font-weight: bold\">}</span>\n",
       "</pre>\n"
      ],
      "text/plain": [
       "\u001b[1m{\u001b[0m\u001b[32m'value'\u001b[0m: \u001b[32m'sadness'\u001b[0m, \u001b[32m'question_name'\u001b[0m: \u001b[32m'emotion'\u001b[0m, \u001b[32m'type'\u001b[0m: \u001b[32m'model'\u001b[0m, \u001b[32m'agent'\u001b[0m: \u001b[32m'llama-3.2-1b-instruct-q8_0.gguf'\u001b[0m\u001b[1m}\u001b[0m\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Process the pending records\n",
    "result = next(\n",
    "    labeller.process(\n",
    "        [\n",
    "            {\n",
    "                \"record\": record,\n",
    "                \"fields\": dataset.fields,\n",
    "                \"question\": dataset.questions[0],\n",
    "                \"guidelines\": dataset.guidelines,\n",
    "            }\n",
    "            for record in pending_records\n",
    "        ]\n",
    "    )\n",
    ")\n",
    "suggestion = result[0][\"suggestion\"]\n",
    "\n",
    "print(suggestion)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Above, we can see that the labeler has labelled the records with the LLM. We could then take this suggestion and add it back to Argilla to be reviwed by a human."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "record = pending_records[0]\n",
    "record.suggestions.add(rg.Suggestion(**suggestion[\"suggestion\"]))\n",
    "dataset.records.log(pending_records)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![label dataset argilla labeller](./images/label_datasets%20with_llms_annotation_guidelines_and_distilabel.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Setup an active loop\n",
    "\n",
    "In reality, we would want to set up a parallel loop that continuously fetches pending records from Argilla and labels them with the LLM. Below, we can implement this as a simple while loop."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from time import sleep\n",
    "\n",
    "while True:\n",
    "\n",
    "    # query argilla for records that have been responded to\n",
    "    pending_records = list(\n",
    "        dataset.records(\n",
    "            query=rg.Query(filter=rg.Filter((\"status\", \"==\", \"pending\"))),\n",
    "            limit=1,\n",
    "        )\n",
    "    )\n",
    "\n",
    "    if not pending_records:\n",
    "        sleep(5)\n",
    "        continue\n",
    "\n",
    "    # label the pending records with the LLM based on the dataset settings\n",
    "    results = next(\n",
    "        labeller.process(\n",
    "            [\n",
    "                {\n",
    "                    \"record\": record,\n",
    "                    \"fields\": dataset.fields,\n",
    "                    \"question\": dataset.questions[0],\n",
    "                    \"guidelines\": dataset.guidelines,\n",
    "                }\n",
    "                for record in pending_records\n",
    "            ]\n",
    "        )\n",
    "    )\n",
    "\n",
    "    # add the suggestions to the records\n",
    "    for record, suggestion in zip(pending_records, results):\n",
    "        record.suggestions.add(rg.Suggestion(**suggestion[\"suggestion\"]))\n",
    "\n",
    "    # log the records with the suggestions back to argilla\n",
    "    dataset.records.log(pending_records)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 🎉 That's it\n",
    "\n",
    "That's it! We have successfully labelled the datasets with LLMs based annotation guidelines."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": ".env",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.9"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}