samples/python/classification/document-classification-text-embeddings.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Document Classification with Azure AI Document Intelligence and Text Embeddings\n",
    "\n",
    "This sample demonstrates how to classify a document using Azure AI Document Intelligence and text embeddings.\n",
    "\n",
    "![Data Classification](../../../images/classification-embeddings.png)\n",
    "\n",
    "This is achieved by the following process:\n",
    "\n",
    "- Define a list of classifications, with descriptions and keywords.\n",
    "- Create text embeddings for each of the classifications.\n",
    "- Analyze a document using Azure AI Document Intelligence's `prebuilt-layout` model to extract the text from each page.\n",
    "- For each page:\n",
    "  - Create text embeddings.\n",
    "  - Compare the embeddings with the embeddings of each classification.\n",
    "  - Assign the page to the classification with the highest similarity that exceeds a given threshold.\n",
    "\n",
    "## Objectives\n",
    "\n",
    "By the end of this sample, you will have learned how to:\n",
    "\n",
    "- Convert text to embeddings using Azure OpenAI's `text-embedding-3-large` model.\n",
    "- Convert a document's pages to Markdown format using Azure AI Document Intelligence.\n",
    "- Use cosine similarity to compare embeddings of classifications with document pages to classify them.\n",
    "\n",
    "## Useful Tips\n",
    "\n",
    "- Combine this technique with a [page extraction](../extraction/README.md) approach to ensure that you extract the most relevant data from a document's pages."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Setup"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Import modules\n",
    "\n",
    "This sample takes advantage of the following Python dependencies:\n",
    "\n",
    "- **numpy** and **sklearn** for determining the cosine similarity between embeddings.\n",
    "- **azure-ai-documentintelligence** to interface with the Azure AI Document Intelligence API for analyzing documents.\n",
    "- **openai** to interface with the Azure OpenAI API for generating text embeddings.\n",
    "- **azure-identity** to securely authenticate with deployed Azure Services using Microsoft Entra ID credentials.\n",
    "\n",
    "The following local components are also used:\n",
    "\n",
    "- [**classification**](../modules/samples/models/classification.py) to define the classifications.\n",
    "- [**accuracy_evaluator**](../modules/samples/evaluation/accuracy_evaluator.py) to evaluate the output of the classification process with expected results.\n",
    "- [**document_processing_result**](../modules/samples/models/document_processing_result.py) to store the results of the classification process as a file.\n",
    "- [**stopwatch**](../modules/samples/utils/stopwatch.py) to measure the end-to-end execution time for the classification process.\n",
    "- [**app_settings**](../modules/samples/app_settings.py) to access environment variables from the `.env` file."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import sys\n",
    "sys.path.append('../modules/') # Import local modules\n",
    "\n",
    "from IPython.display import display\n",
    "import os\n",
    "import pandas as pd\n",
    "from dotenv import dotenv_values\n",
    "from azure.ai.documentintelligence import DocumentIntelligenceClient\n",
    "from azure.ai.documentintelligence.models import AnalyzeResult, DocumentContentFormat\n",
    "from openai import AzureOpenAI\n",
    "from azure.identity import DefaultAzureCredential, get_bearer_token_provider\n",
    "from concurrent.futures import ThreadPoolExecutor, as_completed\n",
    "import numpy as np\n",
    "from sklearn.metrics.pairwise import cosine_similarity\n",
    "\n",
    "from samples.app_settings import AppSettings\n",
    "from samples.utils.stopwatch import Stopwatch\n",
    "from samples.utils.storage_utils import create_json_file\n",
    "from samples.models.document_processing_result import DataClassificationResult\n",
    "\n",
    "from samples.models.classification import Classifications, Classification\n",
    "from samples.evaluation.accuracy_evaluator import AccuracyEvaluator\n",
    "from samples.evaluation.comparison import get_classification_comparison"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Configure the Azure services\n",
    "\n",
    "To use Azure AI Document Intelligence and Azure OpenAI, their SDKs are used to create client instances using a deployed endpoint and authentication credentials.\n",
    "\n",
    "For this sample, the credentials of the Azure CLI are used to authenticate with the deployed services."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Set the working directory to the root of the repo\n",
    "working_dir = os.path.abspath('../../../')\n",
    "settings = AppSettings(dotenv_values(f\"{working_dir}/.env\"))\n",
    "sample_path = f\"{working_dir}/samples/python/classification/\"\n",
    "sample_name = \"document-classification-text-embeddings\"\n",
    "\n",
    "# Configure the default credential for accessing Azure services using Azure CLI credentials\n",
    "credential = DefaultAzureCredential(\n",
    "    exclude_workload_identity_credential=True,\n",
    "    exclude_developer_cli_credential=True,\n",
    "    exclude_environment_credential=True,\n",
    "    exclude_managed_identity_credential=True,\n",
    "    exclude_powershell_credential=True,\n",
    "    exclude_shared_token_cache_credential=True,\n",
    "    exclude_interactive_browser_credential=True\n",
    ")\n",
    "\n",
    "openai_token_provider = get_bearer_token_provider(credential, 'https://cognitiveservices.azure.com/.default')\n",
    "\n",
    "openai_client = AzureOpenAI(\n",
    "    azure_endpoint=settings.openai_endpoint,\n",
    "    azure_ad_token_provider=openai_token_provider,\n",
    "    api_version=\"2024-12-01-preview\" # Requires the latest API version for structured outputs.\n",
    ")\n",
    "\n",
    "document_intelligence_client = DocumentIntelligenceClient(\n",
    "    endpoint=settings.ai_services_endpoint,\n",
    "    credential=credential\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Establish the expected output\n",
    "\n",
    "To compare the accuracy of the classification process, the expected output of the classification process has been defined in the following code block based on each page of a [Vehicle Insurance Policy](../../assets/vehicle_insurance/policy_1.pdf).\n",
    "\n",
    "The expected output has been defined by a human evaluating the document.\n",
    "\n",
    "> **Note**: Only the `page_number` and `classification` are used in the accuracy evaluation."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "path = f\"{working_dir}/samples/assets/vehicle_insurance/\"\n",
    "pdf_fname = \"policy_1.pdf\"\n",
    "pdf_fpath = f\"{path}{pdf_fname}\"\n",
    "\n",
    "expected = Classifications(classifications=[\n",
    "    Classification(page_number=0, classification=\"Insurance Policy\", similarity=1.0),\n",
    "    Classification(page_number=1, classification=\"Insurance Policy\", similarity=1.0),\n",
    "    Classification(page_number=2, classification=\"Insurance Policy\", similarity=1.0),\n",
    "    Classification(page_number=3, classification=\"Insurance Policy\", similarity=1.0),\n",
    "    Classification(page_number=4, classification=\"Insurance Policy\", similarity=1.0),\n",
    "    Classification(page_number=5, classification=\"Insurance Certificate\", similarity=1.0),\n",
    "    Classification(page_number=6, classification=\"Terms and Conditions\", similarity=1.0),\n",
    "    Classification(page_number=7, classification=\"Terms and Conditions\", similarity=1.0),\n",
    "    Classification(page_number=8, classification=\"Terms and Conditions\", similarity=1.0),\n",
    "    Classification(page_number=9, classification=\"Terms and Conditions\", similarity=1.0),\n",
    "    Classification(page_number=10, classification=\"Terms and Conditions\", similarity=1.0),\n",
    "    Classification(page_number=11, classification=\"Terms and Conditions\", similarity=1.0),\n",
    "    Classification(page_number=12, classification=\"Terms and Conditions\", similarity=1.0)\n",
    "])\n",
    "\n",
    "classification_evaluator = AccuracyEvaluator(match_keys=[\"page_number\"], ignore_keys=[\"page_number\", \"similarity\"])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Define classifications\n",
    "\n",
    "The following code block defines the classifications for a document. Each classification has a name, description, and keywords that will be used to generate embeddings and compare similarity with each page of the document.\n",
    "\n",
    "> **Note**, the classifications have been defined based on expected content in a specific type of document, in this example, [a Vehicle Insurance Policy](../../assets/vehicle_insurance/policy_1.pdf)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "classifications = [\n",
    "    {\n",
    "        \"classification\": \"Insurance Policy\",\n",
    "        \"description\": \"Specific information related to an insurance policy, such as coverage, limits, premiums, and terms, often used for reference or clarification purposes.\",\n",
    "        \"keywords\": [\n",
    "            \"welcome letter\",\n",
    "            \"personal details\",\n",
    "            \"vehicle details\",\n",
    "            \"insured driver details\",\n",
    "            \"policy details\",\n",
    "            \"incident/conviction history\",\n",
    "            \"schedule of insurance\",\n",
    "            \"vehicle damage excesses\"\n",
    "        ]\n",
    "    },\n",
    "    {\n",
    "        \"classification\": \"Insurance Certificate\",\n",
    "        \"description\": \"A document that serves as proof of insurance coverage, often required for legal, regulatory, or contractual purposes.\",\n",
    "        \"keywords\": [\n",
    "            \"certificate of vehicle insurance\",\n",
    "            \"effective date of insurance\",\n",
    "            \"entitlement to drive\",\n",
    "            \"limitations of use\"\n",
    "        ]\n",
    "    },\n",
    "    {\n",
    "        \"classification\": \"Terms and Conditions\",\n",
    "        \"description\": \"The rules, requirements, or obligations that govern an agreement or contract, often related to insurance policies, financial products, or legal documents.\",\n",
    "        \"keywords\": [\n",
    "            \"terms and conditions\",\n",
    "            \"legal statements\",\n",
    "            \"payment instructions\",\n",
    "            \"legal obligations\",\n",
    "            \"covered for\",\n",
    "            \"claim settlement\",\n",
    "            \"costs to pay\",\n",
    "            \"legal responsibility\",\n",
    "            \"personal accident coverage\",\n",
    "            \"medical expense coverage\",\n",
    "            \"personal liability coverage\",\n",
    "            \"windscreen damage coverage\",\n",
    "            \"uninsured motorist protection\",\n",
    "            \"renewal instructions\",\n",
    "            \"cancellation instructions\"\n",
    "        ]\n",
    "    }\n",
    "]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Convert the document pages to Markdown\n",
    "\n",
    "To classify the document pages using embeddings, the text from each page must first be extracted.\n",
    "\n",
    "The following code block converts the document pages to Markdown format using Azure AI Document Intelligence's `prebuilt-layout` model.\n",
    "\n",
    "For the purposes of this sample, we will be classifying each page. The benefit of using Azure AI Document Intelligence for this extraction is that it provides a page-by-page analysis result of the document."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "with Stopwatch() as di_stopwatch:\n",
    "    with open(pdf_fpath, \"rb\") as f:\n",
    "        poller = document_intelligence_client.begin_analyze_document(\n",
    "            model_id=\"prebuilt-layout\",\n",
    "            body=f,\n",
    "            output_content_format=DocumentContentFormat.MARKDOWN,\n",
    "            content_type=\"application/pdf\"\n",
    "        )\n",
    "        \n",
    "    result: AnalyzeResult = poller.result()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "pages_content = []\n",
    "for page in result.pages:\n",
    "    # Extract the entire content for each page of the document based on the span offsets and lengths\n",
    "    content = result.content[page.spans[0]['offset']: page.spans[0]['offset'] + page.spans[0]['length']]\n",
    "    pages_content.append(content)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Create embeddings\n",
    "\n",
    "With the text extracted from the document and the classifications defined, the next step is to create embeddings for each page and classification."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Retrieving embeddings for text\n",
    "\n",
    "The following helper function retrieves embeddings for a given piece of text using Azure OpenAI's `text-embedding-3-large` model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "def get_embedding(text: str):\n",
    "    response = openai_client.embeddings.create(\n",
    "        input=text,\n",
    "        model=settings.text_embedding_model_deployment_name\n",
    "    )\n",
    "    embedding = response.data[0].embedding\n",
    "    return embedding"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Convert the classifications to embeddings\n",
    "\n",
    "The following code block takes each classification and generates the embeddings for the keywords."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "def process_classification(classification):\n",
    "    combined_text = f\"{', '.join(classification['keywords'])}\"\n",
    "    classification['embedding'] = get_embedding(combined_text)\n",
    "\n",
    "with Stopwatch() as ce_stopwatch:\n",
    "    with ThreadPoolExecutor() as executor:\n",
    "        executor.map(process_classification, classifications)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Convert the document pages to embeddings\n",
    "\n",
    "The following code block takes each page of the document and generates the embeddings for the text."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "page_embeddings = [None] * len(pages_content)\n",
    "\n",
    "with Stopwatch() as de_stopwatch:\n",
    "    with ThreadPoolExecutor() as executor:\n",
    "        future_to_idx = {executor.submit(get_embedding, text): idx for idx, text in enumerate(pages_content)}\n",
    "        for future in as_completed(future_to_idx):\n",
    "            idx = future_to_idx[future]\n",
    "            page_embeddings[idx] = future.result()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Classify the document pages\n",
    "\n",
    "The following code block runs the classification process using cosine similarity to compare the embeddings of the document pages with the embeddings of the predefined categories.\n",
    "\n",
    "It performs the following steps iteratively for each page in the document:\n",
    "\n",
    "1. Calculates the cosine similarity between the embeddings of the page and the matrix of embeddings of the predefined categories.\n",
    "2. Finds the best match for the page based on the maximum cosine similarity score.\n",
    "3. If the cosine similarity score is above a certain threshold, the page is classified under the best match category. Otherwise, the page is classified as \"Unclassified\"."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "similarity_threshold = 0.6 # Minimum similarity threshold for classification"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "classification_embeddings = [cls['embedding'] for cls in classifications]\n",
    "classification_matrix = np.array(classification_embeddings)\n",
    "\n",
    "with Stopwatch() as classify_stopwatch:\n",
    "    document_classifications = Classifications(classifications=[])\n",
    "    for idx, page_emb in enumerate(page_embeddings):\n",
    "        if not page_emb:\n",
    "            classification = \"Unclassified\"\n",
    "            similarity = 0.0\n",
    "        else:\n",
    "            page_vector = np.array(page_emb).reshape(1, -1)\n",
    "            similarities = cosine_similarity(page_vector, classification_matrix)[0]\n",
    "            best_match_idx = np.argmax(similarities)\n",
    "            best_similarity = similarities[best_match_idx]\n",
    "\n",
    "            if best_similarity >= similarity_threshold:\n",
    "                classification = classifications[best_match_idx]['classification']\n",
    "            else:\n",
    "                classification = f\"\"\"Unclassified ({classifications[best_match_idx]['classification']})\"\"\"\n",
    "                \n",
    "        document_classifications.classifications.append(\n",
    "            Classification(\n",
    "                page_number=idx,\n",
    "                classification=classification,\n",
    "                similarity=best_similarity\n",
    "            )\n",
    "        )"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Calculate the accuracy\n",
    "\n",
    "The following code block calculates the accuracy of the classification process by comparing the actual classifications with the predicted classifications."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [],
   "source": [
    "expected_dict = expected.model_dump()\n",
    "classifications_dict = document_classifications.model_dump()\n",
    "\n",
    "accuracy = classification_evaluator.evaluate(expected=expected_dict, actual=classifications_dict)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Visualize the outputs\n",
    "\n",
    "To provide context for the execution of the code, the following code blocks visualize the outputs of the classification process.\n",
    "\n",
    "This includes:\n",
    "\n",
    "- The accuracy of the classification process comparing the expected output with the result of comparing the embeddings.\n",
    "- The execution time of the end-to-end process.\n",
    "- The classification results for each page in the document.\n",
    "\n",
    "### Understanding Similarity\n",
    "\n",
    "Cosine similarity is a metric used to measure how similar two vectors are. Embeddings are numerical representations of text. By converting a document page and classification keywords to embeddings, we can compare the similarity between the two using this technique.\n",
    "\n",
    "Similarity scores close to 1 indicate that the two vectors share similar characteristics, while scores closer to 0 or negative values indicate that the two vectors are dissimilar."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Gets the total execution time of the classification process.\n",
    "total_elapsed = di_stopwatch.elapsed + ce_stopwatch.elapsed + de_stopwatch.elapsed + classify_stopwatch.elapsed"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Save the output of the data classification result.\n",
    "classification_result = DataClassificationResult(classifications_dict, accuracy, total_elapsed)\n",
    "\n",
    "create_json_file(f\"{sample_path}/{sample_name}.{pdf_fname}.json\", classification_result)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Accuracy</th>\n",
       "      <th>Execution Time</th>\n",
       "      <th>Document Intelligence Execution Time</th>\n",
       "      <th>Classification Embedding Execution Time</th>\n",
       "      <th>Document Embedding Execution Time</th>\n",
       "      <th>Classification Execution Time</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>61.54%</td>\n",
       "      <td>9.28 seconds</td>\n",
       "      <td>7.37 seconds</td>\n",
       "      <td>1.27 seconds</td>\n",
       "      <td>0.63 seconds</td>\n",
       "      <td>0.02 seconds</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "  Accuracy Execution Time Document Intelligence Execution Time  \\\n",
       "0   61.54%   9.28 seconds                         7.37 seconds   \n",
       "\n",
       "  Classification Embedding Execution Time Document Embedding Execution Time  \\\n",
       "0                            1.27 seconds                      0.63 seconds   \n",
       "\n",
       "  Classification Execution Time  \n",
       "0                  0.02 seconds  "
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<table>\n",
       "<thead>\n",
       "<tr><th style=\"text-align: right;\">  Page</th><th>Expected             </th><th>Extracted                          </th><th style=\"text-align: right;\">  Similarity</th></tr>\n",
       "</thead>\n",
       "<tbody>\n",
       "<tr><td style=\"text-align: right;\">     0</td><td>Insurance Policy     </td><td>Unclassified (Insurance Policy)    </td><td style=\"text-align: right;\">    0.588678</td></tr>\n",
       "<tr><td style=\"text-align: right;\">     1</td><td>Insurance Policy     </td><td>Unclassified (Insurance Policy)    </td><td style=\"text-align: right;\">    0.5812  </td></tr>\n",
       "<tr><td style=\"text-align: right;\">     2</td><td>Insurance Policy     </td><td>Insurance Policy                   </td><td style=\"text-align: right;\">    0.661986</td></tr>\n",
       "<tr><td style=\"text-align: right;\">     3</td><td>Insurance Policy     </td><td>Unclassified (Insurance Policy)    </td><td style=\"text-align: right;\">    0.522973</td></tr>\n",
       "<tr><td style=\"text-align: right;\">     4</td><td>Insurance Policy     </td><td>Unclassified (Insurance Policy)    </td><td style=\"text-align: right;\">    0.596999</td></tr>\n",
       "<tr><td style=\"text-align: right;\">     5</td><td>Insurance Certificate</td><td>Insurance Certificate              </td><td style=\"text-align: right;\">    0.669797</td></tr>\n",
       "<tr><td style=\"text-align: right;\">     6</td><td>Terms and Conditions </td><td>Terms and Conditions               </td><td style=\"text-align: right;\">    0.607947</td></tr>\n",
       "<tr><td style=\"text-align: right;\">     7</td><td>Terms and Conditions </td><td>Terms and Conditions               </td><td style=\"text-align: right;\">    0.634872</td></tr>\n",
       "<tr><td style=\"text-align: right;\">     8</td><td>Terms and Conditions </td><td>Unclassified (Terms and Conditions)</td><td style=\"text-align: right;\">    0.562734</td></tr>\n",
       "<tr><td style=\"text-align: right;\">     9</td><td>Terms and Conditions </td><td>Terms and Conditions               </td><td style=\"text-align: right;\">    0.60846 </td></tr>\n",
       "<tr><td style=\"text-align: right;\">    10</td><td>Terms and Conditions </td><td>Terms and Conditions               </td><td style=\"text-align: right;\">    0.606074</td></tr>\n",
       "<tr><td style=\"text-align: right;\">    11</td><td>Terms and Conditions </td><td>Terms and Conditions               </td><td style=\"text-align: right;\">    0.636422</td></tr>\n",
       "<tr><td style=\"text-align: right;\">    12</td><td>Terms and Conditions </td><td>Terms and Conditions               </td><td style=\"text-align: right;\">    0.631899</td></tr>\n",
       "</tbody>\n",
       "</table>"
      ],
      "text/plain": [
       "'<table>\\n<thead>\\n<tr><th style=\"text-align: right;\">  Page</th><th>Expected             </th><th>Extracted                          </th><th style=\"text-align: right;\">  Similarity</th></tr>\\n</thead>\\n<tbody>\\n<tr><td style=\"text-align: right;\">     0</td><td>Insurance Policy     </td><td>Unclassified (Insurance Policy)    </td><td style=\"text-align: right;\">    0.588678</td></tr>\\n<tr><td style=\"text-align: right;\">     1</td><td>Insurance Policy     </td><td>Unclassified (Insurance Policy)    </td><td style=\"text-align: right;\">    0.5812  </td></tr>\\n<tr><td style=\"text-align: right;\">     2</td><td>Insurance Policy     </td><td>Insurance Policy                   </td><td style=\"text-align: right;\">    0.661986</td></tr>\\n<tr><td style=\"text-align: right;\">     3</td><td>Insurance Policy     </td><td>Unclassified (Insurance Policy)    </td><td style=\"text-align: right;\">    0.522973</td></tr>\\n<tr><td style=\"text-align: right;\">     4</td><td>Insurance Policy     </td><td>Unclassified (Insurance Policy)    </td><td style=\"text-align: right;\">    0.596999</td></tr>\\n<tr><td style=\"text-align: right;\">     5</td><td>Insurance Certificate</td><td>Insurance Certificate              </td><td style=\"text-align: right;\">    0.669797</td></tr>\\n<tr><td style=\"text-align: right;\">     6</td><td>Terms and Conditions </td><td>Terms and Conditions               </td><td style=\"text-align: right;\">    0.607947</td></tr>\\n<tr><td style=\"text-align: right;\">     7</td><td>Terms and Conditions </td><td>Terms and Conditions               </td><td style=\"text-align: right;\">    0.634872</td></tr>\\n<tr><td style=\"text-align: right;\">     8</td><td>Terms and Conditions </td><td>Unclassified (Terms and Conditions)</td><td style=\"text-align: right;\">    0.562734</td></tr>\\n<tr><td style=\"text-align: right;\">     9</td><td>Terms and Conditions </td><td>Terms and Conditions               </td><td style=\"text-align: right;\">    0.60846 </td></tr>\\n<tr><td style=\"text-align: right;\">    10</td><td>Terms and Conditions </td><td>Terms and Conditions               </td><td style=\"text-align: right;\">    0.606074</td></tr>\\n<tr><td style=\"text-align: right;\">    11</td><td>Terms and Conditions </td><td>Terms and Conditions               </td><td style=\"text-align: right;\">    0.636422</td></tr>\\n<tr><td style=\"text-align: right;\">    12</td><td>Terms and Conditions </td><td>Terms and Conditions               </td><td style=\"text-align: right;\">    0.631899</td></tr>\\n</tbody>\\n</table>'"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Display the outputs of the classification process.\n",
    "df = pd.DataFrame([\n",
    "    {\n",
    "        \"Accuracy\": f\"{accuracy['overall'] * 100:.2f}%\",\n",
    "        \"Execution Time\": f\"{total_elapsed:.2f} seconds\",\n",
    "        \"Document Intelligence Execution Time\": f\"{di_stopwatch.elapsed:.2f} seconds\",\n",
    "        \"Classification Embedding Execution Time\": f\"{ce_stopwatch.elapsed:.2f} seconds\",\n",
    "        \"Document Embedding Execution Time\": f\"{de_stopwatch.elapsed:.2f} seconds\",\n",
    "        \"Classification Execution Time\": f\"{classify_stopwatch.elapsed:.2f} seconds\"\n",
    "    }\n",
    "])\n",
    "\n",
    "display(df)\n",
    "display(get_classification_comparison(expected, document_classifications))"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.12.8"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}