docs/pinecone-quickstart.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "8KNEuM_G83vi"
   },
   "source": [
    "[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/pinecone-io/examples/blob/master/docs/pinecone-quickstart.ipynb) [![Open nbviewer](https://raw.githubusercontent.com/pinecone-io/examples/master/assets/nbviewer-shield.svg)](https://nbviewer.org/github/pinecone-io/examples/blob/master/docs/pinecone-quickstart.ipynb)\n",
    "\n",
    "# Pinecone Database quickstart\n",
    "\n",
    "This notebook shows you how to set up and use Pinecone Database for high-performance semantic search."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "mjJUDk6xAA_r"
   },
   "source": [
    "## Install an SDK\n",
    "\n",
    "Pinecone provides [SDKs](https://docs.pinecone.io/reference/pinecone-sdks) in multiple languages.\n",
    "\n",
    "For this quickstart, install the Python SDK and a library that makes it easy to sign up with Pinecone:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "id": "4SudLike98WL"
   },
   "outputs": [],
   "source": [
    "!pip install -qU \\\n",
    "    pinecone==6.0.2 \\\n",
    "    pinecone-notebooks"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "WoDXUiIkD1U7"
   },
   "source": [
    "## Get an API key\n",
    "\n",
    "You need an API key to make calls to your Pinecone project.\n",
    "\n",
    "Use the widget below to generate a key. If you don't have a Pinecone account, the widget will sign you up for the free Starter plan."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 247
    },
    "id": "89S8G8oP61-t",
    "outputId": "8cf57515-28e1-4953-b86d-d23dad4ea9fe"
   },
   "outputs": [],
   "source": [
    "import os\n",
    "\n",
    "if not os.environ.get(\"PINECONE_API_KEY\"):\n",
    "    from pinecone_notebooks.colab import Authenticate\n",
    "    Authenticate()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "sbJFp5DO5ryT"
   },
   "source": [
    "## Initialize a client\n",
    "\n",
    "Use the generated API key to intialize a Pinecone client:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "e9rr_u6ZIvZ-"
   },
   "outputs": [],
   "source": [
    "# Import the Pinecone library\n",
    "from pinecone import Pinecone\n",
    "\n",
    "# Initialize a Pinecone client with your API key\n",
    "api_key = os.environ.get(\"PINECONE_API_KEY\")\n",
    "pc = Pinecone(api_key=api_key)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "yfnoFFihfoY4"
   },
   "source": [
    "## Create an index\n",
    "\n",
    "In Pinecone, there are two types of indexes for storing vector data: [Dense indexes](https://docs.pinecone.io/guides/indexes/understanding-indexes#dense-indexes) store dense vectors for semantic search, and [sparse indexes](https://docs.pinecone.io/guides/indexes/understanding-indexes#sparse-indexes) store sparse vectors for lexical/keyword search.\n",
    "\n",
    "For this quickstart, create a dense index that is integrated with an [embedding model hosted by Pinecone](https://docs.pinecone.io/guides/inference/understanding-inference#embedding-models). With integrated models, you upsert and search with text and have Pinecone generate vectors automatically.\n",
    "\n",
    "**Note:** If you prefer to use external embedding models, see [Bring your own vectors](https://docs.pinecone.io/guides/indexes/understanding-indexes#bring-your-own-vectors)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "id": "FMyeNo6Afh4z"
   },
   "outputs": [],
   "source": [
    "# Create a dense index with integrated embedding\n",
    "\n",
    "index_name = \"dense-index\"\n",
    "\n",
    "if not pc.has_index(name=index_name):\n",
    "    pc.create_index_for_model(\n",
    "        name=index_name,\n",
    "        cloud=\"aws\",\n",
    "        region=\"us-east-1\",\n",
    "        embed={\n",
    "            \"model\":\"llama-text-embed-v2\",\n",
    "            \"field_map\":{\"text\": \"chunk_text\"}\n",
    "        }\n",
    "    )\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "bN9Rl7GP258C"
   },
   "source": [
    "## Upsert records\n",
    "\n",
    "Prepare a sample dataset of factual statements from different domains like history, physics, technology, and music. Format the data as records with an ID, text, and category. These objects are expected to contain a `chunk_text` key because of the `field_map` we specified when creating the index above. \n",
    "\n",
    "Other fields which are not mapped in the field mapping, like `category`, will become metadata on the upserted records."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "id": "ZIclo2UK3NFE"
   },
   "outputs": [],
   "source": [
    "# Define your dataset\n",
    "records = [\n",
    "    { \"_id\": \"rec1\", \"chunk_text\": \"The Eiffel Tower was completed in 1889 and stands in Paris, France.\", \"category\": \"history\" },\n",
    "    { \"_id\": \"rec2\", \"chunk_text\": \"Photosynthesis allows plants to convert sunlight into energy.\", \"category\": \"science\" },\n",
    "    { \"_id\": \"rec3\", \"chunk_text\": \"Albert Einstein developed the theory of relativity.\", \"category\": \"science\" },\n",
    "    { \"_id\": \"rec4\", \"chunk_text\": \"The mitochondrion is often called the powerhouse of the cell.\", \"category\": \"biology\" },\n",
    "    { \"_id\": \"rec5\", \"chunk_text\": \"Shakespeare wrote many famous plays, including Hamlet and Macbeth.\", \"category\": \"literature\" },\n",
    "    { \"_id\": \"rec6\", \"chunk_text\": \"Water boils at 100°C under standard atmospheric pressure.\", \"category\": \"physics\" },\n",
    "    { \"_id\": \"rec7\", \"chunk_text\": \"The Great Wall of China was built to protect against invasions.\", \"category\": \"history\" },\n",
    "    { \"_id\": \"rec8\", \"chunk_text\": \"Honey never spoils due to its low moisture content and acidity.\", \"category\": \"food science\" },\n",
    "    { \"_id\": \"rec9\", \"chunk_text\": \"The speed of light in a vacuum is approximately 299,792 km/s.\", \"category\": \"physics\" },\n",
    "    { \"_id\": \"rec10\", \"chunk_text\": \"Newton’s laws describe the motion of objects.\", \"category\": \"physics\" },\n",
    "    { \"_id\": \"rec11\", \"chunk_text\": \"The human brain has approximately 86 billion neurons.\", \"category\": \"biology\" },\n",
    "    { \"_id\": \"rec12\", \"chunk_text\": \"The Amazon Rainforest is one of the most biodiverse places on Earth.\", \"category\": \"geography\" },\n",
    "    { \"_id\": \"rec13\", \"chunk_text\": \"Black holes have gravitational fields so strong that not even light can escape.\", \"category\": \"astronomy\" },\n",
    "    { \"_id\": \"rec14\", \"chunk_text\": \"The periodic table organizes elements based on their atomic number.\", \"category\": \"chemistry\" },\n",
    "    { \"_id\": \"rec15\", \"chunk_text\": \"Leonardo da Vinci painted the Mona Lisa.\", \"category\": \"art\" },\n",
    "    { \"_id\": \"rec16\", \"chunk_text\": \"The internet revolutionized communication and information sharing.\", \"category\": \"technology\" },\n",
    "    { \"_id\": \"rec17\", \"chunk_text\": \"The Pyramids of Giza are among the Seven Wonders of the Ancient World.\", \"category\": \"history\" },\n",
    "    { \"_id\": \"rec18\", \"chunk_text\": \"Dogs have an incredible sense of smell, much stronger than humans.\", \"category\": \"biology\" },\n",
    "    { \"_id\": \"rec19\", \"chunk_text\": \"The Pacific Ocean is the largest and deepest ocean on Earth.\", \"category\": \"geography\" },\n",
    "    { \"_id\": \"rec20\", \"chunk_text\": \"Chess is a strategic game that originated in India.\", \"category\": \"games\" },\n",
    "    { \"_id\": \"rec21\", \"chunk_text\": \"The Statue of Liberty was a gift from France to the United States.\", \"category\": \"history\" },\n",
    "    { \"_id\": \"rec22\", \"chunk_text\": \"Coffee contains caffeine, a natural stimulant.\", \"category\": \"food science\" },\n",
    "    { \"_id\": \"rec23\", \"chunk_text\": \"Thomas Edison invented the practical electric light bulb.\", \"category\": \"inventions\" },\n",
    "    { \"_id\": \"rec24\", \"chunk_text\": \"The moon influences ocean tides due to gravitational pull.\", \"category\": \"astronomy\" },\n",
    "    { \"_id\": \"rec25\", \"chunk_text\": \"DNA carries genetic information for all living organisms.\", \"category\": \"biology\" },\n",
    "    { \"_id\": \"rec26\", \"chunk_text\": \"Rome was once the center of a vast empire.\", \"category\": \"history\" },\n",
    "    { \"_id\": \"rec27\", \"chunk_text\": \"The Wright brothers pioneered human flight in 1903.\", \"category\": \"inventions\" },\n",
    "    { \"_id\": \"rec28\", \"chunk_text\": \"Bananas are a good source of potassium.\", \"category\": \"nutrition\" },\n",
    "    { \"_id\": \"rec29\", \"chunk_text\": \"The stock market fluctuates based on supply and demand.\", \"category\": \"economics\" },\n",
    "    { \"_id\": \"rec30\", \"chunk_text\": \"A compass needle points toward the magnetic north pole.\", \"category\": \"navigation\" },\n",
    "    { \"_id\": \"rec31\", \"chunk_text\": \"The universe is expanding, according to the Big Bang theory.\", \"category\": \"astronomy\" },\n",
    "    { \"_id\": \"rec32\", \"chunk_text\": \"Elephants have excellent memory and strong social bonds.\", \"category\": \"biology\" },\n",
    "    { \"_id\": \"rec33\", \"chunk_text\": \"The violin is a string instrument commonly used in orchestras.\", \"category\": \"music\" },\n",
    "    { \"_id\": \"rec34\", \"chunk_text\": \"The heart pumps blood throughout the human body.\", \"category\": \"biology\" },\n",
    "    { \"_id\": \"rec35\", \"chunk_text\": \"Ice cream melts when exposed to heat.\", \"category\": \"food science\" },\n",
    "    { \"_id\": \"rec36\", \"chunk_text\": \"Solar panels convert sunlight into electricity.\", \"category\": \"technology\" },\n",
    "    { \"_id\": \"rec37\", \"chunk_text\": \"The French Revolution began in 1789.\", \"category\": \"history\" },\n",
    "    { \"_id\": \"rec38\", \"chunk_text\": \"The Taj Mahal is a mausoleum built by Emperor Shah Jahan.\", \"category\": \"history\" },\n",
    "    { \"_id\": \"rec39\", \"chunk_text\": \"Rainbows are caused by light refracting through water droplets.\", \"category\": \"physics\" },\n",
    "    { \"_id\": \"rec40\", \"chunk_text\": \"Mount Everest is the tallest mountain in the world.\", \"category\": \"geography\" },\n",
    "    { \"_id\": \"rec41\", \"chunk_text\": \"Octopuses are highly intelligent marine creatures.\", \"category\": \"biology\" },\n",
    "    { \"_id\": \"rec42\", \"chunk_text\": \"The speed of sound is around 343 meters per second in air.\", \"category\": \"physics\" },\n",
    "    { \"_id\": \"rec43\", \"chunk_text\": \"Gravity keeps planets in orbit around the sun.\", \"category\": \"astronomy\" },\n",
    "    { \"_id\": \"rec44\", \"chunk_text\": \"The Mediterranean diet is considered one of the healthiest in the world.\", \"category\": \"nutrition\" },\n",
    "    { \"_id\": \"rec45\", \"chunk_text\": \"A haiku is a traditional Japanese poem with a 5-7-5 syllable structure.\", \"category\": \"literature\" },\n",
    "    { \"_id\": \"rec46\", \"chunk_text\": \"The human body is made up of about 60% water.\", \"category\": \"biology\" },\n",
    "    { \"_id\": \"rec47\", \"chunk_text\": \"The Industrial Revolution transformed manufacturing and transportation.\", \"category\": \"history\" },\n",
    "    { \"_id\": \"rec48\", \"chunk_text\": \"Vincent van Gogh painted Starry Night.\", \"category\": \"art\" },\n",
    "    { \"_id\": \"rec49\", \"chunk_text\": \"Airplanes fly due to the principles of lift and aerodynamics.\", \"category\": \"physics\" },\n",
    "    { \"_id\": \"rec50\", \"chunk_text\": \"Renewable energy sources include wind, solar, and hydroelectric power.\", \"category\": \"energy\" }\n",
    "]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "f5vNb1pugnR5"
   },
   "source": [
    "[Upsert](https://docs.pinecone.io/guides/data/upsert-data) the sample dataset into a new namespace in your index.\n",
    "\n",
    "Because your index is integrated with an embedding model, you provide the textual statements and Pinecone converts them to dense vectors automatically."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "id": "WqDOcyz5gp1Z"
   },
   "outputs": [],
   "source": [
    "# Get an Index client for the index we created\n",
    "dense_index = pc.Index(name=index_name)\n",
    "\n",
    "# Upsert records into a namespace\n",
    "dense_index.upsert_records(\n",
    "    namespace=\"example-namespace\", \n",
    "    records=records\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "cxM2hdTjg1tS"
   },
   "source": [
    "## Check index stats\n",
    "\n",
    "Pinecone is eventually consistent, so there can be a slight delay before new or changed records are visible to queries. You can [view index stats](https://docs.pinecone.io/guides/data/check-data-freshness#verify-record-counts) to check if the current vector count matches the number of vectors you upserted (50):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "id": "z3B4RkEqg1US"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'dimension': 1024,\n",
       " 'index_fullness': 0.0,\n",
       " 'metric': 'cosine',\n",
       " 'namespaces': {'example-namespace': {'vector_count': 50}},\n",
       " 'total_vector_count': 50,\n",
       " 'vector_type': 'dense'}"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import time\n",
    "\n",
    "# Wait for the upserted vectors to be indexed\n",
    "time.sleep(10)\n",
    "\n",
    "# View stats for the index\n",
    "stats = dense_index.describe_index_stats()\n",
    "stats"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "VpgIIsLlJGFf"
   },
   "source": [
    "## Semantic search\n",
    "\n",
    "[Search the dense index](https://docs.pinecone.io/guides/data/query-data#semantic-search) for ten records that are most semantically similar to the query, `Famous historical structures and monuments`.\n",
    "\n",
    "Again, because your index is integrated with an embedding model, you provide the query as text and Pinecone converts the text to a dense vector automatically."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "id": "Buo2K1h8O_fN"
   },
   "outputs": [],
   "source": [
    "def print_results(search_results):\n",
    "    for hit in search_results['result']['hits']:\n",
    "        print(f\"id: {hit['_id']:<5} | score: {round(hit['_score'], 3):<5} | category: {hit['fields']['category']:<10} | text: {hit['fields']['chunk_text']:<50}\")\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "id: rec17 | score: 0.252 | category: history    | text: The Pyramids of Giza are among the Seven Wonders of the Ancient World.\n",
      "id: rec5  | score: 0.186 | category: literature | text: Shakespeare wrote many famous plays, including Hamlet and Macbeth.\n",
      "id: rec38 | score: 0.186 | category: history    | text: The Taj Mahal is a mausoleum built by Emperor Shah Jahan.\n",
      "id: rec50 | score: 0.098 | category: energy     | text: Renewable energy sources include wind, solar, and hydroelectric power.\n",
      "id: rec15 | score: 0.096 | category: art        | text: Leonardo da Vinci painted the Mona Lisa.          \n",
      "id: rec26 | score: 0.084 | category: history    | text: Rome was once the center of a vast empire.        \n",
      "id: rec1  | score: 0.078 | category: history    | text: The Eiffel Tower was completed in 1889 and stands in Paris, France.\n",
      "id: rec47 | score: 0.072 | category: history    | text: The Industrial Revolution transformed manufacturing and transportation.\n",
      "id: rec7  | score: 0.072 | category: history    | text: The Great Wall of China was built to protect against invasions.\n",
      "id: rec21 | score: 0.061 | category: history    | text: The Statue of Liberty was a gift from France to the United States.\n"
     ]
    }
   ],
   "source": [
    "from pinecone import SearchQuery, SearchRerank\n",
    "\n",
    "# Define the query\n",
    "query = \"Famous historical structures and monuments\"\n",
    "\n",
    "# Search the dense index\n",
    "results = dense_index.search(\n",
    "    namespace=\"example-namespace\",\n",
    "    query=SearchQuery(\n",
    "        top_k=10,\n",
    "        inputs={'text': query}\n",
    "    )\n",
    ")\n",
    "\n",
    "print_results(results)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "SFXzoE1thnXa"
   },
   "source": [
    "Notice that most of the results are about historical structures and monuments. However, a few unrelated statements are included as well and are ranked high in the list, for example, statements about Shakespeare and renewable energy.\n",
    "\n",
    "To get a more accurate ranking, search again but this time [rerank the initial results](https://docs.pinecone.io/guides/data/query-data#rerank-results) based on their relevance to the query."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {
    "id": "MaqbcsI4I1gU"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "id: rec1  | score: 0.107 | category: history    | text: The Eiffel Tower was completed in 1889 and stands in Paris, France.\n",
      "id: rec38 | score: 0.064 | category: history    | text: The Taj Mahal is a mausoleum built by Emperor Shah Jahan.\n",
      "id: rec7  | score: 0.063 | category: history    | text: The Great Wall of China was built to protect against invasions.\n",
      "id: rec21 | score: 0.019 | category: history    | text: The Statue of Liberty was a gift from France to the United States.\n",
      "id: rec17 | score: 0.015 | category: history    | text: The Pyramids of Giza are among the Seven Wonders of the Ancient World.\n",
      "id: rec26 | score: 0.011 | category: history    | text: Rome was once the center of a vast empire.        \n",
      "id: rec15 | score: 0.008 | category: art        | text: Leonardo da Vinci painted the Mona Lisa.          \n",
      "id: rec5  | score: 0.0   | category: literature | text: Shakespeare wrote many famous plays, including Hamlet and Macbeth.\n",
      "id: rec47 | score: 0.0   | category: history    | text: The Industrial Revolution transformed manufacturing and transportation.\n",
      "id: rec50 | score: 0.0   | category: energy     | text: Renewable energy sources include wind, solar, and hydroelectric power.\n"
     ]
    }
   ],
   "source": [
    "# Search the dense index and rerank results\n",
    "reranked_results = dense_index.search(\n",
    "    namespace=\"example-namespace\",\n",
    "    query=SearchQuery(\n",
    "        top_k=10,\n",
    "        inputs={'text': query}\n",
    "    ),\n",
    "    rerank=SearchRerank(\n",
    "        model=\"bge-reranker-v2-m3\",\n",
    "        top_n=10,\n",
    "        rank_fields=[\"chunk_text\"]\n",
    "    )\n",
    ")\n",
    "\n",
    "print_results(reranked_results)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "lfaATEz7hvqC"
   },
   "source": [
    "Notice that all of the most relevant results about historical structures and monuments are now ranked highest."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "AsVqrR2YipPM"
   },
   "source": [
    "## Improve results\n",
    "\n",
    "Reranking results is one of the most effective ways to improve search accuracy and relevance, but there are many other techniques to consider. For example:\n",
    "\n",
    "* [Filtering by metadata](https://docs.pinecone.io/guides/data/query-data#filter-by-metadata): When records contain additional metadata, you can limit the search to records matching a filter expression.\n",
    "\n",
    "* [Hybrid search](https://docs.pinecone.io/guides/data/query-data#hybrid-search): You can add lexical search to capture precise keyword matches (e.g., product SKUs, email addresses, domain-specific terms) in addition to semantic matches.\n",
    "\n",
    "* [Chunking strategies](https://www.pinecone.io/learn/chunking-strategies/): You can chunk your content in different ways to get better results. Consider factors like the length of the content, the complexity of queries, and how results will be used in your application."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "DavgcphWjwWw"
   },
   "source": [
    "## Clean up\n",
    "\n",
    "When you no longer need your example index, you can delete it to save resources. After you delete an index, you cannot use it again or recover it."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "id": "1iHV2Y0ujy0y"
   },
   "outputs": [],
   "source": [
    "pc.delete_index(name=index_name)"
   ]
  }
 ],
 "metadata": {
  "colab": {
   "provenance": []
  },
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.12.9"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}