docs/read-units-demonstrated.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "m-mQMwATVolD"
   },
   "source": [
    "[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/pinecone-io/examples/blob/master/docs/read-units-demonstrated.ipynb) [![Open nbviewer](https://raw.githubusercontent.com/pinecone-io/examples/master/assets/nbviewer-shield.svg)](https://nbviewer.org/github/pinecone-io/examples/blob/master/docs/read-units-demonstrated.ipynb)\n",
    "\n",
    "# Demonstrating the retrieval of new `Read Units` (RUs)\n",
    "\n",
    "[Pinecone serverless](https://www.pinecone.io/blog/serverless/) (`pinecone-client` >= 3.0.0) has an entirely new infrastructure. Included in this major change is a [novel pricing structure](https://docs.pinecone.io/docs/understanding-cost), based on the [serverless model](https://www.pinecone.io/blog/serverless-architecture/). Now, reads and writes have separate cost structures.\n",
    "\n",
    "This notebook will take users through building a Pinecone serverless index, populating that index, and retrieving the related Read Units (RUs) associated with different types of queries.\n",
    "\n",
    "## Install the latest Pinecone client\n",
    "\n",
    "Read about the [newest Pinecone client](https://docs.pinecone.io/docs/new-api).\n",
    "\n",
    "Specifically, we'll be installing the [gRPC version](https://docs.pinecone.io/docs/upsert-data#grpc-python-client) of the newest Pinecone client to maximize performance of upserts and other data operations."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "e_WhqC0fVuSj",
    "outputId": "3423fdc4-a3bd-4b41-b9cd-f1a6992e0455"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Requirement already satisfied: pinecone-client==3.0.0 in /Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/site-packages (from pinecone-client[grpc]==3.0.0) (3.0.0)\n",
      "Requirement already satisfied: certifi>=2019.11.17 in /Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/site-packages (from pinecone-client==3.0.0->pinecone-client[grpc]==3.0.0) (2023.7.22)\n",
      "Requirement already satisfied: tqdm>=4.64.1 in /Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/site-packages (from pinecone-client==3.0.0->pinecone-client[grpc]==3.0.0) (4.66.1)\n",
      "Requirement already satisfied: typing-extensions>=3.7.4 in /Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/site-packages (from pinecone-client==3.0.0->pinecone-client[grpc]==3.0.0) (4.7.1)\n",
      "Requirement already satisfied: urllib3>=1.26.0 in /Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/site-packages (from pinecone-client==3.0.0->pinecone-client[grpc]==3.0.0) (1.26.16)\n",
      "Requirement already satisfied: googleapis-common-protos>=1.53.0 in /Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/site-packages (from pinecone-client[grpc]==3.0.0) (1.61.0)\n",
      "Requirement already satisfied: grpc-gateway-protoc-gen-openapiv2==0.1.0 in /Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/site-packages (from pinecone-client[grpc]==3.0.0) (0.1.0)\n",
      "Requirement already satisfied: grpcio>=1.44.0 in /Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/site-packages (from pinecone-client[grpc]==3.0.0) (1.57.0)\n",
      "Requirement already satisfied: lz4>=3.1.3 in /Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/site-packages (from pinecone-client[grpc]==3.0.0) (4.3.2)\n",
      "Requirement already satisfied: protobuf<3.21.0,>=3.20.0 in /Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/site-packages (from pinecone-client[grpc]==3.0.0) (3.20.3)\n"
     ]
    }
   ],
   "source": [
    "!pip install -qU \\\n",
    "    \"pinecone-client[grpc]==3.1.0\" \\\n",
    "    pinecone-notebooks==0.1.1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "tO2OebWez5La",
    "outputId": "5ef643ba-02a2-4b1c-fcc1-68aed979bfba"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Python 3.10.12\n"
     ]
    }
   ],
   "source": [
    "# Note version of Python this NB is tested with:\n",
    "\n",
    "!python --version"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "v6IAjLhOWRMw"
   },
   "source": [
    "## Connect to Pinecone and create an index"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import os\n",
    "\n",
    "# initialize connection to pinecone (orget API key at app.pinecone.io)\n",
    "if not os.environ.get(\"PINECONE_API_KEY\"):\n",
    "    from pinecone_notebooks.colab import Authenticate\n",
    "    Authenticate()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "PQFSLS6NVBLb",
    "outputId": "302580e4-2586-47f4-c8dc-f4d9b8aa9bdb"
   },
   "outputs": [],
   "source": [
    "from pinecone.grpc import PineconeGRPC\n",
    "\n",
    "api_key = os.environ.get(\"PINECONE_API_KEY\")\n",
    "\n",
    "# configure GRPC client\n",
    "pc = PineconeGRPC(api_key=api_key)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we setup our index specification, this allows us to define the cloud provider and region where we want to deploy our index. You can find a list of all [available providers and regions here](https://docs.pinecone.io/docs/projects)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from pinecone import ServerlessSpec\n",
    "\n",
    "cloud = os.environ.get('PINECONE_CLOUD') or 'aws'\n",
    "region = os.environ.get('PINECONE_REGION') or 'us-east-1'\n",
    "\n",
    "spec = ServerlessSpec(cloud=cloud, region=region)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "vN5d_io6eVGo",
    "outputId": "379012aa-bc88-4818-9168-a49505bb8467"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Index name:  rus-demo\n"
     ]
    }
   ],
   "source": [
    "# Input the name of the index you want to create\n",
    "index_name = \"read-units-demonstrated\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "lkJaCUdIURne",
    "outputId": "f4fdf62e-2563-4828-88da-7ffe733dff38"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Creating index \"rus-demo\"...\n",
      "\n",
      "Successfully created index \"rus-demo\"!\n"
     ]
    }
   ],
   "source": [
    "import time\n",
    "\n",
    "if index_name not in pc.list_indexes().names():\n",
    "    print(f'Creating index \\\"{index_name}\\\"...\\n')\n",
    "    pc.create_index(\n",
    "        name=index_name,\n",
    "        dimension=1536,\n",
    "        metric=\"cosine\",\n",
    "        spec=spec\n",
    "    )\n",
    "    if index_name in pc.list_indexes().names():\n",
    "        print(f'Successfully created index \\\"{index_name}\\\"!')\n",
    "else:\n",
    "    print(f'An index named \\\"{index_name}\\\" already exists!')\n",
    "\n",
    "# wait for index to be ready before connecting\n",
    "while not pc.describe_index(index_name).status['ready']:\n",
    "    time.sleep(1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "id": "dUFFg6wcspKh"
   },
   "outputs": [],
   "source": [
    "# Instantiate index object with your index_name\n",
    "\n",
    "index = pc.Index(index_name)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "JEfkJ_DE5_Ri",
    "outputId": "8d7bf5c6-11b6-4ea6-8ef1-8aa7e23ebbdc"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'dimension': 1536,\n",
       " 'index_fullness': 0.0,\n",
       " 'namespaces': {'': {'vector_count': 0}},\n",
       " 'total_vector_count': 0}"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "index.describe_index_stats()  # Great, we have created our index, and it currently have no vectors in it; we have no namespaces yet"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "5tx6yDfYWiNc"
   },
   "source": [
    "## [Skip this section if your index exists already]\n",
    "\n",
    "## Batch [upsert](https://docs.pinecone.io/docs/upsert-data) vectors into different namespaces\n",
    "\n",
    "We'll create and populate three [namespaces](https://docs.pinecone.io/docs/namespaces) with 50k, 100k, and 200k vectors, respectively. Namespaces are optional, but they are a best practice for limiting queries to relevant records, which both speeds up queries and reduces the RUs consumed."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "TJvERgi4YkW2",
    "outputId": "1b51a9bb-ef99-43d5-de1b-32afdfac2358"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Populating namespace \"50k\":\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████████████████████████████████████████████████████████████████████████████| 5/5 [03:34<00:00, 42.93s/it]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Populating namespace \"100k\":\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      " 60%|████████████████████████████████████████████████▌                                | 6/10 [04:16<02:50, 42.63s/it]"
     ]
    }
   ],
   "source": [
    "# Takes ~2 mins to run on Google colab; longer if running locally\n",
    "\n",
    "import uuid\n",
    "import numpy as np\n",
    "from tqdm import tqdm\n",
    "\n",
    "# Define sizes of namespaces as a list of tuples. Each tuple contains a name and a size.\n",
    "NAMESPACE_SIZES = [('50k', 50_000), ('100k', 100_000), ('200k', 200_000)]\n",
    "BATCH_SIZE = 100  # Number of items in each batch\n",
    "BATCHES = 100  # Number of batches\n",
    "DIMS = 1536  # Assuming a dimension for the random values\n",
    "\n",
    "# Loop through each namespace and its corresponding size\n",
    "for namespace, size in NAMESPACE_SIZES:\n",
    "    print(f'Populating namespace \"{namespace}\":')\n",
    "\n",
    "    # Calculate the total number of iterations needed\n",
    "    total_iterations = size // (BATCH_SIZE * BATCHES)\n",
    "\n",
    "    # Process data in batches using tqdm for progress indication\n",
    "    for _ in tqdm(range(total_iterations)):\n",
    "\n",
    "        # Initialize an empty list to store all batches\n",
    "        all_batches = []\n",
    "\n",
    "        # Outer loop to create multiple batches\n",
    "        for _ in range(BATCHES):\n",
    "\n",
    "            # Initialize an empty list for a single batch\n",
    "            single_batch = []\n",
    "\n",
    "            # Inner loop to create each item in the batch\n",
    "            for _ in range(BATCH_SIZE):\n",
    "\n",
    "                # Generate a random vector value using numpy\n",
    "                random_vector_value = np.random.rand(DIMS)\n",
    "\n",
    "                # Create a tuple with a unique ID, the random vector value, and some toy metadata\n",
    "                item = (str(uuid.uuid4()), random_vector_value, {'metadata': 'some toy metadata'})\n",
    "\n",
    "                # Add the created item to the single batch\n",
    "                single_batch.append(item)\n",
    "\n",
    "            # Add the single batch to the list of all batches\n",
    "            all_batches.append(single_batch)\n",
    "\n",
    "        # Upsert (update/insert) each batch asynchronously and collect future objects\n",
    "        async_results = []\n",
    "        for batch in all_batches:\n",
    "            async_result = index.upsert(vectors=batch, async_req=True, namespace=namespace)\n",
    "            async_results.append(async_result)\n",
    "\n",
    "        # Wait for all asynchronous operations to complete\n",
    "        responses = [async_result.result() for async_result in async_results]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "ce77Hjalt8cM"
   },
   "source": [
    "3. Validate everything looks as expected."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "5kL1lcNQrTys",
    "outputId": "2d0233f6-9873-4146-9e32-951a110d50f5"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'dimension': 1536,\n",
       " 'index_fullness': 0.0,\n",
       " 'namespaces': {'100k': {'vector_count': 100000},\n",
       "                '200k': {'vector_count': 200000},\n",
       "                '50k': {'vector_count': 50000}},\n",
       " 'total_vector_count': 350000}"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Now we can see our namespaces and how many vectors they hold (\"50k\" has 50k, \"100k\" has 100k, and \"200k\" has 200k):\n",
    "index.describe_index_stats()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "U5ZvPERHYf1g"
   },
   "source": [
    "## Inspect Read Costs\n",
    "\n",
    "We'll now execute a simple query on the first namespace (`'50k'`) and inspect its response.\n",
    "\n",
    "You should see `'usage': {'read_units': 5}` at the way bottom. Those are our `'RUs'`!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "metadata": {
    "id": "uHKWL5AUVkFD"
   },
   "outputs": [],
   "source": [
    "# Create a dummy query to pass to our vector DB\n",
    "\n",
    "dummy_vectorized_query = [0] * 1536  # Must have same dimensions as our indexed vectors (1536)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "4WhfQPXHHyTg",
    "outputId": "3982b1eb-4668-46b1-96a9-3833cbf6d07e"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'matches': [{'id': '7ba8b4aa-f883-4c4e-b0b9-b7fc43179750',\n",
       "              'metadata': None,\n",
       "              'score': 0.0,\n",
       "              'sparse_values': {'indices': [], 'values': []},\n",
       "              'values': []},\n",
       "             {'id': 'e83c6d3e-4706-4ada-acbe-e825d11bcdf4',\n",
       "              'metadata': None,\n",
       "              'score': 0.0,\n",
       "              'sparse_values': {'indices': [], 'values': []},\n",
       "              'values': []},\n",
       "             {'id': 'd334f5f6-e62d-47ac-ad7b-3124dcc4cb4d',\n",
       "              'metadata': None,\n",
       "              'score': 0.0,\n",
       "              'sparse_values': {'indices': [], 'values': []},\n",
       "              'values': []},\n",
       "             {'id': '303aa521-b7b0-45d8-9cef-8ab3c8c23f9f',\n",
       "              'metadata': None,\n",
       "              'score': 0.0,\n",
       "              'sparse_values': {'indices': [], 'values': []},\n",
       "              'values': []},\n",
       "             {'id': '9fe84421-3ee1-46f5-9af6-a158983791f5',\n",
       "              'metadata': None,\n",
       "              'score': 0.0,\n",
       "              'sparse_values': {'indices': [], 'values': []},\n",
       "              'values': []},\n",
       "             {'id': '0ec0005e-32a0-431a-b0d9-b31fd7d02e7c',\n",
       "              'metadata': None,\n",
       "              'score': 0.0,\n",
       "              'sparse_values': {'indices': [], 'values': []},\n",
       "              'values': []},\n",
       "             {'id': '729eda28-9032-4b7b-b3b3-d3ff0efb603b',\n",
       "              'metadata': None,\n",
       "              'score': 0.0,\n",
       "              'sparse_values': {'indices': [], 'values': []},\n",
       "              'values': []},\n",
       "             {'id': 'f44c5eda-a92c-42dc-8461-aff5bd1cb968',\n",
       "              'metadata': None,\n",
       "              'score': 0.0,\n",
       "              'sparse_values': {'indices': [], 'values': []},\n",
       "              'values': []},\n",
       "             {'id': '6e21d46d-895e-45e2-b4b6-f69e74eac19a',\n",
       "              'metadata': None,\n",
       "              'score': 0.0,\n",
       "              'sparse_values': {'indices': [], 'values': []},\n",
       "              'values': []},\n",
       "             {'id': '522fa5ed-f918-4fe4-b720-cd279134adbd',\n",
       "              'metadata': None,\n",
       "              'score': 0.0,\n",
       "              'sparse_values': {'indices': [], 'values': []},\n",
       "              'values': []}],\n",
       " 'namespace': '50k',\n",
       " 'usage': {'read_units': 5}}"
      ]
     },
     "execution_count": 50,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Issue the query and get response\n",
    "\n",
    "index.query(vector=dummy_vectorized_query, top_k=10, namespace='50k')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "iPoPALAmwXHW"
   },
   "source": [
    "Since every query consumes read units, every query's response will have a `usage` field. This `usage` field contains the exact number of `RUs` your query incurred.\n",
    "\n",
    "We can drill down to _only_ our query's corresponding cost in `RUs` by doing the following:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "CoAF1TyQwWpo",
    "outputId": "beaaed48-dfdb-45cb-a9fc-db952ef4fc1a"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "5"
      ]
     },
     "execution_count": 51,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Grab only RUs:\n",
    "index.query(vector=dummy_vectorized_query, top_k=10, namespace='50k')['usage']['read_units']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "yD-7MihWYwsg"
   },
   "source": [
    "Querying the `\"50k\"` namespace consumed `5` `RUs`, which is the minimum value a query can use.\n",
    "\n",
    "Let's query the `\"100k\"` namespace to see how the result changes:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 53,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "VpEVPPRFYwUJ",
    "outputId": "11ba439d-ce0f-4459-aa53-db18f6987cb8"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "6"
      ]
     },
     "execution_count": 53,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "index.query(vector=dummy_vectorized_query, top_k=10, namespace='100k')['usage']['read_units']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "4D68y5fyZXlA"
   },
   "source": [
    "When we queried the `\"50k\"` namespace, we consumed `5` `RUs`. When we now query a namespace that has `2x` the vectors (the `\"100k\"` namespace), we see that we only consumed `1` extra `RU`.\n",
    "\n",
    "\n",
    "Let's see what happens when we `2x` the size again, querying the `\"200k\"` namesapce:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 55,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "ABkKs2LLY2bp",
    "outputId": "edf1b2dc-a4e6-45d1-8cf9-a2902cdca061"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "8"
      ]
     },
     "execution_count": 55,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "index.query(vector=dummy_vectorized_query, top_k=10, namespace='200k')['usage']['read_units']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "yLmUofY8ZhJO"
   },
   "source": [
    "When we query the `\"200k\"` namespace, our `RU` cost goes from `6` to `8`. Note that this is sub-linear scaling in action!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "tOlTSCrfywaD"
   },
   "source": [
    "### Toggling `top_k`\n",
    "\n",
    "Now let's stay querying the \"200k\" namespace, but increase our `top_k` value from `10` to `100` to see its effect:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 56,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "wblZKa_YZgJO",
    "outputId": "313d8d81-312d-4c91-ceb1-832b777a8d3c"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "8"
      ]
     },
     "execution_count": 56,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "index.query(vector=dummy_vectorized_query, top_k=100, namespace='200k')['usage']['read_units']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "Jb7cFasIyLa3"
   },
   "source": [
    "Increasing our `top_k` from `10` to `100` in the `\"200k\"` namespace has _not_ changed the number of `RUs` incurred.\n",
    "\n",
    "This is because Pinecone's initial scan of the `\"200k\"` namespace was enough to produce the IDs of the `top_k` results for both `10` and `100`.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "vPDDVfDSy-lC"
   },
   "source": [
    "### Toggling `include_metadata`\n",
    "\n",
    "\n",
    "But what if we set `include_metadata` to `True`? This _should_ trigger a \"post-scan\" `Fetch` stage with an additional cost of `1` `RU` per `10` items in our result set:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 57,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "P51LoE1syJXm",
    "outputId": "090a1b2a-f8cb-4afc-c298-a11ebf0d14dd"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "18"
      ]
     },
     "execution_count": 57,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "index.query(vector=dummy_vectorized_query, top_k=100, namespace='200k', include_metadata=True)['usage']['read_units']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "g-m5xD5CyUcT"
   },
   "source": [
    "Looks like that worked! By including metadata in our query's response (`include_metadata=True`), we went from a cost of `8` `RUs` to a cost of `18` `RUs`, because we added `1` `RU` per `10` items returned (and we returned `100` items by setting our `top_k` to `100`). Our original cost of `8` `RUs` plus our new overhead of `10` `RUs` (`1` additional `RU` for every `10` items out of our total of `100` items), equals `18` `RUs`.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "YXd-kHzdzFJ4"
   },
   "source": [
    "### Putting it all together\n",
    "\n",
    "Now let's increase the `top_k` even more to see how it affects the RU cost:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 58,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "1ROn-JJayT0a",
    "outputId": "78c0475f-1a0c-4ea6-b6e4-ac722cb255ec"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "108"
      ]
     },
     "execution_count": 58,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "index.query(vector=dummy_vectorized_query, top_k=1000, namespace='200k', include_metadata=True)['usage']['read_units']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "rsX5B1XLbd5-"
   },
   "source": [
    "By increasing our `top_k` from `100` to `1000`, and continuing to include metadata in our response, we are now at a cost of `108` `RUs`.\n",
    "\n",
    "\n",
    "---\n",
    "\n",
    "\n",
    "\n",
    "[Play with the cost of your queries](https://docs.pinecone.io/docs/managing-cost) on your own and [let us know](https://community.pinecone.io/) what you find!"
   ]
  }
 ],
 "metadata": {
  "colab": {
   "provenance": []
  },
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}