QA System 1.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Problem: Answer questions about my knowledge base\n",
    "\n",
    "Over the last three years, I've been curating a personal knowledge bases of my notes in Obsidian, academic assignments in PDFs, and the content from textbooks. Considering the HC #scienceoflearning , I should have used spaced repetition to cement crucial knowledge. Admittedly, I usually only review my notes when forgetfulness knocks. Further, navigating my evolving and extensive knowledge database is becoming more of a challenge.\n",
    "\n",
    "In contrast, the LLMs, especially ChatGPT, have consistently exhibited agility in responding to inquiries. However, their limitations surface when confronted with intricate domains like mathematics, compelling me to seek supplementary information from alternative sources, a gap that my meticulously maintained notes aim to fill.\n",
    "\n",
    "Recognizing the ability to understand language of LLMs (e.g. quickly extracting information and and summarizing text) and the wealth and details of my own knowledge base, I want to combine their strengths. In this assignment, I am building a question answering system that can answer questions about my knowledge base. In the next assignment, I will improve upon this system.\n",
    "\n",
    "By leveraging the language understanding prowess of LLMs and the detailed insights within my knowledge base, this question-answering system is poised to offer precise and contextually rich responses, without me going over many notes to find the answer."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Data collection\n",
    "\n",
    "To focus on technical details of the models, I will use a subset of my knowledge base, which is a collection of my notes (Markdown files) from the last three years. I collected the assignments and textbooks (PDFs) but will not use them in this assignment. \n",
    "\n",
    "The dataset is loaded and counted below."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Assignments & Textbooks: 58\n",
      "Notes: 982\n"
     ]
    }
   ],
   "source": [
    "from pathlib import Path\n",
    "\n",
    "DATA_DIR = Path(\"./data/raw\")\n",
    "\n",
    "# Count the number of .pdf and .md files\n",
    "pdf_count = sum(1 for _ in DATA_DIR.rglob(\"*.pdf\"))\n",
    "md_count = sum(1 for _ in DATA_DIR.rglob(\"*.md\"))\n",
    "\n",
    "print(f\"Assignments & Textbooks: {pdf_count}\\nNotes: {md_count}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Data Ingestion & Preprocessing\n",
    "In this section, I perform two main steps: 1) ingest and clean markdown files, and 2) load them inside LlamaIndex.\n",
    "\n",
    "1. Ingest and clean markdown files: I modified and modularized the code from the previous assignment, which is imported as `Markdown2Text` class. The code strips off markdown elements and native Obsidian elements, and returns the cleaned text. \n",
    "2. Load them inside LlamaIndex: LlamaIndex is a framework aimed at helping build applications with large language models (LLMs) by providing tools for data ingestion, structuring, and many more with various application frameworks. Loading my data as the `Document` object is the first step to use LlamaIndex later on. Each `Document` object contains the cleaned text and metadata (file name, domain)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Filled Notes: 957\n",
      "Preview a document: proust effect\n",
      "Doc ID: e3716fc5-6e85-43b1-ba0a-b7a892bcc7cd\n",
      "Text: Past memories are triggered through sensory stimuli, especially\n",
      "scents Humans have around 1000 different genes that encode distinct\n",
      "scent receptors (as a comparison, vision has merely four distinct\n",
      "receptors), which enable us to recognize as many as 10,000 different\n",
      "scent combinations. We can recognize scents previously encountered\n",
      "even after ve...\n"
     ]
    }
   ],
   "source": [
    "from typing import List\n",
    "\n",
    "from llama_index.schema import Document\n",
    "\n",
    "# Modify the Markdown2Text classes from the previous pipeline\n",
    "from src.processing.strip_markdown import Markdown2Text\n",
    "\n",
    "# Load the .md files\n",
    "md_loader = Markdown2Text(\n",
    "    markdown_dir=DATA_DIR,\n",
    ")\n",
    "md_docs: List[Document] = md_loader.load_markdown_dir(return_documents=True)\n",
    "\n",
    "# validation checks\n",
    "assert (len(md_docs) <= md_count) and (len(md_docs) > 0)\n",
    "assert all([isinstance(doc, Document) for doc in md_docs])\n",
    "assert all([hasattr(doc, \"text\") for doc in md_docs])\n",
    "assert all([hasattr(doc, \"metadata\") for doc in md_docs])\n",
    "assert all([doc.text for doc in md_docs])\n",
    "\n",
    "# print the first document\n",
    "print(f\"Filled Notes: {len(md_docs)}\")\n",
    "print(f\"Preview a document: {md_docs[0].metadata['file_name']}\")\n",
    "print(md_docs[0])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Approach: Retrieval-based Question Answering System\n",
    "\n",
    "Retrieval augmented generation (RAG) is a powerful AI framework that enhances the quality of responses generated by large language models (LLMs) by grounding the model on external knowledge. This framework is shown to be particularly useful for question answering systems as it combines the strengths of a knowledge base with those of an LLM to\n",
    "\n",
    "- produce more accurate and contextually relevant responses, especially for questions outside the LLM's trained domains\n",
    "- address the limitations of static training data and enable the model to generate more reliable and up-to-date responses.\n",
    "\n",
    "![](https://miro.medium.com/v2/resize:fit:1400/1*LNVVvuZPLgdOWakC0hQcig.png)\n",
    "\n",
    "At a high level, RAG works by retrieving documents from an external dataset, which serve as the context for answering a given question. The framework consists of two main components\n",
    "\n",
    "- Retriever: responsible for finding and fetching the most relevant information to the question, to feed to the LLM for generating the response\n",
    "- Generator (transformer-based LLM): responsible for generating the response based on the original question, the retrieved information (additional context), and the instructions from prompt.\n",
    "\n",
    "Without retriever, the process is back to the vanilla: the LLM generates the response based on the prompt from the user only.\n",
    "\n",
    "In this section, I will attempt to explain these two components from the ground up, assuming the knowledge of Linear Algebra (vectors), and RNNs."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## RAG over Fine-tuning\n",
    "\n",
    "Before diving into the details of RAG, it is important to understand the difference between fine-tuning and RAG.\n",
    "\n",
    "- Fine-tuning allows the model to learn domain-specific knowledge and adjust its behavior, writing style, or responses to specific nuances, tones, or terminologies. It is effective for tasks requiring strong domain affinity. However, exclusive fine-tuning may not be practical for applications that require responses to be synced with a dynamic pool of information, as the model will need regular retraining for evolving data\n",
    "- RAG is proficient in data retrieval and can provide access to dynamic external data sources. It is ideal for applications that heavily rely on external data sources, as it continuously queries external sources, ensuring that the information remains up-to-date without frequent model retraining. \n",
    "\n",
    "Therefore, while both of these frameworks are useful, RAG is more suitable for my application considering that my knowledge base is dynamic and constantly evolving. Further, a big constraint is that I do not have the datasets and computational resources to fine-tune a large language model.\n",
    "\n",
    "However, RAG and fine-tuning are not mutually exclusive. Fine-tuning a RAG model can further improve the quality of responses, especially for domain-specific questions. I will explore that option in the next assignment."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Retriever: LlamaIndex's embedding-based retriever"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Retriever Building Block 1: Embedding & Vector database\n",
    "\n",
    "Unlike humans, machine learning models, including LLMs, do not have the ability to read and understand natural language text. Instead, they require numerical inputs to operate. To convert text into numbers, we use embedding to map smaller chunks of text (e.g. words, sentences, paragraphs) into dense, high-dimensional vectors of real numbers. Embedding is useful because it preserves the semantic relationships between words, which is crucial for the LLM to understand the context of the text.\n",
    "\n",
    "Embedding is a mapping from a discrete space (e.g. words, sentences) to a continuous space (e.g. vectors), learned after training on a large corpus of text. The embedding space is a vector space, where each dimension represents a feature of the word. For example, the word \"spider\" can be represented as a vector of 2 dimensions (-1.53, 0.41) below, where each dimension represents a feature of the word \"spider\" (e.g. negative/positive, concrete/abstract). Notice how the embedding of the word \"spider\" is similar to that of the word \"snake\", but different from that of the word \"puppy\" (e.g. the words \"spider\" and \"snake\" are similar in meaning and usually more negative, while the word \"puppy\" is usually more positive). In general, the distance between vectors corresponds to the similarity between the corresponding text data. \n",
    "\n",
    "![](https://i.imgur.com/U3cJH4S.png)\n",
    "Figure 2. Embedding of the words \"spider\", \"snake\", and \"puppy\" in a 2-dimensional space. Source: [Natural Language Processing Specialization Course 2](https://www.tensorflow.org/tutorials/text/word_embeddings)\n",
    "\n",
    "The resulting word or sentence vectors are then concatenated or averaged to produce a single vector representation of the entire text data.\n",
    "\n",
    "In the context of RAG, embedding is used to convert both the source data and the user queries into numerical representations that can be compared and matched by the retrieval model. The retrieval model then returns the most relevant documents to the user query. Usually to conserve more information, the sentence embedding is used. \n",
    "\n",
    "And the vector database is a collection of sentence vectors from the source data."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Retriever Building Block 2: Context Window in Dense Passage Retrieval\n",
    "\n",
    "Embedding, which is typically used for smaller text chunks, may not provide sufficient context for the LLM. Instead of embedding just one sentence at a time to the vector base, we can embed a window of text around sentence (e.g. three sentences including the main one). \n",
    "\n",
    "![](https://pbs.twimg.com/media/F7SO4PCa0AAQKKr?format=jpg&name=4096x4096)\n",
    "Figure 3. Although there might be only one relevant sentence, the context window (three sentences) provides more information to the LLM. Source: [Jerry Liu](https://pbs.twimg.com/media/F7SO4PCa0AAQKKr?format=jpg&name=4096x4096)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Retriever Building Block 3: Similarity search  \n",
    "\n",
    "As mentioned above, the retrieval model returns the most relevant documents to the user query. To do so, it compares the user query (a given embedding vector) with the vector database and searchs for the most similar documents (embedding vectors). The similarity between two vectors $\\vec{u}$ and $\\vec{v}$ can be measured by dot product\n",
    "\n",
    "$$\\vec{u} \\cdot \\vec{v} = \\vec{u}^ {\\top}\\times \\vec{v}$$\n",
    "\n",
    "or cosine similarity (more commonly used). Cosine similarity is the dot product of the vectors over the product of vector magnitudes. \n",
    "\n",
    "$$\\text{sim}(\\vec{u},\\vec{v}) = \\cos(\\theta) = \\frac{\\vec{u} \\cdot \\vec{v}}{\\|\\vec{u}\\|\\|\\vec{v}\\|}$$\n",
    "\n",
    "It has values between -1 and 1, where 1 means the vectors are identical (best), 0 means the vectors are orthogonal, and -1 means the vectors are opposite (worst)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Retrival Advanced 1: Embedding-based Reranking\n",
    "\n",
    "While retrieved documents are usually relevant to the user query, they may not ordered by the most relevant. To improve the performance, we introduce reranking that is trained to rank the documents based on their relevance to the question, ensuring that the most relevant information is presented to the LLM for response generation. The main steps are as follows:\n",
    "\n",
    "1. Train the reranking model on a dataset of question-document pairs (where the question represents the user query and the document is the retrieved document) and learns to assess the relevance of documents to specific queries. This training enables the model to discern and prioritize the most relevant documents for a given query.\n",
    "2. Once the initial retrieval phase is complete, the reranking model scores the retrieved documents based on their relevance to the user query. Subsequently, it reorders the documents, promoting the most relevant ones to higher positions in the ranking.\n",
    "3. Reintegrate the reranking model into the retrieval model, which now returns the most relevant documents to the user query, ordered by relevance."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Retrival last note: LLM-based reranking\n",
    "\n",
    "Above, we have discussed the retrieval model that uses embedding to convert text into dense vectors and then compares the user query with the vector database to find the most similar documents. However, we can also use LLMs for reranking to return documents that are more relevant (by human standard) than embedding-based method, with the tradeoff being much higher latency and cost.\n",
    "\n",
    "![](https://miro.medium.com/v2/resize:fit:1400/format:webp/1*xGWz2V-s5sQ6fWDM05auzA.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Generator: GPT-3.5"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Generator Building Block 1: Transformer basics\n",
    "\n",
    "As we learned in class, Recurrent Neural Networks (RNNs) and Long Short-Term Memory (LSTM) have some limitations. They are sequential models, which means they process input data one element at a time, making them slow and computationally expensive. Additionally, they struggle to handle long-range dependencies (i.e. capture the relationship between words/phrases/sentences far apart), which are common in natural language processing tasks.\n",
    "\n",
    "Unlike RNNs and LSTMs, the Transformer model ([Vaswani, et al., 2017](https://arxiv.org/abs/1706.03762)) uses self-attention to process input data in parallel, making it faster and more efficient. Self-attention allows the model to focus on different parts of the input sequence and capture long-range dependencies more effectively. The Transformer model also uses positional encoding to preserve the order of the input sequence. \n",
    "\n",
    "At a high-level, Transformers uses the encoder-decoder architecture, where the encoder processes the input sequence and the decoder generates the output sequence. Later simplified Transformer was shown to achieve great performance in language modeling tasks, like in encoder-only BERT or decoder-only GPT.\n",
    "\n",
    "![](https://vaclavkosar.com/images/transformer-full-model.png)\n",
    "\n",
    "\n",
    "Figure 4. Transformer model architecture. Source: [(Vaswani et al., 2017)](https://arxiv.org/pdf/1706.03762v5)\n",
    "\n",
    "### Generator Building Block 2: Encoder-Decoder Architecture Deep Dive\n",
    "#### Input Embedding & Output Embedding\n",
    "\n",
    "Each input sequence (e.g. sentences) is first converted into a sequence of vectors using an embedding layer. The embedding layer maps each word in the input sequence to a vector in the embedding space. The output of the embedding layer is then passed to the encoder.\n",
    "\n",
    "The Input Embedding $\\mathbf{X}\\in \\mathbb{R}^{L\\times d}$  has \n",
    "each element mapped into an embedding vector of shape $d$, same as the model size, and has another dimension of segment length of input sequence $L$.\n",
    "\n",
    "The Output Embedding $\\mathbf{Y}\\in \\mathbb{R}^{L\\times d}$ is obtained from the encoder component.\n",
    "\n",
    "#### Positional Encoding (Sinusoidal)\n",
    "\n",
    "The Transformer does not inherently capture the sequential order of input tokens like recurrent neural network, as it processes them in parallel. To incorporate positional information, positional encodings are added to the input embeddings. These positional encodings provide the model with information about the relative or absolute position of tokens in the input sequence.\n",
    "\n",
    "For a given position $\\text{pos}=1, \\ldots L$ and dimension $i = 1,\\ldots, d$, the positional encoding $PE(pos,i)$​ is computed using sine and cosine functions:\n",
    "\n",
    "$$\\begin{align*}\n",
    "\n",
    "\\text{PE}(\\text{pos}, 2i) &= \\sin\\left(\\frac{\\text{pos}}{10000^{2i/d}}\\right)\\\\\n",
    "\n",
    "\\text{PE}(\\text{pos}, 2i+1) &= \\cos\\left(\\frac{\\text{pos}}{10000^{2i/d}}\\right)\n",
    "\n",
    "\\end{align*}$$\n",
    "\n",
    "- $d$ = the dimensionality of the model \n",
    "- $L$ = the input sequence length\n",
    "\n",
    "This results in a positional encoding matrix $\\mathbf{P}\\in \\mathbb{R}^{L\\times d}$, where the $i$ row $\\mathbf{p}_{i}$ is the positional encoding for input element $\\mathbf{x}_{i}$.\n",
    "\n",
    "#### Multi-head Attention (Encoder)\n",
    "\n",
    "The encoder consists of multiple layers, each of which consists of a multi-head self-attention layer and a feedforward neural network. The multi-head self-attention layer allows the model to focus on different parts of the input sequence for each head. The attention mechanism calculates attention scores for each position based on its relation to other positions. It uses $h$ attention heads to capture diverse information.\n",
    "\n",
    "$$\\begin{align*}\n",
    "\n",
    "&\\text{MultiHead}(Q, K, V) = \\text{Concat}(\\text{head}_1, \\ldots, \\text{head}_h) \\cdot W_O \\\\ &\\text{head}_i = \\text{Attention}(Q \\cdot W_{i}^{Q}, K \\cdot W_{i}^{K}, V \\cdot W_{i}^{V})\n",
    "\n",
    "\\end{align*}$$\n",
    "\n",
    "- Q = the current position-word vector in the input sequence\n",
    "- K = all the position-word vectors in the input sequence\n",
    "- V = all the position-word vectors in the input sequence\n",
    "- $W_{Q{i}}, W_{Ki}, W_{Vi}, W_{O}$ are learned linear projection for the $i^{th}$ query, key, value, and output transformations, respectively.\n",
    "\n",
    "I will elaborate on the attention mechanism in the next section.\n",
    "\n",
    "#### Add & Norm\n",
    "\n",
    "This is a residual connection followed by layer normalization. The residual connection allows the model to learn identity functions, which can be help flow the information throughout the deep neural networks. Layer normalization normalizes the output of the previous layer by subtracting the mean and dividing by the standard deviation.\n",
    "\n",
    "For an input $X$ and a sub-layer output $\\text{SubLayer}(X)$, it is computed as follows:\n",
    "\n",
    "$$\\text{AddNorm}(X, \\text{SubLayer}(X)) = \\text{LayerNorm}(X + \\text{SubLayer}(X))$$\n",
    "\n",
    "#### Feedforward Neural Network\n",
    "\n",
    "The feed-forward layer consists of two linear transformations with a ReLU activation in between. It introduces non-linearity by applying a feedforward neural network to each position independently  \n",
    "\n",
    "For an input $X$, this layer is computed as follows:\n",
    "$$\\begin{align*}\n",
    "\n",
    "\\text{FFN}(X) &= \\text{ReLU}(X \\cdot W_{1} + b_{1}) \\cdot W_{2}+b_{2} \\\\\n",
    "\n",
    "&= \\max(0, X \\cdot W_1 + b_1) \\cdot W_2 + b_2\n",
    "\n",
    "\\end{align*}$$\n",
    "\n",
    "Here, $W_{1}, W_{2}, b_{1}, b_{2}$ are learnable parameters\n",
    "#### Masked Multi-head Attention (Decoder)\n",
    "\n",
    "This is similar to multi-head attention in the encoder, except that it uses a mask to prevent the decoder from looking at future positions in the sequence during training. It makes the decoder predict the next positions by previous positions learned from encoder only.\n",
    "\n",
    "#### Multi-head Attention (Decoder)\n",
    "\n",
    "The decoder uses multi-head attention, similar to the encoder, to focus on relevant parts within the decoder input (previous positions generated by the decoder). However, it also attends to the encoder's output to obtain information about the input sequence.\n",
    "\n",
    "#### Linear\n",
    "\n",
    "This is a linear transformation layer that projects the output of the decoder to the output vocabulary size \n",
    "\n",
    "$$\\text{Linear}(X) = X \\cdot W + b$$\n",
    "\n",
    "#### Softmax\n",
    "\n",
    "This softmax activation layer that converts the output of the linear layer into a probability distribution over the output vocabulary.\n",
    "\n",
    "$$\\text{Softmax}(X) = \\frac{\\exp(X)}{\\sum_{i=1}^{V} \\exp(X_i)}$$\n",
    "\n",
    "### Generator Building Block 3: Attention mechanism \n",
    "\n",
    "([Raschka, 2023](https://sebastianraschka.com/blog/2023/self-attention-from-scratch.html)) ([Weng, 2023](https://lilianweng.github.io/posts/2023-01-27-the-transformer-family-v2/))\n",
    "\n",
    "**Attention** is a mechanism in neural network that a model can learn to make predictions by selectively paying attention to a specific parts of the input. The amount of attention is quantified by learned weights  and thus the output is usually formed as a weighted average.\n",
    "\n",
    "#### Self-attention\n",
    "\n",
    "**Self-attention** is a type of attention mechanism where the model makes prediction for one part of a data sample using other parts of the observation about the same sample. \n",
    "\n",
    "Self-attention utilizes three weight matrices, query weight matrix $\\mathbf{W}^{q}\\in \\mathbb{R}^{d\\times d_{k}}$, key weight matrix $\\mathbf{W}^{k}\\in \\mathbb{R}^{d\\times d_{k}}$, and value weight matrix $\\mathbf{W}^{v}\\in \\mathbb{R}^{d\\times d_{v}}$ (usually $d_{k}=d$, the model size). These matrices are adjusted as model parameters during training, and used to project the inputs $\\mathbf{X} \\in \\mathbb{R}^{L\\times d}$ into query, key, value components of the sequence, respectively.\n",
    "\n",
    "- query matrix $\\mathbf{Q}\\in \\mathbb{R}^{L\\times d_{k}} = \\mathbf{X}\\mathbf{W}^{q}$ with query vector (column vector) $\\mathbf{q}_{i}\\in \\mathbb{R}^{L}$  \n",
    "- key matrix $\\mathbf{K}\\in \\mathbb{R}^{L\\times d_{k}} = \\mathbf{X}\\mathbf{W}^{k}$\n",
    "- value matrix $\\mathbf{V} \\in \\mathbb{R}^{L\\times d_v}= \\mathbf{X}\\mathbf{W}^{v}$\n",
    "\n",
    "where: \n",
    "\n",
    "- $L$ = The segment length of input sequence \n",
    "- $d_{k}= d$ = The model size, the embedding vector size\n",
    "- $d_{v}$ = The size of the value vector, which might be arbitrary\n",
    "- $d_{k}$ = Vector dimension of vectors \n",
    "- $i$ = index of the element/token in the input sequence\n",
    "\n",
    "#### Calculate unnormalized attention scores with dot product\n",
    "\n",
    "![](https://sebastianraschka.com/images/blog/2023/self-attention-from-scratch/omega.png)\n",
    "\n",
    "Once we obtain the vectors $\\mathbf{q}_{i}, \\mathbf{k}_{i}, \\mathbf{v}_{i}$ are used to compute *alignment weights*, which determine how much a particular element (position $i$) is similar to another element (position $j$) in the sequence, by computing the dot product between query and key sequences \n",
    "$$w_{ij}=\\mathbf{q}_{i} \\mathbf{k}_{j}^\\top$$\n",
    "\n",
    "#### Get Alignment score with scaled dot product and softmax\n",
    "\n",
    "![](https://sebastianraschka.com/images/blog/2023/self-attention-from-scratch/attention-scores.png)\n",
    "\n",
    "The embedding for an element is usually high-dimensional. To prevent the attention weights from becoming too small or too large, which could lead to numerical instability or affect the model’s ability to converge during training, we scale the *alignment weights* down by $\\frac{1}{\\sqrt{d_{k}}}$.   \n",
    "\n",
    "We also pass them through a softmax function, so that the resulting *alignment scores* $a_{ij}$ follow a probability distribution.\n",
    "\n",
    "$$a_{ij} = \\text{softmax}\\left(\\frac{\\mathbf{q}_{i} \\mathbf{k}_{j}^\\top}{\\sqrt{d_{k}}}\\right)$$\n",
    "\n",
    "Again, this score $a_{ij}$ quantifies a particular element (position $i$) is similar to another element (position $j$) in the sequence.\n",
    "\n",
    "#### Compute attention scores by multiplying with value Vector \n",
    "\n",
    "![](https://i.imgur.com/o19qALi.png)\n",
    "\n",
    "Until now, the calculated attention scores revolve around the current element $i$ and an arbitrary element $j$. To account for ALL other input elements $j$ around this element, we calculate the weighted sum of alignment scores $a_{ij}$ and multiplying with the value matrix $\\mathbf{V}$, to obtain the *context vector* $\\mathbf{z}_i$ around the current element $i$\n",
    "\n",
    "$$\\mathbf{z}_{i}=\\sum\\limits_{j=1}^{d}a_{ij} \\mathbf{v}_{j}$$\n",
    "\n",
    "#### Single head attention \n",
    "To generalize, the formula for self-attention mechanism is\n",
    "\n",
    "$$\\begin{align*}\n",
    "\\text{Attention}(\\mathbf{Q}, \\mathbf{K}, \\mathbf{V}) &= \\text{softmax}\\left(\\frac{\\mathbf{QK}^\\top}{\\sqrt{d_{k}}}\\right)\\cdot\\mathbf{V} \n",
    "\\end{align*}$$\n",
    "\n",
    "1. Create queries $\\mathbf{Q}$, keys $\\mathbf{K}$, and values $\\mathbf{V}$\n",
    "2. $\\frac{QK^{\\top}}{\\sqrt{d_{k}}}$: Compute [[scaled dot product]] of matrices $\\mathbf{Q}$ and $\\mathbf{K}$ to get the alignment weights. This encodes the similarity between the input and the target output \n",
    "3. $\\text{softmax}()$: Pass the alignment scores computed previously to a [[softmax]] function, so that they follow a [[probability distribution]]: every score is between 0 and 1, a valid [[probability]].\n",
    "4. $\\cdot \\mathbf{V}$: Matrix-multiply by the matrix $\\mathbf{V}$ (values) to calculate attention scores with context. Higher attention score assigned to a certain word means that it will have stronger influence on the next word in the decoder's output.\n",
    "\n",
    "#### Multi-head attention\n",
    "\n",
    "![](https://machinelearningmastery.com/wp-content/uploads/2022/03/dotproduct_1.png)\n",
    "\n",
    "Multi-head attention involves $h$ heads of self-attention mechanism described above, each consisting of query, key, and value matrices. The output of each head is concatenated and linearly transformed to obtain the final output."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Generator Advanced 1: From Transformer to GPT-3.5\n",
    "\n",
    "GPT-3.5 builds upon the Transformer model described above, by incorporating several improvements and innovations that lead to better performance and more accurate text generation, including:\n",
    "\n",
    "1. **Scale**: GPT-3.5 has 175 billion parameters, which is over 10 times the size of its predecessor, GPT-2, and more than 45 times the size of the original Transformer model. This increased size allows GPT-3.5 to capture more complex patterns and representations in the text data, enabling it to generate more accurate and contextually relevant responses.\n",
    "\n",
    "2. **Training Data**: GPT-3.5 has been trained on a massive amount of text data, including almost all of the public web, amounting to around 45 TB of text data. This extensive training enables GPT-3.5 to learn more advanced patterns and generate more human-like text.\n",
    "\n",
    "3. **Attention Mechanism**: While the original Transformer model uses self-attention, GPT-3.5 incorporates a more advanced attention mechanism that allows the model to selectively focus on segments of input text it predicts to be the most relevant. This improved attention mechanism contributes to the better performance of GPT-3.5 compared to the original Transformer model.\n",
    "\n",
    "4. **Decoder-only Architecture**: GPT-3.5, like its predecessors, uses a decoder-only architecture. This means that it generates text based on the input it receives, without using an encoder to process the entire input sequence. This approach simplifies the model and allows for better control over the generated text."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# RAG pipeline: Hybrid retrieval and reranking\n",
    "\n",
    "To summarize, the step-by-step process of RAG is as follows:\n",
    "\n",
    "1. Prepare the source data by converting it into a vector database of dense embeddings. These dense embeddings contain the semantic information of the sentences and context around them.\n",
    "2. Connect the vector database to the retrieval model, which uses the embeddings to find the most relevant documents to the user query.\n",
    "3. Connect the retrieval model to the reranking model, which reorders the retrieved documents based on their relevance to the user query.\n",
    "4. Connect the reranking model to the LLM, which generates the response based on the user query and the retrieved documents.\n",
    "5. Send the query through several steps of the pipeline, from embedding to retrieval-reranking to response generation, to produce the final response.\n",
    "6. Repeat the process (step 2-5) for each user query and evaluate the generated responses. Evaluations include comparing with the human answers (cosine similarity) and human qualitative assessment."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## LLM: GPT-3.5 through OpenAI API"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "import openai\n",
    "from llama_index.llms import OpenAI\n",
    "from llama_index.llms.llama_utils import completion_to_prompt, messages_to_prompt\n",
    "\n",
    "from src.utils.get_keys import get_openai_api_key\n",
    "\n",
    "openai.api_key = get_openai_api_key()\n",
    "\n",
    "llm = OpenAI(model=\"gpt-3.5-turbo\", temperature=0.1, max_tokens=500)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Sentence embedding: Turn sentences into dense, high-dimensional vectors of numbers"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "from llama_index.embeddings import HuggingFaceEmbedding\n",
    "\n",
    "embedding = HuggingFaceEmbedding(model_name=\"sentence-transformers/all-MiniLM-L12-v2\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Sentence Window Retrieval: Get context window around the sentences, parse to nodes and index to a vector database\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "from llama_index.node_parser import SentenceWindowNodeParser\n",
    "from llama_index import (\n",
    "    VectorStoreIndex,\n",
    "    StorageContext,\n",
    "    ServiceContext,\n",
    "    load_index_from_storage,\n",
    ")\n",
    "\n",
    "INDEX_DIR = Path(\"./data/final/sentence_index\")\n",
    "\n",
    "\n",
    "def build_sentence_window_index(\n",
    "    documents: List[Document],\n",
    "    window_size: int = 3,\n",
    "    llm=llm,\n",
    "    embed_model: HuggingFaceEmbedding = embedding,\n",
    "    index_dir: Path | str = INDEX_DIR,\n",
    ") -> VectorStoreIndex:\n",
    "    \"\"\"\n",
    "    Get the sentences and context around them from the list of documents and build a sentence index.\n",
    "\n",
    "    Args:\n",
    "        documents (List[Document]):\n",
    "            list of documents to extract sentences from\n",
    "        window_size (int, default = 3):\n",
    "            number of sentences to include around the target sentence,\n",
    "            defaults to 3, meaning 1 sentence before and 1 sentence after.\n",
    "        llm:\n",
    "            language model to use for sentence generation\n",
    "        embed_model (HuggingFaceEmbedding, default = sentence-transformers/all-mpnet-base-v2):\n",
    "            embedding model to use for sentence similarity\n",
    "        index_dir (Path | str):\n",
    "            directory to save the index to\n",
    "    \"\"\"\n",
    "\n",
    "    # create the sentence window node parser w/ default settings\n",
    "    sent_context_parser = SentenceWindowNodeParser.from_defaults(\n",
    "        window_size=window_size,\n",
    "        window_metadata_key=\"window\",\n",
    "        original_text_metadata_key=\"original_text\",\n",
    "    )\n",
    "\n",
    "    sentence_context = ServiceContext.from_defaults(\n",
    "        llm=llm,\n",
    "        embed_model=embed_model,\n",
    "        node_parser=sent_context_parser,\n",
    "    )\n",
    "\n",
    "    # if the index doesn't exist, build it\n",
    "    if not index_dir.exists():\n",
    "        sentence_index = VectorStoreIndex.from_documents(\n",
    "            documents, service_context=sentence_context\n",
    "        )\n",
    "\n",
    "        sentence_index.storage_context.persist(persist_dir=index_dir)\n",
    "    # otherwise, load it from storage\n",
    "    else:\n",
    "        sentence_index = load_index_from_storage(\n",
    "            StorageContext.from_defaults(persist_dir=index_dir),\n",
    "            service_context=sentence_context,\n",
    "        )\n",
    "    return sentence_index\n",
    "\n",
    "\n",
    "sentence_index = build_sentence_window_index(\n",
    "    md_docs, window_size=3, index_dir=INDEX_DIR\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Retrieval & Reranking: Find $k$ most relevant text chunks to the question, then rerank and narrow down to $n$ results"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this code cell, I do several steps:\n",
    "\n",
    "1. Set up the retriever and reranker to find seven relevant chunks, and rerank them to three most relevant chunks to the question.\n",
    "2. Set up the LLM to generate the response based on the question, the prompt, and the context of three relevant chunks. I allow the LLM to use its own knowledge to generate the response *only when* my knowledge base does not have the answer. However, it has to flag the response as \"generated\" so that I can review it later.\n",
    "3. Set up structured output, so that I can retrieve the response, the \"generated\" flag, and the source of the chunks."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {},
   "outputs": [],
   "source": [
    "import nest_asyncio\n",
    "\n",
    "from llama_index import ServiceContext, VectorStoreIndex, StorageContext\n",
    "from llama_index.indices.postprocessor import MetadataReplacementPostProcessor\n",
    "from llama_index.indices.postprocessor import SentenceTransformerRerank\n",
    "from llama_index.prompts import PromptTemplate\n",
    "from pydantic import BaseModel\n",
    "\n",
    "nest_asyncio.apply()\n",
    "\n",
    "text_qa_template = \"\"\"You are an AI assistant that answers questions in a friendly and truthful manner, based on the given the context information below and the format requested.\\n\n",
    "Context information is {context_str}\\n\n",
    "You always search for the answer to the question {query_str} in the given source documents first. When the context isn't helpful, you clearly state that \"The answer is not available in your knowledge base. I will use my own knowledge to answer the question,\" then you must answer the question using your knowledge.\\n\n",
    "\"\"\"\n",
    "\n",
    "\n",
    "class Answer(BaseModel):\n",
    "    \"\"\"\n",
    "    Answer object to format the response from the model\n",
    "    \"\"\"\n",
    "\n",
    "    file_names: List[str]\n",
    "    is_model_generated: bool\n",
    "    answer: str\n",
    "\n",
    "\n",
    "def get_sentence_window_query_engine(\n",
    "    sentence_index,\n",
    "    similarity_top_k=7,\n",
    "    rerank_top_n=3,\n",
    "    text_qa_template=text_qa_template,\n",
    "):\n",
    "    # define postprocessors\n",
    "    postproc = MetadataReplacementPostProcessor(target_metadata_key=\"window\")\n",
    "    rerank = SentenceTransformerRerank(\n",
    "        top_n=rerank_top_n, model=\"cross-encoder/ms-marco-MiniLM-L-12-v2\"\n",
    "    )\n",
    "\n",
    "    sentence_window_engine = sentence_index.as_query_engine(\n",
    "        output_cls=Answer,\n",
    "        response_mode=\"compact\",\n",
    "        text_qa_template=PromptTemplate(text_qa_template),\n",
    "        similarity_top_k=similarity_top_k,\n",
    "        node_postprocessors=[postproc, rerank],\n",
    "    )\n",
    "    return sentence_window_engine\n",
    "\n",
    "\n",
    "query_engine = get_sentence_window_query_engine(sentence_index=sentence_index)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Question & Answering\n",
    "\n",
    "I prepared a list of 11 questions to test the system. The questions are divided into two categories: \n",
    "1. Can be answered using the knowledge base only\n",
    "    1. Has answer found in < 5 documents AND contains keyword in the metadata (file name, domain)\n",
    "    2. Has answer found in < 5 documents AND does not contain keyword in the metadata (file name, domain)\n",
    "    3. Must summarize or infer across multiple documents\n",
    "2. Cannot be answered using the knowledge base only but can be answered using LLM. This checks if the LLM flags the response as \"no answer found in the knowledge base.\"\n",
    "\n",
    "We input the questions into LLM (along with the relevant exeternal information) and evaluate the responses later."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['From my notes, what did I learn in the business classes?',\n",
       " 'What is a probability?',\n",
       " 'In marketing 4P, what factors affect price?',\n",
       " 'Provide an overview of most important mechanisms for transformers.',\n",
       " 'How should I evaluate a synthetic control case study?',\n",
       " 'How to identify which critical point is local minimum or maximum?',\n",
       " 'How do I interpret regression coefficients?',\n",
       " 'Why do we need to change the basis?',\n",
       " 'What are some methods to measure how well a Bayesian model estimates the posterior?',\n",
       " 'What does microenomics concern?',\n",
       " 'Based on the NLP techniques I already learned, what should I learn next?']"
      ]
     },
     "execution_count": 44,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import json\n",
    "\n",
    "with open(\"./src/tests/eval_qa_pairs.json\", \"r\") as f:\n",
    "    eval_qa_pairs = json.load(f)[\"questions\"]\n",
    "\n",
    "# get the questions and answers\n",
    "questions = [pair[\"question\"] for pair in eval_qa_pairs]\n",
    "human_answers = [pair[\"answer\"] for pair in eval_qa_pairs]\n",
    "\n",
    "eval_results = {\n",
    "    \"questions\": questions,\n",
    "    \"human_answers\": human_answers,\n",
    "    \"model_answers\": [],\n",
    "    \"is_model_generated\": [],\n",
    "    \"contexts\": [],\n",
    "}\n",
    "eval_results[\"questions\"]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "metadata": {},
   "outputs": [],
   "source": [
    "import time\n",
    "\n",
    "\n",
    "def ask(question: str) -> str:\n",
    "    \"\"\"\n",
    "    Ask a question to the model\n",
    "    \"\"\"\n",
    "\n",
    "    response = query_engine.query(question)\n",
    "    return response.answer, response.is_model_generated, response.file_names\n",
    "\n",
    "\n",
    "# Set the delay between requests (in seconds)\n",
    "delay_between_requests = 8\n",
    "\n",
    "# Send the questions to the model\n",
    "for question in questions:\n",
    "    answer, is_model_generated, file_names = ask(question)\n",
    "    eval_results[\"model_answers\"].append(answer)\n",
    "    eval_results[\"is_model_generated\"].append(is_model_generated)\n",
    "    eval_results[\"contexts\"].append(file_names)\n",
    "    time.sleep(delay_between_requests)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Evaluation\n",
    "\n",
    "## Cosine similarity\n",
    "\n",
    "Here, we compare the LLM-generated answers with the human answers using cosine similarity introduced above. That means we have to convert both types of answers into embedding (using the same embedding used above for consistency)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAjcAAAGzCAYAAADT4Tb9AAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjguMCwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy81sbWrAAAACXBIWXMAAA9hAAAPYQGoP6dpAAA5yklEQVR4nO3dfVxUZf7/8feAMoAKaiIoIShqeJOiqERWammkadktaq6EN9mWppLrTTeSuUq7pUtftfxaqd3Yat7kbmVWknYn5QpamZKaoq4m3iUoghic3x/+mG/joM7I4MDx9Xw85hFzneuc8zkz2Lw557rOWAzDMAQAAGASXp4uAAAAwJ0INwAAwFQINwAAwFQINwAAwFQINwAAwFQINwAAwFQINwAAwFQINwAAwFQINwAAwFQIN0A5LBaLnnvuOU+XcVERERF6+OGH3brN84970aJFslgsysnJcet+unfvru7duzvV9+GHH1ZERIRb9++M0tJStW3bVtOnT7/i+8aVMWDAAD344IOeLgOVgHCDKu+XX37RyJEj1axZM/n6+iogIEBdu3bVyy+/rMLCQk+X53Y//vij7r//foWHh8vX11ehoaHq1auXZs+e7enSKs3Bgwf13HPPacuWLZ4uxeaf//yn9u/fr1GjRnm6FNM4c+aMJk6cqMaNG8vPz0+xsbH67LPPnFr3ueeek8VicXj4+vo69M3Ly9OECRPUokUL+fn5KTw8XMOGDdO+ffvs+k2cOFErVqzQ999/75bjQ9VRw9MFABfz0Ucf6YEHHpDVatWQIUPUtm1bFRcX6+uvv9Zf/vIX/fTTT5o/f77b91tYWKgaNa78P48NGzaoR48eatKkiUaMGKGQkBDt379f3377rV5++WWNHj3a1vfnn3+Wl5d7/z65Usf96aef2j0/ePCgpk6dqoiICEVHR9ste+2111RaWlrpNZ3vxRdf1IABAxQYGHjF921WDz/8sJYvX66xY8eqRYsWWrRokfr06aN169bppptucmobr776qmrXrm177u3tbbe8tLRUvXr10rZt2/TYY4+pZcuW2rVrl1555RV98skn2r59u+rUqSNJ6tChgzp16qSZM2fqrbfect+BwvMMoIravXu3Ubt2bSMqKso4ePCgw/KdO3caaWlpHqis8vTp08cICgoyfvvtN4dlubm5V7yehQsXGpKMPXv2uGV7BQUF5bb/5z//MSQZCxcudMt+KiorK8uQZKxdu9bTpVxQYWGhUVJS4ukynPbdd98ZkowXX3zR1lZYWGhERkYacXFxl1w/JSXFkGQcOXLkov2++eYbQ5IxZ84cu/YFCxYYkoyVK1fatb/00ktGrVq1jJMnT7pwNKjquCyFKuvvf/+7Tp06pTfeeEONGjVyWN68eXONGTPG9vz333/XtGnTFBkZKavVqoiICD311FM6c+aM3XqbNm1SfHy8GjRoID8/PzVt2lRDhw6163P+2JOyU+K7du3Sww8/rLp16yowMFBJSUk6ffq0Q23vvPOOYmJi5Ofnp/r162vAgAHav3//JY/5l19+UZs2bVS3bl2HZQ0bNrR7fv6Ym7LxMV9//bWeeOIJBQUFqW7duho5cqSKi4t14sQJDRkyRPXq1VO9evU0YcIEGYZx0eMuz7/+9S/deeedaty4saxWqyIjIzVt2jSVlJTY9evevbvatm2rzMxM3XLLLfL399dTTz1lW1Y25mb9+vXq3LmzJCkpKcl2uWHRokWSyh9zU1paqrS0NLVp00a+vr4KDg7WyJEj9dtvv9n1c+a9Ls+qVavk4+OjW265xa597969euyxx3TdddfJz89P11xzjR544AG7MUmbNm2SxWLRm2++6bDdTz75RBaLRR9++KGt7cCBAxo6dKiCg4NltVrVpk0bLViwwG699evXy2KxaMmSJXrmmWcUGhoqf39/5efn6/jx4xo/fryuv/561a5dWwEBAerdu3e5l1r27t2ru+66S7Vq1VLDhg01btw4W03r16+36/vdd9/pjjvuUGBgoPz9/dWtWzd98803DtvMzs52uNxTnuXLl8vb21uPPPKIrc3X11fDhg1TRkaGU/8+JMkwDOXn5zv87pbJz8+XJAUHB9u1l/0/xM/Pz669V69eKigocPryGKoHLkuhyvrggw/UrFkz3XjjjU71Hz58uN58803df//9evLJJ/Xdd98pNTVV27dv1/vvvy9JOnz4sG6//XYFBQVp0qRJqlu3rnJycrRy5Uqn9vHggw+qadOmSk1NVVZWll5//XU1bNhQf/vb32x9pk+frmeffVYPPvighg8friNHjmj27Nm65ZZbtHnz5nKDS5nw8HBlZGRo69atatu2rVM1nW/06NEKCQnR1KlT9e2332r+/PmqW7euNmzYoCZNmmjGjBlavXq1XnzxRbVt21ZDhgxxafuLFi1S7dq1lZycrNq1a+vzzz/XlClTlJ+frxdffNGu77Fjx9S7d28NGDBAgwcPdvjAkaRWrVrp+eef15QpU/TII4/o5ptvlqSLvu8jR47UokWLlJSUpCeeeEJ79uzRnDlztHnzZn3zzTeqWbNmhd7rDRs2qG3btqpZs6Zd+3/+8x9t2LBBAwYM0LXXXqucnBy9+uqr6t69u7Zt2yZ/f3916tRJzZo103vvvafExES79ZcuXap69eopPj5ekpSbm6sbbrhBFotFo0aNUlBQkD7++GMNGzZM+fn5Gjt2rN3606ZNk4+Pj8aPH68zZ87Ix8dH27Zt06pVq/TAAw+oadOmys3N1f/+7/+qW7du2rZtmxo3bixJKigo0K233qpff/1VY8aMUUhIiN59912tW7fO4fg///xz9e7dWzExMUpJSZGXl5cWLlyoW2+9VV999ZW6dOli9/5169bNIRydb/PmzWrZsqUCAgLs2su2tWXLFoWFhV10G5LUrFkznTp1SrVq1VL//v01c+ZMu9+rTp06qVatWnr22WdVv359XXfdddq1a5cmTJigzp07q2fPnnbba926tfz8/PTNN9/onnvuueT+UU14+MwRUK68vDxDknH33Xc71X/Lli2GJGP48OF27ePHjzckGZ9//rlhGIbx/vvvG5KM//znPxfdniQjJSXF9rzslPjQoUPt+t1zzz3GNddcY3uek5NjeHt7G9OnT7fr9+OPPxo1atRwaD/fp59+anh7exve3t5GXFycMWHCBOOTTz4xiouLHfqGh4cbiYmJtudll5Di4+ON0tJSW3tcXJxhsViMRx991Nb2+++/G9dee63RrVu3ix53eZelTp8+7VDLyJEjDX9/f6OoqMjW1q1bN0OSMW/ePIf+3bp1s9v3xS5LJSYmGuHh4bbnX331lSHJWLx4sV2/NWvW2LU7+16X59prrzXuu+8+h/byjj0jI8OQZLz11lu2tsmTJxs1a9Y0jh8/bms7c+aMUbduXbvfoWHDhhmNGjUyjh49arfNAQMGGIGBgbb9rVu3zpBkNGvWzKGGoqIih8tTe/bsMaxWq/H888/b2mbOnGlIMlatWmVrKywsNKKiogxJxrp16wzDMIzS0lKjRYsWDr9Hp0+fNpo2bWr06tXLbl+SHH6PytOmTRvj1ltvdWj/6aefLvh78kdpaWnGqFGjjMWLFxvLly83xowZY9SoUcNo0aKFkZeXZ9f3ww8/NBo1amRIsj3i4+MveOmpZcuWRu/evS95DKg+uCyFKqns1HLZwL9LWb16tSQpOTnZrv3JJ5+UdG5gsiTbWZMPP/xQZ8+edbmuRx991O75zTffrGPHjtnqXblypUpLS/Xggw/q6NGjtkdISIhatGhR7l/Jf9SrVy9lZGTorrvu0vfff6+///3vio+PV2hoqP797387VeOwYcNksVhsz2NjY2UYhoYNG2Zr8/b2VqdOnbR7925nD93mj6f1T548qaNHj+rmm2/W6dOnlZ2dbdfXarUqKSnJ5X1czLJlyxQYGKhevXrZvcYxMTGqXbu27TWuyHt97Ngx1atXz6H9j8d+9uxZHTt2TM2bN1fdunWVlZVlW5aQkKCzZ8/anSX69NNPdeLECSUkJEg6d3llxYoV6tevnwzDsDuW+Ph45eXl2W1TkhITEx0uq1itVtvA8pKSEh07dky1a9fWddddZ7f+mjVrFBoaqrvuusvW5uvrqxEjRthtb8uWLdq5c6cGDRqkY8eO2WoqKCjQbbfdpi+//NJugLdhGJc8ayOdG6xutVod2stmO11q5uOYMWM0e/ZsDRo0SPfdd5/S0tL05ptvaufOnXrllVfs+gYFBalDhw6aPn26Vq1apeeee05fffXVBX8X69Wrp6NHj17yGFB9EG5QJZWduj558qRT/ffu3SsvLy81b97crj0kJER169bV3r17JUndunXTfffdp6lTp6pBgwa6++67tXDhQodxORfSpEkTu+dlH4BlYz127twpwzDUokULBQUF2T22b9+uw4cPX3IfnTt31sqVK/Xbb79p48aNmjx5sk6ePKn7779f27Ztc7nGstk+55/yDwwMdBij4oyffvpJ99xzjwIDAxUQEKCgoCANHjxY0rkpuH8UGhoqHx8fl/dxMTt37lReXp4aNmzo8BqfOnXK9hpX9L02yhnTUVhYqClTpigsLExWq1UNGjRQUFCQTpw4YXfs7du3V1RUlJYuXWprW7p0qRo0aKBbb71VknTkyBGdOHFC8+fPdziOsg/h839fmjZt6lBTaWmp/vGPf6hFixZ2Nf3www92Ne3du1eRkZF2wVeSw7+ZnTt3SjoXpM6v6/XXX9eZM2cc3mdn+Pn5lfvaFxUV2Za7atCgQQoJCdHatWttbbt371aPHj00dOhQPfXUU7r77ruVkpKiV155RcuXL9fHH3/ssB3DMBxeF1RvjLlBlRQQEKDGjRtr69atLq13qf9BWSwWLV++XN9++60++OADffLJJxo6dKhmzpypb7/91m6KaXnOn3ZapuyDsLS0VBaLRR9//HG5fS+1/T/y8fFR586d1blzZ7Vs2VJJSUlatmyZUlJSLqvG8trL+wC/mBMnTqhbt24KCAjQ888/r8jISPn6+iorK0sTJ050mLJ9OR9Yl1JaWqqGDRtq8eLF5S4PCgqSVLH3+pprrik3+I0ePVoLFy7U2LFjFRcXp8DAQFksFg0YMMDh2BMSEjR9+nQdPXpUderU0b///W8NHDjQNtW+rP/gwYMdxuaUadeund3z8l7PGTNm6Nlnn9XQoUM1bdo01a9fX15eXho7duxlTaEvW+fFF190mJZfxpXf4zKNGjXSgQMHHNp//fVXSbKNDXJVWFiYjh8/bnu+aNEiFRUVqW/fvnb9ys5YffPNN+rdu7fdst9++00tWrS4rP2jaiLcoMrq27ev5s+fr4yMDMXFxV20b3h4uEpLS7Vz5061atXK1p6bm6sTJ04oPDzcrv8NN9ygG264QdOnT9e7776rhx56SEuWLNHw4cMrVHNkZKQMw1DTpk3VsmXLCm3rjzp16iTp/z4IPGX9+vU6duyYVq5caTeTaM+ePRXarit/NUdGRmrt2rXq2rWrU+Hpct7rqKioco9p+fLlSkxM1MyZM21tRUVFOnHihEPfhIQETZ06VStWrFBwcLDy8/M1YMAA2/KgoCDVqVNHJSUlDoNcXbF8+XL16NFDb7zxhl37iRMn1KBBA9vz8PBwbdu2zeEsxa5du+zWi4yMlHTuD4yK1HW+6OhorVu3Tvn5+XaDir/77jvbclcZhqGcnBx16NDB1pabmyvDMBxm75Vdmvz999/t2n///Xft37/f7nIdqj8uS6HKmjBhgmrVqqXhw4crNzfXYfkvv/yil19+WZLUp08fSVJaWppdn1mzZkmS7rzzTknn/kI7/2xF2f9Unb1ccTH33nuvvL29NXXqVIf9GIahY8eOXXT9devWlXs2pWxM0XXXXVfhGiui7OzPH2ssLi52GPPgqlq1aklSuSHhfA8++KBKSko0bdo0h2W///67bRsVea/j4uK0detWh37e3t4O25w9e7bDB6l0bhbR9ddfr6VLl2rp0qVq1KiRXSD09vbWfffdpxUrVpR7hvLIkSMXrfFiNS1btszhLEl8fLwOHDhgN3arqKhIr732ml2/mJgYRUZG6qWXXtKpU6cuWZezU8Hvv/9+lZSU2N1088yZM1q4cKFiY2PtLpvu27fPYfxWea/Hq6++qiNHjuiOO+6wtbVs2VKGYei9996z6/vPf/5TkuyCkCRt27ZNRUVFTs/KRPXAmRtUWZGRkXr33XeVkJCgVq1a2d2heMOGDVq2bJntPi/t27dXYmKi5s+fb7t0snHjRr355pvq37+/evToIUl688039corr+iee+5RZGSkTp48qddee00BAQG2gFTRmv/6179q8uTJysnJUf/+/VWnTh3t2bNH77//vh555BGNHz/+guuPHj1ap0+f1j333KOoqCjbsS5dulQRERFuH5zrqhtvvFH16tVTYmKinnjiCVksFr399tsuX946X2RkpOrWrat58+apTp06qlWrlmJjY8sdY9KtWzeNHDlSqamp2rJli26//XbVrFlTO3fu1LJly/Tyyy/r/vvvr9B7fffdd2vatGn64osvdPvtt9va+/btq7fffluBgYFq3bq1MjIytHbtWl1zzTXlbichIUFTpkyx3c/l/DtKv/DCC1q3bp1iY2M1YsQItW7dWsePH1dWVpbWrl1rd7nlQvr27avnn39eSUlJuvHGG/Xjjz9q8eLFatasmV2/kSNHas6cORo4cKDGjBmjRo0aafHixbYBvWVnc7y8vPT666+rd+/eatOmjZKSkhQaGqoDBw5o3bp1CggI0AcffGDbrrNTwWNjY/XAAw9o8uTJOnz4sJo3b64333xTOTk5DmedhgwZoi+++MLu9yo8PFwJCQm6/vrr5evrq6+//lpLlixRdHS0Ro4caev38MMP66WXXtLIkSO1efNmtWnTxnbbhjZt2jhM9/7ss8/k7++vXr16XfK1RjVy5SZmAZdnx44dxogRI4yIiAjDx8fHqFOnjtG1a1dj9uzZdlOPz549a0ydOtVo2rSpUbNmTSMsLMyYPHmyXZ+srCxj4MCBRpMmTQyr1Wo0bNjQ6Nu3r7Fp0ya7feoCU8HPvzvqhe7gu2LFCuOmm24yatWqZdSqVcuIiooyHn/8cePnn3++6LF+/PHHxtChQ42oqCijdu3aho+Pj9G8eXNj9OjRDncovtBU8POnPl+o9sTERKNWrVoXPe7yju+bb74xbrjhBsPPz89o3Lixbbq6/jCd2DDOTfdu06ZNucd5/lRwwzCMf/3rX0br1q2NGjVq2E0LP38qeJn58+cbMTExhp+fn1GnTh3j+uuvNyZMmGC7m7Wz7/WFtGvXzhg2bJhd22+//WYkJSUZDRo0MGrXrm3Ex8cb2dnZDu9FmZ07d9qmIn/99dfl7ic3N9d4/PHHjbCwMKNmzZpGSEiIcdtttxnz58+39SmbCr5s2TKH9YuKiownn3zSaNSokeHn52d07drVyMjIKPc13r17t3HnnXcafn5+RlBQkPHkk08aK1asMCQZ3377rV3fzZs3G/fee69xzTXXGFar1QgPDzcefPBBIz093a6fnJwKbhjnpp6PHz/eCAkJMaxWq9G5c2djzZo1Dv3KbiPwR8OHDzdat25t1KlTx6hZs6bRvHlzY+LEiUZ+fr7D+v/973+NoUOHGk2bNjV8fHyMRo0aGSNGjCj37saxsbHG4MGDnaof1YfFMCr4JxcAmNDbb7+txx9/XPv27bvojReru7S0NI0bN07//e9/FRoa6ulyrqgtW7aoY8eOysrKuqwxP6i6CDcAUI7S0lK1a9dOAwcO1NNPP+3pctyisLDQbhB2UVGROnTooJKSEu3YscODlXlG2Sy388fnoPoj3ADAVaJ3795q0qSJoqOjlZeXp3feeUc//fSTFi9erEGDBnm6PMBtGFAMAFeJ+Ph4vf7661q8eLFKSkrUunVrLVmyxHbXZMAsPDoV/Msvv1S/fv3UuHFjWSwWrVq16pLrrF+/Xh07dpTValXz5s1t3xwMALi4sWPHauvWrTp16pQKCwuVmZlJsIEpeTTcFBQUqH379po7d65T/ffs2aM777xTPXr00JYtWzR27FgNHz5cn3zySSVXCgAAqosqM+bGYrHo/fffV//+/S/YZ+LEifroo4/sbng1YMAAnThxQmvWrLkCVQIAgKquWo25ycjIcLgdeHx8vMaOHXvBdc6cOWN3l9HS0lIdP35c11xzDV+UBgBANWEYhk6ePKnGjRs73BDzfNUq3Bw6dEjBwcF2bWXf2XL+FMcyqampmjp16pUqEQAAVKL9+/fr2muvvWifahVuLsfkyZOVnJxse56Xl6cmTZpo//79dl/eBgAAqq78/HyFhYWpTp06l+xbrcJNSEiIwxco5ubmKiAg4ILfDmy1WmW1Wh3aAwICCDcAAFQzzgwpqVbfCh4XF6f09HS7ts8++0xxcXEeqggAAFQ1Hg03p06d0pYtW7RlyxZJ56Z6b9myRfv27ZN07pLSkCFDbP0fffRR7d69WxMmTFB2drZeeeUVvffeexo3bpwnygcAAFWQR8PNpk2b1KFDB3Xo0EGSlJycrA4dOmjKlCmSpF9//dUWdCSpadOm+uijj/TZZ5+pffv2mjlzpl5//XXFx8d7pH4AAFD1VJn73Fwp+fn5CgwMVF5eHmNuAACoJlz5/K5WY24AAAAuhXADAABMhXADAABMhXADAABMhXADAABMhXADAABMhXADAABMhXADAABMpVp9cSYAczp9+rSys7MrvJ3CwkLl5OQoIiLigl+m66qoqCj5+/u7ZVsArgzCDQCPy87OVkxMjKfLKFdmZqY6duzo6TIAuIBwA8DjoqKilJmZWeHtbN++XYMHD9Y777yjVq1auaGyc7UBqF4INwA8zt/f361nR1q1asXZFuAqxoBiAABgKoQbAABgKoQbAABgKoQbAABgKoQbAABgKoQbAABgKoQbAABgKoQbAABgKoQbAABgKoQbAABgKoQbAABgKoQbAABgKoQbAABgKoQbAABgKoQbAABgKoQbAABgKoQbAABgKoQbAABgKoQbAABgKoQbAABgKjU8XQCA6mvnzp06efKkp8uw2b59u91/q4o6deqoRYsWni4DuGoQbgBclp07d6ply5aeLqNcgwcP9nQJDnbs2EHAAa4Qwg2Ay1J2xuadd95Rq1atPFzNOYWFhcrJyVFERIT8/Pw8XY6kc2eRBg8eXKXOcAFmR7gBUCGtWrVSx44dPV2GTdeuXT1dAgAPY0AxAAAwFcINAAAwFcINAAAwFcINAAAwFcINAAAwFcINAAAwFcINAAAwFcINAAAwFcINAAAwFcINAAAwFcINAAAwFcINAAAwFcINAAAwFcINAAAwFcINAAAwFcINAAAwFcINAAAwFcINAAAwFcINAAAwFcINAAAwFcINAAAwFcINAAAwFcINAAAwFY+Hm7lz5yoiIkK+vr6KjY3Vxo0bL9o/LS1N1113nfz8/BQWFqZx48apqKjoClULAACqOo+Gm6VLlyo5OVkpKSnKyspS+/btFR8fr8OHD5fb/91339WkSZOUkpKi7du364033tDSpUv11FNPXeHKAQBAVeXRcDNr1iyNGDFCSUlJat26tebNmyd/f38tWLCg3P4bNmxQ165dNWjQIEVEROj222/XwIEDL3m2BwAAXD08Fm6Ki4uVmZmpnj17/l8xXl7q2bOnMjIyyl3nxhtvVGZmpi3M7N69W6tXr1afPn0uuJ8zZ84oPz/f7gEAAMyrhqd2fPToUZWUlCg4ONiuPTg4WNnZ2eWuM2jQIB09elQ33XSTDMPQ77//rkcfffSil6VSU1M1depUt9YOAACqLo8PKHbF+vXrNWPGDL3yyivKysrSypUr9dFHH2natGkXXGfy5MnKy8uzPfbv338FKwYAAFeax87cNGjQQN7e3srNzbVrz83NVUhISLnrPPvss/rTn/6k4cOHS5Kuv/56FRQU6JFHHtHTTz8tLy/HrGa1WmW1Wt1/AAAAoEry2JkbHx8fxcTEKD093dZWWlqq9PR0xcXFlbvO6dOnHQKMt7e3JMkwjMorFgAAVBseO3MjScnJyUpMTFSnTp3UpUsXpaWlqaCgQElJSZKkIUOGKDQ0VKmpqZKkfv36adasWerQoYNiY2O1a9cuPfvss+rXr58t5AAAgKubR8NNQkKCjhw5oilTpujQoUOKjo7WmjVrbIOM9+3bZ3em5plnnpHFYtEzzzyjAwcOKCgoSP369dP06dM9dQgAAKCKsRhX2fWc/Px8BQYGKi8vTwEBAZ4uB6i2srKyFBMTo8zMTHXs2NHT5VRZvE6Ae7jy+V2tZksBAABcCuEGAACYCuEGAACYCuEGAACYCuEGAACYCuEGAACYCuEGAACYCuEGAACYCuEGAACYCuEGAACYCuEGAACYCuEGAACYCuEGAACYCuEGAACYCuEGAACYCuEGAACYCuEGAACYCuEGAACYCuEGAACYCuEGAACYCuEGAACYCuEGAACYSg1PFwCg+gqpbZHfiR3SQf5OuhC/EzsUUtvi6TKAqwrhBsBlGxnjo1ZfjpS+9HQlVVcrnXudAFw5hBsAl+1/M4uVMGWRWkVFebqUKmt7drb+d+Yg3eXpQoCrCOEGwGU7dMpQYd2WUuNoT5dSZRUeKtWhU4anywCuKlwoBwAApkK4AQAApkK4AQAApkK4AQAApkK4AQAApkK4AQAApkK4AQAApkK4AQAApkK4AQAApkK4AQAApkK4AQAApkK4AQAApkK4AQAApkK4AQAApkK4AQAApkK4AQAApkK4AQAApkK4AQAApkK4AQAApkK4AQAApkK4AQAApkK4AQAApkK4AQAApkK4AQAApkK4AQAApuJyuNm9e3dl1AEAAOAWLoeb5s2bq0ePHnrnnXdUVFRUGTUBAABcNpfDTVZWltq1a6fk5GSFhIRo5MiR2rhxY2XUBgAA4DKXw010dLRefvllHTx4UAsWLNCvv/6qm266SW3bttWsWbN05MiRyqgTAADAKZc9oLhGjRq69957tWzZMv3tb3/Trl27NH78eIWFhWnIkCH69ddf3VknAACAUy473GzatEmPPfaYGjVqpFmzZmn8+PH65Zdf9Nlnn+ngwYO6++673VknAACAU2q4usKsWbO0cOFC/fzzz+rTp4/eeust9enTR15e53JS06ZNtWjRIkVERLi7VgAAgEty+czNq6++qkGDBmnv3r1atWqV+vbtaws2ZRo2bKg33njDqe3NnTtXERER8vX1VWxs7CUHJ584cUKPP/64GjVqJKvVqpYtW2r16tWuHgYAADApl8/cfPbZZ2rSpIlDoDEMQ/v371eTJk3k4+OjxMTES25r6dKlSk5O1rx58xQbG6u0tDTFx8fr559/VsOGDR36FxcXq1evXmrYsKGWL1+u0NBQ7d27V3Xr1nX1MAAAgEm5HG4iIyP166+/OoSP48ePq2nTpiopKXF6W7NmzdKIESOUlJQkSZo3b54++ugjLViwQJMmTXLov2DBAh0/flwbNmxQzZo1JYnLXwAAwI7Ll6UMwyi3/dSpU/L19XV6O8XFxcrMzFTPnj3/rxgvL/Xs2VMZGRnlrvPvf/9bcXFxevzxxxUcHKy2bdtqxowZFw1UZ86cUX5+vt0DAACYl9NnbpKTkyVJFotFU6ZMkb+/v21ZSUmJvvvuO0VHRzu946NHj6qkpETBwcF27cHBwcrOzi53nd27d+vzzz/XQw89pNWrV2vXrl167LHHdPbsWaWkpJS7TmpqqqZOnep0XQAAoHpzOtxs3rxZ0rkzNz/++KN8fHxsy3x8fNS+fXuNHz/e/RX+QWlpqRo2bKj58+fL29tbMTExOnDggF588cULhpvJkyfbgpkk5efnKywsrFLrBAAAnuN0uFm3bp0kKSkpSS+//LICAgIqtOMGDRrI29tbubm5du25ubkKCQkpd51GjRqpZs2a8vb2trW1atVKhw4dUnFxsV3gKmO1WmW1WitUKwAAqD5cHnOzcOHCCgcb6dzZnpiYGKWnp9vaSktLlZ6erri4uHLX6dq1q3bt2qXS0lJb244dO9SoUaNygw0AALj6OHXm5t5779WiRYsUEBCge++996J9V65c6fTOk5OTlZiYqE6dOqlLly5KS0tTQUGBbfbUkCFDFBoaqtTUVEnSn//8Z82ZM0djxozR6NGjtXPnTs2YMUNPPPGE0/sEAADm5lS4CQwMlMVisf3sLgkJCTpy5IimTJmiQ4cOKTo6WmvWrLENMt63b5/d/XTCwsL0ySefaNy4cWrXrp1CQ0M1ZswYTZw40W01AQCA6s1iXGhudznKbtQXFBQkPz+/yqyr0uTn5yswMFB5eXluubwGXK2ysrIUExOjzMxMdezY0dPlVFm8ToB7uPL57dKYG8Mw1Lx5c/33v/+tUIEAAACVxaVw4+XlpRYtWujYsWOVVQ8AAECFuDxb6oUXXtBf/vIXbd26tTLqAQAAqBCXv1tqyJAhOn36tNq3by8fHx+HsTfHjx93W3EAAACucjncpKWlVUIZAAAA7uFyuElMTKyMOgAAANzC5XDzR0VFRSouLrZrY3o1AADwJJcHFBcUFGjUqFFq2LChatWqpXr16tk9AAAAPMnlcDNhwgR9/vnnevXVV2W1WvX6669r6tSpaty4sd56663KqBEAAMBpLl+W+uCDD/TWW2+pe/fuSkpK0s0336zmzZsrPDxcixcv1kMPPVQZdQIAADjF5TM3x48fV7NmzSSdG19TNvX7pptu0pdffune6gAAAFzkcrhp1qyZ9uzZI0mKiorSe++9J+ncGZ26deu6tTgAAABXuRxukpKS9P3330uSJk2apLlz58rX11fjxo3TX/7yF7cXCAAA4AqXx9yMGzfO9nPPnj2VnZ2tzMxMNW/eXO3atXNrcQAAAK6q0H1uJCk8PFzh4eHuqAUAAKDCnAo3//M//+P0Bp944onLLgYAAKCinAo3//jHP5zamMViIdwAAACPcirclM2OAgAAqOpcni0FAABQlTl15iY5OVnTpk1TrVq1lJycfNG+s2bNckthAAAAl8OpcLN582adPXvW9vOFWCwW91QFAABwmZwKN+vWrSv3ZwAAgKqGMTcAAMBUXL6JX1FRkWbPnq1169bp8OHDKi0ttVuelZXltuIAAABc5XK4GTZsmD799FPdf//96tKlC+NsAABAleJyuPnwww+1evVqde3atTLqAQAAqBCXx9yEhoaqTp06lVELAABAhbkcbmbOnKmJEydq7969lVEPAABAhbh8WapTp04qKipSs2bN5O/vr5o1a9otP378uNuKAwAAcJXL4WbgwIE6cOCAZsyYoeDgYAYUAwCAKsXlcLNhwwZlZGSoffv2lVEPAABAhbg85iYqKkqFhYWVUQsAAECFuRxuXnjhBT355JNav369jh07pvz8fLsHAACAJ7l8WeqOO+6QJN1222127YZhyGKxqKSkxD2VAQAAXAaXww1fnAkAAKoyl8NNt27dKqMOAAAAt3Aq3Pzwww9q27atvLy89MMPP1y0b7t27dxSGAAAwOVwKtxER0fr0KFDatiwoaKjo2WxWGQYhkM/xtwAAABPcyrc7NmzR0FBQbafAQAAqiqnwk14eHi5PwMAAFQ1Tt/nZseOHdq4caNdW3p6unr06KEuXbpoxowZbi8OAADAVU6Hm4kTJ+rDDz+0Pd+zZ4/69esnHx8fxcXFKTU1VWlpaZVRIwAAgNOcngq+adMmTZgwwfZ88eLFatmypT755BNJ52ZJzZ49W2PHjnV7kQAAAM5y+szN0aNHde2119qer1u3Tv369bM97969u3JyctxaHAAAgKucDjf169fXr7/+KkkqLS3Vpk2bdMMNN9iWFxcXlzs9HAAA4EpyOtx0795d06ZN0/79+5WWlqbS0lJ1797dtnzbtm2KiIiohBIBAACc5/SYm+nTp6tXr14KDw+Xt7e3/ud//ke1atWyLX/77bd16623VkqRAAAAznI63ERERGj79u366aefFBQUpMaNG9stnzp1qt2YHAAAAE9w6Ysza9Soofbt25e77ELtAAAAV5LTY24AAACqA8INAAAwFcINAAAwFcINAAAwlcsKN1999ZUGDx6suLg4HThwQNK5qeBff/21W4sDAABwlcvhZsWKFYqPj5efn582b96sM2fOSJLy8vL4ZnAAAOBxLoebv/71r5o3b55ee+011axZ09betWtXZWVlubU4AAAAV7kcbn7++WfdcsstDu2BgYE6ceKEO2oCAAC4bC6Hm5CQEO3atcuh/euvv1azZs3cUhQAAMDlcjncjBgxQmPGjNF3330ni8WigwcPavHixRo/frz+/Oc/V0aNAAAATnM53EyaNEmDBg3SbbfdplOnTumWW27R8OHDNXLkSI0ePfqyipg7d64iIiLk6+ur2NhYbdy40an1lixZIovFov79+1/WfgEAgPm4HG4sFouefvppHT9+XFu3btW3336rI0eOaNq0aZdVwNKlS5WcnKyUlBRlZWWpffv2io+P1+HDhy+6Xk5OjsaPH6+bb775svYLAADM6bJv4ufj46PWrVurS5cuql279mUXMGvWLI0YMUJJSUlq3bq15s2bJ39/fy1YsOCC65SUlOihhx7S1KlTGecDAADsuPSt4JJUUFCgF154Qenp6Tp8+LBKS0vtlu/evdvpbRUXFyszM1OTJ0+2tXl5ealnz57KyMi44HrPP/+8GjZsqGHDhumrr7666D7OnDljuxePJOXn5ztdHwAAqH5cDjfDhw/XF198oT/96U9q1KiRLBbLZe/86NGjKikpUXBwsF17cHCwsrOzy13n66+/1htvvKEtW7Y4tY/U1FRNnTr1smsEAADVi8vh5uOPP9ZHH32krl27VkY9F3Xy5En96U9/0muvvaYGDRo4tc7kyZOVnJxse56fn6+wsLDKKhEAAHiYy+GmXr16ql+/vlt23qBBA3l7eys3N9euPTc3VyEhIQ79f/nlF+Xk5Khfv362trLLYjVq1NDPP/+syMhIu3WsVqusVqtb6gUAAFWfywOKp02bpilTpuj06dMV3rmPj49iYmKUnp5uaystLVV6erri4uIc+kdFRenHH3/Uli1bbI+77rpLPXr00JYtWzgjAwAAXD9zM3PmTP3yyy8KDg5WRESE3fdLSXL5+6WSk5OVmJioTp06qUuXLkpLS1NBQYGSkpIkSUOGDFFoaKhSU1Pl6+urtm3b2q1ft25dSXJoBwAAVyeXw427b5iXkJCgI0eOaMqUKTp06JCio6O1Zs0a2yDjffv2ycvrsmesAwCAq4zL4SYlJcXtRYwaNUqjRo0qd9n69esvuu6iRYvcXg8AAKi+OCUCAABMxakzN/Xr19eOHTvUoEED1atX76L3tjl+/LjbigMAAHCVU+HmH//4h+rUqWP7uSI37gMAAKhMToWbxMRE288PP/xwZdUCAABQYS6PucnKytKPP/5oe/6vf/1L/fv311NPPaXi4mK3FgcAAOAql8PNyJEjtWPHDknnviQzISFB/v7+WrZsmSZMmOD2AgEAAFzhcrjZsWOHoqOjJUnLli1Tt27d9O6772rRokVasWKFu+sDAABwicvhxjAM2/c5rV27Vn369JEkhYWF6ejRo+6tDgAAwEUuh5tOnTrpr3/9q95++2198cUXuvPOOyVJe/bssd1VGAAAwFNcDjdpaWnKysrSqFGj9PTTT6t58+aSpOXLl+vGG290e4EAAACucPnrF9q1a2c3W6rMiy++KG9vb7cUBaDqO336tCTXvyy3MhUWFionJ0cRERHy8/PzdDmSpO3bt3u6BOCq43K4KZOZmWn7R9u6dWt17NjRbUUBqPqys7MlSSNGjPBwJdVD2Y1QAVQ+l8PN4cOHlZCQoC+++EJ169aVJJ04cUI9evTQkiVLFBQU5O4aAVRB/fv3lyRFRUXJ39/fs8X8f9u3b9fgwYP1zjvvqFWrVp4ux6ZOnTpq0aKFp8sArhouh5vRo0fr1KlT+umnn2z/89i2bZsSExP1xBNP6J///KfbiwRQ9TRo0EDDhw/3dBnlatWqFWeTgauYy+FmzZo1Wrt2rd1fRa1bt9bcuXN1++23u7U4AAAAV7k8W6q0tFQ1a9Z0aK9Zs6bt/jcAAACe4nK4ufXWWzVmzBgdPHjQ1nbgwAGNGzdOt912m1uLAwAAcJXL4WbOnDnKz89XRESEIiMjFRkZqaZNmyo/P1+zZ8+ujBoBAACc5vKYm7CwMGVlZWnt2rW2qaCtWrVSz5493V4cAACAqy7rPjcWi0W9evVSr1693F0PAABAhTh9Werzzz9X69atlZ+f77AsLy9Pbdq00VdffeXW4gAAAFzldLhJS0vTiBEjFBAQ4LAsMDBQI0eO1KxZs9xaHAAAgKucDjfff/+97rjjjgsuv/3225WZmemWogAAAC6X0+EmNze33PvblKlRo4aOHDnilqIAAAAul9PhJjQ0VFu3br3g8h9++EGNGjVyS1EAAACXy+lw06dPHz377LMqKipyWFZYWKiUlBT17dvXrcUBAAC4yump4M8884xWrlypli1batSoUbruuuskSdnZ2Zo7d65KSkr09NNPV1qhAAAAznA63AQHB2vDhg3685//rMmTJ8swDEnn7nkTHx+vuXPnKjg4uNIKBQAAcIZLN/ELDw/X6tWr9dtvv2nXrl0yDEMtWrRQvXr1Kqs+AAAAl1zWHYrr1aunzp07u7sWAACACnP5izMBAACqMsINAAAwFcINAAAwFcINAAAwFcINAAAwFcINAAAwFcINAAAwFcINAAAwFcINAAAwFcINAAAwFcINAAAwFcINAAAwFcINAAAwFcINAAAwFcINAAAwFcINAAAwFcINAAAwFcINAAAwFcINAAAwFcINAAAwFcINAAAwFcINAAAwFcINAAAwFcINAAAwFcINAAAwFcINAAAwFcINAAAwFcINAAAwlSoRbubOnauIiAj5+voqNjZWGzduvGDf1157TTfffLPq1aunevXqqWfPnhftDwAAri4eDzdLly5VcnKyUlJSlJWVpfbt2ys+Pl6HDx8ut//69es1cOBArVu3ThkZGQoLC9Ptt9+uAwcOXOHKAQBAVWQxDMPwZAGxsbHq3Lmz5syZI0kqLS1VWFiYRo8erUmTJl1y/ZKSEtWrV09z5szRkCFDLtk/Pz9fgYGBysvLU0BAQIXrB1B1ZGVlKSYmRpmZmerYsaOnywHgRq58fnv0zE1xcbEyMzPVs2dPW5uXl5d69uypjIwMp7Zx+vRpnT17VvXr1y93+ZkzZ5Sfn2/3AAAA5uXRcHP06FGVlJQoODjYrj04OFiHDh1yahsTJ05U48aN7QLSH6WmpiowMND2CAsLq3DdAACg6vL4mJuKeOGFF7RkyRK9//778vX1LbfP5MmTlZeXZ3vs37//ClcJAACupBqe3HmDBg3k7e2t3Nxcu/bc3FyFhIRcdN2XXnpJL7zwgtauXat27dpdsJ/VapXVanVLvQAAoOrz6JkbHx8fxcTEKD093dZWWlqq9PR0xcXFXXC9v//975o2bZrWrFmjTp06XYlSAQBANeHRMzeSlJycrMTERHXq1EldunRRWlqaCgoKlJSUJEkaMmSIQkNDlZqaKkn629/+pilTpujdd99VRESEbWxO7dq1Vbt2bY8dBwAAqBo8Hm4SEhJ05MgRTZkyRYcOHVJ0dLTWrFljG2S8b98+eXn93wmmV199VcXFxbr//vvttpOSkqLnnnvuSpYOAACqII+HG0kaNWqURo0aVe6y9evX2z3Pycmp/IIAAEC1Va1nSwEAAJyPcAMAAEyFcAMAAEyFcAMAAEyFcAMAAEyFcAMAAEyFcAMAAEyFcAMAAEyFcAMAAEyFcAMAAEyFcAMAAEyFcAMAAEyFcAMAAEyFcAMAAEyFcAMAAEyFcAMAAEyFcAMAAEyFcAMAAEyFcAMAAEyFcAMAAEyFcAMAAEyFcAMAAEyFcAMAAEyFcAMAAEyFcAMAAEyFcAMAAEyFcAMAAEyFcAMAAEyFcAMAAEyFcAMAAEyFcAMAAEyFcAMAAEyFcAMAAEyFcAMAAEyFcAMAAEyFcAMAAEyFcAMAAEyFcAMAAEyFcAMAAEyFcAMAAEyFcAMAAEyFcAMAAEyFcAMAAEyFcAMAAEyFcAMAAEyFcAMAAEyFcAMAAEyFcAMAAEyFcAMAAEyFcAMAAEyFcAMAAEyFcAMAAEyFcAMAAEyFcAMAAEyFcAMAAEyFcAMAAEyFcAMAAEyFcAMAAEyFcAMAAEylSoSbuXPnKiIiQr6+voqNjdXGjRsv2n/ZsmWKioqSr6+vrr/+eq1evfoKVQoAAKo6j4ebpUuXKjk5WSkpKcrKylL79u0VHx+vw4cPl9t/w4YNGjhwoIYNG6bNmzerf//+6t+/v7Zu3XqFKwcAAFWRxTAMw5MFxMbGqnPnzpozZ44kqbS0VGFhYRo9erQmTZrk0D8hIUEFBQX68MMPbW033HCDoqOjNW/evEvuLz8/X4GBgcrLy1NAQID7DgSAx2VlZSkmJkaZmZnq2LGjp8sB4EaufH579MxNcXGxMjMz1bNnT1ubl5eXevbsqYyMjHLXycjIsOsvSfHx8Rfsf+bMGeXn59s9AACAeXk03Bw9elQlJSUKDg62aw8ODtahQ4fKXefQoUMu9U9NTVVgYKDtERYW5p7iAQBAleTxMTeVbfLkycrLy7M99u/f7+mSAABAJarhyZ03aNBA3t7eys3NtWvPzc1VSEhIueuEhIS41N9qtcpqtbqnYAAAUOV5NNz4+PgoJiZG6enp6t+/v6RzA4rT09M1atSocteJi4tTenq6xo4da2v77LPPFBcXdwUqBlAZTp8+rezs7ApvZ/v27Xb/dYeoqCj5+/u7bXsAKp9Hw40kJScnKzExUZ06dVKXLl2UlpamgoICJSUlSZKGDBmi0NBQpaamSpLGjBmjbt26aebMmbrzzju1ZMkSbdq0SfPnz/fkYQCogOzsbMXExLhte4MHD3bbtph5BVQ/Hg83CQkJOnLkiKZMmaJDhw4pOjpaa9assQ0a3rdvn7y8/m9o0I033qh3331XzzzzjJ566im1aNFCq1atUtu2bT11CAAqKCoqSpmZmRXeTmFhoXJychQRESE/Pz83VHauNgDVi8fvc3OlcZ8bAACqn2pznxsAAAB3I9wAAABTIdwAAABTIdwAAABTIdwAAABTIdwAAABTIdwAAABTIdwAAABTIdwAAABTIdwAAABTIdwAAABTIdwAAABTIdwAAABTqeHpAq60si9Bz8/P93AlAADAWWWf22Wf4xdz1YWbkydPSpLCwsI8XAkAAHDVyZMnFRgYeNE+FsOZCGQipaWlOnjwoOrUqSOLxeLpcgC4UX5+vsLCwrR//34FBAR4uhwAbmQYhk6ePKnGjRvLy+vio2quunADwLzy8/MVGBiovLw8wg1wFWNAMQAAMBXCDQAAMBXCDQDTsFqtSklJkdVq9XQpADyIMTcAAMBUOHMDAABMhXADAABMhXADAABMhXADAABMhXADAABMhXADoNr78ssv1a9fPzVu3FgWi0WrVq3ydEkAPIhwA6DaKygoUPv27TV37lxPlwKgCrjqvhUcgPn07t1bvXv39nQZAKoIztwAAABTIdwAAABTIdwAAABTIdwAAABTIdwAAABTYbYUgGrv1KlT2rVrl+35nj17tGXLFtWvX19NmjTxYGUAPMFiGIbh6SIAoCLWr1+vHj16OLQnJiZq0aJFV74gAB5FuAEAAKbCmBsAAGAqhBsAAGAqhBsAAGAqhBsAAGAqhBsAAGAqhBsAAGAqhBsAAGAqhBsAAGAqhBsAAGAqhBsAAGAqhBsAAGAq/w9pN+ARkykUMAAAAABJRU5ErkJggg==",
      "text/plain": [
       "<Figure size 640x480 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "from typing import List\n",
    "\n",
    "import matplotlib.pyplot as plt\n",
    "import numpy as np\n",
    "from sentence_transformers import SentenceTransformer\n",
    "from sentence_transformers.util import cos_sim\n",
    "\n",
    "embed_model = SentenceTransformer(\"all-MiniLM-L12-v2\")\n",
    "\n",
    "\n",
    "def get_similarity(\n",
    "    human_answers: str | List[str],\n",
    "    model_answers: str | List[str],\n",
    "    embed_model: SentenceTransformer = embed_model,\n",
    ") -> List[float]:\n",
    "    \"\"\"\n",
    "    Get the cosine similarity between the human answer and the model answer for each pair of answers in the list\n",
    "\n",
    "    Args:\n",
    "        human_answers (str or List[str]):\n",
    "            list of human answers\n",
    "        model_answers (str or List[str]):\n",
    "            list of model answers\n",
    "        model (SentenceTransformer, default: all-MiniLM-L12-v2):\n",
    "            embedding model to use for sentence similarity\n",
    "\n",
    "    Returns:\n",
    "        similarities (float or List[float]):\n",
    "            list of cosine similarities between the human answers and the model answers\n",
    "    \"\"\"\n",
    "    similarities = []\n",
    "    for human_answer, model_answer in zip(human_answers, model_answers):\n",
    "        # get the embeddings for the answers\n",
    "        human_answer_embedding = embed_model.encode(\n",
    "            human_answer, convert_to_tensor=True, show_progress_bar=False\n",
    "        )\n",
    "        model_answer_embedding = embed_model.encode(\n",
    "            model_answer, convert_to_tensor=True, show_progress_bar=False\n",
    "        )\n",
    "\n",
    "        # calculate the cosine similarity between the human answer and the model answer\n",
    "        similarities.append(\n",
    "            cos_sim(human_answer_embedding, model_answer_embedding).item()\n",
    "        )\n",
    "\n",
    "    return similarities\n",
    "\n",
    "\n",
    "cosine_sims = get_similarity(human_answers, eval_results[\"model_answers\"])\n",
    "eval_results[\"cosine_similarities\"] = cosine_sims\n",
    "avg_sim = np.mean(cosine_sims)\n",
    "\n",
    "# plot the distribution of cosine similarities\n",
    "plt.boxplot(cosine_sims)\n",
    "plt.title(f\"Cosine Similarities (average: {avg_sim:.2f})\")\n",
    "plt.ylabel(\"Cosine Similarity\")\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The average cosine similarity is roughly 0.6, which is not bad considering the fact that the LLM is not trained on my knowledge base. However, this performance varies among 11 responses: the best is 0.95 and the worst is negative. In the next section, I discuss the reasons behind this variation: which type of question does the LLM tend to perform well on and which type it does not."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Qualitative evaluation"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/markdown": [
       "### 1. **Cosine Similarity**: 0.9529\n",
       "- **Question**: What does microenomics concern?\n",
       "- **Generated by model?**: False\n",
       "- **Human Answer**: Microeconomics is the study of how individuals, households, and firms make decisions about using limited resources—not just buying, selling, or producing them\n",
       "\n",
       "- **Model Answer**: Microeconomics concerns the study of how individuals, households, and firms make decisions about using limited resources.\n"
      ],
      "text/plain": [
       "<IPython.core.display.Markdown object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/markdown": [
       "### 2. **Cosine Similarity**: 0.9399\n",
       "- **Question**: In marketing 4P, what factors affect price?\n",
       "- **Generated by model?**: True\n",
       "- **Human Answer**: Several factors affect price: supply costs, discounts, competitors' prices. However, the optimal approach should be customer-based. We want to quantify the range of willingness to pay and change prices by segments. If we just do cost-based pricing, we miss out on customer needs.\n",
       "\n",
       "- **Model Answer**: Several factors affect price: supply costs, discounts, competitors' prices. However, the optimal approach should be customer-based. We want to quantify the range of willingness to pay and change prices by segments. If we just do cost-based pricing, we miss out on customer needs. Other pricing strategies include outcome-based pricing, value-based pricing, bundling, upselling, and cross-selling.\n"
      ],
      "text/plain": [
       "<IPython.core.display.Markdown object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/markdown": [
       "### 3. **Cosine Similarity**: 0.8892\n",
       "- **Question**: Why do we need to change the basis?\n",
       "- **Generated by model?**: False\n",
       "- **Human Answer**: The change of basis is necessary when we want to work with a different linear transformation or when we want to represent vectors in a different basis. It allows us to convert matrices and vectors from one basis to another. \n",
       "\n",
       "- **Model Answer**: Changing the basis is necessary in linear algebra for various reasons. One common reason is to simplify calculations and analysis. By changing the basis, we can often find a new set of basis vectors that make certain operations or transformations easier to understand and compute. Additionally, changing the basis can help us gain insight into the structure and properties of a vector space or linear transformation. It allows us to view vectors and transformations from different perspectives, which can be useful in solving problems and proving theorems. Overall, changing the basis is a powerful tool in linear algebra that helps us analyze and manipulate vectors and linear transformations in a more convenient and insightful way.\n"
      ],
      "text/plain": [
       "<IPython.core.display.Markdown object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/markdown": [
       "### 4. **Cosine Similarity**: 0.8840\n",
       "- **Question**: Provide an overview of most important mechanisms for transformers.\n",
       "- **Generated by model?**: True\n",
       "- **Human Answer**: The most important mechanisms for transformers include self-attention mechanisms, multi-head attention, positional encoding, and feedforward neural networks. These mechanisms enable transformers to handle sequence data effectively, by learning the relationships between different elements of the sequence and the relative positions of these elements, and by paying attention to relevant parts of the sequence when making predictions.\n",
       "\n",
       "- **Model Answer**: The most important mechanisms for transformers are:\n",
       "1. Attention Mechanism: Transformers use attention mechanism to compute attention scores for each word in the sequence, determining how much it should focus on other words.\n",
       "2. Self-attention: Transformers use self-attention to weigh the importance of different parts of the input when making predictions.\n",
       "3. Input Embedding & Output Embedding: Each input sequence is tokenized as a word embedding, which is a dense vector of real numbers.\n",
       "4. Residual Connection: Transformers use residual connection to help with the flow of information through the network.\n",
       "5. Layer Normalization: Transformers use layer normalization to normalize the activations and stabilize training.\n",
       "6. Masked Multi-Head Attention: Transformers use masked multi-head attention in the decoder to prevent it from looking at future positions in the sequence during training.\n",
       "\n",
       "These mechanisms are key components of the Transformer architecture, which is a model of self-supervised learning that can handle sequence data effectively.\n"
      ],
      "text/plain": [
       "<IPython.core.display.Markdown object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/markdown": [
       "### 5. **Cosine Similarity**: 0.7640\n",
       "- **Question**: How do I interpret regression coefficients?\n",
       "- **Generated by model?**: False\n",
       "- **Human Answer**: Slope b indicates that, for every unit increase in the predictor, we would expect b unit increase in the dependent variable.\n",
       " Intercept a is the value of dependent variable when all independent variables are set to 0. \n",
       "Error term (residual): the difference between the predicted and actual value (aka the uncertainty of the regression model. \n",
       " Interaction term indicate whether there is a joint effect of some predictors on the variable.\n",
       "\n",
       "- **Model Answer**: The interpretation of regression coefficients depends on the specific context and variables involved in the regression analysis. In general, the slope coefficient represents the change in the dependent variable for a one-unit increase in the independent variable, while holding all other independent variables constant. The intercept represents the estimated value of the dependent variable when all independent variables are set to zero. The error term (residual) represents the difference between the observed and predicted values of the dependent variable. The prediction interval is a range of values within which we expect the true value of the dependent variable to fall with a certain level of confidence. It is important to note that correlation does not imply causation, and hypothesis testing is used to determine if a coefficient is statistically significant, indicating a true correlation between variables. The p-value is the probability of observing such a substantial association between the predictor and the response by chance, assuming the null hypothesis is true. If the null hypothesis is rejected, it can be concluded that the coefficient is statistically significant.\n"
      ],
      "text/plain": [
       "<IPython.core.display.Markdown object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/markdown": [
       "### 6. **Cosine Similarity**: 0.7130\n",
       "- **Question**: How should I evaluate a synthetic control case study?\n",
       "- **Generated by model?**: False\n",
       "- **Human Answer**: We can conduct placebo studies to ensure that the treatment effect is only observed by the treated unit in the specific time. This can be done through in-time placebo (reapplying the treatment to a different time period before the intervention) or in-space placebo (reapplying the treatment to a comparison unit).\n",
       " Further, we can perform robustness tests by iteratively excluding each comparison unit with positive weights (leave-one-out), reconstructing the synthetic control, and re-estimating the treatment effect to see if the conclusion still holds.\n",
       "\n",
       "- **Model Answer**: To evaluate a synthetic control case study, there are several steps you can take:\n",
       "\n",
       "1. Assess the validity of the synthetic control by checking the similarity of the treated unit's pre-treatment characteristics to the synthetic control.\n",
       "\n",
       "2. Compare the actual post-treatment outcome of the treated unit with the counterfactual outcome predicted by the synthetic control.\n",
       "\n",
       "3. Refine the synthetic control by adjusting the set of covariates, exploring alternative weighting schemes, or incorporating additional control units.\n",
       "\n",
       "4. Conduct placebo studies to ensure that the treatment effect is only observed by the treated unit in the specific time. This can be done through in-time placebo studies, where the treatment is reapplied to a different time period before the intervention, or in-space placebo studies, where the treatment is reapplied to a comparison unit.\n",
       "\n",
       "5. Perform robustness tests by iteratively excluding each comparison unit with positive weights (leave-one-out), reconstructing the synthetic control, and re-estimating the treatment effect. This helps determine if the conclusion still holds when specific comparison units are excluded.\n",
       "\n",
       "6. Consider the tradeoff between the number of comparison units and the goodness of fit. Using weighted combinations of units generally increases the goodness of fit, but it's important to find a balance.\n",
       "\n",
       "7. Calculate the p-value, which is the sum of squared difference between the synthetic control and treated unit pre-treatment.\n",
       "\n",
       "These steps can help evaluate the effectiveness and reliability of a synthetic control case study.\n"
      ],
      "text/plain": [
       "<IPython.core.display.Markdown object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/markdown": [
       "### 7. **Cosine Similarity**: 0.4775\n",
       "- **Question**: Based on the NLP techniques I already learned, what should I learn next?\n",
       "- **Generated by model?**: False\n",
       "- **Human Answer**: Delve deeper into advanced transformer models beyond BERT\n",
       "Learn how to fine-tune pre-trained models like BERT or GPT for specific downstream tasks\n",
       "Explore models that handle multiple modalities, such as images and text. Models like CLIP (Contrastive Language-Image Pre-training) and ViT (Vision Transformer) can provide insights into cross-modal representations.\n",
       "Gain an understanding of ethical considerations and biases in NLP models. Explore methods to detect and mitigate biases in language models.\n",
       "\n",
       "- **Model Answer**: Based on the NLP techniques you already learned, you should learn about the attention mechanism in sequence-to-sequence models. The attention mechanism allows the model to focus on relevant parts of the input sequence when generating the output sequence. This can improve the performance of sequence-to-sequence models, especially when dealing with long sequences. Additionally, you can explore named entity recognition techniques using libraries such as NLTK, SpaCy, or Polyglot. Named entity recognition is a useful technique for identifying important named entities in text, which can be helpful for various NLP tasks.\n"
      ],
      "text/plain": [
       "<IPython.core.display.Markdown object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/markdown": [
       "### 8. **Cosine Similarity**: 0.4067\n",
       "- **Question**: What is a probability?\n",
       "- **Generated by model?**: False\n",
       "- **Human Answer**: A naive definition: Probability presents how likely an event is to occur with a number between 0 and 1, where, roughly speaking, 0 indicates impossibility of the event and 1 indicates certainty. The higher the probability of an event, the more likely it is that the event will occur.\n",
       "\n",
       "\n",
       "- **Model Answer**: The probability of an event occurring given another event occurred is called conditional probability. It is denoted as P(A|B), which represents the probability of event A happening given that event B has already occurred. The conditional probability can be calculated using the formula P(A|B) = P(A and B) / P(B), where P(A and B) is the probability of both events A and B happening, and P(B) is the probability of event B happening. In the given example of drawing cards, the conditional probability would be the probability of drawing a red card as the second card, given that the first card drawn is a heart.\n"
      ],
      "text/plain": [
       "<IPython.core.display.Markdown object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/markdown": [
       "### 9. **Cosine Similarity**: 0.2894\n",
       "- **Question**: From my notes, what did I learn in the business classes?\n",
       "- **Generated by model?**: False\n",
       "- **Human Answer**: In marketing, you learned about customer-centric thinking, such as customer segmentation and customer lifetime value, and popular frameworks, such as five competitive forces and marketing 4Ps.\n",
       "In accounting, you learned about three financial statements that are used to evaluate a company's financial performance: balance sheet, income statement, and cash flow statement.\n",
       " In finance, you learned about the time value of money, valuation, options, and risk management. You also learned the basics of economics.\n",
       "\n",
       "- **Model Answer**: The accounts that would be increased with a Credit are Revenue, Notes Payable, and Additional Paid-in-Capital. Paying cash to reduce a liability would be represented by a journal entry that debits the liability account and credits the cash account.\n"
      ],
      "text/plain": [
       "<IPython.core.display.Markdown object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/markdown": [
       "### 10. **Cosine Similarity**: 0.0480\n",
       "- **Question**: What are some methods to measure how well a Bayesian model estimates the posterior?\n",
       "- **Generated by model?**: False\n",
       "- **Human Answer**: 1. Model evidence: Quantifies the probability of observed data under the model, indicating how well a Bayesian model fits the observed data.\n",
       " 2. Kullback-Leibler divergence: Quantifies the difference between two probability distributions, typically a true/observed distribution and an estimated model distribution.\n",
       " 3. Log Point-wise Predictive Density (LPPD): Represents the difference between the KL divergence of the target and model and the KL divergence of the target and itself. It is a measure of how well the model fits the data.\n",
       "\n",
       "- **Model Answer**: The answer is not available and I will use my own knowledge to answer the question.\n"
      ],
      "text/plain": [
       "<IPython.core.display.Markdown object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/markdown": [
       "### 11. **Cosine Similarity**: -0.0136\n",
       "- **Question**: How to identify which critical point is local minimum or maximum?\n",
       "- **Generated by model?**: False\n",
       "- **Human Answer**: We can use the second derivative test. If the second derivative is positive, the critical point is a local minimum. If the second derivative is negative, the critical point is a local maximum. If the second derivative is zero, the test is inconclusive.\n",
       "\n",
       "- **Model Answer**: The answer is not available and I will use my own knowledge to answer the question.\n"
      ],
      "text/plain": [
       "<IPython.core.display.Markdown object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "from IPython.display import Markdown, display\n",
    "\n",
    "# Sort the indices of the eval_results by cosine similarity\n",
    "sorted_indices = np.argsort(eval_results[\"cosine_similarities\"])[::-1]\n",
    "\n",
    "# Display eval_reults in a numbered list\n",
    "for i, index in enumerate(sorted_indices):\n",
    "    cos_sim = eval_results[\"cosine_similarities\"][index]\n",
    "    question = eval_results[\"questions\"][index]\n",
    "    use_context = eval_results[\"is_model_generated\"][index]\n",
    "    human_answer = eval_results[\"human_answers\"][index]\n",
    "    model_answer = eval_results[\"model_answers\"][index]\n",
    "    display(\n",
    "        Markdown(\n",
    "            f\"### {i+1}. **Cosine Similarity**: {cos_sim:.4f}\\n\"\n",
    "            f\"- **Question**: {question}\\n\"\n",
    "            f\"- **Generated by model?**: {use_context}\\n\"\n",
    "            f\"- **Human Answer**: {human_answer}\\n\\n\"\n",
    "            f\"- **Model Answer**: {model_answer}\\n\"\n",
    "        )\n",
    "    )"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- The system tends to perform very well (cosine similarity > 0.9) on questions that contain keywords in the metadata. For question “How to identify which critical point is local minimum or maximum?” I have a note called “critical point,” and for “In marketing 4P, what factors affect price?,” I have a note called “marketing 4Ps.” The retriever manages to grab the exact passage I copy my human answer from.\n",
    "- It performs less well (cosine similarity: 0.7-0.8) when the LLM has to summarize across longer documents (question 3-4-6). \n",
    "- It performs quite bad (cosine similarity: 0.2 - 0.4) when it is asked to summarize across my documents and infer based on its knowledge (e.g. give advice) in questions 7 and 9. It is understandable though, but I can engineer prompt further.\n",
    "- The LLM-generated answer to the question \"What is a probability?\" is about conditional probability, but the cosine similarity is medium (0.6).  This poses two problems:\n",
    "\t- Cosine similarity lacks as the sole metric to evaluate the performance of the system. The similarity between “probability” and “conditional probability” might have increased the score, although the answer is completely irrelevant. I need another systematic approach to quantify the relevance, potentially through `trulens_eval` with the use of LLM. \n",
    "\t- Embedding-based retrieval (and even reranking) might not capture the context as well as the LLM. It’s more obvious when I run several times through the “What are some methods to measure how well a Bayesian model estimates the posterior?.” Sometimes the model can capture the keyword well, or sometimes it cannot. More advanced techniques are needed to increase the accuracy of the response.\n",
    "- Even after engineering the prompt, I sometimes can’t get the model to answer the question outside of my knowledge base.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Conclusion\n",
    "In this assignment, I build a question answering system that can answer questions about my knowledge base (particularly my notes is Markdown), based on the retrieval augmented generation (RAG) framework. I evaluate the system using cosine similarity and qualitative assessment, comparing between my expected answer and the model answer for each of 11 questions. The system performs well on questions that contain keywords in the metadata, but less well on questions that require summarization across multiple documents. Further, the system performs poorly on questions that require the LLM to infer based on its knowledge.\n",
    "\n",
    "## Limitations & Future work directions\n",
    "\n",
    "- The system is not robust to questions that require the LLM to infer based on its knowledge. I can engineer the prompt further to improve the performance.\n",
    "- I will improve further on this RAG pipeline, by incorporating more advanced techniques such as big-to-small retrieval, LLM-based reranking and LLM-based retrieval.\n",
    "- Although I do iterate this pipeline by changing prompt and hyperparameters such as context window size, I did not have a systemized approach to do so. Additionally, cosine similarity is not a good sole metric to evaluate the performance of the system. In the next assignment, I will use LLM-powered `trulens_eval` to evaluate and record the performance of the system for each set of hyperparameters.\n",
    "- I will experiment with an open-source, local LLM, to avoid sending private data to OpenAI \n",
    "- I focus on explaining the math behind transformer and only go over the high-level details of retriever. In the next assignment, I plan to elaborate more on the math behind the RAG framework, especially the retrieval model and the reranking model.\n",
    "- And if I have more time, I will add more data to my knowledge base, such as PDFs (assignments & textbooks)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Executive summary\n",
    "\n",
    "In this assignment, I built a question answering system that can answer questions about my knowledge base. The system is built on the retrieval augmented generation (RAG) framework, which combines the strengths of a knowledge base with those of an LLM to produce more accurate and contextually relevant responses. I justify the advantages of RAG over fine-tuning, since I have a growing knowledge base. I then explain in details the building blocks for two components of RAG, retriever and generator, including the concepts of embedding, vector database, context window, transformer architecture, and attention mechanism. \n",
    "\n",
    "After building the RAG pipeline, I generate 11 question-answer pairs, and evaluate the model answers using cosine similarity and qualitative assessment. It achieves an average cosine similarity of 0.58; however, this performance varies among different types of questions. The system performs well on questions that contain keywords in the metadata, but less well on questions that require summarization across multiple documents. Further, it performs poorly on questions that require the LLM to infer based on its knowledge. I discuss the limitations of the system and future work directions to improve the performance."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# AI Statement\n",
    "\n",
    "I use Github Copilot as the pair programmer for basic syntax (RAG pipeline is beyond its training scope). I also use a combination of external source and perplexity.ai to write the explanation for the transformer architecture and attention mechanism. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# References\n",
    "Alammar, J. (2019). _The illustrated transformer_. http://jalammar.github.io/illustrated-transformer/\n",
    "\n",
    "AssemblyAI. (2021, November 27). _Transformers for beginners | What are they and how do they work_ [Video]. YouTube. https://www.youtube.com/watch?v=_UVfwBqcnbM\n",
    "\n",
    "Lewis, P., Perez, E., Piktus, A., Petroni, F., Karpukhin, V., Goyal, N., Küttler, H., Lewis, M., Yih, W., Rocktäschel, T., Riedel, S., & Kiela, D. (2020). Retrieval-Augmented Generation for Knowledge-Intensive NLP tasks. _arXiv (Cornell University)_. https://arxiv.org/pdf/2005.11401\n",
    "\n",
    "Liu, J. (2023, August 16). Using LLM’s for retrieval and reranking - LlamaIndex blog. _Medium_. https://blog.llamaindex.ai/using-llms-for-retrieval-and-reranking-23cf2d3a14b6\n",
    "\n",
    "Liu, J., & Datta, A. (2023, November 29). _Building and Evaluating Advanced RAG with LlamaIndex and TruEra_. deeplearning.ai. https://www.deeplearning.ai/short-courses/building-evaluating-advanced-rag/\n",
    "\n",
    "LlaMaIndex. (n.d.). _LlaMAIndex 0.9.9_. LlaMAIndex Documentation. https://docs.llamaindex.ai/en/stable\n",
    "\n",
    "NexaNodeLabs. (2023, October 30). Fine-Tuning vs RAG: Choosing the Right Approach for Your Question-Answering System. _Medium_. https://medium.com/@nexanodelabs/fine-tuning-vs-rag-choosing-the-right-approach-for-your-question-answering-system-7c7fbf5d4131\n",
    "\n",
    "Raschka, S. (2023, February 9). _Understanding and coding the Self-Attention Mechanism of large Language models from scratch_. Sebastian Raschka, PhD. https://sebastianraschka.com/blog/2023/self-attention-from-scratch.html\n",
    "\n",
    "SBERT. (2022). _Retrieve & Re-Rank_. Sentence-Transformers Documentation. https://www.sbert.net/examples/applications/retrieve_rerank/README.html\n",
    "\n",
    "SBERT. (2023). _sentence-transformers/all-mpnet-base-v2_. Hugging Face. https://huggingface.co/sentence-transformers/all-mpnet-base-v2\n",
    "\n",
    "Serrano, L., [Serrano.Academy]. (2023, August 31). _The math behind Attention: Keys, Queries, and Values matrices_ [Video]. YouTube. https://www.youtube.com/watch?v=UPtG_38Oq8o\n",
    "\n",
    "Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, Ł., & Polosukhin, I. (2017). Attention is All you Need. _arXiv (Cornell University)_, _30_, 5998–6008. https://arxiv.org/pdf/1706.03762v5\n",
    "\n",
    "Weng, L. (2023, January 27). The Transformer Family Version 2.0. _Lil’Log_. https://lilianweng.github.io/posts/2023-01-27-the-transformer-family-v2/\n",
    "\n",
    "Yan, E. (2023, July 30). _Patterns for building LLM-based systems & products_. eugeneyan.com. https://eugeneyan.com/writing/llm-patterns/#retrieval-augmented-generation-to-add-knowledge\n"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "venv",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.12"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}