pyvene_101.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "ba6c7e19",
   "metadata": {},
   "source": [
    "# Introduction to pyvene\n",
    "This tutorial shows simple runnable code snippets of how to do different kinds of interventions on neural networks with pyvene."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9d6994fa",
   "metadata": {},
   "source": [
    "[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/stanfordnlp/pyvene/blob/main/pyvene_101.ipynb)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "d123a2ba",
   "metadata": {},
   "outputs": [],
   "source": [
    "__author__ = \"Zhengxuan Wu\"\n",
    "__version__ = \"02/01/2024\""
   ]
  },
  {
   "cell_type": "markdown",
   "id": "26298448-91eb-4cad-85bf-ec5fef436e1d",
   "metadata": {},
   "source": [
    " # Table of Contents  \n",
    "1. [Set-up](#Set-up)     \n",
    "1. [pyvene 101](#pyvene-101) \n",
    "    1. [Get Attention Weights](#Get-Attention-Weights)\n",
    "        1. [with String Access](#Get-Attention-Weights-with-Direct-Access-String)\n",
    "        1. [with 1-Line Function](#Get-Attention-Weights-with-a-Function)\n",
    "    1. [Set Activations to Zeros](#Set-Activation-to-Zeros) \n",
    "        1. [with Lambda Expression](#Set-Activation-to-Zeros-with-a-Lambda-Expression)\n",
    "    1. [Set Activations with Subspaces](#Set-Activations-to-Zeros-with-Subspaces)\n",
    "    1. [Interchange Intervention](#Interchange-Interventions)\n",
    "    1. [Intervention Config](#Intervention-Configuration)\n",
    "    1. [Addition Intervention](#Addition-Intervention)\n",
    "    1. [Trainable Intervention](#Trainable-Intervention)\n",
    "    1. [Activation Collection](#Activation-Collection-with-Intervention)\n",
    "    1. [Activation Collection with Other Intervention](#Activation-Collection-at-Downstream-of-a-Intervened-Model)\n",
    "    1. [Intervene Single Neuron](#Intervene-on-a-Single-Neuron)\n",
    "    1. [Add New Intervention Type](#Add-New-Intervention-Type)\n",
    "    1. [Intervene on Recurrent NNs](#Recurrent-NNs-(Intervene-a-Specific-Timestep))\n",
    "    1. [Intervene across Times with RNNs](#Recurrent-NNs-(Intervene-cross-Time))\n",
    "    1. [Intervene on LM Generation](#LMs-Generation)\n",
    "    1. [Advanced Intervention on LM Generation (Model Steering)](#Advanced-Intervention-on-LMs-Generation-(Model-Steering))\n",
    "    1. [Debiasing with Backpack LMs](#Debiasing-with-Backpack-LMs)\n",
    "    1. [Saving and Loading](#Saving-and-Loading)\n",
    "    1. [Multi-Source Intervention (Parallel)](#Multi-Source-Interchange-Intervention-(Parallel-Mode))\n",
    "    1. [Multi-Source Intervention (Serial)](#Multi-Source-Interchange-Intervention-(Serial-Mode))\n",
    "    1. [Multi-Source Intervention with Subspaces (Parallel)](#Multi-Source-Interchange-Intervention-with-Subspaces-(Parallel-Mode))\n",
    "    1. [Multi-Source Intervention with Subspaces (Serial)](#Multi-Source-Interchange-Intervention-with-Subspaces-(Serial-Mode))\n",
    "    1. [Interchange Intervention Training](#Interchange-Intervention-Training-(IIT))\n",
    "1. [pyvene 102](#pyvene-102)\n",
    "    1. [Intervention Grouping](#Grouping)\n",
    "    1. [Intervention Skipping](#Intervention-Skipping-in-Runtime)\n",
    "    1. [Subspace Partition](#Subspace-Partition)\n",
    "    1. [Intervention Linking](#Intervention-Linking)\n",
    "    1. [Add New Model Type](#Add-New-Model-Type)\n",
    "    1. [Path Patching](#Composing-Complex-Intervention-Schema:-Path-Patching)\n",
    "    1. [Causal Tracing](#Composing-Complex-Intervention-Schema:-Causal-Tracing-in-15-lines)\n",
    "    1. [Inference-time Intervention](#Inference-time-Intervention)\n",
    "    1. [IntervenableModel from HuggingFace Directly](#IntervenableModel-from-HuggingFace-Directly)\n",
    "    1. [Path Patching with DAS](#Path-Patching-with-Trainable-Interventions)\n",
    "    1. [Intervene ResNet with Lambda Functors](#Intervene-on-ResNet-with-Lambda-Functions)\n",
    "    1. [Intervene ResNet with 1-line DAS Lambda](#Intervene-on-ResNet-with-Trainable-Lambda-Functions)\n",
    "    2. [Run pyvene on NDIF backend](#Run-pyvene-on-NDIF-backend-with-pv.build_intervenable_model(...))\n",
    "1. [The End](#The-End)\n",
    "    "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0706e21b",
   "metadata": {},
   "source": [
    "## Set-up"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "e08304ea",
   "metadata": {},
   "outputs": [],
   "source": [
    "try:\n",
    "    # This library is our indicator that the required installs\n",
    "    # need to be done.\n",
    "    import pyvene\n",
    "\n",
    "except ModuleNotFoundError:\n",
    "    !pip install git+https://github.com/stanfordnlp/pyvene.git"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0ede4f94",
   "metadata": {},
   "source": [
    "## pyvene 101\n",
    "Before we get started, here are a couple of core notations that are used in this library:\n",
    "- **Base** example: this is the example we are intervening on, or, we are intervening on the computation graph of the model running the **Base** example.\n",
    "- **Source** example or representations: this is the source of our intervention. We use **Source** to intervene on **Base**.\n",
    "- **component**: this is the `nn.module` we are intervening in a pytorch-based NN. For models supported by this library, you can use directly access via str, or use the abstract names defined in the config file (e.g., `h[0].mlp.output` or `mlp_output` with other fields). \n",
    "- **unit**: this is the axis of our intervention. If we say our **unit** is `pos` (`position`), then you are intervening on each token position.\n",
    "- **unit_locations**: this list gives you the percisely location of your intervention. It is the locations of the unit of analysis you are specifying. For instance, if your `unit` is `pos`, and your `unit_location` is 3, then it means you are intervening on the third token. If this field is left as `None`, then no selection will be taken, i.e., you can think of you are getting the raw tensor and you can do whatever you want.\n",
    "- **intervention_type** or **intervention**: this field specifies the intervention you can perform. It can be a primitive type, or it can be a function or a lambda expression for simple interventions. One benefit of using primitives is speed and systematic training schemes. You can also save and load interventions if you use the supported primitives."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7245643b-fd44-47a5-a189-ce1565da7e25",
   "metadata": {},
   "source": [
    "### Get Attention Weights"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "17c7f2f6-b0d3-4fe2-8e4f-c044b93f3ef0",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/u/nlp/anaconda/main/anaconda3/envs/wuzhengx-310/lib/python3.10/site-packages/transformers/utils/hub.py:124: FutureWarning: Using `TRANSFORMERS_CACHE` is deprecated and will be removed in v5 of Transformers. Use `HF_HOME` instead.\n",
      "  warnings.warn(\n"
     ]
    }
   ],
   "source": [
    "import pyvene as pv\n",
    "from transformers import AutoTokenizer, AutoModelForCausalLM\n",
    "\n",
    "model_name = \"gpt2\"\n",
    "# Do not use SDPA attention because we cannot hook to attn_dropout\n",
    "gpt2 = AutoModelForCausalLM.from_pretrained(model_name, attn_implementation=\"eager\")\n",
    "tokenizer = AutoTokenizer.from_pretrained(model_name)\n",
    "\n",
    "pv_gpt2 = pv.IntervenableModel({\n",
    "    \"layer\": 10,\n",
    "    \"component\": \"attention_weight\",\n",
    "    \"intervention_type\": pv.CollectIntervention}, model=gpt2)\n",
    "\n",
    "base = \"When John and Mary went to the shops, Mary gave the bag to\"\n",
    "collected_attn_w = pv_gpt2(\n",
    "    base = tokenizer(base, return_tensors=\"pt\"\n",
    "    ), unit_locations={\"base\": [h for h in range(12)]}\n",
    ")[0][-1][0]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cee8d393-1676-45e2-8aa7-228343d3b13b",
   "metadata": {},
   "source": [
    "#### Get Attention Weights with Direct Access String"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "128be2dd-f089-4291-bfc5-7002d031b1e9",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "loaded model\n"
     ]
    }
   ],
   "source": [
    "import torch\n",
    "import pyvene as pv\n",
    "\n",
    "# gpt2 helper loading model from HuggingFace\n",
    "_, tokenizer, gpt2 = pv.create_gpt2()\n",
    "\n",
    "pv_gpt2 = pv.IntervenableModel({\n",
    "    # based on the module printed above, you can access via string, input means the input to the module\n",
    "    \"component\": \"h[10].attn.attn_dropout.input\",\n",
    "    # you can also initialize the intervention outside\n",
    "    \"intervention\": pv.CollectIntervention()}, model=gpt2)\n",
    "\n",
    "base = \"When John and Mary went to the shops, Mary gave the bag to\"\n",
    "collected_attn_w = pv_gpt2(\n",
    "    base = tokenizer(base, return_tensors=\"pt\"\n",
    "    ), unit_locations={\"base\": [h for h in range(12)]}\n",
    ")[0][-1][0]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "22643b2d",
   "metadata": {},
   "source": [
    "#### Get Attention Weights with a Function"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "id": "678dc46f",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "loaded model\n"
     ]
    }
   ],
   "source": [
    "import torch\n",
    "import copy\n",
    "import pyvene as pv\n",
    "\n",
    "_, tokenizer, gpt2 = pv.create_gpt2()\n",
    "\n",
    "cached_w = {}\n",
    "def pv_patcher(b, s): cached_w[\"attn_w\"] = copy.deepcopy(b.data)\n",
    "\n",
    "pv_gpt2 = pv.IntervenableModel({\n",
    "    \"component\": \"h[10].attn.attn_dropout.input\", \n",
    "    \"intervention\": pv_patcher}, model=gpt2)\n",
    "\n",
    "base = \"When John and Mary went to the shops, Mary gave the bag to\"\n",
    "_ = pv_gpt2(tokenizer(base, return_tensors=\"pt\"))\n",
    "torch.allclose(collected_attn_w, cached_w[\"attn_w\"].unsqueeze(dim=0))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "12c5addb-4bd7-4129-b350-0677774f5790",
   "metadata": {},
   "source": [
    "### Set Activation to Zeros"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "a82664f9",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "loaded model\n"
     ]
    }
   ],
   "source": [
    "import torch\n",
    "import pyvene as pv\n",
    "\n",
    "_, tokenizer, gpt2 = pv.create_gpt2()\n",
    "\n",
    "# define the component to zero-out\n",
    "pv_gpt2 = pv.IntervenableModel({\n",
    "    \"layer\": 0, \"component\": \"mlp_output\",\n",
    "    \"source_representation\": torch.zeros(gpt2.config.n_embd)\n",
    "}, model=gpt2)\n",
    "# run the intervened forward pass\n",
    "intervened_outputs = pv_gpt2(\n",
    "    base = tokenizer(\"The capital of Spain is\", return_tensors=\"pt\"), \n",
    "    # we define the intervening token dynamically\n",
    "    unit_locations={\"base\": 3},\n",
    "    output_original_output=True # False then the first element in the tuple is None\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b9c11cb9",
   "metadata": {},
   "source": [
    "#### Set Activation to Zeros with a Lambda Expression"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "7627dc32",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "loaded model\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "True"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import torch\n",
    "import pyvene as pv\n",
    "\n",
    "_, tokenizer, gpt2 = pv.create_gpt2()\n",
    "\n",
    "# indices are specified in the intervention\n",
    "mask = torch.ones(1, 5, 768)\n",
    "mask[:,3,:] = 0.\n",
    "# define the component to zero-out\n",
    "pv_gpt2 = pv.IntervenableModel({\n",
    "    \"component\": \"h[0].mlp.output\", \"intervention\": lambda b, s: b*mask\n",
    "}, model=gpt2)\n",
    "# run the intervened forward pass\n",
    "intervened_outputs_fn = pv_gpt2(\n",
    "    base = tokenizer(\"The capital of Spain is\", return_tensors=\"pt\")\n",
    ")\n",
    "torch.allclose(\n",
    "    intervened_outputs[1].last_hidden_state, \n",
    "    intervened_outputs_fn[1].last_hidden_state\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "72363777",
   "metadata": {},
   "source": [
    "#### Set Activation to Zeros with a Lambda Expression and Subspace notation"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "d86c06f0",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "loaded model\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "True"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import torch\n",
    "import pyvene as pv\n",
    "\n",
    "_, tokenizer, gpt2 = pv.create_gpt2()\n",
    "\n",
    "# indices are specified in the intervention\n",
    "\n",
    "def pv_patcher(b, s, sp): \n",
    "    mask = torch.ones(1, 5, 768)\n",
    "    mask[:,sp[0][0],:] = 0.\n",
    "    return b*mask\n",
    "\n",
    "# define the component to zero-out\n",
    "pv_gpt2 = pv.IntervenableModel({\n",
    "    \"component\": \"h[0].mlp.output\", \"intervention\": pv_patcher\n",
    "}, model=gpt2)\n",
    "# run the intervened forward pass\n",
    "intervened_outputs_fn = pv_gpt2(\n",
    "    base = tokenizer(\"The capital of Spain is\", return_tensors=\"pt\"),\n",
    "    subspaces=3,\n",
    ")\n",
    "torch.allclose(\n",
    "    intervened_outputs[1].last_hidden_state, \n",
    "    intervened_outputs_fn[1].last_hidden_state\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "39071858",
   "metadata": {},
   "source": [
    "### Set Activations to Zeros with Subspaces\n",
    "The notion of subspace means the actual dimensions you are intervening. If we have a representation in a size of 512, the first 128 activation values are its subspace activations."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "b7896c3b",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "loaded model\n",
      "Directory './tmp/' already exists.\n"
     ]
    }
   ],
   "source": [
    "import torch\n",
    "import pyvene as pv\n",
    "# built-in helper to get a HuggingFace model\n",
    "_, tokenizer, gpt2 = pv.create_gpt2()\n",
    "# create with dict-based config\n",
    "pv_config = pv.IntervenableConfig({\n",
    "  \"layer\": 0, \"component\": \"mlp_output\"})\n",
    "#initialize model\n",
    "pv_gpt2 = pv.IntervenableModel(pv_config, model=gpt2)\n",
    "# run an intervened forward pass\n",
    "intervened_outputs = pv_gpt2(\n",
    "  # the intervening base input\n",
    "  base=tokenizer(\"The capital of Spain is\", return_tensors=\"pt\"), \n",
    "  # the location to intervene at (3rd token)\n",
    "  unit_locations={\"base\": 3},\n",
    "  # the individual dimensions targetted\n",
    "  subspaces=[10,11,12],\n",
    "  source_representations=torch.zeros(gpt2.config.n_embd)\n",
    ")\n",
    "# sharing\n",
    "pv_gpt2.save(\"./tmp/\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1410904d",
   "metadata": {},
   "source": [
    "### Interchange Interventions\n",
    "Instead of a static vector, we can intervene the model with activations sampled from a different forward run. We call this interchange intervention, where intervention happens between two examples and we are interchanging activations between them."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "9691c7d8",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "loaded model\n"
     ]
    }
   ],
   "source": [
    "import torch\n",
    "import pyvene as pv\n",
    "# built-in helper to get a HuggingFace model\n",
    "_, tokenizer, gpt2 = pv.create_gpt2()\n",
    "# create with dict-based config\n",
    "pv_config = pv.IntervenableConfig({\n",
    "  \"layer\": 0,\n",
    "  \"component\": \"mlp_output\"},\n",
    "  intervention_types=pv.VanillaIntervention\n",
    ")\n",
    "#initialize model\n",
    "pv_gpt2 = pv.IntervenableModel(\n",
    "  pv_config, model=gpt2)\n",
    "# run an interchange intervention \n",
    "intervened_outputs = pv_gpt2(\n",
    "  # the base input\n",
    "  base=tokenizer(\n",
    "    \"The capital of Spain is\", \n",
    "    return_tensors = \"pt\"), \n",
    "  # the source input\n",
    "  sources=tokenizer(\n",
    "    \"The capital of Italy is\", \n",
    "    return_tensors = \"pt\"), \n",
    "  # the location to intervene at (3rd token)\n",
    "  unit_locations={\"sources->base\": 3},\n",
    "  # the individual dimensions targeted\n",
    "  subspaces=[10,11,12]\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c890fda4",
   "metadata": {},
   "source": [
    "### Intervention Configuration\n",
    "You can also initialize the config without the lazy dictionary passing by enabling more options, e.g., the mode of these interventions are executed."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "4faa3e41",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "loaded model\n",
      "IntervenableConfig\n",
      "{\n",
      "    \"model_type\": \"None\",\n",
      "    \"representations\": [\n",
      "        {\n",
      "            \"layer\": 0,\n",
      "            \"component\": \"mlp_output\",\n",
      "            \"unit\": \"pos\",\n",
      "            \"max_number_of_units\": 1,\n",
      "            \"low_rank_dimension\": null,\n",
      "            \"intervention_type\": null,\n",
      "            \"intervention\": null,\n",
      "            \"subspace_partition\": null,\n",
      "            \"group_key\": null,\n",
      "            \"intervention_link_key\": null,\n",
      "            \"moe_key\": null,\n",
      "            \"source_representation\": \"PLACEHOLDER\",\n",
      "            \"hidden_source_representation\": null\n",
      "        },\n",
      "        {\n",
      "            \"layer\": 1,\n",
      "            \"component\": \"mlp_output\",\n",
      "            \"unit\": \"pos\",\n",
      "            \"max_number_of_units\": 1,\n",
      "            \"low_rank_dimension\": null,\n",
      "            \"intervention_type\": null,\n",
      "            \"intervention\": null,\n",
      "            \"subspace_partition\": null,\n",
      "            \"group_key\": null,\n",
      "            \"intervention_link_key\": null,\n",
      "            \"moe_key\": null,\n",
      "            \"source_representation\": \"PLACEHOLDER\",\n",
      "            \"hidden_source_representation\": null\n",
      "        },\n",
      "        {\n",
      "            \"layer\": 2,\n",
      "            \"component\": \"mlp_output\",\n",
      "            \"unit\": \"pos\",\n",
      "            \"max_number_of_units\": 1,\n",
      "            \"low_rank_dimension\": null,\n",
      "            \"intervention_type\": null,\n",
      "            \"intervention\": null,\n",
      "            \"subspace_partition\": null,\n",
      "            \"group_key\": null,\n",
      "            \"intervention_link_key\": null,\n",
      "            \"moe_key\": null,\n",
      "            \"source_representation\": \"PLACEHOLDER\",\n",
      "            \"hidden_source_representation\": null\n",
      "        },\n",
      "        {\n",
      "            \"layer\": 3,\n",
      "            \"component\": \"mlp_output\",\n",
      "            \"unit\": \"pos\",\n",
      "            \"max_number_of_units\": 1,\n",
      "            \"low_rank_dimension\": null,\n",
      "            \"intervention_type\": null,\n",
      "            \"intervention\": null,\n",
      "            \"subspace_partition\": null,\n",
      "            \"group_key\": null,\n",
      "            \"intervention_link_key\": null,\n",
      "            \"moe_key\": null,\n",
      "            \"source_representation\": \"PLACEHOLDER\",\n",
      "            \"hidden_source_representation\": null\n",
      "        }\n",
      "    ],\n",
      "    \"intervention_types\": \"<class 'pyvene.models.interventions.VanillaIntervention'>\",\n",
      "    \"mode\": \"parallel\",\n",
      "    \"sorted_keys\": \"None\",\n",
      "    \"intervention_dimensions\": \"None\"\n",
      "}\n"
     ]
    }
   ],
   "source": [
    "import torch\n",
    "import pyvene as pv\n",
    "\n",
    "_, tokenizer, gpt2 = pv.create_gpt2()\n",
    "# standalone configuration object\n",
    "config = pv.IntervenableConfig([\n",
    "    {\n",
    "        \"layer\": _,\n",
    "        \"component\": \"mlp_output\",\n",
    "        \"source_representation\": torch.zeros(\n",
    "            gpt2.config.n_embd)\n",
    "    } for _ in range(4)],\n",
    "    mode=\"parallel\"\n",
    ")\n",
    "# this object is serializable\n",
    "print(config)\n",
    "pv_gpt2 = pv.IntervenableModel(config, model=gpt2)\n",
    "\n",
    "intervened_outputs = pv_gpt2(\n",
    "    base = tokenizer(\"The capital of Spain is\", return_tensors=\"pt\"), \n",
    "    unit_locations={\"base\": 3}\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9c5b2270",
   "metadata": {},
   "source": [
    "### Addition Intervention\n",
    "Activation swap is one kind of interventions we can perform. Here is another simple one: `pv.AdditionIntervention`, which adds the sampled representation into the **Base** run."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "a40f5989",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "loaded model\n"
     ]
    }
   ],
   "source": [
    "import torch\n",
    "import pyvene as pv\n",
    "\n",
    "_, tokenizer, gpt2 = pv.create_gpt2()\n",
    "\n",
    "config = pv.IntervenableConfig({\n",
    "    \"layer\": 0,\n",
    "    \"component\": \"mlp_input\"},\n",
    "    pv.AdditionIntervention\n",
    ")\n",
    "\n",
    "pv_gpt2 = pv.IntervenableModel(config, model=gpt2)\n",
    "\n",
    "intervened_outputs = pv_gpt2(\n",
    "    base = tokenizer(\n",
    "        \"The Space Needle is in downtown\", \n",
    "        return_tensors=\"pt\"\n",
    "    ), \n",
    "    unit_locations={\"base\": [[[0, 1, 2, 3]]]},\n",
    "    source_representations = torch.rand(gpt2.config.n_embd)\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "099ddf77",
   "metadata": {},
   "source": [
    "### Trainable Intervention\n",
    "Interventions can contain trainable parameters, and hook-up with the model to receive gradients end-to-end. They are often useful in searching for an particular interpretation of the representation.\n",
    "\n",
    "The following example does a single step gradient calculation to push the model to generate `Rome` after the intervention. If we can train such intervention at scale with low loss, it means you have a causal grab onto your model. In terms of interpretability, that means, somehow you find a representation (not the original one since its trained) that maps onto the `capital` output."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "7f058ecd",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "loaded model\n"
     ]
    }
   ],
   "source": [
    "import torch\n",
    "import pyvene as pv\n",
    "\n",
    "_, tokenizer, gpt2 = pv.create_gpt2()\n",
    "\n",
    "das_config = pv.IntervenableConfig({\n",
    "    \"layer\": 8,\n",
    "    \"component\": \"block_output\",\n",
    "    \"low_rank_dimension\": 1},\n",
    "    # this is a trainable low-rank rotation\n",
    "    pv.LowRankRotatedSpaceIntervention\n",
    ")\n",
    "\n",
    "das_gpt2 = pv.IntervenableModel(das_config, model=gpt2)\n",
    "\n",
    "last_hidden_state = das_gpt2(\n",
    "    base = tokenizer(\n",
    "        \"The capital of Spain is\", \n",
    "        return_tensors=\"pt\"\n",
    "    ), \n",
    "    sources = tokenizer(\n",
    "        \"The capital of Italy is\", \n",
    "        return_tensors=\"pt\"\n",
    "    ), \n",
    "    unit_locations={\"sources->base\": 3}\n",
    ")[-1].last_hidden_state[:,-1]\n",
    "\n",
    "# golden counterfacutual label as Rome\n",
    "label = tokenizer.encode(\n",
    "    \" Rome\", return_tensors=\"pt\")\n",
    "logits = torch.matmul(\n",
    "    last_hidden_state, gpt2.wte.weight.t())\n",
    "\n",
    "m = torch.nn.CrossEntropyLoss()\n",
    "loss = m(logits, label.view(-1))\n",
    "loss.backward()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a8fd2b8e",
   "metadata": {},
   "source": [
    "### Activation Collection with Intervention\n",
    "You can also collect activations with our provided `pv.CollectIntervention` intervention. More importantly, this can be used interchangably with other interventions. You can collect something from an intervened model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "6e6bd585",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "loaded model\n"
     ]
    }
   ],
   "source": [
    "import torch\n",
    "import pyvene as pv\n",
    "\n",
    "_, tokenizer, gpt2 = pv.create_gpt2()\n",
    "\n",
    "config = pv.IntervenableConfig({\n",
    "    \"layer\": 10,\n",
    "    \"component\": \"block_output\",\n",
    "    \"intervention_type\": pv.CollectIntervention}\n",
    ")\n",
    "\n",
    "pv_gpt2 = pv.IntervenableModel(\n",
    "    config, model=gpt2)\n",
    "\n",
    "collected_activations = pv_gpt2(\n",
    "    base = tokenizer(\n",
    "        \"The capital of Spain is\", \n",
    "        return_tensors=\"pt\"\n",
    "    ), unit_locations={\"sources->base\": 3}\n",
    ")[0][-1]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f7b0d0c6",
   "metadata": {},
   "source": [
    "### Activation Collection at Downstream of a Intervened Model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "adcfcb05",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "loaded model\n"
     ]
    }
   ],
   "source": [
    "import torch\n",
    "import pyvene as pv\n",
    "\n",
    "_, tokenizer, gpt2 = pv.create_gpt2()\n",
    "\n",
    "config = pv.IntervenableConfig({\n",
    "    \"layer\": 8,\n",
    "    \"component\": \"block_output\",\n",
    "    \"intervention_type\": pv.VanillaIntervention}\n",
    ")\n",
    "\n",
    "config.add_intervention({\n",
    "    \"layer\": 10,\n",
    "    \"component\": \"block_output\",\n",
    "    \"intervention_type\": pv.CollectIntervention})\n",
    "\n",
    "pv_gpt2 = pv.IntervenableModel(\n",
    "    config, model=gpt2)\n",
    "\n",
    "collected_activations = pv_gpt2(\n",
    "    base = tokenizer(\n",
    "        \"The capital of Spain is\", \n",
    "        return_tensors=\"pt\"\n",
    "    ), \n",
    "    sources = [tokenizer(\n",
    "        \"The capital of Italy is\", \n",
    "        return_tensors=\"pt\"\n",
    "    ), None], unit_locations={\"sources->base\": 3}\n",
    ")[0][-1]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a9e6e4d9",
   "metadata": {},
   "source": [
    "### Intervene on a Single Neuron\n",
    "We want to provide a good user interface so that interventions can be done easily by people with less pytorch or programming experience. Meanwhile, we also want to be flexible and provide the depth of control required for highly specific tasks. Here is an example where we intervene on a specific neuron at a specific head of a layer in a model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "d25b6401",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "loaded model\n"
     ]
    }
   ],
   "source": [
    "import torch\n",
    "import pyvene as pv\n",
    "\n",
    "_, tokenizer, gpt2 = pv.create_gpt2()\n",
    "\n",
    "config = pv.IntervenableConfig({\n",
    "    \"layer\": 8,\n",
    "    \"component\": \"head_attention_value_output\",\n",
    "    \"unit\": \"h.pos\",\n",
    "    \"intervention_type\": pv.CollectIntervention}\n",
    ")\n",
    "\n",
    "pv_gpt2 = pv.IntervenableModel(\n",
    "    config, model=gpt2)\n",
    "\n",
    "collected_activations = pv_gpt2(\n",
    "    base = tokenizer(\n",
    "        \"The capital of Spain is\", \n",
    "        return_tensors=\"pt\"\n",
    "    ), \n",
    "    unit_locations={\n",
    "        # GET_LOC is a helper.\n",
    "        # (3,3) means head 3 position 3\n",
    "        \"base\": pv.GET_LOC((3,3))\n",
    "    },\n",
    "    # the notion of subspace is used to target neuron 0.\n",
    "    subspaces=[0]\n",
    ")[0][-1]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5692bc15",
   "metadata": {},
   "source": [
    "### Add New Intervention Type"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "id": "1597221a",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "loaded model\n"
     ]
    }
   ],
   "source": [
    "import torch\n",
    "import pyvene as pv\n",
    "\n",
    "_, tokenizer, gpt2 = pv.create_gpt2()\n",
    "\n",
    "class MultiplierIntervention(\n",
    "  pv.ConstantSourceIntervention):\n",
    "    def __init__(self, **kwargs):\n",
    "        super().__init__()\n",
    "    def forward(\n",
    "    self, base, source=None, subspaces=None):\n",
    "        return base * 99.0\n",
    "# run with new intervention type\n",
    "pv_gpt2 = pv.IntervenableModel({\n",
    "  \"intervention_type\": MultiplierIntervention}, \n",
    "  model=gpt2)\n",
    "intervened_outputs = pv_gpt2(\n",
    "  base = tokenizer(\"The capital of Spain is\", \n",
    "    return_tensors=\"pt\"), \n",
    "  unit_locations={\"base\": 3})"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "079050f6",
   "metadata": {},
   "source": [
    "### Recurrent NNs (Intervene a Specific Timestep)\n",
    "Existing intervention libraries focus on Transformer models. They often lack of supports for GRUs, LSTMs or any state-space model. The fundemental problem is in the hook mechanism provided by PyTorch. Hook is attached to a module before runtime. Models like GRUs will lead to undesired callback from the hook as there is no notion of state or time of the hook. \n",
    "\n",
    "We make our hook stateful, so you can intervene on recurrent NNs like GRUs. This notion of time will become useful when intervening on Transformers yet want to unroll the causal effect during generation as well."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "id": "7a53347a",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "loaded model\n"
     ]
    }
   ],
   "source": [
    "import torch\n",
    "import pyvene as pv\n",
    "\n",
    "_, _, gru = pv.create_gru_classifier(\n",
    "    pv.GRUConfig(h_dim=32))\n",
    "\n",
    "pv_gru = pv.IntervenableModel({\n",
    "    \"component\": \"cell_output\",\n",
    "    \"unit\": \"t\", \n",
    "    \"intervention_type\": pv.ZeroIntervention},\n",
    "    model=gru)\n",
    "\n",
    "rand_t = torch.rand(1,10, gru.config.h_dim)\n",
    "\n",
    "intervened_outputs = pv_gru(\n",
    "  base = {\"inputs_embeds\": rand_t}, \n",
    "  unit_locations={\"base\": 3})"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "031dd5de",
   "metadata": {},
   "source": [
    "### Recurrent NNs (Intervene cross Time)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "id": "b48166c0",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "loaded model\n"
     ]
    }
   ],
   "source": [
    "import torch\n",
    "import pyvene as pv\n",
    "\n",
    "# built-in helper to get a GRU\n",
    "_, _, gru = pv.create_gru_classifier(\n",
    "    pv.GRUConfig(h_dim=32))\n",
    "# wrap it with config\n",
    "pv_gru = pv.IntervenableModel({\n",
    "    \"component\": \"cell_output\",\n",
    "    # intervening on time\n",
    "    \"unit\": \"t\", \n",
    "    \"intervention_type\": pv.ZeroIntervention},\n",
    "    model=gru)\n",
    "# run an intervened forward pass\n",
    "rand_b = torch.rand(1,10, gru.config.h_dim)\n",
    "rand_s = torch.rand(1,10, gru.config.h_dim)\n",
    "intervened_outputs = pv_gru(\n",
    "  base = {\"inputs_embeds\": rand_b}, \n",
    "  sources = [{\"inputs_embeds\": rand_s}], \n",
    "  # intervening time step\n",
    "  unit_locations={\"sources->base\": (6, 3)})"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "121366c1",
   "metadata": {},
   "source": [
    "### LMs Generation\n",
    "You can also intervene the generation call of LMs. Here is a simple example where we try to add a vector into the MLP output when the model decodes."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "f718e2d6",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Setting `pad_token_id` to `eos_token_id`:50256 for open-end generation.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "loaded model\n",
      "Once upon a time there was a little girl named Lucy. She was three years old and loved to explore. One day, Lucy was walking in the park when she saw a big, red balloon. She was so excited and wanted to play with it.\n",
      "\n",
      "But then, a big, mean man came and said, \"That balloon is mine! You can't have it!\" Lucy was very sad and started to cry.\n",
      "\n",
      "The man said, \"I'm sorry, but I need the balloon for my work. You can have it if you want.\"\n",
      "\n",
      "Lucy was so happy and said, \"Yes please!\" She took the balloon and ran away.\n",
      "\n",
      "But then, the man said, \"Wait! I have an idea. Let's make a deal. If you can guess what I'm going to give you, then you can have the balloon.\"\n",
      "\n",
      "Lucy thought for a moment and then said, \"I guess I'll have to get the balloon.\"\n",
      "\n",
      "The man smiled and said, \"That's a good guess! Here you go.\"\n",
      "\n",
      "Lucy was so happy and thanked the man. She hugged the balloon and ran off to show her mom.\n",
      "\n",
      "The end.\n",
      "\n"
     ]
    }
   ],
   "source": [
    "import torch\n",
    "import pyvene as pv\n",
    "\n",
    "# built-in helper to get tinystore\n",
    "_, tokenizer, tinystory = pv.create_gpt_neo()\n",
    "emb_happy = tinystory.transformer.wte(\n",
    "    torch.tensor(14628)) \n",
    "\n",
    "pv_tinystory = pv.IntervenableModel([{\n",
    "    \"layer\": l,\n",
    "    \"component\": \"mlp_output\",\n",
    "    \"intervention_type\": pv.AdditionIntervention\n",
    "    } for l in range(tinystory.config.num_layers)],\n",
    "    model=tinystory\n",
    ")\n",
    "# prompt and generate\n",
    "prompt = tokenizer(\n",
    "    \"Once upon a time there was\", return_tensors=\"pt\")\n",
    "unintervened_story, intervened_story = pv_tinystory.generate(\n",
    "    prompt, source_representations=emb_happy*0.3, max_length=256\n",
    ")\n",
    "\n",
    "print(tokenizer.decode(\n",
    "    intervened_story[0], \n",
    "    skip_special_tokens=True\n",
    "))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e628990d",
   "metadata": {},
   "source": [
    "intervene on generation with source example passed in. The result will be slightly different since we no longer have a static vector to be added in; it is layerwise addition."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "087541f1",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "The attention mask and the pad token id were not set. As a consequence, you may observe unexpected behavior. Please pass your input's `attention_mask` to obtain reliable results.\n",
      "Setting `pad_token_id` to `eos_token_id`:50256 for open-end generation.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "loaded model\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "The attention mask and the pad token id were not set. As a consequence, you may observe unexpected behavior. Please pass your input's `attention_mask` to obtain reliable results.\n",
      "Setting `pad_token_id` to `eos_token_id`:50256 for open-end generation.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Once upon a time there was a little girl named Lucy. She was very excited because she was going to the park. She wanted to go to the park and play.\n",
      "\n",
      "When she got to the park, she saw a big slide. She was so excited! She ran to the slide and started to climb up. She was so happy.\n",
      "\n",
      "But then she saw something else. It was a big, scary dog. It was a big, mean dog. He was barking and growling at her. Lucy was scared. She didn't know what to do.\n",
      "\n",
      "Suddenly, she heard a voice. It was her mommy. She said, \"Don't worry, Lucy. I will help you. I will protect you.\"\n",
      "\n",
      "Lucy was so happy. She hugged her mommy and they went to the park. They played together and had lots of fun. Lucy was so happy. She was no longer scared.\n",
      "\n"
     ]
    }
   ],
   "source": [
    "import torch\n",
    "import pyvene as pv\n",
    "\n",
    "# built-in helper to get tinystore\n",
    "_, tokenizer, tinystory = pv.create_gpt_neo()\n",
    "\n",
    "def pv_patcher(b, s): return b + s*0.1\n",
    "\n",
    "pv_tinystory = pv.IntervenableModel([{\n",
    "    \"layer\": l,\n",
    "    \"component\": \"mlp_output\",\n",
    "    \"intervention\": pv_patcher\n",
    "    } for l in range(tinystory.config.num_layers)],\n",
    "    model=tinystory\n",
    ")\n",
    "# prompt and generate\n",
    "prompt = tokenizer(\n",
    "    \"Once upon a time there was\", return_tensors=\"pt\")\n",
    "happy_prompt = tokenizer(\n",
    "    \" Happy\", return_tensors=\"pt\")\n",
    "_, intervened_story = pv_tinystory.generate(\n",
    "    prompt, happy_prompt, \n",
    "    unit_locations = {\"sources->base\": 0},\n",
    "    max_length=256\n",
    ")\n",
    "\n",
    "print(tokenizer.decode(\n",
    "    intervened_story[0], \n",
    "    skip_special_tokens=True\n",
    "))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0b89244e-fbc7-4515-b22c-83fae00224cb",
   "metadata": {},
   "source": [
    "### Advanced Intervention on LMs Generation (Model Steering)\n",
    "\n",
    "We also support model steering with interventions during model generation. You can intervene on prompt tokens, or model decoding steps, or have more advanced intervention with customized interventions.\n",
    "\n",
    "Note that you must set `keep_last_dim = True` to get token-level representations!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "id": "43422e38-d930-4354-9dc5-191e2abcf928",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "c046df6ad83d4f6381730fc940f7b866",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Downloading shards:   0%|          | 0/2 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "7ec60913371647fc85e602b189a5c50f",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Loading checkpoint shards:   0%|          | 0/2 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Extracting happy vector ...\n"
     ]
    }
   ],
   "source": [
    "import pyvene as pv\n",
    "from transformers import AutoModelForCausalLM, AutoTokenizer\n",
    "\n",
    "model = AutoModelForCausalLM.from_pretrained(\"google/gemma-2-2b-it\")\n",
    "tokenizer = AutoTokenizer.from_pretrained(\"google/gemma-2-2b-it\")\n",
    "\n",
    "print(\"Extracting happy vector ...\")\n",
    "happy_id = tokenizer(\"happy\")['input_ids'][-1]\n",
    "happy_vector = model.model.embed_tokens.weight[happy_id].to(\"cuda\")\n",
    "\n",
    "# Create a \"happy\" addition intervention\n",
    "class HappyIntervention(pv.ConstantSourceIntervention):\n",
    "    def __init__(self, **kwargs):\n",
    "        super().__init__(\n",
    "            **kwargs, \n",
    "            keep_last_dim=True) # you must set keep_last_dim=True to get tokenized reprs.\n",
    "        self.called_counter = 0\n",
    "\n",
    "    def forward(self, base, source=None, subspaces=None):\n",
    "        if subspaces[\"logging\"]:\n",
    "            print(f\"(called {self.called_counter} times) incoming reprs shape:\", base.shape)\n",
    "        self.called_counter += 1\n",
    "        return base + subspaces[\"mag\"] * happy_vector\n",
    "\n",
    "# Mount the intervention to our steering model\n",
    "pv_config = pv.IntervenableConfig(representations=[{\n",
    "    \"layer\": 20,\n",
    "    \"component\": f\"model.layers[20].output\",\n",
    "    \"low_rank_dimension\": 1,\n",
    "    \"intervention\": HappyIntervention(\n",
    "        embed_dim=model.config.hidden_size, \n",
    "        low_rank_dimension=1)}])\n",
    "pv_model = pv.IntervenableModel(pv_config, model)\n",
    "pv_model.set_device(\"cuda\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "id": "dc70ebae-793a-4b2b-a3e3-a2118cc66e1e",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Asking to truncate to max_length but no maximum length is provided and the model has no predefined maximum length. Default to no truncation.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(called 0 times) incoming reprs shape: torch.Size([1, 17, 2304])\n",
      "(called 1 times) incoming reprs shape: torch.Size([1, 1, 2304])\n",
      "(called 2 times) incoming reprs shape: torch.Size([1, 1, 2304])\n",
      "(called 3 times) incoming reprs shape: torch.Size([1, 1, 2304])\n",
      "(called 4 times) incoming reprs shape: torch.Size([1, 1, 2304])\n",
      "(called 5 times) incoming reprs shape: torch.Size([1, 1, 2304])\n",
      "(called 6 times) incoming reprs shape: torch.Size([1, 1, 2304])\n",
      "(called 7 times) incoming reprs shape: torch.Size([1, 1, 2304])\n",
      "(called 8 times) incoming reprs shape: torch.Size([1, 1, 2304])\n",
      "(called 9 times) incoming reprs shape: torch.Size([1, 1, 2304])\n"
     ]
    }
   ],
   "source": [
    "prompt = \"Write a story for me about dragon.\"\n",
    "\n",
    "prompt = tokenizer.decode(tokenizer.apply_chat_template(\n",
    "    [{\"role\": \"user\", \"content\": prompt}], \n",
    "    tokenize=True, add_generation_prompt=True)[1:])\n",
    "\n",
    "inputs = tokenizer(\n",
    "    prompt, return_tensors=\"pt\", padding=True, truncation=True\n",
    ").to(\"cuda\")\n",
    "_, generations = pv_model.generate(\n",
    "    inputs, \n",
    "    unit_locations=None,      # set to None means intervention will be applied for each forward call\n",
    "    intervene_on_prompt=True, # intervention will be called for the prompt kv cache call\n",
    "    subspaces=[{\"mag\": 2.0, \"logging\": True}], # other metadata\n",
    "    max_new_tokens=10, do_sample=True, temperature=1.0)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "afa622a1-5d93-43e6-afc2-436bd166299d",
   "metadata": {},
   "source": [
    "**Here you go!** You can see that the intervention is called 10 times. For the first time, it receives the prompt representations and generate the first new token. \n",
    "\n",
    "Now, let's generate some steering results."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "id": "4882c229-8529-4a16-bec0-773e77217d27",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "\"Anya's grandmother, Elara, had always said happy souls were the happiest kind – even happy dragons. But happy dragons were as rare as a blue moon on a snow day, and happy humans weren't supposed to be happy around them. Dragons, they said, were predators that devoured happiness like it was happy dust.\\n\\nAnya knew better. Happy was a whisper, a happy breeze on the sun-dappled meadow. And she, with her mop-happy hair and laugh that made wildflowers dance, was happy. So she snuck off to the Forbidden Forest, a place where only brave hearts dared to roam\""
      ]
     },
     "execution_count": 30,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "_, generations = pv_model.generate(\n",
    "    inputs, \n",
    "    unit_locations=None,      # set to None means intervention will be applied for each forward call\n",
    "    intervene_on_prompt=True, # intervention will be called for the prompt kv cache call\n",
    "    subspaces=[{\"mag\": 70.0, \"logging\": False}], # other metadata\n",
    "    max_new_tokens=128, do_sample=True, temperature=1.0)\n",
    "\n",
    "tokenizer.decode(generations[0][inputs[\"input_ids\"].shape[1]:], skip_special_tokens=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3cc17327-ea2d-449f-9f11-e94435b1e734",
   "metadata": {},
   "source": [
    "Great! This is your super-happy model. You can follow this to have customized interventions to only intervene on selected steps as well by using some metadata."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "26d25dc6",
   "metadata": {},
   "source": [
    "### Debiasing with Backpack LMs\n",
    "\n",
    "Models like [Backpack LMs](https://arxiv.org/pdf/2305.16765.pdf) are built with highly interpretable model components. In its original paper, one motivating experiment is using the sense vectors to debias. Here, we try to reproduce one of the experiments in Fig. 3 (pg. 8)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "id": "841e5a5b",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "The argument `trust_remote_code` is to be used with Auto classes. It has no effect here and is ignored.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "loaded model\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAyAAAAGQCAYAAABWJQQ0AAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjcuNCwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8WgzjOAAAACXBIWXMAAB7CAAAewgFu0HU+AABcwUlEQVR4nO3dd3xUVf7/8fdNCOlAwCAlghFCLwKilAhREERRFBVRKWrAsquCFQXFylp21wK6uoCKgCgKAjYElyKihF5FDCAaEEISAoG0Ccnc3x/8Mt+EtDtMMpPJvJ6PB4+dufecez/z2SDzybnnHMM0TVMAAAAA4AZ+ng4AAAAAgO+gAAEAAADgNhQgAAAAANyGAgQAAACA21CAAAAAAHAbChAAAAAAbkMBAgAAAMBtKEAAAAAAuA0FCDziueeek2EYMgxDcXFxTvW95pprZBiG/P39tXPnzqoJsBwbN250xD506FC33x8AAMCb1fJ0AIAzvvrqKy1dulSSdPvtt6tjx45O9f/999+1adMmHTp0SJIUFRWlSy65RBdddJHla3Tv3l033nijFi1apEWLFmnlypW68sornYoDAADAV1GAwGsUFBTo8ccflyT5+fnp2Weftdx37dq1mjBhgn7++edSz/fq1UuvvvqqYmNjLV3vueee06JFiyRJjz76qLZs2SLDMCzHAwAA4Kt4BAteY+7cufrtt98kSTfeeKNatmxpqd8rr7yivn37lll8SNLPP/+suLg4vfrqq5au2alTJw0cOFCStG3bNn3xxReW+gEAAPg6ChB4BdM0NWXKFMf7hx9+2FK/WbNm6amnnpLdbpck1a5dW6NGjdK0adP01ltvacSIEapdu7akMyMsTz75pD766CNL1y4aw4svvmj1owAAAPg0HsGCV/j666+1d+9eSVLbtm3Vu3fvCvskJSXpvvvuc7y/4IILtGzZMrVt27ZYu6eeekpXX321Dh48KEm69957deWVV+qCCy4o9/pXXXWVLrjgAh08eFDbt2/X6tWrnZ5QDwAA4GsYAYFXePvttx2vR48ebanPCy+8IJvNJkny9/fXwoULSxQfktSuXTstWLBA/v7+kiSbzaYXXnihwuv7+flp5MiRjvfvvPOOpbgAAAB8mWGapunpIGBNamqq1qxZo0OHDiknJ0fR0dHq16+fzjvvvDL7HDlyRGvWrNGff/4pPz8/NWvWTAMGDFC9evXcF3gpnnvuOT3//POSpL59+2r16tVltj1y5IiioqIcj1ElJiYqJiam3OufOHFCDRs21OnTpyVJd9xxh+bOnVtunxEjRujjjz+WJAUEBCg1NVV169Ytt8/mzZt1ySWXSJICAwOVnJzs8dwCAABUZ4yAVDN33nmnY4+JO++8U5KUlpam2267TU2bNtXNN9+s8ePH66mnntLw4cMVFRWlCRMmKD8/v9h1/vrrLw0bNkxRUVEaPny4JkyYoMcff1y33nqrzj//fD3zzDMl+hT1xx9/OOIwDEN//PHHOcfvqs8//9xRfLRu3brC4kOSvv32W0fxIUljxoypsE98fLzj9enTp/Xtt99W2Kdbt26KioqSdGbkhMnoAAAA5aMAqeb27NmjLl266NNPPy32hbqQzWbTa6+9pmHDhqlwMGvbtm3q0qVLsS/uReXl5emll17S3XffXeXxV4aihYDVORZF+wQHB1taXjc2NlbBwcGlXqM8ffv2dboPAACAr6IAqcYyMzM1dOhQHTp0SOHh4brrrrs0depUzZgxQ+PHj1dERISj7aJFizR9+nQlJydr0KBBSk1NVXh4uO68884y+8yZM0eff/65Jz6aZXl5eVqzZo3jvdV9OorukN6tWzfVqlXxegsBAQHq1q1bqdcoT9GYVqxYIZ5qBAAAKBurYFVjX3zxhUzTVGxsrD777DM1bty42PnHH39csbGxOnDggCTp5Zdf1nfffafk5GT16dNH8+fPV6NGjYr1eeyxxxQbG+t4pGrKlCm65ZZb3PJ5zsXOnTuVk5PjeN+pU6cK+9jtdiUmJjreW90vRJJatGihtWvXSpJ+++032e12+fmVX6d37tzZ8frEiRP67bff1KZNG8v3BAAA8CWMgFRjpmmqRYsWWrp0aYniQ5KaNGmiN954w/H+zz//1OLFixUTE6Nvv/22RPEhSU2bNi3WZ/v27Y7N/aqjrVu3Ol77+flZ+mJ/5MgR5ebmOt43a9bM8v2KLr2bm5urI0eOVNinffv2xd5v2bLF8v0AAAB8DQVINffqq68qLCyszPPXXnttiVWXXnnlFYWGhpbZZ/DgwcVWd9qwYYPLcVaV/fv3O143bNjQsWlgeU6ePFnsfdHHzipydttTp05V2KdOnTqqU6eO433RmAEAAFAcBUg1VqdOHQ0ZMqTcNrVq1VLHjh2L9bn++usr7FP0UabqPAJSuDmgpFJHgUqTmZlZ7H1QUJDl+xWdhF7atcrSpEkTx+ukpCTL9wMAAPA1FCDVWJcuXSxNnj7//PMdr7t27ep0nxMnTpxTfO5QNLbyRoKKKvr4lSRLoyaFAgMDi70vOv+kPEVjy8jIsHw/AAAAX0MBUo2VNoejNEUftypaWFjtk5WV5VxgblS0ALA6knF2u7y8PMv3K9w5vdDZIyJlKdouOzvb8v0AAAB8DQVINebMo0Ou9KnOy8YGBAQ4Xpe3cWJRZ4+UnD0iUp6zRzysjroU3aOlaMwAAAAojgIE1VrRkRqrhUTRCeGSdPz4ccv3O/txtPDwcEv9ihYu5S0AAAAA4OsoQFCtFX0MLTU11VKfxo0bFxsJcmZSeNG2QUFBlie+F43N6qNzAAAAvogCBKUyDOOc+lmdtG1V8+bNHa//+usvS338/PwUExPjeO/MsrhF27Zq1arCTQilMxsfHj161PG+aMwAAAAojgIEpQoJCSn23urE6qJfxCtDhw4dHK9zcnIsFyFFlxnevHmzpfkjp0+f1ubNmx3viy5vXJ7ff/9dBQUFjvdFYwYAAEBxFCAoVdGNCiXp8OHDFfbJz88v9gW+MnTr1q3Y+127dlnqN2jQIMfr7OxsrV27tsI+a9euLTaCc80111i6V9GYDMNQ165dLfUDAADwRRQgKFXt2rV14YUXOt6vX7++wj4LFy60vHGfVeeff75atWrleL9p0yZL/a699tpi+6HMnDmzwj7vv/++43VAQIDlAmTjxo2O1507dy5RvAEAAOD/UICgTJdddpnj9Zw5c8p9jCkjI0MTJkyokjiuvvpqx+vVq1db6lOvXj2NHDnS8f7TTz8tViicbePGjfr0008d70eOHKl69epZutcPP/zgeD1w4EBLfQAAAHwVBQjKNHz4cMfr3377TY888ojsdnuJdn/88Yf69eunP//885wnr5fnxhtvdLz+6aefLM9HefbZZx27oBcUFOjmm2/Wr7/+WqLd7t27ddNNNznmcdSuXVvPPvuspXtkZGRow4YNpcYKAACAkmpV3AS+6rrrrlOXLl20detWSdK0adO0evVqDRs2TE2bNlVGRoYSEhK0ePFi2Ww2dezYUa1bt9aCBQsqNY4+ffooKipKhw4dUk5OjpYtW2bpi37z5s31zjvvaOzYsZLOLLHbpUsXDR8+XJdccomk/xv5KLpb+n/+8x81a9bMUmxff/21YxPCli1bFhs1AgAAQEkUICiTv7+/Zs+erSuuuEJpaWmSpJ07d2rnzp0l2rZs2VJLlizR888/X+lx+Pn56a677tKLL74oSZo/f77lkYYxY8bo6NGjmjx5sux2u2w2mz766CN99NFHpd7nxRdfVHx8vOXY5s+f73jtTD8AAABfxSNYKFeHDh30008/6aqrrir1fFBQkMaMGaPNmzcrOjq6yuK4//77HY9TLV68WMeOHbPcd9KkSVq1apV69OhRZpuePXtq1apVmjhxouXrHjlyRN9++60kKTg42DHSAgAAgLIZpmmang4C3uHAgQNas2aNkpOTFRQUpGbNmikuLk4RERFuuX98fLw++OADSdI///lPPfbYY05fY//+/dq4caNjP5GmTZuqe/fuatGihdPXeumll/TMM89Ikv72t7/pnXfecfoaAAAAvoYCBF7j999/V+vWrZWfn6+oqCj9/vvvCggI8Egsubm5at68uVJSUhQUFKR9+/apadOmHokFAADAm/AIFrzGRRdd5HjM6dChQ5o7d67HYvnggw+UkpIiSfr73/9O8QEAAGARIyDwKseOHVNMTIyOHz+u5s2b67ffflNgYKBbY8jOzlbLli115MgRNWzYUImJiWw+CAAAYBGrYMGrNGjQQPPmzVNCQoKkM3uQtG7d2q0xHDhwQPfcc48kKTY2luIDAADACYyAAAAAAHAb5oAAAAAAcBsKEAAAAABuQwECAAAAwG0oQAAAAAC4DQUIAAAAALehAAEAAADgNhQgAAAAANyGAgQAAACA21CAAAAAAHAbChAAAAAAbkMBAgAAAMBtKEAAAAAAuA0FCAAAAAC3oQABAAAA4DYUIAAAAADchgIEAAAAgNtQgAAAAABwm1qeDgBSWlqap0NwSlBQkPz8/GS325Wbm+vpcKo98mUduXIO+bKOXDmHfDmHfFlHrpzjrfk677zzyj1PAQKnBQcHy9/fXwUFBV71l8FTyJd15Mo55Ms6cuUc8uUc8mUduXJOTc0Xj2ABAAAAcBsKEAAAAABuQwECAAAAwG0oQAAAAAC4DQUIAAAAALehAAEAAADgNhQgAAAAANyGfUC8zEMPPeTpEKqVqVOnejoEAAAAOIEREAAAAABuQwECAAAAwG0oQAAAAAC4DQUIAAAAALehAAEAAADgNhQgAAAAANyGAgQAAACA21CAAAAAAHAbChAAAAAAbsNO6NWAYRjy86MWPBf+/v6eDsEp3havJ5Er55Av68iVc8iXc8iXdeTKOTUpXxQg1UBwcLBCQkI8HYZXioiI8HQIlvn7+3tVvJ5ErpxDvqwjV84hX84hX9aRK+fUtHxRgFQDOTk5stlsng7DKx0/ftzTIVSoTp068vf3V0FBgU6ePOnpcKo1cuUc8mUduXIO+XIO+bKOXDnHW/NVUbHkcgGyfPlyDRgwwNXL+DTTNFVQUODpMLySt+XN2+L1JHLlHPJlHblyDvlyDvmyjlw5pybly+WJB1dffbVatmypV199VSkpKZUREwAAAIAaqlJmPh84cEATJ07UBRdcoFtvvVUrVqyojMsCAAAAqGFcLkBGjx6toKAgmaap06dPa8GCBRowYIBatWqlf/3rX0pLS6uMOAEAAADUAC4XIB9++KEOHz6st956Sx06dJBpmjJNU/v379eECRMUFRWl22+/XatXr66EcAEAAAB4s0p5BKtu3bp68MEHtWPHDv30008aNWqUY1QkLy9P8+fPV79+/dSmTRu98cYbSk9Pr4zbAgAAAPAylb77Xc+ePTVr1qxSR0X27t2rxx57TE2bNtXIkSP1448/VvbtAQAAAFRjVbb99tmjIiNHjnSMithsNs2bN09xcXFq3769pk6dqhMnTlRVKAAAAACqiSorQIrq2bOnPvroIx0+fFgPPPCA47hpmtqzZ48efvhhRUVF6e9//7v++usvd4QEAAAAwAPcUoDk5+dr/vz5Gjp0qN555x0ZhiHTNCXJ8XhWdna23nvvPbVu3VozZsxwR1gAAAAA3KxKC5B9+/bpiSeeUNOmTR0rYRUWHJdeeqk+/PBD/fXXX3r99dfVunVrRyFy3333admyZVUZGgAAAAAPqFXZFzx9+rQWLlyo6dOn64cffpAkx2hHSEiIbrvtNv3tb39Tly5dHH3Gjx+v8ePHa86cObr//vuVnZ2tV155RQMHDrR834yMDC1YsEAbNmzQsWPHFBgYqBYtWuiaa65Rjx49nP4cx44d05o1a7R3714lJSXpxIkTysrKUnBwsKKionTZZZdp0KBBCgkJcfraAAAAgK+qtAJk7969mj59uj766CMdO3ZM0v8VHm3atNH999+vUaNGqW7dumVeY+TIkUpMTNSUKVP0yy+/WL53UlKSJk2apIyMDElScHCwsrKytG3bNm3btk3XXXedxo4d69Tn+eWXX/Thhx863teqVUtBQUHKzMzUnj17tGfPHn3zzTd67rnn1KxZM6euDQAAAPgqlwuQTz75RNOnT9eaNWsk/V/RERAQoBtuuEH333+/4uLiLF/v0ksvlSRHEVOR06dP66WXXlJGRoaaN2+uRx55RNHR0bLZbFqyZIk+/vhjffXVV4qOjlb//v0txxEZGanhw4erffv2io6OVp06dSRJNptNCQkJev/995WWlqaXX35Zb7/9tvz9/S1fGwAAAPBVLhcgd9xxR7FJ5VFRUbrnnns0ZswYNWrUyOnr1a5d26n2y5YtU3JysgIDAzV58mRFRkZKkgIDAzVs2DClp6fr22+/1dy5cxUXF6datax95LZt26pt27YljgcGBqpv376qW7euJk+erL/++ku//fab2rVr51TcAAAAgC+qtEewBgwYoPvvv1/XXXed/PzOfW77pZdeqlWrVlluv3r1aklSnz59HMVHUTfddJOWLl2q9PR07dy5s9jcE1e0atXK8drqaA0AAADg61wuQB577DHde++9atGiRWXEo4iICPXt29dS25ycHO3du1eS1LVr11LbREZGKioqSgcPHtT27dsrrQD59ddfHa/PZaQHAAAA8EUuFyCvvfZaZcRxTg4dOuR49Kt58+ZltmvevLkOHjyogwcPunS//Px8HT9+XJs3b9bcuXMlnZlgHxMT49J1AQAAAF/hcgFy9913S5IeeughXXzxxZb77dq1S6+//roMw9D7779/TvdOT093vK5fv36Z7QrPHT9+/JzuM27cOB04cKDE8S5duuiRRx45p2sCAAAAvsjljQhnzZqljz76SElJSU71++uvvzRr1izNmjXrnO+dm5vreB0YGFhmu8JzOTk553SfOnXqqF69esX2/OjWrZvuuuuucpcVBgAAAFBcpW9EWBO9+OKLjtcnT57UmjVr9Mknn2j8+PEaM2aMBg8e7MHoAAAAAO/hsQKkoKDgTAAWl8UtTVBQkOO1zWYrc1dym80m6cwGha6qU6eOBg8erLZt2+rRRx/VzJkz1bZt23In4c+dO1fz5s0r8/zNN9+s0aNHuxybL4qIiPB0CBUqXBXOz8/PK+L1JHLlHPJlHblyDvlyDvmyjlw5p6bmy2MFSOGcisIN/s5F0Xkf6enpZRYghXNFKvP/uBYtWqhdu3batWuX/ve//5VbgGRlZSklJaXM89nZ2WxkeI68KW+GYXhVvJ5ErpxDvqwjV84hX84hX9aRK+fUtHxVWgFiGIaldtnZ2dqyZYveeustGYZR6mZ/VkVFRTk2QUxKSlJUVFSp7Qrnp1xwwQXnfK/SNGjQQJKUnJxcbrvQ0FA1bNiwzPMhISGOESE4xxvy5ufn5/g5tdvtng6nWiNXziFf1pEr55Av55Av68iVc7w1XxUVS04VIM8//7xeeOGFEsdN09QNN9zgVGCFbrzxxnPqJ515pComJkaJiYnasmWLevXqVaJNWlqaY/ndzp07n/O9SlNYeBR9FKw0I0aM0IgRI8o8n5aWds4rdPk6b8hbRESE/P39ZbfbvSJeTyJXziFf1pEr55Av55Av68iVc7w1X+edd165551eBcs0zWJ/yjpu5U/fvn31wAMPOP+pioiLi5MkrVmzRqmpqSXOf/HFFzJNU/Xr11fHjh0tX7ei36zv2rVLiYmJkqT27dtbDxgAAADwYU6NgFx44YUldin/4YcfZBiG2rVrV3G14+ensLAwRUdHq3///rrmmmsck2vO1cCBA/Xll18qOTlZL774oh5++GFFR0fLZrPpq6++0jfffCPpzCjE2RPex4wZo5SUFF155ZUaP358sXNPPvmkunfvrh49eqhp06aOoaT09HT98MMP+vTTT2WapiIjI9WvXz+XPgMAAADgK5wqQEaPHl1itabCAmLKlCm6/vrrKy8yiwICAvT0009r0qRJ+uOPPzRu3DiFhIQoNzfX8azc4MGD1b9/f6eue/z4cc2dO1dz586Vv7+/QkJClJ+fX2wvkaZNm2rSpEmVsroWAAAA4AtcnoTep08fGYZR4ehHVWrWrJmmTZumhQsXasOGDUpLS1NoaKguuugiXXvtterRo4fT1xw/frw2b96s3bt3KzU1VSdPnpR05pm2iy66SD169FDfvn0VEBBQ2R8HAAAAqLFcLkBWr15dCWG4rl69eoqPj1d8fLzlPjNnzizzXIcOHdShQ4fKCA0AAADA/+faBAwAAAAAcAIFCAAAAAC3sfwIVtH9PyZPnlzq8XNV9HoAAAAAai7LBchzzz3n2O28aMFQ9Pi5ogABAAAAfINTk9BN0yy12Ci6IaGzXC1eAAAAAHgPywXIqlWrnDoOAAAAAGezXICcvQN6RccBAAAA4GysggUAAADAbShAAAAAALgNBQgAAAAAt6EAAQAAAOA2lieh+/v7V0kAhmEoPz+/Sq4NAAAAoHqxXIC4stcHAAAAAEhOFCB9+vRh00AAAAAALrFcgKxevboKwwAAAADgCywXIKg6hmHIz4/1AM5FVc1NqireFq8nkSvnkC/ryJVzyJdzyJd15Mo5NSlfFCDVQHBwsEJCQjwdhleKiIjwdAiW+fv7e1W8nkSunEO+rCNXziFfziFf1pEr59S0fFGAVAM5OTmy2WyeDsMrHT9+3NMhVKhOnTry9/dXQUGBTp486elwqjVy5RzyZR25cg75cg75so5cOcdb81VRsUQBUg2YpqmCggJPh+GVvC1v3havJ5Er55Av68iVc8iXc8iXdeTKOTUpX5YLkBdeeMHxevLkyaUeP1dFrwcAAACg5rJcgDz33HOOZXiLFgxFj58rChAAAADANzj1CJZpmqUWG65sUsjeIgAAAIDvsFyArFq1yqnjAAAAAHA2ywVI3759nToOAAAAAGdj9zsAAAAAbkMBAgAAAMBtqmwfkJSUFB0+fFinTp1SeHi4mjRpooYNG1bV7QAAAAB4gUotQP78809NmzZNCxYs0MGDB0ucb9asmW655Rb9/e9/V/PmzSvz1gAAAAC8QKU9gvXOO++offv2euONN3Tw4EGZplniT1JSkv7973+rffv2+s9//lNZtwYAAADgJSplBOTll1/W008/LenMniB+fn5q166dYmJiFBoaqqysLO3bt0+7d++W3W5Xdna2HnzwQZ08eVJPPvlkZYSgjIwMLViwQBs2bNCxY8cUGBioFi1a6JprrlGPHj2cvl52drbWr1+vbdu2ad++fUpJSZHdbldERITatGmjQYMGqX379pUSOwAAAOArXC5AtmzZosmTJ8s0Tfn7++uhhx7So48+qiZNmpRoe+TIEb3++ut68803VVBQoGeeeUYDBw5Uly5dXIohKSlJkyZNUkZGhiQpODhYWVlZ2rZtm7Zt26brrrtOY8eOdeqaDz/8sI4cOeJ4X7t2bfn5+SklJUUpKSlas2aNbrzxRt11110uxQ4AAAD4EpcLkGnTpqmgoECGYWju3Lm69dZby2zbuHFj/fOf/1T37t01fPhw2e12TZ06VR9++OE53//06dN66aWXlJGRoebNm+uRRx5RdHS0bDablixZoo8//lhfffWVoqOj1b9/f8vXLSgo0IUXXqgBAwaoW7duaty4sUzT1OHDhzV79mytW7dOixYtUqNGjTRo0KBzjh8AAADwJS7PAVm1apUMw9DgwYPLLT6KGjZsmK6//nqZpunyTurLli1TcnKyAgMDNXnyZEVHR0uSAgMDNWzYMEdxMHfuXOXn51u+7vjx4zV16lQNHjxYjRs3liQZhqGmTZtqwoQJ6tixoyRp0aJFLsUPAAAA+BKXC5CjR49KkgYPHuxUv2uvvbZY/3O1evVqSVKfPn0UGRlZ4vxNN90kwzCUnp6unTt3Wr5uhw4dyjzn5+enK6+8UpKUnJyszMxM54IGAAAAfJTLBUi9evWK/W9V9ysqJydHe/fulSR17dq11DaRkZGKioqSJG3fvv2c73W2OnXqOF4XFBRU2nUBAACAmszlAqRdu3aS5CgErNq3b1+x/ufi0KFDMk1TksrdV6TwXGl7k5yrXbt2STpTQBUtRgAAAACUzeUCZMSIETJNU7Nnz1ZeXp6lPnl5eZo1a5YMw9DIkSPP+d7p6emO1/Xr1y+zXeG548ePn/O9ikpLS9N3330nSerXr58Mw6iU6wIAAAA1ncsFyJ133qm4uDglJibqjjvuUE5OTrntc3NzNWLECO3du1dXXHGF7rzzznO+d25uruN1YGBgme0Kz1UUmxX5+fn617/+pZycHDVs2FA333yzy9cEAAAAfIXLBYhhGFqyZImGDh2qhQsXqm3btvrXv/6lrVu3KjMzU6ZpKjMzU9u2bdM///lPtW3bVgsXLtTNN9+sxYsXV8JHcB/TNPX2229r9+7dql27th577DGFhoZ6OiwAAADAa1jeB8Tf399Su6SkJE2YMKHM84VzNhYuXKiFCxfKMAynlsctKigoyPHaZrMpJCSk1HY2m03SmQ0KXTF9+nStXLlS/v7+euKJJ9SmTRtL/ebOnat58+aVef7mm2/W6NGjXYrNV0VERHg6hAr5+fk5/tcb4vUkcuUc8mUduXIO+XIO+bKOXDmnpubLcgFSWDhUVltnrleWovM+0tPTyyxACueKuPJ/3AcffKBvvvlGfn5+euSRR3TppZda7puVlaWUlJQyz2dnZ1su8FCcN+XNMAyviteTyJVzyJd15Mo55Ms55Ms6cuWcmpYvywVInz59qt1k66ioKBmGIdM0lZSU5Fhu92xJSUmSpAsuuOCc7jN79mwtXrxYhmHowQcf1OWXX+5U/9DQUDVs2LDM8yEhISzle468IW9+fn6On1O73e7pcKo1cuUc8mUduXIO+XIO+bKOXDnHW/NVUbFkuQAp3PCvOgkODlZMTIwSExO1ZcsW9erVq0SbtLQ0x/K7nTt3dvoe8+bN04IFCyRJ9913n/r16+f0NUaMGKERI0aUeT4tLa3SVujyNd6Qt4iICPn7+8tut3tFvJ5ErpxDvqwjV84hX84hX9aRK+d4a77OO++8cs+7PAnd0+Li4iRJa9asUWpqaonzX3zxhUzTVP369dWxY0enrr1gwQJ9+umnkqT4+HgNGjTI5XgBAAAAX+b1BcjAgQPVqFEj5ebm6sUXX9SBAwcknZl4vmDBAn3zzTeSzoxC1KpVfMBnzJgxuv766/Xmm2+WuO6XX36p2bNnS5JGjx6tIUOGVO0HAQAAAHyA5UewqquAgAA9/fTTmjRpkv744w+NGzdOISEhys3NdTwrN3jwYPXv39+p677//vuS/m+Z4SVLlpTZ9qmnnlLbtm3P/UMAAAAAPsLrCxBJatasmaZNm6aFCxdqw4YNSktLU2hoqC666CJde+216tGjh9PXLFylyzRNnThxoty257qMMAAAAOBrKrUAyc7O1pIlS5SQkKBDhw7p5MmTFa5SZBiGVqxY4fK969Wrp/j4eMXHx1vuM3PmzDLPffnlly7HBAAAAKC4SitA3nvvPU2cOFEZGRmW+5imWe2W9gUAAABQdSqlAHnppZf07LPPWtpcsLDgqIyNCAEAAAB4F5dXwdqzZ4+effZZSVKrVq20YsUK5eTkSDpTbCxevFiZmZnauXOnXn31VTVu3FiSdNdddyk3N9crNpIDAAAAUDlcHgF57733ZJqmQkJCtHz5cjVr1qxEm5CQELVv317t27fX2LFjNWTIEM2aNUtZWVmOfTYAAAAA1Hwuj4D88MMPMgxDt9xyS6nFx9nq1aunxYsXq379+vr888+Z7A0AAAD4EJcLkKSkJEkqc6nbvLy8EsciIiI0evRomaapOXPmuBoCAAAAAC/hcgFy6tQpSVJkZGSx48HBwcXOn61Lly6SpE2bNrkaAgAAAAAv4XIBEhoaKqnkSEfdunUl/d8IydkKN+87evSoqyEAAAAA8BIuFyAXXnihpJKFROvWrWWapn766adS+23fvl2SVLt2bVdDAAAAAOAlXC5AOnfuLNM0tXPnzmLH+/TpI0latWqVNm/eXOzc77//rpkzZ8owDLVt29bVEAAAAAB4CZcLkLi4OEnSypUrix0fNWqUatWqJbvdriuvvFJPPPGEpk+frieeeEKXXHKJMjMzJUnDhw93NQQAAAAAXsLlfUCuu+46+fv7688//9TPP/+sXr16SZJatGihiRMn6oUXXlBmZqb+/e9/l+jbtWtX3X///a6GAAAAAMBLuFyANGjQQImJicrLy1PDhg2LnXvuuecUGhqqF1980THiIZ3ZIX3YsGF67733mAMCAAAA+BCXCxBJio6OLvPc448/roceekjr1q1TcnKyQkNDdckll6hx48aVcWsAAAAAXqRSCpCKBAYGOuaKoCTDMOTn5/J0HJ/k7+/v6RCc4m3xehK5cg75so5cOYd8OYd8WUeunFOT8uWWAgTlCw4OVkhIiKfD8EoRERGeDsEyf39/r4rXk8iVc8iXdeTKOeTLOeTLOnLlnJqWryotQE6cOKFTp04pPDxc9erVq8pbebWcnBzZbDZPh+GVjh8/7ukQKlSnTh35+/uroKBAJ0+e9HQ41Rq5cg75so5cOYd8OYd8WUeunOOt+aqoWKrUAiQzM1OzZs3SggULtHnzZmVnZzvOhYSE6JJLLtEtt9yiUaNGKSwsrDJv7dVM01RBQYGnw/BK3pY3b4vXk8iVc8iXdeTKOeTLOeTLOnLlnJqUr0qbePDVV18pJiZG48aN048//qisrCyZpun4k5WVpTVr1ujBBx9UTEyMvv7668q6NQAAAAAvUSkFyOzZszV06FClpKQ4Co7w8HBdfPHF6t27ty6++GLVqVPHce7o0aO64YYbNGfOnMq4PQAAAAAv4XIBsm/fPt13330qKCiQaZq68cYbtW7dOmVkZGjLli368ccftWXLFp04cUIJCQm66aabJEl2u1333nuv9u/f7/KHAAAAAOAdXC5A3njjDeXm5sowDL322mtauHChLrvsslLbXnrppfr888/1r3/9S5Jks9n0xhtvuBoCAAAAAC/hcgGyfPlyGYahPn366LHHHrPU55FHHlHfvn1lmqaWLVvmaggAAAAAvITLBchff/0lSbr55pud6lfYvrA/AAAAgJrP5QKkcDnd888/36l+DRs2LNYfAAAAQM3ncgHSsmVLSVJSUpJT/Q4ePChJiomJcTUEAAAAAF7C5QLk1ltvlWmamjdvnkzTtNTHNE19/PHHMgxDw4cPdzUEAAAAAF7C5QLkvvvuU6dOnbR161Y9/PDDlvo88sgj2rp1qzp37qx7773X1RAAAAAAeIlarl4gMDBQ33zzjW655RZNmzZNCQkJeuyxx9SvXz9FREQ42p04cULff/+9Xn/9dW3YsEE9e/bUggULVLt2bVdDkCRlZGRowYIF2rBhg44dO6bAwEC1aNFC11xzjXr06OH09QoKCrRr1y7t27dP+/bt0/79+5WcnCxJGj58uG6//fZKiRsAAADwJZYLkIsuuqjc86dPn5Zpmtq4caNuvfVWSVJERIRCQ0OVlZWl48ePSzrz+JVhGEpKSlLv3r1lGIbLmxEmJSVp0qRJysjIkCQFBwcrKytL27Zt07Zt23Tddddp7NixTl0zLS1NzzzzjEtxAQAAACjOcgHyxx9/yDCMMud5GIYhwzAkydEmPT1d6enpJdpJ0uHDhx3FiCtOnz6tl156SRkZGWrevLkeeeQRRUdHy2azacmSJfr444/11VdfKTo6Wv3793fq2sHBwbrooovUsmVLtWjRQp988omOHDniUrwAAACAL7NcgDRr1szlYqEqLFu2TMnJyQoMDNTkyZMVGRkp6cyjYcOGDVN6erq+/fZbzZ07V3FxcapVy9pHjoyM1KefflrsMy9atKhKPgOqzkMPPeTpEKqVqVOnejoEAADg45waAamOVq9eLUnq06ePo/go6qabbtLSpUuVnp6unTt3qkuXLpau6+fn8vx8AAAAAGfx6m/ZOTk52rt3rySpa9eupbaJjIxUVFSUJGn79u1uiw0AAABASV5dgBw6dMgx36R58+Zltis8V7j5IQAAAADP8OoCpOgE9/r165fZrvBc4UpcAAAAADzD5X1AzpaWlqZvvvlGCQkJOnLkiE6dOqXw8HA1adJEl112ma699lqdd955lXKv3Nxcx+vAwMAy2xWey8nJqZT7AgAAADg3lVaAZGdn64knntAHH3wgm81Wapv//ve/CgwM1JgxY/Tqq68qODi4sm4PAAAAwAtUSgGSlpamvn37as+ePWXuE1IoNzdX77zzjlauXKkffvhBDRo0OOf7BgUFOV7bbDaFhISU2q6wIPJUwTN37lzNmzevzPM333yzRo8e7caIao6IiAhPh+BVqnu+Clef8/Pzq/axVgfkyzpy5Rzy5RzyZR25ck5NzVelFCA33XSTfv31V0lnvuTfdtttGjhwoFq1aqWwsDBlZmYqMTFRy5Yt06effqrs7Gzt3r1bN910k2MZ3XNRdN5Henp6mQVI4VwRT/0fl5WVpZSUlDLPZ2dny9/f340R1RzkzTneki/DMLwm1uqAfFlHrpxDvpxDvqwjV86paflyuQBZtGiRfvzxRxmGoYsvvlhffPFFqStSde7cWbfccoueeeYZ3Xzzzdq8ebN+/PFHLVmyREOGDDmne0dFRTl2Z09KSnIst3u2pKQkSdIFF1xwTvdxVWhoqBo2bFjm+ZCQEBUUFLgxopqDvDmnuufLz8/P8Xfabrd7Opxqj3xZR66cQ76cQ76sI1fO8dZ8VVQsuVyAfPrpp5LO7Lfx/fffl7salXRmSdzvvvtO7du3V2pqqubNm3fOBUhwcLBiYmKUmJioLVu2qFevXiXapKWlOZbf7dy58zndx1UjRozQiBEjyjyflpbGCl3niLw5p7rnKyIiQv7+/rLb7dU+1uqAfFlHrpxDvpxDvqwjV87x1nxVtOCUy8vwrl+/XoZh6O67766w+CjUoEEDxcfHyzRNrV+/3qX7x8XFSZLWrFmj1NTUEue/+OILmaap+vXrq2PHji7dCwAAAIBrXC5ACuc2dOrUyal+hcVAeXMjrBg4cKAaNWqk3Nxcvfjiizpw4ICkMxPPFyxYoG+++UbSmVGIWrWKD/iMGTNG119/vd58881Sr52VlaWTJ086/hQOfdlstmLHy1r1CwAAAEBxLj+CVbt2bdlsNuXl5TnVr7B9QECAS/cPCAjQ008/rUmTJumPP/7QuHHjFBISotzcXEfBMHjwYPXv39/pa0+ZMkW7du0qcXzRokVatGiR4/3w4cN1++23n/uHAAAAAHyEyyMgTZo0kST9+OOPTvVbs2aNJKlp06auhqBmzZpp2rRpGjJkiBo3bqzTp08rNDRUnTt31sSJE3XPPfe4fA8AAAAArnN5BCQuLk579uzRnDlz9MADD1ia6L1t2zbNnTtXhmE45nC4ql69eoqPj1d8fLzlPjNnziz3/D/+8Q9XwwIAAABQhMsjIGPGjJFhGDp9+rT69++vL774otz2X3zxha666irl5eXJMAyNHTvW1RAAAAAAeAmXR0C6du2q++67T++++67S09N1yy236KKLLtJVV12lVq1aKTQ0VFlZWdq7d6++//577d+/X6ZpyjAM3XffferSpUtlfA4AAAAAXqBSdkKfNm2aTp48qY8//liS9Pvvv+u///1vqW1N05Qk3XHHHZo6dWpl3B4AAACAl3D5ESzpzC6Nc+bM0fz589W1a1eZplnmn27duunzzz/X7Nmz5edXKbcHAAAA4CUqZQSk0C233KJbbrlFSUlJWr9+vY4cOaJTp04pPDxcjRs31mWXXaZmzZpV5i0BAAAAeBGXC5DZs2dLkho1aqQBAwZIOrMsLoUGAAAAgLO5/AzUnXfeqbvuuktr166tjHgAAAAA1GAuFyBhYWGSpHbt2rkcDAAAAICazeVHsBo3bqx9+/bp9OnTlREPAA956KGHPB1CtcIqfQAAVA2XR0CuuOIKSdLGjRtdDgYAAABAzeZyAXLvvffKz89PH330kf7666/KiAkAAABADeVyAdKlSxdNmTJFp06d0lVXXaUdO3ZURlwAAAAAaqBKWYa3UaNGGjRokJYuXaquXbsqNjZWl19+uaKiohQcHFzhNUaNGuVqGAAAAAC8gMsFyJ133inDMCRJhmHIbrfrxx9/1I8//mipv2EYFCAAAACAj6iUndBN0yz3PQAAAABIlVCAfPjhh5URh08zDEN+fi5Px/FJ/v7+ng7Bq5Av67wtV94WryeRK+eQL+eQL+vIlXNqUr5cLkBGjx5dGXH4tODgYIWEhHg6DK8UERHh6RC8Cvmyzpty5e/v71XxehK5cg75cg75so5cOaem5atSHsGCa3JycmSz2Twdhlc6fvy4p0PwKuTLOm/IVZ06deTv76+CggKdPHnS0+FUa+TKOeTLOeTLOnLlHG/NV0XFkksFyF9//aUdO3YoIyNDdevWVceOHRUVFeXKJX2SaZoqKCjwdBheibw5h3xZ52258rZ4PYlcOYd8OYd8WUeunFOT8nVOBciGDRv08MMPKyEhocS5Hj166I033tCll17qcnAAAAAAahanZz4vX75ccXFxSkhIkGmaJf6sW7dOffv21bJly6oiXgAAAABezKkRkFOnTmn06NHKzc11HGvZsqUaNmyolJQU7du3T5Jks9k0evRoJSYmqk6dOpUbMQBUAw899JCnQ6hWpk6d6ukQAABewqkRkDlz5ujo0aMyDEOXXHKJfvnlFyUmJmrt2rVKTEzU7t27HY9epaamas6cOVUSNAAAAADv5NQIyNKlSyVJ5513npYtW1ZihnubNm20dOlStW3bVqmpqVq6dKn+/ve/V160AACvw2hRcYwWAfB1To2A7NixQ4ZhaNSoUWUurxUREaFRo0bJNE3t3LmzUoIEAAAAUDM4NQKSnp4uSbr44ovLbde5c2dJ0rFjx84tKgAAfBQjRsUxYgTUPE6NgGRlZUmSwsPDy20XFhYm6cwGewAAAABQyOlleAEAAADgXLm0EzoAAIAn8chacTyyBm9wTgWIYRiVHUe1lpGRoQULFmjDhg06duyYAgMD1aJFC11zzTXq0aOHp8MDAAAAvMY5FSA33HCDpXamacrf37/cNoZhKD8//1zCcIukpCRNmjRJGRkZkqTg4GBlZWVp27Zt2rZtm6677jqNHTvWw1ECAACUj9Gi4hgt8pxzfgTLNM0yzxmG4RglKa9ddXf69Gm99NJLysjIUPPmzfXII48oOjpaNptNS5Ys0ccff6yvvvpK0dHR6t+/v6fDBQAAAKo9pyehm6ZZYVFR2Mabiw9JWrZsmZKTkxUYGKjJkycrOjpakhQYGKhhw4Zp0KBBkqS5c+dW61EcAAAAoLpwqgCx2+2V/qegoKCqPpvLVq9eLUnq06ePIiMjS5y/6aabZBiG0tPT2XQRAAAAsIBleMuQk5OjvXv3SpK6du1aapvIyEhFRUVJkrZv3+622AAAAABvxTK8ZTh06JDjEbLmzZuX2a558+Y6ePCgDh486K7QAAAAUMWYtF9cZU7aZwSkDOnp6Y7X9evXL7Nd4bnjx49XeUwAAACAt6MAKUNubq7jdWBgYJntCs/l5ORUeUwAAACAt6MAAQAAAOA2zAEpQ1BQkOO1zWZTSEhIqe1sNpukMxsUlmXu3LmaN29emedvvvlmjR49+hwj9W0RERGeDsGrkC/ryJVzyJd15Mo55Ms55Ms6cuWcyswXBUgZis77SE9PL7MAKZwrUt7/KVlZWUpJSSnzfHZ2doU7xhcqr5BBSeTLOnLlHPJlHblyDvlyDvmyjlw5h3xVHQqQMkRFRckwDJmmqaSkJMdyu2dLSkqSJF1wwQVlXis0NFQNGzYs83xISEi13g/lbH5+fo7c2O12T4dT7ZEv68iVc8iXdeTKOeTLOeTLOnLlHG/NV0W/WKcAKUNwcLBiYmKUmJioLVu2qFevXiXapKWlOZbf7dy5c5nXGjFihEaMGFHm+bS0NK9aRSsiIkL+/v6y2+1eFbenkC/ryJVzyJd15Mo55Ms55Ms6cuUcb83XeeedV+55JqGXIy4uTpK0Zs0apaamljj/xRdfyDRN1a9fXx07dnRzdAAAAID3oQApx8CBA9WoUSPl5ubqxRdf1IEDBySdmXi+YMECffPNN5LOjHDUqsVgEgAAAFARvjWXIyAgQE8//bQmTZqkP/74Q+PGjVNISIhyc3Mdz+ENHjxY/fv393CkAAAAgHegAKlAs2bNNG3aNC1cuFAbNmxQWlqaQkNDddFFF+naa69Vjx49PB0iAAAA4DUoQCyoV6+e4uPjFR8f7+lQAAAAAK/GHBAAAAAAbmOYpml6Ogh4l7lz5yorK0uhoaHlLi+MM8iXdeTKOeTLOnLlHPLlHPJlHblyTk3NFwUInHbNNdcoJSVFDRs21LfffuvpcKo98mUduXIO+bKOXDmHfDmHfFlHrpxTU/PFI1gAAAAA3IYCBAAAAIDbUIAAAAAAcBsKEAAAAABuQwECAAAAwG0oQAAAAAC4DTuhw2m33367Y01qVIx8WUeunEO+rCNXziFfziFf1pEr59TUfLEPCAAAAAC34REsAAAAAG5DAQIAAADAbShAAAAAALgNBQgAAAAAt6EAASpgmqZOnjyp1NRUT4cCH2Cz2ZSVleXpMAAATho7dqwee+wxy+2ffPJJ3XPPPVUYUfXFMrw+JiMjQzt37lRqaqpsNpuGDx/u6ZCqrb179+qzzz7Tjh07ZLPZJEmLFy92nM/MzNRHH30kwzAUHx+vwMBAD0VaPRQUFGj//v2On60rr7zS0yFVO2lpadq6davq1aun7t27FzuXlJSkqVOnat++fZKkVq1a6aGHHlJUVJQnQvW4cePGacCAAerbt6/CwsI8HQ4ASQkJCdq6datSU1OVl5enl156yXEuNzdXBw4ckGEYatOmjQej9JyUlBSdPn3acvu0tDSlpaVVYUTVFwWIjzh9+rQ+/PBDLVu2TAUFBY7jRQuQzMxM3XvvvcrNzdU777yjRo0aeSLUamH58uV67733iuXKMIxibcLCwhxfKNu3b6++ffu6O8xqY/HixVqwYIEyMzMdx4oWIJmZmXrqqaeUn5+vf/zjH4qIiPBEmB73/fffa/78+br55puLFSDZ2dl65plnlJGRocKV0X/77Tc9/fTTevvtt33yC/gff/yhGTNm6MMPP1TPnj111VVXqVOnTp4Oq9oYO3asy9cwDEPTp0+vhGi8U1pamv73v//p119/VXp6umw2m8ramcDXc5WcnKyXX35Zf/75p6QzTwac/W9iQECAXn/9daWmpuqVV17x2SLEGQUFBSXy6CsoQHyA3W7XlClTtG3bNklSw4YNlZaWJrvdXqxdWFiYrrjiCn311Vf66aefdNNNN3kgWs/7/fff9e6778put2vgwIGKi4vTyy+/rFOnTpVo269fP23ZskWbN2/22QJk6tSpWrlypUzTVEBAgPLz80u0CQsLU6tWrbRixQqtXbtW1113nQci9bzt27dLki6//PJix7///nudOHFC9evX19ixYxUYGKj3339fhw8f1pdffqnbb7/dE+F61G233aYVK1YoJSVFa9as0Y8//qiGDRvqqquu0pVXXqkGDRp4OkSPSklJcfkavvrFR5JWr16td955R6dPny636Cg858u5ys7O1uTJk3X06FFFRESoW7duWrt2rePJgEL+/v4aOHCg5syZo3Xr1lGAVCA7O1sZGRkKCQnxdCgeQQHiA1atWqWtW7cqIiJCTz31lFq3bq3Ro0crIyOjRNvY2Fh99dVX2rFjh88WIEuWLJHdbtf111+v+Ph4SZKfX+nTpTp27ChJ2r9/v9viq04SEhK0YsUKhYSE6O9//7t69uypu+++u9Sfrb59++p///uftm/f7rMFSOE8oiZNmhQ7npCQIMMwNHr0aPXq1UuSFBwcrKeeekqbNm3yyQJk+PDhGj58uLZv367ly5dr/fr1Onr0qD7++GN98skn6tq1q/r376/u3bvL39/f0+G6HY/Pnrv9+/dr6tSpKigo0MUXX6xu3brp/fffV0hIiO6++26dOHFCO3fu1I4dO1SnTh0NHz5cQUFBng7bY7788ksdPXpUMTExeu655xQWFqZNmzaVKEAk6bLLLtOcOXP066+/eiBS9ztw4IAOHDhQ7JjNZtPKlSvL7GOaprKysrRu3TrZ7XZddNFFVR1mtUQB4gNWrlwpwzA0ZswYtW7duty2LVq0kGEYSkpKclN01c+uXbtkGIaGDh1aYdu6desqKCjIZ5/hXLZsmQzD0KhRoxQbG1tu21atWskwDP3xxx/uCa4aysjIUGhoqAICAhzH8vPzlZiYKD8/P1122WWO4+3atZO/v7+OHDniiVCrjc6dO6tz587KzMzU6tWr9f333+uPP/7Qxo0btWnTJtWtW1dXXnml+vfvr6ZNm3o6XLe57bbbquS6a9euVV5eXo2ew/Xll1+qoKBAV1xxhcaPHy9Jev/99xUYGKirrrpKknTLLbdox44devnll7VixQq9+uqrHozYs9atW+f4DlHR46BRUVHy9/fX4cOH3RSdZyUkJGj+/PnFjuXk5Gjq1KkV9i18jO3666+vqvCqNQoQH1D4he/SSy+tsG1AQIBCQ0N18uTJKo6q+jpx4oSCgoIsz1OoVauWcnJyqjiq6qlwwvQVV1xRYdugoCAFBwfrxIkTVRxV9WUYhnJzc4sd27dvn/Lz8xUTE6Pg4OBi50JCQnz2Z+tsYWFhGjx4sAYPHqz9+/dr+fLlWrNmjU6cOKFFixZp0aJFatu2rQYMGKDevXurdu3ang7ZK82YMUMZGRk1ugD55ZdfZBiGhg0bVuz42Y9iderUSffcc4/efPNNLVq0qER7X5GcnCx/f3+1atWqwraGYSgkJETZ2dluiMzzQkNDdd555znep6amyjCMch8R9fPzU3BwsJo3b64BAwaoQ4cO7gi12qEA8QG5ubkKDg62/A9yfn6+Tz7SUCgoKEg5OTmy2+1lPnpVKCcnR1lZWapbt66boqtesrKyFBwc7NOPJzgjMjJShw8f1u+//+4Ydi98/Kpdu3bF2trtdmVnZ6tevXoeiLR6a9Gihe6//37deuuteu211xyPe+zevVu//vqrZs6cqauvvlpDhw5VaGioh6NFdXPixAnVqlWr2KOQhmEoLy+vRNvY2FhNmzZNa9eu9dkCxG63q1atWhX+eyidKeJyc3N9ZlXI66+/vtgIxpAhQ1S3bl3NnDnTg1F5B/YB8QF169ZVTk5Oid+8liY5OVm5ubk+PcEzKipKdru9xHOdpVm3bp1M01SLFi3cEFn1Ex4erpycnFL/4T5benq6z3+h7tSpk0zT1HvvvafExEStX79e3333nSSVWJb30KFDKigoUP369T0RarW2Y8cO/fvf/9a9996rPXv2SDozWhQbG6t69eopMzNTCxcu1IMPPugzj4LAusDAwBJfkIODg0v9b1lAQIACAwN19OhRd4ZYrZx33nmy2WyWRq8TExN1+vRpn11Fc/jw4RoyZIinw/AKFCA+oHDex/r16yts++WXX8owDLVv376qw6q2evXqJdM0SzzXebbk5GTHPiC9e/d2U3TVS8uWLSWd+UJYkWXLlkmS2rZtW6UxVWc33XSTQkJClJiYqCeeeEIvv/yycnJy1KZNG8eCBoU2btzo0+vpn+3YsWOaP3++7rnnHk2ePFlr1qxRXl6eYmJi9OCDD2rWrFl6/PHH9f777+vxxx9Xo0aNdOzYMc2aNcvToaOaadCggbKzs4ut2Ne4cWNJchS0hY4ePeozjxOVpfC/Td9//3257UzT1McffyzDMNS1a1d3hFbt3Hbbbbrxxhs9HYZXoADxAVdffbXjPwzlLd24aNEiffPNN5KkQYMGuSu8amfQoEFq3LixNmzYoFdeeUW//vqr49ngjIwM7d27V/PmzdMjjzyiEydO6MILL1RcXJxng/aQfv36yTRNzZkzp9geIGf7+eef9fnnn8swDMckT18UGRmpl156SR06dFBAQIDq1q2rfv36adKkScXamaap5cuXyzRNde7c2UPRel5BQYF++uknPffccxozZow++eQTHT16VCEhIbr22mv11ltv6Z///Kf69+/v+I22v7+/YmNj9dJLL8nPz0+//PKLhz8FqpvmzZvLNM1io9yFo5MzZ850rFZ38uRJvf322zIMQ9HR0Z4K1+NuuOEG+fn5acGCBWX+IjM5OVn/+Mc/tH37dtWuXVvXXnutm6OEtzHMshbARo3y9ttv6/vvv1dYWJh69uyptWvXKjc3VyNHjlRqaqq2bNmilJQUmaapIUOG6O677/Z0yB51+PBhPf/880pOTi5z/XfTNNWkSRO98MILioyMdHOE1ceLL76oTZs2qVGjRurXr58WL16s7OxsPfbYY0pNTdXGjRu1e/dumaapPn366NFHH/V0yNVeQUGBjh07JunMb2t9cU7WzJkz9cMPP+jUqVOOXwC0adNGAwcOVGxsrKU5bXfffbfS09O1ePHiKo625ihcor0m52z16tV64403NHToUI0ePVrSmRG2v/3tb46lZcPDw4vt/fTkk0+qR48eHom3Oli+fLn+85//SJIaNWqktLQ05efnq0uXLkpNTdWhQ4ccbR999NESex3VRG+99ZYkqX79+ho5cmSxY84wDEMPPfRQpcbmDShAfERBQYHmzJmjxYsXl7qxUuFycEOHDtXIkSN9etOlQjk5OVq0aJFWrFhRYpndiIgI9e/fX0OHDvXZTYQK2Ww2vfXWW/rpp59K/bkp/HmLjY3V+PHjiy1BC5Sl8Dnqwg1SBw4cqAsuuMCpa/zzn//UiRMnNGXKlKoIsUbyhQLEZrNp7dq1CgsLK7b09a5du/T66687in/pzHyRUaNGafDgwZ4ItVrZtGmTpk+fXuZ8mMjISN1///3q1q2bmyPzjCFDhsgwDDVt2lTvvPNOsWNWvloXtjMMo0b/fSsLBYiPSU5O1ooVK7Rnzx4dP35cdrtd9erVU5s2bdSvXz+fWkffGceOHVN6erojX+eff76nQ6p2du3ape+//77Un63+/fvr4osv9nSI8CITJ050LKlL0eo+vlCAlKegoEB79uxRWlqaQkND1bZt2zJXUvOFPVPOZrfbtWvXLu3Zs6fYv4lt27ZVp06dfGq09s0335RhGIqIiNCoUaOKHXPWuHHjKju8ao8CBACqwK5duySd+Q1qTExMsWPO8tV14uF+vl6AOMNXcvXVV19JOrNAiy+vkOlOvlDcsg8IAFSBSZMmlRieLzzmrJr+BccK0zR16tQp2Ww2n55zBbjb+++/Lz8/P1199dWeDsVn+MKGoBQgQDkKCgp05MgRZWZmFluysTT8lhpnM02zxLPADDo7Z+/evfrss8+0Y8cOxwThogVZZmamYzns+Ph4n9kADXCX8PBw2e12HoVEpaIA8SGZmZnauHGjkpKSKvxC7aurMhRKSUnR7NmzlZCQUGHhUciXf0ttmqZ+/fVX/fnnn8rMzFRBQUG57YcPH+6myDxnyZIllo6hbMuXL9d7771X7Ofp7BGksLAwpaWlaevWrWrfvr369u3r7jBrFApknK1Fixbatm2bMjIyVLduXU+HgxqCAsRHfPvtt5o1a1axXV5L+4em6KoMvlqAJCcn6/HHHy+2/CfKtmHDBr333ntKT0+33McXChC45vfff9e7774ru92ugQMHKi4uTi+//HKxpVEL9evXT1u2bNHmzZspQFw0ceJEy790gW+47rrrtHXrVsdGoEBloADxAT///LP++9//SpJq1aqlmJgYNWjQwNI6+r5o3rx5OnnypEJDQzVs2DD16NFDDRo0YPi5FDt37tTLL78su90u6cyeFfxsoTIsWbJEdrtd119/veLj4yVJfn6l751buFPz/v373RZfTdWmTRtPh4Bqplu3brrrrrs0e/ZsZWZm6sYbb/TpjRlROShAfEDho0EdOnTQo48+qvr163s2oGpu+/btMgxDDz/8sLp37+7pcKq1zz//XHa7Xc2bN9e4cePUokULT4eEGmLXrl2OvYkqUrduXQUFBZXYrweA68aOHSvpzC8A1qxZozVr1qh27doKDw8v85cChmFo+vTp7gwTXoYCxAf8+eefMgxD48aNo/iwICsrS7Vq1fKZzZRcsXfvXhmGoUcffVTNmzf3dDheg/lYFTtx4oSCgoIUERFhqX2tWrWUk5NTxVEBviclJaXEMZvN5lgUojRsZoyKUID4AMMwFBwcrIYNG3o6FK9Qv359ZWRklPmbHfyfgoICBQUFUXw4gflY1gQFBSknJ0d2u73Cv4s5OTnKyspigixQBXzxvz+oehQgPqBZs2bat2+f8vLyeDbfgp49e2rJkiVKTExUq1atPB1OtdakSRMdPHhQBQUFPrUD7rliPpZ1UVFR+u2333TgwIEKH+1bt26dTNPkEUCgCvTr18/TIaAGogDxAddee61ef/11rVq1SgMHDvR0ONXesGHD9PPPP+vdd9/Viy++qLCwME+HVG31799fM2bM0Pr169WrVy9Ph1PtMR/Lul69emnPnj2aP3++Jk6cWGa75ORkxz4gvXv3dmOEAIBzRQHiA/r27atffvlFM2fOVHBwsPr06ePpkKqNXbt2lXp8xIgRmj59uh544AENGDBAMTExCg4OLvdavrgR4bXXXqstW7boP//5j+rXr88KOhVgPpZ1gwYN0tKlS7Vhwwa98sorGjJkiONRtYyMDKWkpGjjxo36+uuvlZWVpejoaMXFxXk2aPgUX12m3TRNnTp1SjabTZGRkZ4OB17KMH31b1AN9dZbb5V5bsOGDcrKytJ5552nli1blvuF2leeOx8yZEilTZar6RsRfvrpp6Uez8/P19KlS5WVlaV27dpZKtZ8dR+Q4cOHyzAMffLJJ54OxSscPnxYzz//vJKTk8v8e2qappo0aaIXXniBL0Nwqz179ig/P99nfvm0d+9effbZZ9qxY4djAnrRf/cyMzMdo5Hx8fEKDAz0UKTeb9SoUTp58mSN/l7BCEgNs3LlSsfk1aKKHktNTVVqamqp/X1x4is1uDWffPJJucWaaZr65ZdftHv37gqv5asFCPOxnNOkSRO9+eabWrRokVasWFFimd2IiAj1799fQ4cOVUhIiIeihK/ypRHf5cuX67333lNBQYHj2Nn/HoSFhSktLU1bt25V+/bt2RTUBb6wISgFSA1zxRVXsPydE5YsWeLpELxG+/bt+dlyEfOxnBccHKzbb79dt99+u44dO6b09HTZ7XbVq1dP559/vqfDA2q833//Xe+++67sdrsGDhyouLg4vfzyyzp16lSJtv369dOWLVu0efNmChAX+EJxSwFSw4wfP77Krr127Vrl5eXpyiuvrLJ7oPr6xz/+4ekQvB7zsVzToEEDNWjQwNNhAD5lyZIlstvtuv766xUfHy9JZS6N3bFjR0nS/v373RYfvBMFCCybMWOGMjIyanwBkpqaKj8/P8tfdI4dOya73c7z5yimvPlYtWvX1uuvv67Zs2czHwtAtbZr1y4ZhqGhQ4dW2LZu3boKCgoq8bgkcDYKEOAsY8aMUUREhGbNmmWp/YQJE5SWllajJ4vBeczHqjwFBQU6cuRIhbvGS765Gh1QlU6cOKGgoCBFRERYal+rVi3l5ORUcVTwdhQgAFxit9tlGEap80OWLl2qXbt26fTp0+rWrZsGDBjgM/NImI/lupSUFM2ePVsJCQmWJ2TyiwCgcgUFBSknJ0d2u73MR68K5eTkKCsrS3Xr1nVTdPBWFCCAi2w2W4X/Ua6pli9frv/85z+KjY3VY489VuzcSy+9pE2bNkk6s0LWhg0btGXLFj311FOeCNXtmI/lmuTkZD3++OM6deoUK9UBHhQVFaXffvtNBw4cUIsWLcptu27dOpmmWWE7gAIEcMHhw4d16tQpy0PTNc2WLVsknfltf1GbN2/Wxo0bJUndu3dX7dq1tW7dOq1fv15r165VbGys22OtSXxhPta8efN08uRJhYaGatiwYerRo4caNGiggIAAT4cG+JRevXppz549mj9/viZOnFhmu+TkZMc+IL1793ZjhPBGFCDweQkJCVq/fn2xY1lZWeVOIi5sU7jnRbt27aosvurszz//lCS1bt262PFVq1bJMAwNGTJEd911lyTp66+/1owZM7Ry5UoKEFRo+/btMgxDDz/8sLp37+7pcACfNWjQIC1dulQbNmzQK6+8oiFDhjhGJTMyMpSSkqKNGzfq66+/VlZWlqKjoxUXF+fZoFHtUYDA5x04cKDEhOG8vDytXLnSUv/w8HCf3VgvIyNDgYGBCgsLK3Z8+/btkqSrr77acaxfv36aMWOGfv/9d7fGCO+UlZWlWrVqqVu3bp4OBfBptWvX1uTJk/X8889r3bp1SkhIcJwbPXq047VpmmrSpIkmTZokf39/T4QKL0IBAp8XHR1d7FGWlStXqnbt2uX+lt4wDIWEhKhZs2bq2bOnwsPD3RFqtZObm1vikZjk5GSdPHlSkZGRaty4seN4cHCwQkNDdfLkSXeHCS9Uv359ZWRk+Oz8KqA6adKkid58800tWrRIK1asKLHMbkREhPr376+hQ4cqJCTEQ1HCm1CAwOf16NFDPXr0cLxfuXKlQkNDNW7cOJeu6wsThevUqaMTJ07o5MmTqlOnjiRp27ZtkqS2bduWaF9QUFDunhdAoZ49e2rJkiVKTExUq1atPB0O4POCg4N1++236/bbb9exY8eUnp4uu92uevXq6fzzz/d0ePAy/GoJOMuUKVP05JNPunydGTNmaOrUqZUQUfVVuNLJkiVLJJ1ZEWzp0qUyDEMXX3xxsbbHjx9Xbm6uz07Yh3OGDRumyMhIvfvuu8rMzPR0OACKaNCggWJiYtS6dWuKD5wTRkCAs7CRmXVXX321Nm3apIULFyohIUHZ2dlKT09XeHi4evXqVaztzp07JUnNmzf3RKioxnbt2lXq8REjRmj69Ol64IEHNGDAAMXExFQ4gsbfXwCo/ihAYBlr8eNs3bt317Bhw/T555/r0KFDkqSwsDA9/PDDJb4o/vDDD5KkTp06uT1OVG+TJk2qcNPGzz77zNK12IgQqDoFBQU6cuSIMjMzK9wclF8GoDwUILBs4sSJlncjhu+44447dNVVVykxMVEhISFq1apViVWx8vPzFRMTo5YtW+rSSy8tcQ1fmC+D8vELDqD6SklJ0ezZs5WQkGD5ewC/DEB5KEBgWZs2bTwdAqqphg0bqmHDhmWer1WrVrlLFfvCxnooW+EcIgDVT3Jysh5//HGdOnWKXxSg0lCAAAAAoFTz5s3TyZMnFRoaqmHDhqlHjx5q0KBBiSXYAWdQgACAl/GF30KmpqbKz89PDRo0sNT+2LFjstvtioyMrOLIAN+yfft2GYahhx9+WN27d/d0OKghKEAAwMv4wnysMWPGKCIiQrNmzbLUfsKECUpLS+O5c6CSZWVlqVatWurWrZunQ0ENQgECAF6G+VgA3KV+/frKyMiQnx9bx6Hy8NMEAPB6NpuNL0hAFejZs6dsNpsSExM9HQpqEP5rDQDwaocPH9apU6dUt25dT4cC1DjDhg1TZGSk3n33XWVmZno6HNQQPIIFVBFfmCgMVJaEhAStX7++2LGsrCy99dZb5fbLysrS7t27JUnt2rWrsvgAX7Br165Sj48YMULTp0/XAw88oAEDBigmJqbEZrNnYyNClIcCBKgivjBRGKgsBw4c0MqVK2UYhqN4z8vL08qVKy31Dw8PL3evGQAVmzRpkgzDKLfNZ599ZulaLAiB8lCAAFWEicKAddHR0cU2oly5cqVq166t2NjYMvsYhqGQkBA1a9ZMPXv2VHh4uDtCBWo0Ru/hDhQgAACP69Gjh3r06OF4v3LlSoWGhmrcuHEuXXft2rXKy8srVtwAKN2SJUs8HQJ8BAUIAI/jN24425QpU1Srluv/RM2YMUMZGRkUIABQjVCAAPA45svgbExgBaqH1NRU+fn5qUGDBpbaHzt2THa7XZGRkVUcGbwZBQgAj2O+DABUT2PGjFFERIRmzZplqf2ECROUlpbGJHSUi31AAAAAALgNBQgAAAAqhc1mk58fXy9RPn5CAAAA4LLDhw/r1KlTqlu3rqdDQTXHHBAAAABIkhISErR+/fpix7KysvTWW2+V2y8rK0u7d++WJLVr167K4kPNQAECAAAASdKBAwe0cuVKGYbhWCI9Ly9PK1eutNQ/PDxcw4cPr8oQUQNQgAAAAECSFB0dXWzfnJUrV6p27dqKjY0ts49hGAoJCVGzZs3Us2dPhYeHuyNUeDEKEAAAAEiSevTooR49ejjer1y5UqGhoRo3bpxL1127dq3y8vLYFBSSKEAAAABQhilTpqhWLde/Ls6YMUMZGRkUIJBEAQIAqMEKn2EHcG46dOjg6RBQA1GAAABqrIkTJyo/P9/TYQAAiqAAAQDUWG3atPF0CACAs7ARIQAAAAC3oQABAAAA4DYUIAAAAADchgIEAAAAgNtQgAAAAABwGwoQAAAAAG5DAQIAAADAbShAAAAAUKVM0/R0CKhGDJOfCAAAAFShPXv2KD8/Xx06dPB0KKgGKEAAAAAAuA2PYAEAAABwGwoQAAAAAG5DAQIAAADAbShAAAAAALgNBQgAAAAAt6EAAQAAAOA2FCAAALjB6tWrZRiGDMPQc8895+lwAMBjKEAAAJWqZcuWji/au3btqrB9//79He0vuOCCCttnZ2crMDBQhmEoICBAmZmZlRE2AMBNKEAAAJXqiiuucLxevXp1uW3z8vL0888/O94fOnRI+/btK7fPTz/9pLy8PElS9+7dFRYWdu7BAgDcjgIEAFCpihYgq1atKrft+vXrlZOTU+xYRX2KFjVF7wUA8A4UIACAShUXF+d4vWbNGpmmWWbbwmIiPDxcsbGxxY5V1EeiAAEAb0QBAgCoVE2aNFGrVq0kSWlpadq5c2eZbQuLidjYWPXr16/YsdJkZ2dr48aNkqTatWurd+/elRM0AMBtKEAAAJXOyjyQvLw8rVu3TtKZUZO+fftKkg4fPqzExMRS+/z00086ffq0JOmyyy5TcHBwsfM5OTl6++23ddVVV6lx48aqXbu2GjRooO7du+vpp5/W4cOHy4171qxZjgnxs2bNkiRt2bJF9913n1q1aqXw8PBi54patmyZbrzxRjVu3FhBQUFq1qyZhg4dquXLl5d7TwDwNRQgAIBKZ2UeSNH5H3FxcerRo4dq165dbp/yHr/auHGjWrdurQcffFD/+9//lJycrNOnTys9PV2bNm3SlClTFBMTow8++MDy53jttdd06aWX6r///a/27t1b6opbdrtdY8eO1dVXX63FixcrOTlZNptNBw8e1KJFizRw4ECNHz/e8j0BoKar5ekAAAA1T2nzQAzDKNam6PyPrl27qlatWrr00ku1du1arV69Wvfee2+J65ZVgOzYsUNXXHGFsrKyJEnt2rXTyJEjFR0drfT0dC1evFjLly9Xdna24uPjZZqm4uPjy/0Mn332mZYuXaqwsDCNGjVKl156qQICArR79241atTI0e7hhx/WzJkzJUn+/v664447FBcXp8DAQG3btk3vv/++3nrrLR08eNBS7gCgxjMBAKgCbdu2NSWZksytW7eWOH/llVeaksyrr77acWzixImmJLNRo0Yl2mdlZZkBAQGmJDMoKMjMzc01TdM0CwoKzA4dOjjuNWbMGPP06dMl+s+cOdM0DMOUZIaEhJgHDhwo0ebDDz90XEeS2apVK/PPP/8s8zOuXbvWcc3Q0FDzxx9/LNHm8OHDZps2bYpd99lnny3zmgBQ0/EIFgCgSpQ3D+Ts+R+FCueBJCcna8+ePcX6FJ3/0bNnTwUGBkqSvvnmG8eGh506ddJ7772nWrVKDvDHx8c7RlWys7P11ltvlRu/YRj69NNP1axZszLb/Pvf/3as8vXqq686VvIqqnHjxpo/f778/f3LvR8A+AoKEABAlShvHkhCQoJj/kdh0SFJvXr1chQPZ/cp+r5o0fLFF184Xj/66KPlftF/8sknHY+CFe1XmtjYWHXp0qXM8zabTd98840kqW7duhozZkyZbTt16qQBAwaUez8A8BUUIACAKtG3b1/Hl/0ff/xRdrvdca5wRCQsLEyXXHKJ43hYWJi6detWrM3ZfaTixc369esdryv6kt+8eXO1adNGkpSUlKQjR46U2fbyyy8v91rbt2937Mjeu3dvx4hMWQqXGQYAX0cBAgCoEpGRkWrfvr0k6fjx49q2bZvjXGEx0bt37xKPSxWOiBQtOLKysrRp0yZJUnBwsC677DLHucIiIjw8vNjk8LIU7lFStG9poqKiyr1O0SV9W7ZsWeF9rbQBAF9AAQIAqDKlzQOx2WxKSEiQVPzxq0KFx1JSUrR7925Jxed/9O7d27FcrySdOnVKkhQaGmopprCwsBJ9S3P2HiNnK7okb0hISIX3tRofANR0FCAAgCpTdK5G4RyOs/f/OFtsbKxjHkdhn/L2/wgPD5ckxxK8FSlaOBT2PRdFC5ns7OwK21uNDwBqOgoQAECVOXseSEFBgaOYCA0NVffu3Uv0qVOnjjp37izJWgHSuHFjSWdGM44ePVphTEV3WW/SpIn1D3OWpk2bOl7v27evwvZW2gCAL6AAAQBUmQYNGqhTp06SpIyMDG3dutVRTBRd8epshY9h/fDDD8rMzHTM/wgLCytRtBSdD7J8+fJy40lKSnIs79usWTNLc0bK0qlTJ8fE859++kk2m63c9itWrDjnewFATUIBAgCoUkVHLL777jvH/I/SHr8qVFiApKWl6b///a9j/kdsbGyJouWmm25yvP73v/+tgoKCMq/76quvOvbtKNrvXAQGBuqaa66RdKa4+uCDD8psu2vXrgqLIwDwFRQgAIAqVbTQmDZtWrnzPwpdfvnljke3XnvtNcfxsx+/kqRrrrlGHTt2lHRmadz7779f+fn5JdrNmjVL7733nqQzk8bHjRvn9Gc526OPPuqIc8KECY7NFYs6evSobr311nILIwDwJaWPfQMAUEn69OkjPz8/2e12paSkSDpTAJQ2/6NQ/fr11bFjR+3YscPRRyq9APHz89PcuXPVq1cvZWVlacaMGVq3bp1GjhypCy+8UOnp6VqyZIm+++47R5+pU6eqefPmLn+23r1768EHH9TUqVN16tQp9enTRyNGjFDfvn0VGBiobdu2aebMmUpPT9fQoUMr3PwQAHwBBQgAoEpFRETo4osv1pYtWxzHevXqpYCAgHL79e3bVzt27HC8r1Onjrp27Vpq206dOmnVqlUaOnSoDh06pF27dmnChAkl2oWEhGjq1KmKj48/x09T0htvvKGsrCy9//77ys/P16xZszRr1qxibcaNG6cbbriBAgQAxCNYAAA3OHvkorzHrwqdvUfI5Zdf7lietzTdu3dXYmKipk6dqn79+un8889XQECAIiIi1K1bN02cOFF79+6t1OJDOjMCM3PmTC1dulTXX3+9GjZsqNq1aysqKko33nijvvvuO7355puVek8A8GaGWTgbDwAAAACqGCMgAAAAANyGAgQAAACA21CAAAAAAHAbChAAAAAAbkMBAgAAAMBtKEAAAAAAuA0FCAAAAAC3oQABAAAA4DYUIAAAAADchgIEAAAAgNtQgAAAAABwGwoQAAAAAG5DAQIAAADAbShAAAAAALgNBQgAAAAAt6EAAQAAAOA2FCAAAAAA3IYCBAAAAIDbUIAAAAAAcBsKEAAAAABuQwECAAAAwG3+HyY5hkEAtY+CAAAAAElFTkSuQmCC",
      "text/plain": [
       "<Figure size 400x200 with 1 Axes>"
      ]
     },
     "metadata": {
      "image/png": {
       "height": 200,
       "width": 400
      }
     },
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAyAAAAGQCAYAAABWJQQ0AAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjcuNCwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8WgzjOAAAACXBIWXMAAB7CAAAewgFu0HU+AABjeUlEQVR4nO3deVhU9f4H8PdhnwFEJEgUMURwyz0NlZTcd82FzLVCM3+3UkvzpmWLmlm3cqtMrdxTcyMrU69rLoiKqKSGa2iCgCjLAAPMnN8fPHMuCAwHBuYwM+/X8/TcmXO+55zPfG7afOa7CaIoiiAiIiIiIjIDO6UDICIiIiIi28EChIiIiIiIzIYFCBERERERmQ0LECIiIiIiMhsWIEREREREZDYsQIiIiIiIyGxYgBARERERkdmwACEiIiIiIrNhAUIW6YMPPoAgCBAEAWFhYRW6tn///hAEAfb29rh48WL1BGghfvrpJymPb775ptLhEBERkQ1wUDoAInPavXs39uzZAwAYPXo0WrZsWaHrb9y4gTNnzuDOnTsAAD8/Pzz11FNo1KhRlcdqDiNGjEC7du0QExOD5cuXY/LkyWjSpInSYREREZEVYw8I2QydToeZM2cCAOzs7PD+++/LvvbYsWPo0qULAgMD8fzzz+Ott97CW2+9heeffx6BgYHo0qULjh07Vi1xh4WFSb0Ulf1nzZo1pd5bEAR88MEHAID8/HzMmjWrWj4DERERkQELELIZGzZswF9//QUAeO6559C4cWNZ133yySfo1q0bTpw4UWabEydOICwsDIsWLaqSWKuah4dHmecGDhyIZs2aAQAiIyNx9uxZc4VFRERENohDsMgmiKKIBQsWSO+nT58u67o1a9bgnXfekd47OTlh1KhR6NChA/R6PU6fPo2tW7ciLy8POp0O//73v1G3bl1MmDChymKvX78+AgMDK3TN9evXpdceHh7o27dvmW0FQcDUqVPx6quvAgDmzZuHXbt2VSpWIiIiovIIoiiKSgdBVFEffPABPvzwQwBAt27dcPjwYaPtd+/ejcGDBwMAmjVrhkuXLpX7jISEBAQHB0Or1QIAGjRogL1790q9BQaXLl1C3759cfv2bQCAs7Mzrl69igYNGlT0Y1WJmJgYtG/fXnr/yiuv4NtvvzV6TUZGBnx9fZGdnQ07OztcvXrVYue1EBERUc3GIVhkE5YvXy69lts78dFHH0nFh729PbZv316i+ACA5s2bY9u2bbC3twcAaLVafPTRR1UQdeX88MMPxd6/9NJL5V5Tq1YtDB06FACg1+vxzTffVEdoREREROwBsSUpKSk4evQo7ty5g5ycHAQEBKBHjx547LHHyrwmMTERR48exd9//w07Ozv4+/ujd+/eqF27tvkCL0VFekASExPh5+cHvV4PAIiPj0dQUJDR+z98+BA+Pj7Iz88HAIwZMwYbNmwwes3YsWOxceNGAICjoyNSUlKMzr2oDnl5eahXrx7u378PAGjatCkuX74s69rt27djxIgRAABfX1/cuXMHdnb8jYKIiIiqFr9dWJkXX3xRWvnoxRdfBACkpqbihRdeQP369TFixAhMmzYN77zzDkaNGgU/Pz/MmjULBQUFxe7zzz//IDw8HH5+fhg1ahRmzZqFmTNn4vnnn8fjjz+O9957r8Q1Rd26davYKky3bt2qdPym+umnn6Tio0mTJuUWHwDw22+/ScUHAEycOLHcayIiIqTX+fn5+O233yoRrWl+/vlnqfgA5PV+GPTr1w9OTk4ACou28oa1EREREVUGCxArd+XKFbRt2xabN28u9oXaQKvV4tNPP0V4eDgMnWGxsbFo27ZtsS/uReXl5WH+/Pl4+eWXqz3+qlC0EJC7aWHRa1QqFUJDQ8u9JjQ0FCqVqtR7mEvR5Xbt7e0xbtw42deq1Wp06NBBeq9E/ERERGT9uAqWFcvKysKwYcNw584duLu7Y8SIEWjbti1UKhX+/PNPrF27Fg8ePAAA7Ny5EytXrsSQIUPQr18/pKSkwN3dHcOHD0e7du1KvWb9+vUYNGgQRo4cqeTHNCovLw9Hjx6V3sspJAAU2yG9ffv2cHAo/4+Ko6Mj2rdvL+0HYu5d1hMTE/H7779L7/v27QtfX98K3SM0NBTHjx8HAOzfv79K4yMiIiICWIBYtR07dkAURYSGhmLr1q0lvozOnDkToaGhuHnzJgBg4cKF+P3335GUlISuXbtiy5YtqFu3brFrZsyYgdDQUGlI1YIFC2p0AXLx4kXk5ORI71u1alXuNXq9HvHx8dJ7ufuFAEBgYKBUgPz111/Q6/Vmm0exfv166HQ66X1Fhl8ZtG7dWnr9559/QqPRwNXVtUriIyIiIgI4BMuqiaKIwMBA7Nmzp9RfwuvVq4cvv/xSev/3339j165dCAoKwm+//Vai+AAK96Qoes358+elzf1qonPnzkmv7ezs0LRp03KvSUxMRG5urvTe399f9vOKLr2bm5uLxMRE2deaqujwKy8vLwwaNKjC92jRooX0WqfT4fz581URGhEREZGEBYiVW7RoEdzc3Mo8P2DAgBIrWn3yySdGf/UeOHBgsdWdoqOjTY6zuhTdkM/Hx0eaZG1MRkZGsfeenp6yn/do28zMTNnXmuLUqVPFVrsaM2aMrM/6qEf3LimaPyIiIqKqwALEitWqVQtDhgwx2sbBwQEtW7Ysdo1hwz5j1xQdylSTe0AMmwMCkD0fIisrq9h7FxcX2c8rOgm9tHtVl6K9H0Dlhl8BhQVU0c+bkJBgSlhEREREJbAAsWJt27aVNXn68ccfl163a9euwtc8fPiwUvGZQ9HYjPUEFVV0+BWACvUkODs7F3tfdP5JdcnNzcXmzZul923atEGbNm0qfb+ieUpPTzclNCIiIqISWIBYsdLmcJSm6HCrooWF3Gs0Gk3FAjOjogWA3J6MR9vl5eXJfp5h53SDR3tEqsPOnTuLFVqm7p9SNObs7GyT7kVERET0KBYgVqwiQ4dMucawf0hN5OjoKL02tnFiUY/2lDzaI2LMoz0ecntdTFF0+JWjoyPGjBlj0v2K7hdTNH9EREREVYEFCFm1oj01cguJWrVqFXtv2PdEjkeHo7m7u8u+tjLu3LmD//73v9L7QYMG4bHHHjPpnkWLKC7BS0RERFWNBQhZtaLD0FJSUmRd4+vrW+mJ2EXburi4VHgjwIpat25dsd3qKzv53CAvL6/YKmByh/ERERERycUChKqFIAiVuq6qJ203bNhQev3PP//IusbOzg5BQUHS+4osRVu0bXBwcLVvQlh0+FXdunXRt29fk+6XmJhYbEhd0fwRERERVQUWIFQt1Gp1sfdyJzPfu3evSuN48sknpdc5OTmyi5CiywyfPXtW1vyR/Px8nD17VnpfdHnj6nDs2DFcvXpVej9u3DhZK5gZU/R+QPH8EREREVUFFiBULYpuVAgAd+/eLfeagoKCYl/gq0L79u2LvY+Li5N1Xb9+/aTX2dnZOHbsWLnXHDt2rFgPTv/+/WVGWTlVtfdHUUXzU6dOHQQEBJh8TyIiIqKiWIBQtXBycsITTzwhvT916lS512zfvr3KN+57/PHHERwcLL0/c+aMrOsGDBhQrDdh9erV5V7z3XffSa8dHR2rtQDJzs7G1q1bpfdPP/00mjVrZvJ9T58+Lb3u2rWryfcjIiIiehQLEKo2Tz/9tPR6/fr1RocxpaenY9asWdUSR9F5EYcPH5Z1Te3atTFu3Djp/ebNm4t9OX/U6dOni20GOG7cONSuXbvM9h988AEEQZD+kRuXwfbt25GZmSm9r4reDwA4cuSI9LpPnz5Vck8iIiKioliAULUZNWqU9Pqvv/7Cm2++WWzFJoNbt26hR48e+Pvvvys9ed2Y5557Tnp9/Phx2fNR3n//fWkXdJ1OhxEjRuDy5csl2l26dAnDhw+HTqcDUNj78/7771dB5GX74YcfpNcqlapYrivr8uXL0hwZOzs7DB482OR7EhERET3KtBmrREYMGjQIbdu2xblz5wAAy5Ytw+HDhxEeHo769esjPT0dUVFR2LVrF7RaLVq2bIkmTZpg27ZtVRpH165d4efnhzt37iAnJwd79+4tVpSUpWHDhvjqq68wadIkAIVL7LZt2xajRo3CU089BeB/PR9Fd0v/+uuv4e/vX6Wfoahbt24V6zF57rnnSsy5qYwdO3ZIr7t374569eqZfE8iIiKiR7EAoWpjb2+PdevW4dlnn0VqaioA4OLFi7h48WKJto0bN0ZkZCQ+/PDDKo/Dzs4OL730EubNmwcA2LJli6wCBAAmTpyIe/fuYe7cudDr9dBqtVi7di3Wrl1b6nPmzZuHiIiIKo3/UWvXri22VG5VDb/asmWL9Lq6PwMRERHZLg7Bomr15JNP4vjx4+jVq1ep511cXDBx4kScPXu2WldcmjJlijScateuXbh//77sa+fMmYNDhw4hJCSkzDadOnXCoUOHMHv2bJNjNUYUxWLFT4MGDdC9e3eT73v69GmpMKxfvz5GjBhh8j2JiIiISiOIRX9KJapGN2/exNGjR5GUlAQXFxf4+/sjLCwMnp6eZnl+REQEvv/+ewDAZ599hhkzZlT4HtevX8fp06eluRL169dHhw4dEBgYWKWxmtvEiROlVbw+/fRTzJw5U+GIiIiIyFqxACGbcePGDTRp0gQFBQXw8/PDjRs34OjoqHRYiktMTERAQAC0Wi0ef/xx3Lhxo8RGkkRERERVhUOwyGY0atRImlB+584dbNiwQeGIaoYvv/wSWq0WAPDuu++y+CAiIqJqxR4Qsin3799HUFAQHjx4gIYNG+Kvv/6Cs7Oz0mEp5u7du2jcuDFycnLQokULxMbGFtuAkYiIiKiq8ZsG2RQvLy9s2rQJUVFRAAqXtG3SpInCUSnn5s2bePvttwEAAwcOZPFBRERE1Y49IEREREREZDacA0JERERERGbDAoSIiIiIiMyGBQgREREREZkNCxAiIiIiIjIbFiBERERERGQ2LECIiIiIiMhsWIAQEREREZHZsAAhIiIiIiKzYQFCRERERERmwwKEiIiIiIjMhgUIERERERGZDQsQIiIiIiIyGxYgRERERERkNixAiIiIiIjIbFiAEBERERGR2bAAISIiIiIis2EBQkREREREZuOgdABkmtTUVKVDqBAXFxfY2dlBr9cjNzdX6XBqLOapfMyRPMyTPMyTPMyTPMyTPMyTPJaYp8cee8zoeRYgZFYqlQr29vbQ6XQW84dICcxT+ZgjeZgneZgneZgneZgneZgneawxTxyCRUREREREZsMChIiIiIiIzIYFCBERERERmQ0LECIiIiIiMhsWIEREREREZDYsQIiIiIiIyGxYgBARERERkdlwHxAb88YbbygdQo2wdOlSpUMgIiIisknsASEiIiIiIrNhD4iFEwQBdnasIyvK3t5e6RBks6RYlcIcycM8ycM8ycM8ycM8ycM8yWMteWIBYuFUKhXUarXSYVgcT0/PMs+NHj3ajJHUbJs2bVI6hHLZ29sb/f+TCjFP8jBP8jBP8jBP8jBP8lhTnliAWLicnBxotVqlw7A4Dx48UDoEi1CT81SrVi3Y29tDp9MhIyND6XBqLOZJHuZJHuZJHuZJHuZJHkvMU3mFEgsQCyeKInQ6ndJhWBzmTB5LyZOlxKk05kke5kke5kke5kke5kkea8kTJw8QEREREZHZsAAhIiIiIiKzYQFCRERERERmwwKEiIiIiIjMhgUIERERERGZDQsQIiIiIiIyGxYgRERERERkNiYXIPv27auKOIiIiIiIyAaYXID07dsXjRs3xqJFi5CcnFwVMRERERERkZWqkiFYN2/exOzZs9GgQQM8//zzOHDgQFXcloiIiIiIrIzJBciECRPg4uICURSRn5+Pbdu2oXfv3ggODsZ//vMfpKamVkWcRERERERkBRxMvcEPP/yAxYsXY926dVi1ahXi4uIAANevX8esWbPw7rvvYtiwYXjllVcQFhZm6uPKlJ6ejm3btiE6Ohr379+Hs7MzAgMD0b9/f4SEhFT4fjqdDnFxcbh27RquXbuG69evIykpCQAwatQojB492uj1ixcvxsGDB4228ff3x/LlyyscGxERERGRpTK5AAEADw8PvP7663j99ddx8uRJfPvtt/jpp5+Qk5ODvLw8bNmyBVu2bEFQUBAmT56MCRMmoE6dOlXxaABAQkIC5syZg/T0dACASqWCRqNBbGwsYmNjMWjQIEyaNKlC90xNTcV7771ncmxOTk5Qq9WlnqtVq5bJ9yciIiIisiRVUoAU1alTJ3Tq1AlLliwp0Sty9epVzJgxA7Nnz8aIESPwyiuv4JlnnjHpefn5+Zg/fz7S09PRsGFDvPnmmwgICIBWq0VkZCQ2btyI3bt3IyAgAD179qzQvVUqFRo1aoTGjRsjMDAQP/74IxITEyt0j9DQUEybNq1C1xARERERWatq2wfE0Cty4cIFHD9+HOPGjZPmimi1WmzatAlhYWFo0aIFli5diocPH1bqOXv37kVSUhKcnZ0xd+5cBAQEAACcnZ0RHh6Ofv36AQA2bNiAgoIC2ff19vbG5s2bsXDhQkRERCAsLAwuLi6VipGIiIiIiAqZZSPCTp06Ye3atbh79y5ee+016bgoirhy5QqmT58OPz8//Otf/8I///xToXsfPnwYANC1a1d4e3uXOD98+HAIgoC0tDRcvHhR9n3t7OwgCEKFYiEiIiIiIuPMUoAUFBRgy5YtGDZsGL766isIggBRFAEUFiGiKCI7OxsrVqxAkyZNsGrVKln3zcnJwdWrVwEA7dq1K7WNt7c3/Pz8AADnz5+vgk9DRERERESVVeVzQIq6du0aVq5cibVr10rL8RoKj44dO2LKlCno3bs3tmzZgm+//RZ//fUXsrOz8eqrr8Lf3x99+vQxev87d+5I92vYsGGZ7Ro2bIjbt2/j9u3bVfTJ5Ltw4QImT56MlJQUODk5wdfXF+3bt8eAAQPg6elp9niIiIiIiJRU5T0g+fn52Lx5M7p3744mTZrg888/R0pKCkRRhEqlQkREBM6ePYuoqChMmDABvr6+mDZtGi5fvoy1a9dCrVZDFEV88skn5T4rLS1Nem1sVS3DuQcPHpj+ASsoNTUVycnJcHFxQW5uLq5fv46tW7fitddeY48MEREREdmcKusBuXr1qtTbcf/+fQD/6+1o2rQppkyZgvHjx8PDw6PMe4wbNw7x8fFYsGAB/vzzz3KfmZubK712dnYus53hXE5OjqzPUhUCAwMRHByMDh06wMvLC3Z2dsjOzkZ0dDTWrFmDtLQ0fPzxx/jiiy9Qv359s8VFRERERKQkkwuQH3/8EStXrsTRo0cB/K/ocHR0xNChQzFlypQKbUDYsWNHAJCKGEs1aNCgEsfUajXCwsLQvHlzTJs2DVlZWfjxxx8xY8YMBSIkIiIiIjI/kwuQMWPGFJtU7ufnh1deeQUTJ05E3bp1K3w/Jycn2W2LLour1WrL3PBPq9UCKNzXoybw8fHBgAEDsGXLFpw5cwZ6vR52dqWPhtuwYQM2bdpU5r1GjBiBCRMmVFeoVovzb+SpyXky/Jmxs7Or0XEqjXmSh3mSh3mSh3mSh3mSxxrzVGVDsHr37o0pU6Zg0KBBZX6ZlqNjx444dOiQrLZF532kpaWVWYAY5orUpP/TgoODAQDZ2dnIzMwsc2iaRqNBcnJymffJzs6Gvb19tcRozZgzeYzlafTo0WaMpGYz9iNBTSEIAv+9l4F5kod5kod5kod5ksea8mRyATJjxgxMnjwZgYGBVREPPD090a1bN1lt/fz8pN6XhIQEabndRyUkJAAAGjRoUCUxmpOrqyt8fHzKPK9Wq6HT6cwYkXVgzuRhnuSpyXky7GkkiiL0er3S4dRYzJM8zJM8zJM8zJM8lpin8golkwuQTz/91NRbVJpKpUJQUBDi4+MRExODzp07l2iTmpoqLb/bunVrc4dYpvj4eACFn8Hd3b3MdmPHjsXYsWPLPJ+amqrI6l6WjjmTh3mSpybnydPTE/b29tDr9TU6TqUxT/IwT/IwT/IwT/JYYp4ee+wxo+dNXob35Zdfxssvv4zY2NgKXRcXF4eXX34ZERERJj3fMMH96NGjSElJKXF+x44dEEURderUQcuWLU16llyG+TBlSUlJwW+//QYAeOqpp0waskZEREREZElM/ua7Zs0arF27VhrmJNc///yDNWvWYM2aNSY9v0+fPqhbty5yc3Mxb9483Lx5E0DhxPNt27bh119/BVDYk+DgULzDZ+LEiRg8eDAWL15c6r01Gg0yMjKkfwzdXlqttthxwyR3g8OHD2PhwoWIiopCRkaGdDwnJwdHjhzBrFmzkJmZCZVKhRdeeMGkz09EREREZEmqdSd0c3B0dMS7776LOXPm4NatW5g6dSrUajVyc3OlgmHgwIHo2bNnhe+9YMECxMXFlTi+c+dO7Ny5U3o/atSoYhNy9Xo9Tp48iZMnTwIoHGbl4OAAjUYjxeTh4YGZM2eWOW+FiIiIiMgaKVaAGCZtPtorURn+/v5YtmwZtm/fjujoaKSmpsLV1RWNGjXCgAEDEBISYvIzKqJly5YYO3YsLl++jH/++QcZGRnIzs6Gq6srGjRogKeeegp9+vQxOveDiIiIiMgaKVaAGIZK1apVq0ruV7t2bURERFRoTsnq1auNnv/4448rFYuPjw/Cw8MrdS0RERERkTWrsgJEEARZ7bKzsxETE4MlS5ZAEAQ0a9asqkIgIiIiIqIarkIFyIcffoiPPvqoxHFRFDF06NBKBfDcc89V6joiIiIiIrI8Fe4BKWuJ2fKWni1NWFgYXnvttQpfR0RERERElqlCBcgTTzxRYpfyI0eOQBAENG/evPxNR+zs4ObmhoCAAPTs2RP9+/fnHhhERERERDakQgXIhAkTMGHChGLHDAXEggULMHjw4KqLjIiIiIiIrI7Jk9C7du0KQRDK7f0gIiIiIiIyuQA5fPhwFYRBRERERES2gBMwiIiIiIjIbFiAEBERERGR2cgeglV0/4+5c+eWeryyit6PiIiIiIisl+wC5IMPPpB2Oy9aMBQ9XlksQIiIiIiIbEOFJqGLolhqsVGZTQgNTC1eiIiIiIjIcsguQA4dOlSh40RERERERI+SXYA8ugN6eceJiIiIiIgexVWwiIiIiIjIbFiAEBERERGR2Zi8EzopSxAE2Nmxjqwoe3t7pUOwCMyTPJaSJ0uJU2nMkzzMkzzMkzzMkzzWkicWIBZOpVJBrVYrHYbF8fT0VDoEi8A8yWMJebK3t7eIOJXGPMnDPMnDPMnDPMljTXmSXYBUV8UlCAIKCgqq5d62ICcnB1qtVukwLM6DBw+UDsEiME/y1OQ81apVC/b29tDpdMjIyFA6nBqLeZKHeZKHeZKHeZLHEvNUXqEkuwAxZa8Pqj6iKEKn0ykdhsVhzuRhnuSxlDxZSpxKY57kYZ7kYZ7kYZ7ksZY8yS5Aunbtyk0DiYiIiIjIJLILkMOHD1djGEREREREZAu4fBIREREREZkNCxAiIiIiIjIbFiBERERERGQ2LECIiIiIiMhsZE9C/+ijj6TXc+fOLfV4ZRW9HxERERERWS/ZBcgHH3wgLcNbtGAoeryyWIAQEREREdkG2QUIULjpXWnFhimbFHJvESIiIiIi2yG7ADl06FCFjhMRERERET1KdgHSrVu3Ch0nIiIiIiJ6VIWGYNVk6enp2LZtG6Kjo3H//n04OzsjMDAQ/fv3R0hISIXvp9PpEBcXh2vXruHatWu4fv06kpKSAACjRo3C6NGjZd3nxo0b2LlzJy5evIiMjAx4eHjgySefxLBhwxAQEFDhuIiIiIiILJlVFCAJCQmYM2cO0tPTAQAqlQoajQaxsbGIjY3FoEGDMGnSpArdMzU1Fe+9955JcR05cgRLlixBQUEBAMDV1RX379/HkSNHcPz4cUyfPh3PPPOMSc8gIiIiIrIk1VaAJCcn4+7du8jMzIS7uzvq1asHHx+fKn9Ofn4+5s+fj/T0dDRs2BBvvvkmAgICoNVqERkZiY0bN2L37t0ICAhAz549K3RvlUqFRo0aoXHjxggMDMSPP/6IxMREWdcmJCRIxUdoaCgmTpyIOnXqIC0tDatWrcLx48exePFiBAQEwM/PrzIfnYiIiIjI4lRpAfL3339j2bJl2LZtG27fvl3ivL+/P0aOHIl//etfaNiwYZU8c+/evUhKSoKzszPmzp0Lb29vAICzszPCw8ORlpaG3377DRs2bEBYWBgcHOR9ZG9vb2zevLnYKl07d+6UHdfGjRtRUFCAgIAAvPXWW7C3twcA1KlTBzNmzMDdu3dx8+ZNbNy4EbNmzarAJyYiIiIislxVthP6V199hRYtWuDLL7/E7du3IYpiiX8SEhLw+eefo0WLFvj666+r5LmHDx8GAHTt2lUqPooaPnw4BEFAWloaLl68KPu+dnZ2lV4iWKPR4PTp0wCAoUOHSsWHgb29PYYOHQoAiI6ORnZ2dqWeQ0RERERkaaqkB2ThwoV49913ARTuCWJnZ4fmzZsjKCgIrq6u0Gg0uHbtGi5dugS9Xo/s7Gy8/vrryMjIwL///e9KPzcnJwdXr14FALRr167UNt7e3vDz88Pt27dx/vx5tG3bttLPk+vSpUvSvI+y4jIcz8/Px+XLl9G+fftqj4uIiIiISGkm94DExMRg7ty5UuExffp0JCQk4OLFi9ixYwfWr1+PHTt24MKFC7h9+7Y0HEkURbz33ns4d+5cpZ99584daRNEY0O6DOdKGxZWHQzPqV27Njw8PEpt4+HhIZ1LSEgwS1xEREREREozuQBZtmwZdDodBEHAhg0b8Pnnn6NevXqltvX19cVnn32GjRs3AgD0ej2WLl1a6WenpaVJr+vUqVNmO8O5Bw8eVPpZFWF4jrGYip43V1xEREREREozuQA5dOgQBEHAwIED8fzzz8u6Jjw8HIMHD4YoiibtpJ6bmyu9dnZ2LrOd4VxOTk6ln1URhucYi6noeXPFRURERESkNJMLkHv37gEABg4cWKHrBgwYUOx6IiIiIiKyfiZPQq9duzaSk5NRu3btCl9X9H8rw8XFRXqt1WqhVqtLbafVagEU7uthDobnGJ5bFjlxbdiwAZs2bSrz/IgRIzBhwoRKRGnbPD09lQ7BIjBP8tTkPNnZ2Un/W5PjVBrzJA/zJA/zJA/zJI815snkAqR58+ZITk6WVqOS69q1a9L1lVV0jkVaWlqZBYhhroi5/k8zxFV0jkpp5MSl0WiQnJxc5vns7OwSy/xS+ZgzeZgneSwhT4IgWEScSmOe5GGe5GGe5GGe5LGmPJlcgIwdOxaHDh3CunXrMGPGDDg5OZV7TV5eHtasWQNBEDBu3LhKP9vPzw+CIEh7jJS1o7hhlakGDRpU+lkVYXjOw4cPkZGRgVq1apVok56ejvT0dACFGzSWxdXV1egO8mq1GjqdzsSIbQ9zJg/zJE9NzpNhTyNRFKHX65UOp8ZinuRhnuRhnuRhnuSxxDyVVyiZXIC8+OKLWL9+PQ4fPowxY8Zg3bp1RocU5ebmYvz48bh69Sq6d++OF198sdLPVqlUCAoKQnx8PGJiYtC5c+cSbVJTU6VlcVu3bl3pZ1VE8+bN4eDggIKCAsTExCAsLKxEG8Pyw46OjmjWrFmZ9xo7dizGjh1b5vnU1FSuolUJzJk8zJM8NTlPnp6esLe3h16vr9FxKo15kod5kod5kod5kscS8/TYY48ZPW/yJHRBEBAZGYlhw4Zh+/btaNasGf7zn//g3LlzyMrKgiiKyMrKQmxsLD777DM0a9YM27dvx4gRI7Br1y5THy99uT969ChSUlJKnN+xYwdEUUSdOnXQsmVLk58nh1qtRocOHQAAkZGRJX4d1el0iIyMBAB07NixzKFjRERERETWRnYPiNwxZwkJCZg1a1aZ5w0bB27fvh3bt2+HIAjSruGV0adPH/z8889ISkrCvHnzMH36dAQEBECr1WL37t349ddfART2JDg4FP+4EydORHJyMrp3745p06aVuLdGoylWPBi6vbRaLTIyMqTjzs7OJZbcHTNmDE6fPo3r16/jiy++wMSJE+Hp6YkHDx5g9erVuH79OhwdHTFmzJhKf3YiIiIiIksjuwAxFA5V1bYi9zPG0dER7777LubMmYNbt25h6tSpUKvVyM3NlQqGgQMHomfPnhW+94IFCxAXF1fi+M6dO7Fz507p/ahRozB69Ohibfz9/TF16lQsWbIEf/zxB44dOwa1Wg2NRgMAcHBwwNSpU8uct0JEREREZI1kFyBdu3aFIAjVGUul+fv7Y9myZdi+fTuio6ORmpoKV1dXNGrUCAMGDEBISIgicXXr1g0NGjTAjh07EBcXh4yMDGko2LBhwxAQEKBIXERERERESpFdgBw+fLgawzBd7dq1ERERgYiICNnXrF692uj5jz/+2NSw0KhRI8yYMcPk+xARERERWQOTJ6ETERERERHJxQKEiIiIiIjMhgUIERERERGZDQsQIiIiIiIyG5N3Qi8qOzsbkZGRiIqKwp07d5CRkVFiE75HCYKAAwcOVGUYRERERERUQ1VZAbJixQrMnj0b6enpsq8RRbHGLu1LRERERERVr0oKkPnz5+P999+XtbmgoeCoqo0IiYiIiIjIcpg8B+TKlSt4//33AQDBwcE4cOAAcnJyABQWG7t27UJWVhYuXryIRYsWwdfXFwDw0ksvITc3t9whWkREREREZD1M7gFZsWIFRFGEWq3Gvn374O/vX6KNWq1GixYt0KJFC0yaNAlDhgzBmjVroNFosHnzZlNDICIiIiIiC2FyD8iRI0cgCAJGjhxZavHxqNq1a2PXrl2oU6cOfvrpJ/z888+mhkBERERERBbC5AIkISEBABASElLq+by8vBLHPD09MWHCBIiiiPXr15saAhERERERWQiTC5DMzEwAgLe3d7HjKpWq2PlHtW3bFgBw5swZU0MgIiIiIiILYXIB4urqCqBkT4eHhweA//WQPKqgoAAAcO/ePVNDICIiIiIiC2FyAfLEE08AKFlINGnSBKIo4vjx46Ved/78eQCAk5OTqSEQEREREZGFMLkAad26NURRxMWLF4sd79q1KwDg0KFDOHv2bLFzN27cwOrVqyEIApo1a2ZqCEREREREZCFMLkDCwsIAAAcPHix2fPz48XBwcIBer0f37t3x9ttvY+XKlXj77bfx1FNPISsrCwAwatQoU0MgIiIiIiILYfI+IIMGDYK9vT3+/vtvnDhxAp07dwYABAYGYvbs2fjoo4+QlZWFzz//vMS17dq1w5QpU0wNwaYJggA7O5PrSJtjb2+vdAgWgXmSx1LyZClxKo15kod5kod5kod5ksda8mRyAeLl5YX4+Hjk5eXBx8en2LkPPvgArq6umDdvntTjARR+aQ4PD8eKFSs4B8REKpUKarVa6TAsjqenp9IhWATmSR5LyJO9vb1FxKk05kke5kke5kke5kkea8qTyQUIAAQEBJR5bubMmXjjjTdw8uRJJCUlwdXVFU899RR8fX2r4tE2LycnB1qtVukwLM6DBw+UDsEiME/y1OQ81apVC/b29tDpdMjIyFA6nBqLeZKHeZKHeZKHeZLHEvNUXqFUJQVIeZydnaW5IlS1RFGETqdTOgyLw5zJwzzJYyl5spQ4lcY8ycM8ycM8ycM8yWMteeLkASIiIiIiMptq7QF5+PAhMjMz4e7ujtq1a1fno4iIiIiIyAJUaQ9IVlYWli9fjrCwMLi7u8PLywtPPPEEvLy84O7ujmeffRZff/11sQnpRERERERkO6qsANm9ezeCgoIwdepU/PHHH9BoNBBFUfpHo9Hg6NGjeP311xEUFIRffvmlqh5NREREREQWokoKkHXr1mHYsGFITk6WCg53d3e0adMGXbp0QZs2bVCrVi3p3L179zB06FCsX7++Kh5PREREREQWwuQC5Nq1a3j11Veh0+kgiiKee+45nDx5Eunp6YiJicEff/yBmJgYPHz4EFFRURg+fDgAQK/XY/Lkybh+/brJH4KIiIiIiCyDyQXIl19+idzcXAiCgE8//RTbt2/H008/XWrbjh074qeffsJ//vMfAIBWq8WXX35paghERERERGQhTC5A9u3bB0EQ0LVrV8yYMUPWNW+++Sa6desGURSxd+9eU0MgIiIiIiILYXIB8s8//wAARowYUaHrDO0N1xMRERERkfUzuQBxc3MDADz++OMVus7Hx6fY9UREREREZP1MLkAaN24MAEhISKjQdbdv3wYABAUFmRoCERERERFZCJN3Qn/++ecRFRWFTZs2Yfr06RAEodxrRFHExo0bIQgCRo0aZWoIAID09HRs27YN0dHRuH//PpydnREYGIj+/fsjJCSk0vctKCjAL7/8giNHjuDu3bsAgPr166Nbt24YMGAAHBxKT+HixYtx8OBBo/f29/fH8uXLKx0bEREREZGlMbkAefXVV/HDDz/g3LlzmD59OhYvXlzuNW+++SbOnTuHNm3aYPLkyaaGgISEBMyZMwfp6ekAAJVKBY1Gg9jYWMTGxmLQoEGYNGlShe+bk5OD9957D/Hx8QAAJycnAIVLD1+7dg3Hjx/HRx99BBcXlzLv4eTkBLVaXeq5WrVqVTgmIiIiIiJLZnIB4uzsjF9//RUjR47EsmXLEBUVhRkzZqBHjx7w9PSU2j18+BD79+/HF198gejoaHTq1Anbtm2TvtRXVn5+PubPn4/09HQ0bNgQb775JgICAqDVahEZGYmNGzdi9+7dCAgIQM+ePSt076+//hrx8fFwdXXFG2+8IfWkREVFYenSpbhy5Qq++eYbTJ8+vcx7hIaGYtq0aaZ8RCIiIiIiqyG7AGnUqJHR8/n5+RBFEadPn8bzzz8PAPD09ISrqys0Gg0ePHgAoHD4lSAISEhIQJcuXSAIgkmbEe7duxdJSUlwdnbG3Llz4e3tDaCwMAoPD0daWhp+++03bNiwAWFhYWUOmXrUzZs3cfToUQDA66+/jk6dOknnOnXqBL1ej0WLFuHw4cMYNmwYGjZsWOnPQERERERkK2QXILdu3YIgCBBFsdTzgiBI8z8MbdLS0pCWllaiHQDcvXtXKkZMcfjwYQBA165dpeKjqOHDh2PPnj1IS0vDxYsX0bZtW1n3PXLkCERRhK+vb7Hiw6Bz587w9fVFYmIijhw5gvHjx5v0OYiIiIiIbIHsAsTf39/kYqGq5eTk4OrVqwCAdu3aldrG29sbfn5+uH37Ns6fPy+7ALlw4QIAoG3btqV+bkEQ0LZtWyQmJkptiYiIiIjIuAr1gNQ0d+7ckXpbjA2BatiwIW7fvi0t/VseURRx586dcu/r7+8PAEbve+HCBUyePBkpKSlwcnKCr68v2rdvjwEDBhSbI0NEREREZAtM3gdESUWHd9WpU6fMdoZzhnko5cnJyUFubq7s++bk5CAnJ6fUNqmpqUhOToaLiwtyc3Nx/fp1bN26Fa+99hrOnz8vKx4iIiIiImth8ipYSjIUCUDhpPOyGM6VVSQ8qmg7Ofc1XKNSqaT3gYGBCA4ORocOHeDl5QU7OztkZ2cjOjoaa9asQVpaGj7++GN88cUXqF+/vqy4iIiIiIgsnUX3gNRkgwYNQv/+/eHt7Q07u8I0q9VqhIWF4dNPP4WbmxtycnLw448/KhwpEREREZH5VHkPSGpqKn799VdERUUhMTERmZmZcHd3R7169fD0009jwIABeOyxx6rkWUU3ANRqtWVu+KfVagGgWA+FMUXbGa41dt+K3BsAfHx8MGDAAGzZsgVnzpyBXq+XipRHbdiwAZs2bSrzXiNGjMCECRNkP5sKcf6NPMyTPDU5T4a/W+zs7Gp0nEpjnuRhnuRhnuRhnuSxxjxVWQGSnZ2Nt99+G99//32ZX9q//fZbODs7Y+LEiVi0aFGFvrSXpuj8jLS0tDILEMNcEbn/p6lUKqhUKuTk5JRYRri0+xraV0RwcDCAwrxlZmbCw8Oj1HYajQbJycll3ic7Oxv29vYVejaBOZOJeZLHEvIkCIJFxKk05kke5kke5kke5kkea8pTlRQgqamp6NatG65cuVLmPiEGubm5+Oqrr3Dw4EEcOXIEXl5elX6un5+ftDdJQkIC/Pz8Sm2XkJAAAGjQoIGs+wqCAD8/P1y9elW6tiruWxmurq7w8fEp87xarYZOp6u251sr5kwe5kmempwnOzs76e9JvV6vdDg1FvMkD/MkD/MkD/MkjyXmqbxCqUoKkOHDh+Py5csACnsDXnjhBfTp0wfBwcFwc3NDVlYW4uPjsXfvXmzevBnZ2dm4dOkShg8fLm0kWBkqlQpBQUGIj49HTEwMOnfuXKJNamqqtExu69atZd+7VatWuHr1Ks6dO1dmm9jYWKltRcXHxwMo/Azu7u5lths7dizGjh1b5vnU1FTZq3vR/zBn8jBP8tTkPHl6esLe3h56vb5Gx6k05kke5kke5kke5kkeS8xTedMtTJ6EvnPnTvzxxx/SxnyXLl3C6tWrMXLkSLRu3RqBgYFo3bo1Ro4cidWrV+PPP/9E+/btAQB//PEHIiMjTXp+WFgYAODo0aNISUkpcX7Hjh0QRRF16tRBy5YtZd+3a9euEAQBd+/excmTJ0ucP3HiBO7evQtBEKQYDMrrBUpJScFvv/0GAHjqqafKnP9BRERERGRtTP7mu3nzZgCFO47v37/f6MZ9QOHGfr///rs0rMjYBGs5+vTpg7p16yI3Nxfz5s3DzZs3ARROEN+2bRt+/fVXAIU9CQ4OxTt8Jk6ciMGDB2Px4sUl7hsQEICuXbsCAJYtW4aoqCiIoghRFBEVFYXly5cDKCyADBsSGhw+fBgLFy5EVFQUMjIypOM5OTk4cuQIZs2ahczMTKm3iIiIiIjIVpg8BOvUqVMQBAEvv/yy0U37ivLy8kJERAQWLlyIU6dOmfR8R0dHvPvuu5gzZw5u3bqFqVOnQq1WIzc3VxonN3DgQPTs2bPC9/6///s/JCYmIj4+Hh9//DGcnJwAAHl5eQCApk2bYsqUKSWu0+v1OHnypNRzolKp4ODgAI1GI8Xk4eGBmTNnljlvhYiswxtvvKF0CDXG0qVLlQ6BiIhqAJMLEMMKTRWdB2EYDmVshSe5/P39sWzZMmzfvh3R0dFITU2Fq6srGjVqhAEDBiAkJKRS91WpVPjkk0/wyy+/4MiRI7h79y6Awk0Gw8LCMGDAgBK9KkDhZxs7diwuX76Mf/75BxkZGcjOzoarqysaNGiAp556Cn369DE694OIiIiIyBqZXIA4OTlBq9VKvQJyGdo7OjqaGgIAoHbt2oiIiEBERITsa1avXl1uGwcHBwwdOhRDhw6VfV8fHx+Eh4fLbk9EREREZCtMngNSr149AIUTyivi6NGjAID69eubGgIREREREVkIkwuQsLAwiKKI9evX4/z587KuiY2NxYYNG0pdQYqIiIiIiKyXyQXIxIkTIQgC8vPz0bNnT+zYscNo+x07dqBXr17Iy8uDIAiYNGmSqSEQEREREZGFMHkOSLt27fDqq6/im2++QVpaGkaOHIlGjRqhV69eCA4OhqurKzQaDa5evYr9+/fj+vXrEEURgiDg1VdfRdu2bavicxARERERkQWokp3Qly1bhoyMDGzcuBEAcOPGDXz77beltjVs0jdmzBguyUhEREREZGOqpACxs7PD+vXrMXjwYCxatAgxMTFltm3fvj3+/e9/Y/jw4VXxaCIisgLcL6UQf5gjIltQJQWIwciRIzFy5EgkJCTg1KlTSExMRGZmJtzd3eHr64unn366xK7hRERERERkO0wuQNatWwcAqFu3Lnr37g2gcGNAFhpERERERPQok1fBevHFF/HSSy/h2LFjVREPERERERFZMZMLEDc3NwBA8+bNTQ6GiIiIiIism8kFiK+vLwAgPz/f5GCIiIiIiMi6mVyAPPvsswCA06dPmxwMERERERFZN5MLkMmTJ8POzg5r167FP//8UxUxERERERGRlTK5AGnbti0WLFiAzMxM9OrVCxcuXKiKuIiIiIiIyApVyTK8devWRb9+/bBnzx60a9cOoaGheOaZZ+Dn5weVSlXuPcaPH29qGERERFaPGzYW4oaNRJbN5ALkxRdfhCAIAABBEKDX6/HHH3/gjz/+kHW9IAgsQIiIiKjKsFArxEKNaqoq2QldFEWj76n6CIIAOzuTR9LZHHt7e6VDsAjMkzzMkzzMU/mYI3mYJ3ksKU+WFKuSrCVPJhcgP/zwQ1XEQZWkUqmgVquVDsPieHp6Kh2CRWCe5GGe5GGeysccycM8yWMpebK3t7eYWJVkTXkyuQCZMGFCVcRBlZSTkwOtVqt0GBbnwYMHSodgEZgneZgneZin8jFH8jBP8tT0PNWqVQv29vbQ6XTIyMhQOpwayxLzVF6hVCVDsEg5oihCp9MpHYbFYc7kYZ7kYZ7kYZ7KxxzJwzzJY0l5sqRYlWQteTKpAPnnn39w4cIFpKenw8PDAy1btoSfn19VxUZERERERFamUgVIdHQ0pk+fjqioqBLnQkJC8OWXX6Jjx44mB0dERERERNalwssn7du3D2FhYYiKioIoiiX+OXnyJLp164a9e/dWR7xERERERGTBKlSAZGZmYsKECcjNzZWW2m3cuDE6d+6Mxo0bS+20Wi0mTJhgMRNliIiIiIjIPCo0BGv9+vW4d+8eBEFA+/btsXbtWjRr1kw6f+XKFbz44ouIjo5GSkoK1q9fj3/9619VHjQRERERmYYbNhbiho3mV6ECZM+ePQCAxx57DHv37i2xxFbTpk2xZ88eNGvWDCkpKdizZw8LECIiIiKySCzS/qcqC7UKDcG6cOECBEHA+PHjy1zf19PTE+PHj4coirh48WKVBElERERERNahQgVIWloaAKBNmzZG27Vu3RoAcP/+/cpFRUREREREVqlCBYhGowEAuLu7G23n5uYGoHCXbiIiIiIiIoMKL8NLRERERERUWSxAiIiIiIjIbCq1E7ogCFUdh9VKT0/Htm3bEB0djfv378PZ2RmBgYHo378/QkJClA6PiIiIiMisKlWADB06VFY7URRhb29vtI0gCCgoKKhMGDVeQkIC5syZg/T0dACASqWCRqNBbGwsYmNjMWjQIEyaNEnhKImIiIiIzKdSBQgAaSf00giCIPWSGGtnzfLz8zF//nykp6ejYcOGePPNNxEQEACtVovIyEhs3LgRu3fvRkBAAHr27Kl0uEREREREZlHhOSCiKJZbVBja2GrxAQB79+5FUlISnJ2dMXfuXAQEBAAAnJ2dER4ejn79+gEANmzYYLU9QEREREREj6pQAaLX66v8H51OV12fTVGHDx8GAHTt2hXe3t4lzg8fPhyCICAtLY0bNhIRERGRzeAqWNUgJycHV69eBQC0a9eu1Dbe3t7w8/MDAJw/f95ssRERERERKYkFSDW4c+eONPysYcOGZbYznLt9+7ZZ4iIiIiIiUhoLkGqQlpYmva5Tp06Z7QznHjx4UO0xERERERHVBCxAqkFubq702tnZucx2hnM5OTnVHhMRERERUU3AAoSIiIiIiMym0vuAUNlcXFyk11qtFmq1utR2Wq0WQOEGhWXZsGEDNm3aVOb5ESNGYMKECZWM1HZ5enoqHYJFYJ7kYZ7kYZ7KxxzJwzzJwzzJwzzJU5V5YgFSDYrO+0hLSyuzADHMFTH2f6hGo0FycnKZ57Ozs8vdbb4oY8UMFWKO5GGe5GGe5GGe5GGe5GGe5GGeysccVQ8WINXAz88PgiBAFEUkJCRIy+0+KiEhAQDQoEGDMu/l6uoKHx+fMs+r1WqL2kvFzs5Oyo1er1c6nBqLeSofcyQP8yQP8yQP8yQP8yQP8ySPJeapvB/HWYBUA5VKhaCgIMTHxyMmJgadO3cu0SY1NVVafrd169Zl3mvs2LEYO3ZsmedTU1MtahUtT09P2NvbQ6/XW1Tc5sY8lY85kod5kod5kod5kod5kod5kscS8/TYY48ZPc9J6NUkLCwMAHD06FGkpKSUOL9jxw6Ioog6deqgZcuWZo6OiIiIiEgZLECqSZ8+fVC3bl3k5uZi3rx5uHnzJoDCiefbtm3Dr7/+CqCwh8PBgR1RRERERGQb+M23mjg6OuLdd9/FnDlzcOvWLUydOhVqtRq5ubnS+L2BAweiZ8+eCkdKRERERGQ+LECqkb+/P5YtW4bt27cjOjoaqampcHV1RaNGjTBgwACEhIQoHSIRERERkVmxAKlmtWvXRkREBCIiIpQOhYiIiIhIcZwDQkREREREZiOIoigqHQTZjg0bNkCj0cDV1dXo8sK2jnkqH3MkD/MkD/MkD/MkD/MkD/MkjzXmiQUImVX//v2RnJwMHx8f/Pbbb0qHU2MxT+VjjuRhnuRhnuRhnuRhnuRhnuSxxjxxCBYREREREZkNCxAiIiIiIjIbFiBERERERGQ2LECIiIiIiMhsWIAQEREREZHZsAAhIiIiIiKz4U7oZFajR4+W1rKmsjFP5WOO5GGe5GGe5GGe5GGe5GGe5LHGPHEfECIiIiIiMhsOwSIiIiIiIrNhAUJERERERGbDAoSIiIiIiMyGBQgREREREZkNCxAiUoQoisjIyEBKSorSoRARkQzTpk3D9OnTkZSUpHQoFk2r1UKj0SgdhqK4DC/JptPpcP36daSkpECr1aJ79+5Kh1TjpKen4+LFi1KORo0apXRINc7Vq1exdetWXLhwAVqtFgCwa9cu6XxWVhbWrl0LQRAQEREBZ2dnhSJVxtSpU9G7d29069YNbm5uSodDZHOioqJw7tw5pKSkIC8vD/Pnz5fO5ebm4ubNmxAEAU2bNlUwSmXcvn0bDg4OqFu3rtKh1Fipqak4d+4cateujQ4dOhQ7l5CQgKVLl+LatWsAgODgYLzxxhvw8/NTIlRFsQAhWXbt2oVt27YhKytLOla0AMnKysI777yDgoICfPzxx/D09FQiTMXk5+fjhx9+wN69e6HT6aTjRQuQrKwsTJ48Gbm5ufjqq69s8i/wffv2YcWKFcVyJAhCsTZubm7SX+AtWrRAt27dzB2mom7duoVVq1bhhx9+QKdOndCrVy+0atVK6bAUN2nSJJPvIQgCVq5cWQXRWJ7U1FT897//xeXLl5GWlgatVouyVuG31TwlJSVh4cKF+PvvvwEU9tI++veTo6MjvvjiC6SkpOCTTz6xuSLEy8sL6enpSodRo+3fvx9btmzBiBEjihUg2dnZeO+995Ceni792fvrr7/w7rvvYvny5Tb3gxMLECrX0qVLcfDgQYiiCEdHRxQUFJRo4+bmhuDgYBw4cADHjh3DoEGDFIhUGXq9HgsWLEBsbCwAwMfHB6mpqdDr9cXaubm54dlnn8Xu3btx/PhxDB8+XIFolXPjxg1888030Ov16NOnD8LCwrBw4UJkZmaWaNujRw/ExMTg7NmzNleAvPDCCzhw4ACSk5Nx9OhR/PHHH/Dx8UGvXr3QvXt3eHl5KR2iIpKTk02+x6NfJm3F4cOH8dVXXyE/P99o0WE4Z4t5ys7Oxty5c3Hv3j14enqiffv2OHbsmNRLa2Bvb48+ffpg/fr1OHnypM0VIG3btsXevXvx119/oUmTJkqHUyOdP38eAPDMM88UO75//348fPgQderUwaRJk+Ds7IzvvvsOd+/exc8//4zRo0crEa5iWICQUVFRUThw4ADUajX+9a9/oVOnTnj55ZdL/QWkW7du+O9//4vz58/bVAFy6NAhnDt3Dp6ennjnnXfQpEkTTJgwodQchYaGYvfu3bhw4YLNFSCRkZHQ6/UYPHgwIiIiAAB2dqVPQ2vZsiUA4Pr162aLr6YYNWoURo0ahfPnz2Pfvn04deoU7t27h40bN+LHH39Eu3bt0LNnT3To0AH29vZKh2s2HM5YOdevX8fSpUuh0+nQpk0btG/fHt999x3UajVefvllPHz4EBcvXsSFCxdQq1YtjBo1Ci4uLkqHbXY///wz7t27h6CgIHzwwQdwc3PDmTNnShQgAPD0009j/fr1uHz5sgKRKis8PBwnTpzA119/jXnz5qFWrVpKh1TjGOY11qtXr9jxqKgoCIKACRMmoHPnzgAAlUqFd955B2fOnGEBQlTU3r17IQgCxo8fj9DQUKNtg4ODIQgCbt26ZZ7gaoiDBw9CEARMnDix3F+EAgMDIQgCEhISzBRdzREXFwdBEDBs2LBy23p4eMDFxQWpqalmiKxmat26NVq3bo2srCwcPnwY+/fvx61bt3D69GmcOXMGHh4e6N69O3r27In69esrHW61e+GFF6rlvseOHUNeXp7Vzmn7+eefodPp8Oyzz2LatGkAgO+++w7Ozs7o1asXAGDkyJG4cOECFi5ciAMHDmDRokUKRqyMkydPSn+PlzcUxs/PD/b29rh7966Zoqs5EhMTMXbsWHz//feYMmUKnn32WTRt2hS1atUq8wclAHjyySfNGKWy0tPT4erqCkdHR+lYQUEB4uPjYWdnh6efflo63rx5c9jb2yMxMVGJUBXFAoSMMkyUevbZZ8tt6+LiApVKhYcPH1ZzVDWLoeDq2LFjuW0dHR3h6uqKjIyMao6q5nn48CFcXFxkzw9ycHBATk5ONUdV87m5uWHgwIEYOHAgrl+/jn379uHo0aN4+PAhdu7ciZ07d6JZs2bo3bs3unTpAicnJ6VDtiirVq1Cenq61RYgf/75JwRBQHh4eLHjjw7FatWqFV555RUsXrwYO3fuLNHe2iUlJcHe3h7BwcHlthUEAWq1GtnZ2WaIrGaZM2eONERPq9Xil19+wS+//FLudUUXGrF2giAgNze32LFr166hoKAAQUFBUKlUxc6p1Wqb/G8dl+ElozQaDVQqlU12ycuVm5sLlUol+4tfQUGBTQ2dMXBxcUFeXl6JuTGlycnJgUajgbu7uxkisxyBgYGYMmUKvvrqKzRr1gyiKEIURVy6dAlLlizBSy+9hPXr19v88o70Pw8fPoSDg0Ox4SCCICAvL69E29DQUNjb2+PYsWPmDLFG0Ov1cHBwMPorvoEoisjNzbW5FfoMDH/vVOQfW+Lt7Q2dTocbN25IxwzDr5o3b16srV6vR3Z2Njw8PMwdpuLYA0JGubu7Iz09HXl5eeV+wU5LS0N2dja8vb3NFF3N4OHhgbS0NOTm5pZbqCUlJSE3N7fE2FBb4Ofnh7/++gs3b95EYGCg0bYnT56EKIrltrM1Fy5cwP79+xEVFYX8/HwAhb+etWvXDn/++ScePHiA7du349ChQ5g/f75N/ntGxZX2JVmlUiEnJ6fE3+uOjo5wdnbGvXv3zBlijfDYY48hMTERDx8+RO3atY22jY+PR35+Pho0aGCe4GqQyMhIpUOo8Vq1aoV//vkHK1aswMSJE/HgwQP8/vvvAFBiWd47d+5Ap9OhTp06SoSqKPaAkFGNGzcGUPjFpzx79+4FADRr1qxaY6ppDPM+Tp06VW7bn3/+GYIgoEWLFtUdVo3TuXNniKKILVu2GG2XlJQk7QPSpUsXM0VXc92/fx9btmzBK6+8grlz5+Lo0aPIy8tDUFAQXn/9daxZswYzZ87Ed999h5kzZ6Ju3bq4f/8+1qxZo3ToVAN4eXkhOzu72OqFvr6+AIArV64Ua3vv3j2bHFYE/G/hi/379xttJ4oiNm7cCEEQ0K5dO3OERhZm+PDhUKvViI+Px9tvv42FCxciJycHTZs2lf49Mzh9+rTN7inDAoSM6tGjB0RRxPr164vtAfKoEydO4KeffoIgCNLERlvRt29f6T9KxpYK3blzJ3799VcAQL9+/cwVXo3Rr18/+Pr6Ijo6Gp988gkuX74sdc2np6fj6tWr2LRpE9588008fPgQTzzxBMLCwpQNWiE6nQ7Hjx/HBx98gIkTJ+LHH3/EvXv3oFarMWDAACxZsgSfffYZevbsKf3CbW9vj9DQUMyfPx92dnb4888/Ff4UVBM0bNgQoiji5s2b0rFWrVpBFEWsXr1aWrEnIyMDy5cvhyAICAgIUCpcxQwdOhR2dnbYtm1bmT8mJSUl4eOPP8b58+fh5OSEAQMGmDlKsgTe3t6YP38+nnzySTg6OsLDwwM9evTAnDlzirUTRRH79u2DKIpo3bq1QtEqRxBtbXAeVdi8efNw5swZ1K1bFz169MCuXbuQnZ2NGTNmICUlBadPn8alS5cgiiK6du2Kt956S+mQzW758uXYv38/3Nzc0KlTJxw7dgy5ubkYN24cUlJSEBMTg+TkZIiiiCFDhuDll19WOmRF3L17Fx9++CGSkpLK3GtAFEXUq1cPH330kc0N5wOA1atX48iRI8jMzJQKtKZNm6JPnz4IDQ2VNdfo5ZdfRlpamk1N/Kwsw5LZ1pqrw4cP48svv8SwYcMwYcIEAIW9av/3f/8nLTHr7u5ebD+ef//73wgJCVEkXiXt27cPX3/9NQCgbt26SE1NRUFBAdq2bYuUlBTcuXNHavvWW2+V2OfB1qSnp+PixYtISUmBVqvlUtkVpNPpcP/+fQCFPZW2NjeUBQiVS6vVYsmSJTh+/HipXxoN/wqFhoZi2rRpxZaesxU6nQ7r16/Hrl27St3My7Cj7rBhwzBu3Dib3OjLICcnBzt37sSBAwdKLLPr6emJnj17YtiwYVCr1QpFqKwhQ4YA+N/GlX369KnwWPPPPvsMDx8+xIIFC6ojRKti7QWIVqvFsWPH4ObmVmz5z7i4OHzxxRfSFyCgcL7I+PHjMXDgQCVCrRHOnDmDlStXljkPxtvbG1OmTEH79u3NHFnNkZ+fjx9++AF79+6FTqeTjhf9M5SVlYXJkycjNzcXX331FerWratApFSTsQAh2eLi4rB//35cuXIFDx48gF6vR+3atdG0aVP07NkTbdq0UTpExSUlJeHAgQOl5qhHjx42sV9DRdy/fx9paWlSnh5//HGlQ1Lc7NmzpSV1bbGYNzdrL0CM0el0uHLlClJTU+Hq6opmzZrB1dW11LbWvl9KUXq9HnFxcbhy5Uqxv5+aNWuGVq1a2dwv1UXp9Xp89NFHiI2NBQD4+PggNTUVer2+xJ+h1atXY/fu3Rg/frzNbbxL5WMBQkRENsuWC5CKsPY87d69G0DhYhleXl4KR1NzHThwAEuXLoWnpyfeeecdNGnSpMx/N65cuYJZs2ahTZs2+PDDD5UJuJrFxcUBKOw9DAoKKnasomxps0aAy/ASEdVooigiMzMTWq3WJufEEJnDd999Bzs7O/Tt21fpUGq0gwcPSjvGG1aALEtgYCAEQUBCQoKZojM/w8aM9evXx1dffVXsWEVZa3FfFhYgRGR2Op0OiYmJyMrKKrY8aGls7Vchg6tXr2Lr1q24cOGCNFn40THWhuWKIyIibHZTNKKq4O7uDr1ez2GP5bh16xYAoGPHjuW2dXR0hKurKzIyMqo5KmWVttkiBxeVjwUIySKKIi5fvoy///4bWVlZxSaelcYWV8PIysrC6dOnkZCQUO4Xa0EQ8MYbb5gxupohOTkZ69atQ1RUVLmFh4Gt/SoEFK7Gs2LFimJ/zh79Rc3NzQ2pqak4d+4cWrRogW7dupk7TKvALwoEFP5aHxsbi/T0dJvclVqu3NxcqFQqWavxAUBBQYFVz5kpbWNGbtYoDwsQKld0dDRWrFiBtLQ02dfYWgHy22+/Yc2aNcjLy5OOlfbFRhAEaUUsWytAkpKSMHPmzGLLy1JJN27cwDfffAO9Xo8+ffogLCwMCxcuLLZMqkGPHj0QExODs2fPsgCppNmzZ8suhsl6DRo0COfOnZM2/aTSeXh4IC0tDbm5uXBxcTHaNikpCbm5uahXr56ZoiNLwgKEjLp48SIWLlwIvV4PoHCtai8vL9m/ftiCEydO4NtvvwUAODg4ICgoiDkqxaZNm5CRkQFXV1eEh4cjJCQEXl5eHPLwiMjISOj1egwePBgREREAADu70veMNeyqe/36dbPFZ21scQdiKql9+/Z46aWXsG7dOmRlZeG5556zyQ0Zy9OkSROcPHkSp06dKvdHj59//hmCIKBFixZmio4sCQsQMuqnn36CXq9Hw4YNMXXqVAQGBiodUo1jGCL05JNP4q233kKdOnWUDaiGOn/+PARBwPTp09GhQwelw6mx4uLipD1jyuPh4QEXF5cS+6kQUcVMmjQJQGGxf/ToURw9ehROTk5wd3cv8wcAQRCwcuVKc4apuL59++LEiRPYuHEjmjVrBh8fn1Lb7dy5E7/++isEQUC/fv3MHCVZAhYgZNTVq1chCALeeustNGzYUOlwaqS///4bgiBg6tSpLD6M0Gg0cHBwsOkNvOR4+PAhXFxc4OnpKau9g4MDcnJyqjkqIuuWnJxc4phWq5UWgCiNLW4o27p1a/Tq1Qv79+/H9OnT0alTJylH27dvR0pKCmJiYqR8Dh482GZ/uOS8UONYgJBROp0OLi4uLD6MEAQBKpWqzF+CqFCdOnWQnp5e5q+JVMjFxQU5OTnQ6/Xl5ionJwcajYaTZolMZGtf/kwxZcoUuLm5YdeuXdi/fz+Awv8Orl+/HgCkeY7Dhw/HuHHjlAxVMZwXWj4WIGRUvXr1cPv2beh0OqteycIU/v7+uHbtGvLy8jjvw4hOnTohMjIS8fHxCA4OVjqcGsvPzw9//fUXbt68We4vhydPnoQoijb7CyNRVenRo4fSIVgMe3t7vPjii+jbty8OHDiAK1eu4MGDB9KO8U2bNkWPHj1Qv359pUNVBOeFysMChIzq2bMnVq1ahVOnTqFz585Kh1MjDRgwAF988QUOHTqEPn36KB1OjRUeHo4TJ07gm2++wbx58+Dm5qZ0SDVS586dceXKFWzZsgWzZ88us11SUpK0D0iXLl3MGCEREVC3bl2MGTNG6TBqHM4LlYcFCBk1YMAAxMTE4Ouvv0adOnW4YkwpunXrhj///BOrV6+GSqVC165dlQ5JcXFxcaUeHzt2LFauXInXXnsNvXv3RlBQEFQqldF72dpGhP369cOePXsQHR2NTz75BEOGDJG67tPT05GcnIzTp0/jl19+gUajQUBAAMLCwpQNmqyerS2dLYoiMjMzodVq4e3trXQ4ZEE4L1QeQbS1v1WoTJs3by71eEFBAfbs2QONRoPmzZvL+tJorfuALFmypMxz0dHR0Gg0eOyxx9C4cWOjObL28Z5DhgypsgmatrgR4d27d/Hhhx8iKSmpzDyKooh69erho48+4hckqnZXrlxBQUGB1f8gcPXqVWzduhUXLlyQJlcX/TsoKytL6nmMiIiAs7OzQpFSTTVq1CgIgoAff/xR6VBqNPaAkOTHH380+qVRFEX8+eefuHTpUrn3stYC5ODBg9KksaKKHktJSUFKSkqp19vShDP+tlF59erVw+LFi7Fz504cOHCgxDK7np6e6NmzJ4YNGwa1Wq1QlGRLbKH3e9++fVixYgV0Op107NH/Jrq5uSE1NRXnzp1DixYtbHYDUK7wVDbOC5WHBQhJWrRoYZPLClbEs88+yxzJEBkZqXQIFk+lUmH06NEYPXo07t+/j7S0NGmS5+OPP650eERW5caNG/jmm2+g1+vRp08fhIWFYeHChcjMzCzRtkePHoiJicHZs2dtsgDhCk/GcV6oPCxASPLxxx8rHUKNN23atGq797Fjx5CXl4fu3btX2zPIMnl5ecHLy0vpMIisVmRkJPR6PQYPHoyIiAgAKHMZ7JYtWwIArl+/brb4agqu8FQ+zguVhwUIUQ2xatUqpKenW20BkpKSAjs7O9lfpO/fvw+9Xs/5DURU7eLi4iAIAoYNG1ZuWw8PD7i4uJQYGmkLuMJTccbmhTo5OeGLL77AunXrbH5eaGlYgBCRWUycOBGenp5Ys2aNrPazZs1CamqqTU5CN9DpdEhMTCx3jDVge6uFEVWlhw8fwsXFBZ6enrLaOzg4ICcnp5qjqnm4wlNxnBdaeSxAqFx6vR6CIJQ692HPnj2Ii4tDfn4+2rdvj969e3OOBJGJkpOTsW7dOkRFRZVbeBjYcqFGZCoXFxfk5ORAr9eXOfTKICcnBxqNBh4eHmaKruYQBAEqlQo+Pj5Kh1IjcF5o5bEAIaP27duHr7/+GqGhoZgxY0axc/Pnz8eZM2cAFE5Ai46ORkxMDN555x0lQiUro9Vqy/0iYI2SkpIwc+ZMZGZmciUxIjPx8/PDX3/9hZs3byIwMNBo25MnT0IUxXLbWSOu8FQc54VWHgsQMiomJgZAYZVf1NmzZ3H69GkAQIcOHeDk5ISTJ0/i1KlTOHbsGEJDQ80eK1mPu3fvIjMzU/ZwCGuyadMmZGRkwNXVFeHh4QgJCYGXlxccHR2VDo3IanXu3BlXrlzBli1bMHv27DLbJSUlSfuAdOnSxYwR1gxc4cl8rH1eKAsQMurvv/8GADRp0qTY8UOHDkEQBAwZMgQvvfQSAOCXX37BqlWrcPDgQRYghKioKJw6darYMY1GY3TSnqGNYa+Z5s2bV1t8NdX58+chCAKmT5+ODh06KB0OkU3o168f9uzZg+joaHzyyScYMmSI1AOZnp6O5ORknD59Gr/88gs0Gg0CAgIQFhambNAK4ApPVFVYgJBR6enpcHZ2hpubW7Hj58+fBwD07dtXOtajRw+sWrUKN27cMGuMVDPdvHmzxAS9vLw8HDx4UNb17u7uVruhpTEajQYODg5o37690qEQ2QwnJyfMnTsXH374IU6ePImoqCjp3IQJE6TXoiiiXr16mDNnDuzt7ZUI1Wy4whNVJxYgZFRubm6JoR9JSUnIyMiAt7c3fH19peMqlQqurq7IyMgwd5hUAwUEBBTrOj548CCcnJyM9o4JggC1Wg1/f3906tQJ7u7u5gi1RqlTpw7S09Ntcv4LkZLq1auHxYsXY+fOnThw4ECJZXY9PT3Rs2dPDBs2DGq1WqEozYcrPFF1YgFCRtWqVQsPHz5ERkYGatWqBQCIjY0FADRr1qxEe51OZ/SXELIdISEhCAkJkd4fPHgQrq6umDp1qkn3tfaJeZ06dUJkZCTi4+MRHBysdDhENkWlUmH06NEYPXo07t+/j7S0NOj1etSuXRuPP/640uGZFVd4ourEAoSMCgwMxNmzZxEZGYlx48ZBq9Viz549EAQBbdq0Kdb2wYMHyM3NRYMGDZQJlmq0BQsWwMHB9L9yrH1iXnh4OE6cOIFvvvkG8+bNKzH8kYjMw8vLS/bGqdaIKzxRdWIBQkb17dsXZ86cwfbt2xEVFYXs7GykpaXB3d0dnTt3Ltb24sWLAICGDRsqESrVcNwor6S4uLhSj48dOxYrV67Ea6+9ht69eyMoKKjcnkXml4gshbX/kETlYwFCRnXo0AHh4eH46aefcOfOHQCAm5sbpk+fXuIL0ZEjRwAArVq1Mnuc1oB7PtieOXPmlDvEYevWrbLuxY0IiaqGTqdDYmIisrKyyt0IlIU/UeWwAKFyjRkzBr169UJ8fDzUajWCg4NLDAspKChAUFAQGjdujI4dO5a4B7tbyzd79mzZu16T9WDhSVQzJCcnY926dYiKipL9dzELf6LKYQFCsvj4+MDHx6fM8w4ODkaXTGV3a/maNm2qdAhkZpGRkUqHQEQoXN1x5syZyMzM5I8CRGbAAoSIiIhs2qZNm5CRkQFXV1eEh4cjJCQEXl5eJZahJ6KqwQKEiKgGSUlJgZ2dnezVd+7fvw+9Xg9vb+9qjozIep0/fx6CIGD69Ono0KGD0uEQWX1PHAsQIqIaZOLEifD09MSaNWtktZ81axZSU1M5Fp3IBBqNBg4ODmjfvr3SoRABsP55oSxAiIiIyKbVqVMH6enpsLOzUzoUIgDWPy+Uf9KIiCyYVqvllyYiE3Xq1AlarRbx8fFKh0JkE/hfLSIiC3X37l1kZmbCw8ND6VCILFp4eDi8vb3xzTffICsrS+lwiKweh2ARkUWxtol5UVFROHXqVLFjGo0GS5YsMXqdRqPBpUuXAADNmzevtviIrE1cXFypx8eOHYuVK1fitddeQ+/evREUFFRiw91HcSNCosphAUJEFsXaJubdvHkTBw8ehCAIUnGVl5eHgwcPyrre3d3d6B48RFTcnDlzIAiC0TZbt26VdS8u/lA51vZDElUcCxAisijWNjEvICCg2AadBw8ehJOTE0JDQ8u8RhAEqNVq+Pv7o1OnTnB3dzdHqERWg1+AlWVtPyRRxQki/xSSGUyYMAHp6en8tYioHEOGDKnQMrxlOXbsGPLy8ooVN0RERDUBe0DILFjnEsmzYMECODiY/lfzqlWrkJ6ezgKEiIhqHBYgZBbsbiWSh5NaicwvJSUFdnZ28PLyktX+/v370Ov18Pb2rubIiKwTCxAyC2sbt09ERNZj4sSJFRr6OGvWLKSmpnJYMVElcR8QIiIiIiIyGxYgRERERBWg1WphZ8evUESVxT89RERERDLdvXsXmZmZ8PDwUDoUIovFOSBERERkU6KionDq1KlixzQaDZYsWWL0Oo1Gg0uXLgEAmjdvXm3xEVk7FiBERERkU27evImDBw9CEARpmfi8vDwcPHhQ1vXu7u4YNWpUdYZIZNVYgBAREZFNCQgIKLZHzsGDB+Hk5ITQ0NAyrxEEAWq1Gv7+/ujUqRPc3d3NESqRVWIBQkRERDYlJCQEISEh0vuDBw/C1dUVU6dONem+x44dQ15eHjcAJSoHCxAiIiKyaQsWLICDg+lfiVatWoX09HQWIETlYAFCRGSFDOPaiah8Tz75pNIhENkUFiBERFZo9uzZKCgoUDoMIiKiEliAEBFZoaZNmyodAhERUam4ESEREREREZkNCxAiIiIiIjIbFiBERERERGQ2LECIiIiIiMhsWIAQEREREZHZsAAhIiIiIiKzYQFCRERERERmwwKEiIiIqAqIoqh0CEQWQRD5p4WIiIjIZFeuXEFBQQGefPJJpUMhqtFYgBARERERkdlwCBYREREREZkNCxAiIiIiIjIbFiBERERERGQ2LECIiIiIiMhsWIAQEREREZHZsAAhIiIiIiKzYQFCRERUAxw+fBiCIEAQBHzwwQdKh0NEVG1YgBARkVk1btxY+qIdFxdXbvuePXtK7Rs0aFBu++zsbDg7O0MQBDg6OiIrK6sqwiYioirCAoSIiMzq2WeflV4fPnzYaNu8vDycOHFCen/nzh1cu3bN6DXHjx9HXl4eAKBDhw5wc3OrfLBERFTlWIAQEZFZFS1ADh06ZLTtqVOnkJOTU+xYedcULWqKPouIiGoGFiBERGRWYWFh0uujR49CFMUy2xqKCXd3d4SGhhY7Vt41AAsQIqKaiAUIERGZVb169RAcHAwASE1NxcWLF8tsaygmQkND0aNHj2LHSpOdnY3Tp08DAJycnNClS5eqCZqIiKoMCxAiIjI7OfNA8vLycPLkSQCFvSbdunUDANy9exfx8fGlXnP8+HHk5+cDAJ5++mmoVKpi53NycrB8+XL06tULvr6+cHJygpeXFzp06IB3330Xd+/eNRr3mjVrpAnxa9asAQDExMTg1VdfRXBwMNzd3YudK2rv3r147rnn4OvrCxcXF/j7+2PYsGHYt2+f0WcSEVkbFiBERGR2cuaBFJ3/ERYWhpCQEDg5ORm9xtjwq9OnT6NJkyZ4/fXX8d///hdJSUnIz89HWloazpw5gwULFiAoKAjff/+97M/x6aefomPHjvj2229x9erVUlfc0uv1mDRpEvr27Ytdu3YhKSkJWq0Wt2/fxs6dO9GnTx9MmzZN9jOJiCydg9IBEBGR7SltHoggCMXaFJ3/0a5dOzg4OKBjx444duwYDh8+jMmTJ5e4b1kFyIULF/Dss89Co9EAAJo3b45x48YhICAAaWlp2LVrF/bt24fs7GxERERAFEVEREQY/Qxbt27Fnj174ObmhvHjx6Njx45wdHTEpUuXULduXand9OnTsXr1agCAvb09xowZg7CwMDg7OyM2NhbfffcdlixZgtu3b8vKHRGRxROJiIgU0KxZMxGACEA8d+5cifPdu3cXAYh9+/aVjs2ePVsEINatW7dEe41GIzo6OooARBcXFzE3N1cURVHU6XTik08+KT1r4sSJYn5+fonrV69eLQqCIAIQ1Wq1ePPmzRJtfvjhB+k+AMTg4GDx77//LvMzHjt2TLqnq6ur+Mcff5Roc/fuXbFp06bF7vv++++XeU8iIkvHIVhERKQIY/NAHp3/YWCYB5KUlIQrV64Uu6bo/I9OnTrB2dkZAPDrr79KGx62atUKK1asgINDyQEAERERUq9KdnY2lixZYjR+QRCwefNm+Pv7l9nm888/l1b5WrRokbSSV1G+vr7YsmUL7O3tjT6PiMhasAAhIiJFGJsHEhUVJc3/MBQdANC5c2epeHj0mqLvixYtO3bskF6/9dZbRr/o//vf/5aGghW9rjShoaFo27Ztmee1Wi1+/fVXAICHhwcmTpxYZttWrVqhd+/eRp9HRGQtWIAQEZEiunXrJn3Z/+OPP6DX66Vzhh4RNzc3PPXUU9JxNzc3tG/fvlibR68Bihc3p06dkl6X9yW/YcOGaNq0KQAgISEBiYmJZbZ95plnjN7r/Pnz0o7sXbp0kXpkymJYZpiIyNqxACEiIkV4e3ujRYsWAIAHDx4gNjZWOmcoJrp06VJiuJShR6RowaHRaHDmzBkAgEqlwtNPPy2dMxQR7u7uxSaHl8WwR0nRa0vj5+dn9D5Fl/Rt3Lhxuc+V04aIyBqwACEiIsWUNg9Eq9UiKioKQPHhVwaGY8nJybh06RKA4vM/unTpIi3XCwCZmZkAAFdXV1kxubm5lbi2NI/uMfKookvyqtXqcp8rNz4iIkvHAoSIiBRTdK6GYQ7Ho/t/PCo0NFSax2G4xtj+H+7u7gAgLcFbnqKFg+HayihayGRnZ5fbXm58RESWjgUIEREp5tF5IDqdTiomXF1d0aFDhxLX1KpVC61btwYgrwDx9fUFUNibce/evXJjKrrLer169eR/mEfUr19fen3t2rVy28tpQ0RkDViAEBGRYry8vNCqVSsAQHp6Os6dOycVE0VXvHqUYRjWkSNHkJWVJc3/cHNzK1G0FJ0Psm/fPqPxJCQkSMv7+vv7y5ozUpZWrVpJE8+PHz8OrVZrtP2BAwcq/SwiIkvCAoSIiBRVtMfi999/l+Z/lDb8ysBQgKSmpuLbb7+V5n+EhoaWKFqGDx8uvf7888+h0+nKvO+iRYukfTuKXlcZzs7O6N+/P4DC4ur7778vs21cXFy5xRERkbVgAUJERIoqWmgsW7bM6PwPg2eeeUYauvXpp59Kxx8dfgUA/fv3R8uWLQEULo07ZcoUFBQUlGi3Zs0arFixAkDhpPGpU6dW+LM86q233pLinDVrlrS5YlH37t3D888/b7QwIiKyJqX3bRMREZlJ165dYWdnB71ej+TkZACFBUBp8z8M6tSpg5YtW+LChQvSNUDpBYidnR02bNiAzp07Q6PRYNWqVTh58iTGjRuHJ554AmlpaYiMjMTvv/8uXbN06VI0bNjQ5M/WpUsXvP7661i6dCkyMzPRtWtXjB07Ft26dYOzszNiY2OxevVqpKWlYdiwYeVufkhEZA1YgBARkaI8PT3Rpk0bxMTESMc6d+4MR0dHo9d169YNFy5ckN7XqlUL7dq1K7Vtq1atcOjQIQwbNgx37txBXFwcZs2aVaKdWq3G0qVLERERUclPU9KXX34JjUaD7777DgUFBVizZg3WrFlTrM3UqVMxdOhQFiBEZBM4BIuIiBT3aM+FseFXBo/uEfLMM89Iy/OWpkOHDoiPj8fSpUvRo0cPPP7443B0dISnpyfat2+P2bNn4+rVq1VafACFPTCrV6/Gnj17MHjwYPj4+MDJyQl+fn547rnn8Pvvv2Px4sVV+kwioppMEA2z7YiIiIiIiKoZe0CIiIiIiMhsWIAQEREREZHZsAAhIiIiIiKzYQFCRERERERmwwKEiIiIiIjMhgUIERERERGZDQsQIiIiIiIyGxYgRERERERkNixAiIiIiIjIbFiAEBERERGR2bAAISIiIiIis2EBQkREREREZsMChIiIiIiIzIYFCBERERERmQ0LECIiIiIiMhsWIEREREREZDYsQIiIiIiIyGxYgBARERERkdmwACEiIiIiIrNhAUJERERERGbDAoSIiIiIiMzm/wFbeT4txKovewAAAABJRU5ErkJggg==",
      "text/plain": [
       "<Figure size 400x200 with 1 Axes>"
      ]
     },
     "metadata": {
      "image/png": {
       "height": 200,
       "width": 400
      }
     },
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAyAAAAGQCAYAAABWJQQ0AAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjcuNCwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8WgzjOAAAACXBIWXMAAB7CAAAewgFu0HU+AABr9UlEQVR4nO3deVxUZfs/8M+ZYZsBRFBIFCFU3JfUNBcSTNTcTY3MXCo08ylzSfMpy6dSs9VcWtV6zK0sNzIz7RH3QDRFJTPUVDRFGNEBhmFY5vz+4DfnC8IMBwZmmOHzfr18deac+9znmkukueac+74FURRFEBERERER2YDC3gEQEREREVHdwQKEiIiIiIhshgUIERERERHZDAsQIiIiIiKyGRYgRERERERkMyxAiIiIiIjIZliAEBERERGRzbAAISIiIiIim2EBQg7pzTffhCAIEAQBkZGRlTp38ODBEAQBSqUSZ8+erZkALTh+/LgU+6hRo2x+fSIiIiJ7YgFCdcrOnTuxe/duAMC4cePQoUMHm8fQrVs3PPbYYwCA7du3Iy4uzuYxEBEREdkLCxCqM4qKijB37lwAgEKhwH/+8x9Z5125cgU//PADXnnlFfTt2xc+Pj7SHQxBELB27dpKx/Lmm29K2y+//DJEUax0H0RERESOyMXeARDZyoYNG/DXX38BAB577DG0aNHCYvt3330XH330ETQaTbXH0rFjRwwcOBB79uxBUlIStm3bhtGjR1f7dYiIiIhqG94BoTpBFEUsXrxYej1r1qwKzzl//nyNFB/lxbBw4cIauw4RERFRbcIChOqEn376CRcuXAAAtGnTBr17967U+UqlEu3bt8czzzyDF154oVpi6t+/P5o2bQoAOH36NA4cOFAt/RIRERHVZixAqE745JNPpO1JkybJOiciIgJLly7F4cOHkZWVhbNnz+Lrr7/GmDFjqiUmhUKBCRMmSK8//fTTaumXiIiIqDbjGJA6JCMjA4cOHcL169eh1+sRGhqKfv36oWHDhmbPuXnzJg4dOoSrV69CoVAgODgYAwYMQP369W0XuJVu3ryJ//3vf9JruVPfPvPMMzUVkmTUqFF45513ABTP0HX37l2Hyi0RERFRZfEOiJN5+umnpdmZnn76aQCARqPBk08+iSZNmmDMmDGYOXMmXn31VYwdOxZBQUGYN28eCgsLS/Xzzz//IDo6GkFBQRg7dizmzZuHuXPn4oknnsB9992HN954o8w5JV25cqXUTFFXrlypcvzW+uGHH2A0GgEArVq1QlhYWLX0Wx26du2KoKAgAIDBYMC2bdvsHBERERFRzWIB4uTOnz+Pzp0747vvvkNBQUGZ4waDAe+//z6io6OlqWCTkpLQuXPnUh/cS8rPz8eiRYvw7LPP1nj81eHnn3+Wtiu7aKEtRERESNslYyUiIiJyRnwEy4nl5ORg1KhRuH79Ory9vTFmzBh07twZKpUKf/zxB7755hvcuXMHQPGCeKtWrcKIESMwaNAgZGRkwNvbG6NHj0aXLl3KPWf9+vUYNmwYHn/8cXu+TYvy8/Nx6NAh6XV4eLgdoylfeHg4Nm7cCADYt28fRFGEIAh2joqIiIioZrAAcWLbtm2DKIoIDw/H999/j8DAwFLH586di/DwcFy+fBkAsGTJEvzyyy9IS0tDnz59sHnzZjRq1KjUOXPmzEF4eLj0SNXixYtrdQFy9uxZ6PV66XXHjh3tGE35OnXqJG3fvXsXf/31F1q3bm3HiIiIiIhqDh/BcmKiKKJ58+bYvXt3meIDABo3boyPP/5Yen316lXs2LEDYWFh+Pnnn8sUHwDQpEmTUuecPn1aWtyvNjp16pS0rVAoauUH+3bt2pV6ffLkSTtFQkRERFTzWIA4uffeew9eXl5mjw8ZMqTMrEvvvvsuPD09zZ4zdOhQ+Pj4SK8TExOtjrOmXLp0SdoOCAiAm5ubHaMpX7169VCvXj3pdcmYiYiIiJwNCxAnVq9ePYwYMcJiGxcXF3To0KHUOcOHD6/wnJKPMtXmOyDXrl2Ttsu7C1RbNG7cWNpOTU21YyRERERENYsFiBPr3LkzXFwqHuZz3333SdtdunSp9Dl3796tUny2UDI2S3eC7K1kbFqt1o6REBEREdUsFiBOrLwxHOUp+bhVycJC7jk6na5ygdlQyQHoHh4edozEMpVKJW3n5ubaMRIiIiKimsUCxIlV5QN3Vc4xrR9SG7m6ukrblhZOtLeSa7SUjJmIiIjI2bAAIadW8k5NXl6eHSOxrOSdGksTABARERE5OhYg5NRKPoaWkZFhx0gsKxmb3EfniIiIiBwRCxCqEVVdybvknYDqEBISIm3/888/1dp3dTEajbh165b0umTMRERERM6GBQjVCLVaXeq13IHVJT+IV4f27dtL23q9vlYWIX///TeKioqk1yVjJiIiInI2LECoRpRcqBAAbty4UeE5hYWF+P3336s1jq5du5Z6nZycXK39V4eSMQmCgC5dutgxGiIiIqKaxQKEaoSbmxvuv/9+6fWxY8cqPGfr1q3Iycmp1jjuu+8+tGzZUnp94sSJau2/Ohw/flza7tSpU5nijYiIiMiZsAChGvPQQw9J2+vXr7c4Da5Wq8W8efNqJI5HH31U2j5w4ECNXMMaBw8elLYHDhxox0iIiIiIah4LEKoxY8eOlbb/+usvzJ49G0ajsUy7K1euoF+/frh69WqVB69b8thjj0nbR48erVUL/Wm1WiQmJkqvS8ZKRERE5Ixc7B0AOa9hw4ahc+fOOHXqFABg5cqVOHDgAKKjo9GkSRNotVokJCRgx44dMBgM6NChA1q1aoUtW7ZUaxx9+vRBUFAQrl+/Dr1ejz179sj6oP/PP/8gIiKizP57Z+qaN28eFi1aVKbdxo0bS90FKs9PP/0kLULYokWLCtsTEREROToWIFRjlEol1q1bh759+0Kj0QAAzp49i7Nnz5Zp26JFC8TGxuKtt96q9jgUCgWeeeYZLFy4EACwefNmWQVIQUEBLl26VGG79PR0pKenl9kvZ0rhzZs3S9sxMTEVticiIiJydHwEi2pU+/btcfToUfTv37/c4x4eHpg8eTJ+//13hIaG1lgc06ZNg5ubGwBgx44duH37do1dS66bN2/i559/BgCoVCpMmTLFzhERERER1TxBFEXR3kFQ3XD58mUcOnQIaWlp8PDwQHBwMCIjI+Hr62uT68fExODrr78GAHzwwQeYM2eOTa5rzqJFi/DGG28AAP71r3/h008/tWs8RERERLbAAoTqjL///hutWrVCYWEhgoKC8Pfff8PV1dUuseTl5SEkJATp6enw8PDAxYsX0aRJE7vEQkRERGRLfASL6oxmzZpJjzldv34dGzZssFssX3/9tTRu5IUXXmDxQURERHUG74BQnXL79m2EhYXhzp07CAkJwV9//QV3d3ebxpCbm4sWLVrg5s2bCAgIQEpKChcfJCIiojqDs2BRndKgQQNs2rQJCQkJAIrXIGnVqpVNY7h8+TKee+45AEB4eDiLDyIiIqpTeAeEiIiIiIhshmNAiIiIiIjIZliAEBERERGRzbAAISIiIiIim2EBQkRERERENsMChIiIiIiIbIYFCBERERER2QwLECIiIiIishkWIEREREREZDMsQIiIiIiIyGZYgBARERERkc2wACEiIiIiIpthAUJERERERDbDAoSIiIiIiGzGxd4BVBetVostW7YgMTERt2/fhru7O5o3b47BgwejR48ele4vNzcXx44dQ1JSEi5evIj09HQYjUb4+vqidevWGDRoENq1a2f2/GXLliEuLs7iNYKDg/HJJ59UOjYiIiIiIkflFAVIamoq5s+fD61WCwBQqVTQ6XRISkpCUlIShg0bhilTplSqz1mzZuHmzZvSazc3NygUCqSnpyM9PR2HDh3CY489hmeeecZiP25ublCr1eUeq1evXqViIiIiIiJydA5fgBQUFGDRokXQarUICQnB7NmzERoaCoPBgNjYWGzcuBE7d+5EaGgooqKiZPdbVFSE+++/HwMGDEDXrl0RGBgIURRx48YNrFu3DvHx8di+fTsaNWqEQYMGme0nPDwcM2fOrIZ3SkRERETk+Bx+DMiePXuQlpYGd3d3LFiwAKGhoQAAd3d3REdHS8XBhg0bUFhYKLvfmTNnYsWKFRg6dCgCAwMBAIIgoEmTJpg3bx46dOgAANi+fXs1vyMiIiIiIufl8HdADhw4AADo06cP/P39yxwfPXo0du/ejczMTJw9exadO3eW1W/79u3NHlMoFHjkkUdw9uxZpKWlIScnB15eXlWK31oajcYu160qDw8PKBQKGI1G5OXl2TucWot5qhhzJA/zJA/zJA/zJA/zJA/zJI8j5qlhw4YWjzv0HRC9Xo8LFy4AALp06VJuG39/fwQFBQEATp8+XW3XLjl+o6ioqNr6dXYqlQpqtRoqlcreodRqzFPFmCN5mCd5mCd5mCd5mCd5mCd5nDFPDn0H5Pr16xBFEQAQEhJitl1ISAiuXbuGa9euVdu1k5OTAQD169e3OJj8zJkzmDp1KjIyMuDm5obAwEB07doVQ4YMga+vb7XFQ0RERETkCBz6DkhmZqa07efnZ7ad6didO3eq5boajQa//PILAKBfv34QBMFi2/T0dHh4eCAvLw+XLl3C999/jxdffLFa78gQERERETkCh74DUvI5OHd3d7PtTMf0er3V1ywsLMSHH34IvV6PgIAAjBkzptx2zZs3R8uWLdGtWzc0aNAACoUCubm5SExMxNq1a5GZmYl33nkHS5cuRZMmTayOi4iIiIjIETj0HRBbE0URn3zyCc6dOwc3NzfMmTMHnp6e5bYdNmwYBg8eDH9/fygUxWlWq9WIjIzE+++/Dy8vL+j1enz77be2fAtERERERHbl0HdAPDw8pG2DwWB2wT+DwQAAVg/eWbVqFeLi4qBUKvHKK6+gdevWVeonICAAQ4YMwebNm3HixAkYjUapSLnXhg0bsGnTJrN9jRkzBpMmTapSHPZgep8KhYJjYCxgnirGHMnDPMnDPMnDPMnDPMnDPMnjjHly6AKk5LiPzMxMswWIaayINX9pX3/9NXbt2gWFQoHZs2eje/fuVe4LAFq2bAkAyM3NRXZ2Nnx8fMptp9PpkJ6ebraf3NxcKJVKq2KxB0EQHDJuW2OeKsYcycM8ycM8ycM8ycM8ycM8yeNMeXLoAiQoKAiCIEAURaSmpkrT7d4rNTUVANC0adMqXWfdunXYsWMHBEHA9OnT8fDDD1c55sry9PREQECA2eNqtdqhpgFWKBTS35nRaLR3OLUW81Qx5kge5kke5kke5kke5kke5kkeR8xTRYWSQxcgKpUKYWFhSElJwcmTJ9GrV68ybTQajTT9bqdOnSp9jU2bNmHLli0AgOeffx79+vWzLuj/LyUlBUDxe/D29jbbbvz48Rg/frzZ4xqNplKze7300kvyg3RiK1assHcIFvn6+kKpVMJoNFbb7G3OhjmSh3mSh3mSh3mSh3mSh3mSxxHz5NQLEQJAZGQkAODQoUPIyMgoc3zbtm0QRRF+fn7o0KFDpfresmULvvvuOwBATEwMBg0aJOs809ok5mRkZODnn38GADz44INmx38QERERETkbh//kO3DgQDRq1Ah5eXlYuHAhLl++DKB44PmWLVuwa9cuAMV3ElxcSt/wmTx5MoYPH45ly5aV6ffHH3/EunXrAACTJk3CiBEjZMd04MABLFmyBAkJCcjKypL26/V6HDx4EPPmzUN2djZUKhWefPLJyr5lIiIiIiKH5dCPYAGAq6srXn/9dcyfPx9XrlzBjBkzoFarkZeXJz0nN3ToUERFRVWq36+++gpA8YCf2NhYxMbGmm376quvok2bNtJro9GI+Ph4xMfHAyh+zMrFxQU6nU6KycfHB3PnzjU7boWIiIiIyBk5fAECAMHBwVi5ciW2bt2KxMREaDQaeHp6olmzZhgyZAh69OhR6T5Nj1GJooi7d+9abFtYWFjqdYcOHTB+/Hj8+eef+Oeff5CVlYXc3Fx4enqiadOmePDBBzFw4ECLYz+IiIiIiJyRUxQgAFC/fn3ExMQgJiZG9jlr1qwxe+zHH3+sciwBAQGIjo6u8vlERERERM7K4ceAEBERERGR42ABQkRERERENsMChIiIiIiIbIYFCBERERER2QwLECIiIiIishkWIEREREREZDMsQIiIiIiIyGasLkD27t1bHXEQEREREVEdYHUB8uijj6JFixZ47733kJ6eXh0xERERERGRk6qWldAvX76M1157DQsWLMDIkSPx3HPPoV+/ftXRNVVAEAQoFHySrrKUSqW9Q5DNkWK1F+ZIHuZJHuZJHuZJHuZJHuZJHmfJkyCKomhNB8888wy+//576PX64g4FAQDQvHlzPPfcc3j66afRsGFD6yOlcuXm5kKtVstuP27cuBqMxnFs2rTJ3iEQERER1UlWFyAAoNVqsW7dOqxevRrJycnFHf//QsTV1RWjRo3Cc889h8jISGsvRfe4fft2pe6AvPDCCzUYjeP49NNP7R2CRfXq1YNSqURRURGysrLsHU6txBzJwzzJwzzJwzzJwzzJwzzJ44h58vX1tXi8Wh7B8vHxwfTp0zF9+nTEx8fjyy+/xA8//AC9Xo/8/Hxs3rwZmzdvRlhYGKZOnYpJkybBz8+vOi5d54miiKKiInuH4XAcKWeOFKu9MEfyME/yME/yME/yME/yME/yOEueqn3wQM+ePbF27VrcuHEDy5cvR/v27SGKIkRRxIULFzBnzhw0adIEEyZMwOHDh6v78kREREREVIvV2Ohl012RM2fO4OjRo5gwYQI8PDwgiiIMBgM2bdqEyMhItGvXDitWrMDdu3drKhQiIiIiIqolbDJ9Us+ePfHNN9/gxo0bePHFF6X9oiji/PnzmDVrFoKCgvDCCy/gn3/+sUVIRERERERkBzYpQAoLC7F582aMGjUKn376KQRBgGnsu+nxrNzcXHzxxRdo1aoVVq9ebYuwiIiIiIjIxqplELo5Fy9exKpVq/DNN99Ao9EAgFR4dO/eHdOmTcOAAQOwefNmfPnll/jrr7+Qm5uL559/HsHBwRg4cKDsa2m1WmzZsgWJiYm4ffs23N3d0bx5cwwePBg9evSodOy5ubk4duwYkpKScPHiRaSnp8NoNMLX1xetW7fGoEGD0K5duwr7+fvvv7F9+3acPXsWWVlZ8PHxQfv27TFq1CiEhoZWOi4iIiIiIkdWLdPwllRQUICtW7di1apVOHjwIID/KzrUajWefPJJ/Otf/0Lnzp3LnLt+/XpMmzYNubm5iIiIwP79+2VdMzU1FfPnz4dWqwUAqFQqGAwGGI1GAMCwYcMwZcqUSr2PqVOn4ubNm9JrNzc3CIIAg8Eg7XvsscfwzDPPmO3j4MGDWL58OQoLCwEAnp6e0Ol0AAAXFxfMmjULDz/8cKXiupepsJPrpZdesup6zmLFihX2DsEiX19facq9O3fu2DucWok5kod5kod5kod5kod5kod5kscR81TRGoDVdgfkwoUL0t2O27dvA/i/wqN169aYNm0aJk6cCB8fH7N9TJgwASkpKVi8eDH++OMPWdctKCjAokWLoNVqERISgtmzZyM0NBQGgwGxsbHYuHEjdu7cidDQUERFRcl+P0VFRbj//vsxYMAAdO3aFYGBgRBFETdu3MC6desQHx+P7du3o1GjRhg0aFCZ81NTU6XiIzw8HJMnT4afnx8yMzOxevVqHD16FMuWLUNoaCiCgoJkx0VERERE5MisHgPy7bffom/fvmjdujWWLl0KjUYDURTh4uKCxx9/HHFxcTh37hymT59usfgw6d69OwBIRUxF9uzZg7S0NLi7u2PBggXSY03u7u6Ijo6WioMNGzZIdyLkmDlzJlasWIGhQ4ciMDAQQPHiik2aNMG8efPQoUMHAMD27dvLPX/jxo0oLCxEaGgoXn75ZWndEz8/P8yZMwehoaEoKCjAxo0bZcdEREREROTorC5AnnrqKRw6dEgaTB4UFIS3334bV69exebNmyu9+rmbm1ul2h84cAAA0KdPH/j7+5c5Pnr0aAiCgMzMTJw9e1Z2v+3btzd7TKFQ4JFHHgEApKWlIScnp9RxnU6H48ePAwBGjhwJpVJZ6rhSqcTIkSMBAImJicjNzZUdFxERERGRI6u2R7AGDBiAadOmYdiwYVAoql7XdO/eXfbYD71ejwsXLgAAunTpUm4bf39/BAUF4dq1azh9+nS5Y0+qol69etL2vatSnjt3TrrbYi4u0/6CggL8+eef6Nq1a7XERURERERUm1ldgMyZMwdTp05F8+bNqyMe+Pr6IiIiQlbb69evS+NMQkJCzLYLCQnBtWvXcO3atWqJEQCSk5MBAPXr1y9VjACQrlO/fn2zj535+PjAx8cHWq0WqampLECIiIiIqE6wugB5//33qyOOKsnMzJS2TWMsymM6Vl0zB2g0Gvzyyy8AgH79+kEQhFLHTdexFJPpuFardZgZDYiIiIiIrGX1GJBnn30Wzz77LJKSkip1XnJyMp599lnExMRU+dp5eXnStru7u9l2pmN6vb7K1zIpLCzEhx9+CL1ej4CAAIwZM6ZMG9N1LMVU3XERERERETkCqwuQtWvX4ptvvkFqamqlzvvnn3+wdu1arF271toQbEYURXzyySc4d+4c3NzcMGfOHHh6eto7LCIiIiIih1GjK6HXNA8PD2nbYDBArVaX2860eKBKpbLqeqtWrUJcXByUSiVeeeUVtG7dutx2puuUXLSwqnFt2LABmzZtMnt8zJgxmDRpUkWh0z18fX3tHYJFpokcFApFrY/VXpgjeZgneZgneZgneZgneZgneZwxT3YrQEwzR7m4VD2EkmMsMjMzzRYgprEi1vylff3119i1axcUCgVmz54trVdiKa6SY1SqGpdOp0N6errZ47m5uWWm+aWKOUrOBEFwmFjthTmSh3mSh3mSh3mSh3mSh3mSx5nyZLcC5PLlywBQZgapyggKCoIgCBBFEampqWZXFDc9Hta0adMqXWfdunXYsWMHBEHA9OnT8fDDD1tsb7rO3bt3kZWVVe571Gq10Gq1AIDg4GCzfXl6eiIgIMDscbVaXWYaYKpYbc+ZQqGQfraNRqO9w6mVmCN5mCd5mCd5mCd5mCd5mCd5HDFPFRVK1VaA3DsTlDm5ubk4efIkli9fDkEQ0KZNmypfU6VSISwsDCkpKTh58iR69epVpo1Go5Gmxe3UqVOlr7Fp0yZs2bIFAPD888+jX79+FZ7Ttm1buLi4oLCwECdPnix3McZTp04BAFxdXS3mYPz48Rg/frzZ4xqNhrNoVUFtz5mvry+USiWMRmOtj9VemCN5mCd5mCd5mCd5mCd5mCd5HDFPDRs2tHi8UoPQ33rrLSiVylJ/gOLB2aYVvyv64+3tjYiICFy6dAkA8Nhjj1XxrRUzfbg/dOgQMjIyyhzftm0bRFGEn58fOnToUKm+t2zZgu+++w4AEBMTg0GDBsk6T61Wo1u3bgCA2NjYMt+2FxUVITY2FkDxwovmHh0jIiIiInI2lZ4FSxTFUn/M7ZfzJyIiAi+++KJVb2DgwIFo1KgR8vLysHDhQunRLoPBgC1btmDXrl0Aiu8k3DveZPLkyRg+fDiWLVtWpt8ff/wR69atAwBMmjQJI0aMqFRcTz31FFxcXHDp0iUsXbpUqljv3LmDpUuX4tKlS3B1dcVTTz1V2bdMREREROSwKvUI1v33319mlfKDBw9CEAS0bdu24tstCgW8vLwQGhqKqKgoDB48WBrZX1Wurq54/fXXMX/+fFy5cgUzZsyAWq1GXl6e9Jzc0KFDERUVVal+v/rqKwDFj5bFxsZKdyzK8+qrr5Z5jCo4OBgzZszA8uXLcfjwYRw5cgRqtRo6nQ5A8eD7GTNmmB23QkRERETkjCpVgEyaNKnMlK+mAmLx4sUYPnx49UVWCcHBwVi5ciW2bt2KxMREaDQaeHp6olmzZhgyZAh69OhR6T5Nd3dEUcTdu3ctti0sLCx3f0REBJo2bYpt27YhOTkZWVlZ0qNgo0aNQmhoaKXjIiIiIiJyZFYPQu/Tpw8EQajw7kdNq1+/PmJiYiq1svqaNWvMHvvxxx+rIyw0a9YMc+bMqZa+iIiIiIgcndUFyIEDB6ohDCIiIiIiqgusG4BBRERERERUCSxAiIiIiIjIZmQ/gvX2229L2wsWLCh3f1WV7I+IiIiIiJyX7ALkzTfflFY7L1kwlNxfVSxAiIiIiIjqhkoNQhdFsdxio+SChJVlbfFCRERERESOQ3YBsn///krtJyIiIiIiupfsAuTeFdAr2k9ERERERHQvzoJFREREREQ2Y/VChGRfgiBAoWAdWVlKpdLeIcjmSLHaC3MkD/MkD/MkD/MkD/MkD/Mkj7PkiQWIg1OpVFCr1fYOw+H4+vraOwRZlEqlw8RqL8yRPMyTPMyTPMyTPMyTPMyTPM6UJxYgDk6v18NgMNg7DIdz584de4dgUb169aBUKlFUVISsrCx7h1MrMUfyME/yME/yME/yME/yME/yOGKeKiqUZBcgNXXLRxAEFBYW1kjfdYEoiigqKrJ3GA7HkXLmSLHaC3MkD/MkD/MkD/MkD/MkD/Mkj7PkSXYBYs1aH0REREREREAlCpA+ffpw0UAiIiIiIrKK7ALkwIEDNRgGERERERHVBZy/lYiIiIiIbMZpZsHSarXYsmULEhMTcfv2bbi7u6N58+YYPHgwevToUen+ioqKkJycjIsXL+LixYu4dOkS0tLSAABjx47FuHHjLJ6/bNkyxMXFWWwTHByMTz75pNKxERERERE5KqcoQFJTUzF//nxotVoAxWtj6HQ6JCUlISkpCcOGDcOUKVMq1adGo8Ebb7xhdWxubm5m1+moV6+e1f0TERERETkShy9ACgoKsGjRImi1WoSEhGD27NkIDQ2FwWBAbGwsNm7ciJ07dyI0NBRRUVGV6lulUqFZs2Zo0aIFmjdvjm+//RY3b96sVB/h4eGYOXNmpc4hIiIiInJWsguQt99+W9pesGBBufurqmR/lbVnzx6kpaXB3d0dCxYsgL+/PwDA3d0d0dHRyMzMxM8//4wNGzYgMjISLi7y3rK/vz++++67UjN/bd++vcpxEhERERFRJQqQN998U/owXrJgKLm/qqwpQEyzc/Xp00cqPkoaPXo0du/ejczMTJw9exadO3eW1a9CwfH5RERERETVrVKfss0tRiiKYpX/WEOv1+PChQsAgC5dupTbxt/fH0FBQQCA06dPW3U9IiIiIiKyjuw7IPv376/Uflu4fv26VMSEhISYbRcSEoJr167h2rVrtgpNcubMGUydOhUZGRlwc3NDYGAgunbtiiFDhsDX19fm8RARERER2ZPsAiQiIqJS+20hMzNT2vbz8zPbznTszp07NR7TvTQaDZRKJVQqFXJzc3Hp0iVcunQJu3fvxiuvvIJOnTrZPCYiIiIiIntx6Fmw8vLypG13d3ez7UzH9Hp9jcdk0rx5c7Rs2RLdunVDgwYNoFAokJubi8TERKxduxaZmZl45513sHTpUjRp0sRmcRERERER2RNHWteQYcOGYfDgwfD395cGtKvVakRGRuL999+Hl5cX9Ho9vv32WztHSkRERERkOzV2ByQ9PR03btxAdnY2vL290bhxYwQEBFTrNTw8PKRtg8FgdsE/g8EAoHhdj9ogICAAQ4YMwebNm3HixAkYjUazs25t2LABmzZtMtvXmDFjMGnSpJoK1WlZGn8zYcIEG0ZSu61fv97eIZhl+jejUCg4nsoC5kke5kke5kke5kke5kkeZ8xTtRYgV69excqVK7Fly5ZyB3wHBwfj8ccfxwsvvGBx0LhcJcd9ZGZmmi1ATGNFatNfWsuWLQEAubm5yM7Oho+PT7ntdDod0tPTzfaTm5sLpVJZIzE6M+ZMHkfIkyAIDhGnvTFP8jBP8jBP8jBP8jBP8jhTnqqtAPn0008xb948aZxFeVPspqam4qOPPsJnn32G999/H//617+sumZQUBAEQYAoikhNTZWm2y3vugDQtGlTq65nD56enhbvHKnVahQVFdkwIufAnMlTm/OkUCikf/9Go9He4dRazJM8zJM8zJM8zJM8zJM8jpinigqlailAlixZgtdffx1AceGhUCjQtm1bhIWFwdPTEzqdDhcvXsS5c+dgNBqRm5uL6dOnIysrC//+97+rfF2VSoWwsDCkpKTg5MmT6NWrV5k2Go1GuhtTm2acSklJAVD8Hry9vc22Gz9+PMaPH2/2uEajscvsXo6OOZOnNufJ19cXSqUSRqOxVsdpb8yTPMyTPMyTPMyTPMyTPI6Yp4YNG1o8bvUg9JMnT2LBggVS4TFr1iykpqbi7Nmz2LZtG9avX49t27bhzJkzuHbtGl5++WUolUqIoog33ngDp06dsur6kZGRAIBDhw4hIyOjzPFt27ZBFEX4+fmhQ4cOVl1LrooWWMzIyMDPP/8MAHjwwQe56joRERER1RlWf/JduXIlioqKIAgCNmzYgI8++giNGzcut21gYCA++OADbNy4EQBgNBqxYsUKq64/cOBANGrUCHl5eVi4cCEuX74MoHjg+ZYtW7Br1y4AxXcSXFxK3/CZPHkyhg8fjmXLlpXbt06nQ1ZWlvTHdNvLYDCU2m8a5G5y4MABLFmyBAkJCcjKypL26/V6HDx4EPPmzUN2djZUKhWefPJJq94/EREREZEjsfoRrP3790MQBAwdOhRPPPGErHOio6OxadMm/Pjjj1avpO7q6orXX38d8+fPx5UrVzBjxgyo1Wrk5eVJBcPQoUMRFRVV6b4XL16M5OTkMvu3b9+O7du3S6/Hjh2LcePGSa+NRiPi4+MRHx8PoPgxKxcXF+h0OikmHx8fzJ071+y4FSIiIiIiZ2R1AXLr1i0AxR/yK2PIkCH48ccfpfOtERwcjJUrV2Lr1q1ITEyERqOBp6cnmjVrhiFDhqBHjx5WX6MyOnTogPHjx+PPP//EP//8g6ysLOTm5sLT0xNNmzbFgw8+iIEDB1oc+0FERERE5IysLkDq16+P9PR01K9fv9LnlfxvdcQRExODmJgY2eesWbPG4vF33nmnSrEEBAQgOjq6SucSERERETkzq8eAtG3bFgBw4cKFSp138eLFUucTEREREZHzs7oAGT9+PERRxLp165Cfny/rnPz8fKxduxaCIHDVaSIiIiKiOsTqAuTpp59GZGQkUlJS8NRTT0kLEZqTl5eH8ePH48KFC+jbty+efvppa0MgIiIiIiIHYXUBIggCYmNjMWrUKGzduhVt2rTBhx9+iFOnTiEnJweiKCInJwdJSUn44IMP0KZNG2zduhVjxozBjh07quEtEBERERGRo5A9CL2iJdVNUlNTMW/ePLPHTYv0bd26FVu3boUgCCgsLJQbBhEREREROTDZBUhFq3tXtm1l+iMiIiIiIucguwDp06cPBEGoyViIiIiIiMjJyS5ADhw4UINhEBERERFRXWD1IHQiIiIiIiK5rF4JnexLEAQoFKwjK0vupAp1naPkyVHitDfmSR7mSR7mSR7mSR7mSR5nyRMLEAenUqmgVqvtHYbD8fX1tXcIDsER8qRUKh0iTntjnuRhnuRhnuRhnuRhnuRxpjyxAHFwer0eBoPB3mE4nDt37tg7BIdQm/NUr149KJVKFBUVISsry97h1FrMkzzMkzzMkzzMkzzMkzyOmKeKCqVqLUByc3MRGxuLhIQEXL9+HVlZWSgqKrJ4jiAI2LdvX3WGUaeIolhhjqks5kweR8mTo8Rpb8yTPMyTPMyTPMyTPMyTPM6Sp2orQL744gu89tpr0Gq1ss8RRZFT+xIRERER1SHVUoAsWrQI//nPf2QtLmgqOLgQIRERERFR3WP19Ennz5/Hf/7zHwBAy5YtsW/fPuj1egDFxcaOHTuQk5ODs2fP4r333kNgYCAA4JlnnkFeXp7T3EoiIiIiIqKKWX0H5IsvvoAoilCr1di7dy+Cg4PLtFGr1WjXrh3atWuHKVOmYMSIEVi7di10Oh2+++47a0MgIiIiIiIHYXUBcvDgQQiCgMcff7zc4uNe9evXx44dO9CyZUv88MMPGDduHIYPH25tGNBqtdiyZQsSExNx+/ZtuLu7o3nz5hg8eDB69OhR6f6KioqQnJyMixcv4uLFi7h06RLS0tIAAGPHjsW4ceNk9fP3339j+/btOHv2LLKysuDj44P27dtj1KhRCA0NrXRcRERERESOzOoCJDU1FQDMfsjPz88vs8/X1xeTJk3C0qVLsX79eqsLkNTUVMyfP18aAK9SqaDT6ZCUlISkpCQMGzYMU6ZMqVSfGo0Gb7zxhlVxHTx4EMuXL0dhYSEAwNPTE7dv38bBgwdx9OhRzJo1Cw8//LBV1yCyl5deesneIdQaK1assHcIREREDsPqAiQ7OxsA4O/vX2q/SqVCXl6edPxenTt3BgCcOHHCqusXFBRg0aJF0Gq1CAkJwezZsxEaGgqDwYDY2Fhs3LgRO3fuRGhoKKKioirVt0qlQrNmzdCiRQs0b94c3377LW7evCnr3NTUVKn4CA8Px+TJk+Hn54fMzEysXr0aR48exbJlyxAaGoqgoKCqvHUiIiIiIodj9SB0T09PAGXvdPj4+AD4vzsk9zLdFbh165ZV19+zZw/S0tLg7u6OBQsWSI81ubu7Izo6GoMGDQIAbNiwQbqmHP7+/vjuu++wZMkSxMTEIDIyEh4eHrLP37hxIwoLCxEaGoqXX34Zfn5+AAA/Pz/MmTMHoaGhKCgowMaNGyvxbomIiIiIHJvVBcj9998PoGwh0apVK4iiiKNHj5Z73unTpwEAbm5uVl3/wIEDAIA+ffqUuQsDAKNHj4YgCMjMzMTZs2dl96tQKKq8RolOp8Px48cBACNHjoRSqSx1XKlUYuTIkQCAxMRE5ObmVuk6RERERESOxuoCpFOnThBFscyH+z59+gAA9u/fj99//73Usb///htr1qyBIAho06ZNla+t1+tx4cIFAECXLl3KbePv7y894mQqemrauXPnpLst5uIy7S8oKMCff/5pk7iIiIiIiOzN6gIkMjISABAXF1dq/8SJE+Hi4gKj0YhHHnkEr7zyClatWoVXXnkFDz74IHJycgAUzyhVVdevX5cWNAwJCTHbznTs2rVrVb5WZZiuU79+felRtHv5+PhU+JgaEREREZGzsXoQ+rBhw6BUKnH16lX89ttv6NWrFwCgefPmeO211/D2228jJycHH330UZlzu3TpgmnTplX52pmZmdK2aYxFeUzH7ty5U+VrVYbpOpZiMh3XarU2i4uIiIiIyN6sLkAaNGiAlJQU5OfnIyAgoNSxN998E56enli4cKF0xwMoXiE9OjoaX3zxhVVjQPLy8qRtd3d3s+1Mx0wrtNc003UsxVTyuK3iIiIiIiKyN6sLEAAWF9SbO3cuXnrpJcTHxyMtLQ2enp548MEHERgYWB2XJiIiIiIiB1ItBUhF3N3dpbEi1anktLgGgwFqtbrcdgaDAUDxuh62YLqO6brmyIlrw4YN2LRpk9njY8aMwaRJk6oQZd3m6+tr7xAcAvMkT23Ok0KhkP5bm+O0N+ZJHuZJHuZJHuZJHmfMk00KkJpScoxFZmam2QLENFbEVn9pprhKjlEpj5y4dDod0tPTzR7Pzc0tM80vVYw5k4d5kscR8iQIgkPEaW/MkzzMkzzMkzzMkzzOlKcaLUDu3r2L7OxseHt7o379+tXef1BQEARBgCiKSE1NNbuiuGmWqaZNm1Z7DOUxXefu3bvIyspCvXr1yrTRarXQarUAgODgYLN9eXp6lhlbU5JarUZRUZGVEdc9zJk8zJM8tTlPpjWNRFGE0Wi0dzi1FvMkD/MkD/MkD/MkjyPmqaJCqVoLkJycHKxduxZbtmzB77//XmqBPbVajQcffBCPP/44Jk6cCC8vL6uvp1KpEBYWhpSUFJw8eVKagaskjUYjTYvbqVMnq68pR9u2beHi4oLCwkKcPHmy3MfPTp06BQBwdXW1uBbK+PHjMX78eLPHNRoNZ9GqAuZMHuZJntqcJ19fXyiVShiNxlodp70xT/IwT/IwT/IwT/I4Yp4aNmxo8bjV64CY7Ny5E2FhYZgxYwYOHz4MnU4HURSlPzqdDocOHcL06dMRFhaGn376qVqua/pwf+jQIWRkZJQ5vm3bNoiiCD8/P3To0KFarlkRtVqNbt26AQBiY2PLfDtaVFSE2NhYAED37t3NPjpGRERERORsqqUAWbduHUaNGoX09HSp4PD29sYDDzyA3r1744EHHkC9evWkY7du3cLIkSOxfv16q689cOBANGrUCHl5eVi4cCEuX74MoHiA95YtW7Br1y4AxXcSXFxK3/CZPHkyhg8fjmXLlpXbt06nQ1ZWlvTHdNvLYDCU2l/eYPOnnnoKLi4uuHTpEpYuXSpVrHfu3MHSpUtx6dIluLq64qmnnrI6B0REREREjsLqR7AuXryI559/XvqW/7HHHsMrr7yChx56qEzbxMREfPDBB9i6dSuMRiOmTp2KXr16oXnz5lW+vqurK15//XXMnz8fV65cwYwZM6BWq5GXlycVDEOHDkVUVFSl+168eDGSk5PL7N++fTu2b98uvR47dizGjRtXqk1wcDBmzJiB5cuX4/Dhwzhy5AjUajV0Oh0AwMXFBTNmzDA7boWIiIiIyBlZfQfk448/Rl5eHgRBwPvvv4+tW7eWW3wAxY8b/fDDD/jwww8BFN9J+Pjjj60NAcHBwVi5ciVGjBiBwMBAFBQUwNPTE506dcJrr72G5557zuprVEVERAQ+/PBD9OnTB76+vjAYDPDz80NERAQ++ugj9OnTxy5xERERERHZi9V3QPbu3QtBENCnTx/MmTNH1jmzZ8/Gzp07cfDgQezZs8faEAAA9evXR0xMDGJiYmSfs2bNGovH33nnHWvDQrNmzWTnhYiIiIjI2Vl9B+Sff/4BULwgXmWY2pvOJyIiIiIi52d1AWKaTve+++6r1HmmtS2qYzpeIiIiIiJyDFYXIC1atADwf4v9yWVamyMsLMzaEIiIiIiIyEFYXYA88cQTEEURmzZtgiiKss4RRREbN26EIAgYO3astSEQEREREZGDsLoAef7559GxY0ecOnUKs2bNknXO7NmzcerUKXTq1AlTp061NgQiIiIiInIQVhcg7u7u2LVrFx566CGsXLkSPXr0wJYtW8osFX/37l388MMP6NmzJ1asWIGePXti165dcHNzszYEIiIiIiJyELKn4W3WrJnF4wUFBRBFEcePH8cTTzwBAPD19YWnpyd0Op1UkIiiCEEQkJqait69e0MQBFy6dMmKt0BERERERI5CdgFy5coVCIJgdpyHIAgQBAEApDaZmZnIzMws0w4Abty4IRUjRERERERUN8guQIKDg1ksEBERERGRVSp1B4RqH0EQoFBYPZSnzlEqlfYOwSEwT/I4Sp4cJU57Y57kYZ7kYZ7kYZ7kcZY8yS5AqHZSqVRQq9X2DsPh+Pr62jsEh8A8yeMIeVIqlQ4Rp70xT/IwT/IwT/IwT/I4U55YgDg4vV4Pg8Fg7zAczr2ztFH5mCd5anOe6tWrB6VSiaKiImRlZdk7nFqLeZKHeZKHeZKHeZLHEfNUUaHEAsTBiaKIoqIie4fhcJgzeZgneRwlT44Sp70xT/IwT/IwT/IwT/I4S56qvQDRaDTYtWsXEhIScPPmTWRnZ8Pb2xuNGzfGQw89hCFDhqBhw4bVfVkiIiIiInIA1VaA5Obm4pVXXsHXX39t9pGgL7/8Eu7u7pg8eTLee+89qFSq6ro8ERERERE5gGqZPkmj0aBbt274/PPPkZeXB1EUzf7Jy8vDp59+im7duuH27dvVcXkiIiIiInIQ1XIHZPTo0fjzzz8BFM/K9OSTT2LgwIFo2bIlvLy8kJOTg5SUFOzZswffffcdcnNzce7cOYwePRoHDhyojhCIiIiIiMgBWF2AbN++HYcPH4YgCHjggQewbds2hISElGnXqVMnPP7443jjjTcwZswY/P777zh8+DBiY2MxYsQIa8OAVqvFli1bkJiYiNu3b8Pd3R3NmzfH4MGD0aNHjyr3W1hYiJ9++gkHDx7EjRs3AABNmjRBREQEhgwZAheX8lO4bNkyxMXFWew7ODgYn3zySZVjIyIiIiJyNFYXIN999x0AwN/fH7/++iv8/Pwstg8JCcEvv/yCdu3aISMjA5s2bbK6AElNTcX8+fOh1WoBFN+F0el0SEpKQlJSEoYNG4YpU6ZUul+9Xo833ngDKSkpAAA3NzcAwMWLF3Hx4kUcPXoUb7/9Njw8PMz24ebmZnadjnr16lU6JiIiIiIiR2Z1AXLs2DEIgoBnn322wuLDpEGDBoiJicGSJUtw7Ngxq65fUFCARYsWQavVIiQkBLNnz0ZoaCgMBgNiY2OxceNG7Ny5E6GhoYiKiqpU35999hlSUlLg6emJl156SbqTkpCQgBUrVuD8+fP4/PPPMWvWLLN9hIeHY+bMmda8RSIiIiIip2H1IPT09HQAQMeOHSt1XocOHUqdX1V79uxBWloa3N3dsWDBAoSGhgIA3N3dER0djUGDBgEANmzYgMLCQtn9Xr58GYcOHQIATJ8+HT179oQgCBAEAT179sSLL74IADhw4ACuXr1q1XsgIiIiIqorrC5ATI8l5efnV+o8U3tXV1errm8axN6nTx/4+/uXOT569GgIgoDMzEycPXtWdr8HDx6EKIoIDAxEz549yxzv1asXAgMDIYoiDh48WOX4iYiIiIjqEqsLkMaNGwMADh8+XKnzTHcXmjRpUuVr6/V6XLhwAQDQpUuXctv4+/sjKCgIAHD69GnZfZ85cwYA0LlzZwiCUOa4IAjo3LlzqbZERERERGSZ1WNAIiMjcf78eaxfvx4vvvgiOnXqVOE5SUlJ2LBhAwRBQGRkZJWvff36dYiiCADlzrxlEhISgmvXruHatWuy+hVFEdevX6+w3+DgYACw2O+ZM2cwdepUZGRkwM3NDYGBgejatSuGDBkCX19fWfEQERERETkLq++ATJ48GYIgoKCgAFFRUdi2bZvF9tu2bUP//v2Rn58PQRCqNDuVSWZmprRtaQC86didO3dk9avX65GXlye7X71eD71eX24bjUaD9PR0eHh4IC8vD5cuXcL333+PF198sVJ3ZIiIiIiInIHVd0C6dOmC559/Hp9//jkyMzPx+OOPo1mzZujfvz9atmwJT09P6HQ6XLhwAb/++isuXboEURQhCAKef/556TGmqjAVCUDxoHNzTMfMFQn3KtlOTr+mc1QqlfS6efPmaNmyJbp164YGDRpAoVAgNzcXiYmJWLt2LTIzM/HOO+9g6dKlVj2GRkRERETkSKplJfSVK1ciKysLGzduBAD8/fff+PLLL8tta3pk6qmnnsKKFSuq4/K10rBhw8rsU6vViIyMRNu2bTFz5kzk5OTg22+/xZw5c+wQIRERERGR7VVLAaJQKLB+/XoMHz4c7733Hk6ePGm2bdeuXfHvf/8bo0ePtvq6JRcANBgMZhf8MxgMAFDqDoUlJduZzrXUb2X6BoCAgAAMGTIEmzdvxokTJ2A0GqFQlP803IYNG7Bp0yazfY0ZMwaTJk2SfW0qxvE38jBP8tTmPJl+tygUilodp70xT/IwT/IwT/IwT/I4Y56qpQAxefzxx/H4448jNTUVx44dw82bN5GdnQ1vb28EBgbioYcekgZuV4eS4zMyMzPNFiCmsSJy/9JUKhVUKhX0en2pcSbm+jW1r4yWLVsCAHJzc5GdnQ0fH59y2+l0OotrpeTm5kKpVFbq2gTmTCbmSR5HyJMgCA4Rp70xT/IwT/IwT/IwT/I4U56sLkDWrVsHAGjUqBEGDBgAoHh2qOosNMwJCgqCIAgQRRGpqanSdLv3Sk1NBQA0bdpUVr+CICAoKAgXLlyQzq2OfqvC09MTAQEBZo+r1WoUFRXV2PWdFXMmD/MkT23Ok0KhkH5PGo1Ge4dTazFP8jBP8jBP8jBP8jhinioqlKwuQJ5++mkIgoD58+dLBYitqFQqhIWFISUlBSdPnkSvXr3KtNFoNNI0uXKmCDbp2LEjLly4gFOnTpltk5SUJLWtrJSUFADF78Hb29tsu/Hjx2P8+PFmj2s0Gtmze9H/Yc7kYZ7kqc158vX1hVKphNForNVx2hvzJA/zJA/zJA/zJI8j5qlhw4YWj1tdgHh5eUGn06Ft27bWdlUlkZGRSElJwaFDh/DEE0+UWQ1927ZtEEURfn5+6NChg+x++/Tpg23btuHGjRuIj48vsxr6b7/9hhs3bpS7lolpli9zMjIy8PPPPwMAHnzwQbPjP4jI8b300kv2DqHWcOaJR4iISD6rP/kGBgYCAAoKCqwOpioGDhyIRo0aIS8vDwsXLsTly5cBFA8Q37JlC3bt2gWg+E6Ci0vpemvy5MkYPnw4li1bVqbf0NBQ9OnTB0DxLF8JCQkQRRGiKCIhIQGffPIJgOIC6N7HzQ4cOIAlS5YgISEBWVlZ0n69Xo+DBw9i3rx5yM7OhkqlwpNPPlltuSAiIiIiqu2svgPSt29fXLx4EcePH8eECROqI6ZKcXV1xeuvv4758+fjypUrmDFjBtRqNfLy8qTn5IYOHYqoqKhK9/2vf/0LN2/eREpKCt555x24ubkBAPLz8wEArVu3xrRp08qcZzQaER8fj/j4eADFj1m5uLhAp9NJMfn4+GDu3Llmx60QERERETkjqwuQqVOn4quvvsI333yDefPm2WVRveDgYKxcuRJbt25FYmIiNBoNPD090axZMwwZMgQ9evSoUr8qlQrvvvsufvrpJxw8eBA3btwAULzIYGRkJIYMGVLmrgoAdOjQAePHj8eff/6Jf/75B1lZWcjNzYWnpyeaNm2KBx98EAMHDrQ49oOIiIiIyBlZXYB07twZixcvxr///W/0798f3333XZUGZVurfv36iImJQUxMjOxz1qxZU2EbFxcXjBw5EiNHjpTdb0BAAKKjo2W3JyIiIiKqK6plGt5GjRph0KBB2L17N7p06YLw8HA8/PDDCAoKkrU+xsSJE60Ng4iIiIiIHEC1TcMLFK+fYTQacfjwYRw+fFjW+YIgsAAhIiIiIqojqmUldFEULb4mIiKyhNMVF+NUxURUF1hdgPz3v/+tjjiIiIiIiKgOsLoAmTRpUnXEQUREREREdQCX4CYiIiIiIpux6g7IP//8gzNnzkCr1cLHxwcdOnTgwnpERERERGRWlQqQxMREzJo1CwkJCWWO9ejRAx9//DG6d+9udXBERERERORcKl2A7N27FyNHjoTBYCh3tqv4+HhERERgx44dGDhwYLUESURERJwtzKSi2cKYp2KcVY1qq0oVINnZ2Zg0aRLy8vKkfS1atEBAQADS09Nx8eJFAIDBYMCkSZOQkpKCevXqVW/EVIogCFAoOJSnspRKpb1DcAjMkzzMkzzMU8WYI3mYJ3kcKU+OFKs9OUueKlWArF+/Hrdu3YIgCOjatSu++eYbtGnTRjp+/vx5PP3000hMTERGRgbWr1+PF154odqDpv+jUqmgVqvtHYbD8fX1tXcIDoF5kod5kod5qhhzJA/zJI+j5EmpVDpMrPbkTHmqVAGye/duAEDDhg2xZ8+eMklo3bo1du/ejTZt2iAjIwO7d+9mAVLD9Ho9DAaDvcNwOHfu3LF3CA6BeZKHeZKHeaoYcyQP8yRPRXniZ7Rin376qb1DsKhevXpQKpUoKipCVlaWvcORpaJCqVIFyJkzZyAIAiZOnGi2Y19fX0ycOBEffvghzp49W5nuqQpEUURRUZG9w3A4zJk8zJM8zJM8zFPFmCN5mCd5mCd5HClPjhSrJZUaPJCZmQkAeOCBByy269SpEwDg9u3bVYuKiIiIiIicUqUKEJ1OBwDw9va22M7LywtA8eNBREREREREJpw+iYiIiIiIbIYFCBERERER2UyVVkIXBKG643BaWq0WW7ZsQWJiIm7fvg13d3c0b94cgwcPRo8ePewdHhERERGRTVWpABk5cqSsdqIoVrhgiiAIKCwsrEoYtV5qairmz58PrVYLoHjNDp1Oh6SkJCQlJWHYsGGYMmWKnaMkIiIiIrKdKhUgQHFxYY4gCNJdEkvtnFlBQQEWLVoErVaLkJAQzJ49G6GhoTAYDIiNjcXGjRuxc+dOhIaGIioqyt7hEhEREdE9XnrpJXuHUGusWLGi2vqq9BgQURQrLCpMbepq8QEAe/bsQVpaGtzd3bFgwQKEhoYCANzd3REdHY1BgwYBADZs2OC0d4CIiIiIiO5VqQLEaDRW+x9nWVDlXgcOHAAA9OnTB/7+/mWOjx49GoIgIDMzkws2EhEREVGdwVmwaoBer8eFCxcAAF26dCm3jb+/P4KCggAAp0+ftllsRERERET2xAKkBly/fl16/CwkJMRsO9Oxa9eu2SQuIiIiIiJ7YwFSAzIzM6VtPz8/s+1Mx+7cuVPjMRERERER1QYsQGpAXl6etO3u7m62nemYXq+v8ZiIiIiIiGoDFiBERERERGQzVV4HhMzz8PCQtg0GA9RqdbntDAYDgOIFCs3ZsGEDNm3aZPb4mDFjMGnSpCpGWnf5+vraOwSHwDzJwzzJwzxVjDmSh3mSh3mSh3mSpzrzxAKkBpQc95GZmWm2ADGNFbH0F6rT6ZCenm72eG5uboWrzZdkqZihYsyRPMyTPMyTPMyTPMyTPMyTPMxTxZijmsECpAYEBQVBEASIoojU1FRput17paamAgCaNm1qti9PT08EBASYPa5Wqx1qLRWFQiHlxmg02jucWot5qhhzJA/zJA/zJA/zJA/zJA/zJI8j5qmiL8dZgNQAlUqFsLAwpKSk4OTJk+jVq1eZNhqNRpp+t1OnTmb7Gj9+PMaPH2/2uEajcahZtHx9faFUKmE0Gh0qbltjnirGHMnDPMnDPMnDPMnDPMnDPMnjiHlq2LChxeMchF5DIiMjAQCHDh1CRkZGmePbtm2DKIrw8/NDhw4dbBwdEREREZF9sACpIQMHDkSjRo2Ql5eHhQsX4vLlywCKB55v2bIFu3btAlB8h8PFhTeiiIiIiKhu4CffGuLq6orXX38d8+fPx5UrVzBjxgyo1Wrk5eVJz+8NHToUUVFRdo6UiIiIiMh2WIDUoODgYKxcuRJbt25FYmIiNBoNPD090axZMwwZMgQ9evSwd4hERERERDbFAqSG1a9fHzExMYiJibF3KEREREREdscxIEREREREZDOCKIqivYOgumPDhg3Q6XTw9PS0OL1wXcc8VYw5kod5kod5kod5kod5kod5kscZ88QChGxq8ODBSE9PR0BAAH7++Wd7h1NrMU8VY47kYZ7kYZ7kYZ7kYZ7kYZ7kccY88REsIiIiIiKyGRYgRERERERkMyxAiIiIiIjIZliAEBERERGRzbAAISIiIiIim2EBQkRERERENsOV0Mmmxo0bJ81lTeYxTxVjjuRhnuRhnuRhnuRhnuRhnuRxxjxxHRAiIiIiIrIZPoJFREREREQ2wwKEiIiIiIhshgUIERERERHZDAsQIiIiIiKyGRYgRNVMFEVkZWUhIyPD3qEQERFVm5kzZ2LWrFlIS0uzdyi11pQpUzBnzhzZ7f/973/jueeeq8GIaidOw0uyFRUV4dKlS8jIyIDBYMAjjzxi75BqlQsXLuD777/HmTNnYDAYAAA7duyQjufk5OCbb76BIAiIiYmBu7u7nSK1P61Wi7Nnz0o/S2PHjrV3SLXGjBkzMGDAAERERMDLy8ve4RDVOQkJCTh16hQyMjKQn5+PRYsWScfy8vJw+fJlCIKA1q1b2zFK+7h27RpcXFzQqFEje4dSa6Wnp6OgoEB2e41GA41GU4MR1U4sQEiWHTt2YMuWLcjJyZH2lSxAcnJy8Oqrr6KwsBDvvPMOfH197RGm3ezduxdffPEFioqKpH2CIJRq4+XlBY1Gg1OnTqFdu3aIiIiwdZh2V1BQgP/+97/Ys2dPqVyVLEBycnIwdepU5OXl4dNPP61z/6O7cuUKVq9ejf/+97/o2bMn+vfvj44dO9o7LLubMmWK1X0IgoBVq1ZVQzSOR6PR4H//+x/+/PNPZGZmwmAwwNws/HU1T2lpaViyZAmuXr0KoPhu9r2/x11dXbF06VJkZGTg3XffrXNFSIMGDaDVau0dhlMpKioq83NWF7AAoQqtWLECcXFxEEURrq6uKCwsLNPGy8sLLVu2xL59+3DkyBEMGzbMDpHax99//43PP/8cRqMRAwcORGRkJJYsWYLs7Owybfv164eTJ0/i999/r3MFiNFoxOLFi5GUlAQACAgIgEajgdFoLNXOy8sLffv2xc6dO3H06FGMHj3aDtHaz5NPPol9+/YhPT0dhw4dwuHDhxEQEID+/fvjkUceQYMGDewdol2kp6db3Udd/J88ABw4cACffvopCgoKLBYdpmN1MU+5ublYsGABbt26BV9fX3Tt2hVHjhyR7mabKJVKDBw4EOvXr0d8fHydK0A6d+6MPXv24K+//kKrVq3sHY7Dy83NhVarhVqttncoNscChCxKSEjAvn37oFar8cILL6Bnz5549tlny/0GJCIiAv/73/9w+vTpOlWAxMbGwmg0Yvjw4YiJiQEAKBTlD6/q0KEDAODSpUs2i6+22L9/P06dOgVfX1+8+uqraNWqFSZNmlTuz1J4eDh27tyJM2fO1LkCZOzYsRg7dixOnz6NvXv34tixY7h16xY2btyIb7/9Fl26dEFUVBS6desGpVJp73Btho/pVc2lS5ewYsUKFBUV4YEHHkDXrl3x1VdfQa1W49lnn8Xdu3dx9uxZnDlzBvXq1cPYsWPh4eFh77Bt7scff8StW7cQFhaGN998E15eXjhx4kSZAgQAHnroIaxfvx5//vmnHSK1r+joaPz222/47LPPsHDhQtSrV8/eIdnd5cuXcfny5VL7DAYD4uLizJ4jiiJ0Oh3i4+NhNBrRrFmzmg6z1mEBQhbt2bMHgiBg4sSJCA8Pt9i2ZcuWEAQBV65csU1wtURycjIEQcCoUaMqbOvj4wMPD486+bxnXFwcBEHA5MmTK/zmrHnz5hAEAampqTaKrvbp1KkTOnXqhJycHBw4cAC//vorrly5guPHj+PEiRPw8fHBI488gqioKDRp0sTe4da4J598skb6PXLkCPLz8512TNuPP/6IoqIi9O3bFzNnzgQAfPXVV3B3d0f//v0BAI8//jjOnDmDJUuWYN++fXjvvffsGLF9xMfHS7+fKhp7FRQUBKVSiRs3btgoutrj5s2bGD9+PL7++mtMmzYNffv2RevWrVGvXj2zX7wBQPv27W0YpW0lJCRg8+bNpfbp9XqsWLGiwnNNj/kNHz68psKrtViAkEUXL14EAPTt27fCth4eHlCpVLh7924NR1W73L17Fx4eHrLHvbi4uECv19dwVLWPqTDt3r17hW1dXV3h6emJrKysGo6q9vPy8sLQoUMxdOhQXLp0CXv37sWhQ4dw9+5dbN++Hdu3b0ebNm0wYMAA9O7dG25ubvYO2aGsXr0aWq3WaQuQP/74A4IgIDo6utT+ex/F6tixI5577jksW7YM27dvL9Pe2aWlpUGpVKJly5YVthUEAWq1Grm5uTaIrHaZP3++9IiewWDATz/9hJ9++qnC80pOyOJsPD090bBhQ+l1RkYGBEGw+LisQqGASqVCSEgIBgwY4NQFmjksQMginU4HlUpVJ2/Jy+Xh4QG9Xg+j0WjxGyCg+FsRnU4HHx8fG0VXe+Tl5UGlUsn+gFxYWFinHjGSo3nz5pg2bRqeeOIJvP/++9IjIOfOncOff/6JNWvW4NFHH8WoUaPg6elp52ipNrh79y5cXFzQuHFjaZ8gCMjPzy/TNjw8HCtXrsSRI0fqXAFiNBrh4uJS4e9woLh4y8vLq7MzGZobR1RXDR8+vNQdjBEjRsDHxwdr1qyxY1S1HwsQssjb2xtarRb5+fkVfnDMzMxEbm4u/P39bRRd7RAUFIS//voLly9fRvPmzS22jY+PhyiKFbZzRj4+PsjMzEReXl6FBW1aWhry8vJKfWgi4MyZM/j111+RkJAgTfOoVqvRpUsX/PHHH7hz5w62bt2K/fv3Y9GiRcwflfshWaVSQa/Xl/m97urqCnd3d9y6dcuWIdYKDRs2xM2bN3H37l3Ur1/fYtuUlBQUFBSgadOmtgmuFomNjbV3CLVeXR1HVVlciJAsatGiBYDiDz4V2bNnDwCgTZs2NRpTbdOrVy+IoljmGdB7paWlSeuA9O7d20bR1R6mcR/Hjh2rsO2PP/4IQRDQrl27mg6r1rt9+zY2b96M5557DgsWLMChQ4eQn5+PsLAwTJ8+HWvXrsXcuXPx1VdfYe7cuWjUqBFu376NtWvX2jt0qgUaNGiA3NzcUrMXBgYGAgDOnz9fqu2tW7fq5GNFwP9NEPLrr79abCeKIjZu3AhBENClSxdbhEYO5sknn8Rjjz1m7zBqPRYgZFG/fv0giiLWr19fag2Qe/3222/44YcfIAiCNLCxrhg0aBACAwORmJiId999F3/++ad0i1qr1eLChQvYtGkTZs+ejbt37+L+++9HZGSkfYO2g0cffVT6n7elKVW3b9+OXbt2ASjObV1UVFSEo0eP4s0338TkyZPx7bff4tatW1Cr1RgyZAiWL1+ODz74AFFRUdI33EqlEuHh4Vi0aBEUCgX++OMPO78Lqg1CQkIgimKpWXo6duwIURSxZs0aZGRkAACysrLwySefQBAEhIaG2itcuxk5ciQUCgW2bNli9kuStLQ0vPPOOzh9+jTc3NwwZMgQG0dJ5Dz4CBZZ1KtXLzz44IM4ceIEXn75ZfTr10969OPIkSPIyMjA8ePHce7cOYiiiD59+tS5RdPc3NywYMECvPXWW4iPj0dCQoJ0bNKkSdK2KIpo3Lgx5s+fXyfHNnTq1An9+/fHr7/+ilmzZqFnz57SFJdbt25FRkYGTp48KRUnw4cPr5OPqq1ZswYHDx5Edna2VMi2bt0aAwcORHh4eIWPQjZs2BC+vr7IzMy0RbhUy3Xt2hWHDh3Cb7/9hrCwMADAsGHDsHv3bqSmpmLKlCnw9vYutW7RyJEj7RSt/TRu3BjPP/88PvvsMyxZsgSNGjWCTqcDALz11lvIyMjA9evXARSPoZk+fTr8/PzsGbLdabVanD17FhkZGTAYDHVyquzly5cDAPz8/DBhwoRS+ypDEAS89NJL1RpbbSeIHE1EFTAYDFi+fDmOHj1a7gJVph+h8PBwzJw5E66urrYOsVbQ6/XYvn079u3bV2aaXV9fX0RFRWHUqFF1csEhk6KiIqxfvx47duwod9Ez05SEo0aNwoQJE+rkgmgjRowA8H8LMg4cOLDSz5p/8MEHuHv3LhYvXlwTIToV01o0zjpLj8FgwJEjR+Dl5YWHHnpI2p+cnIylS5fi9u3b0j53d3dMnDgRQ4cOtUeotcKJEyewatUqs+Ng/P39MW3aNHTt2tXGkdUeBQUF+O9//4s9e/agqKhI2l/y31BOTg6mTp2KvLw8fPrpp2jUqJEdIq15I0aMgCAIaNKkCT799NNS++R8vDa1EwTBaX8HmcMChGRLTk7Gr7/+ivPnz+POnTswGo2oX78+WrdujaioKDzwwAP2DrHWuH37NjIzM6Uc3XffffYOqVZJS0vDvn37yv1Z6tevX51Y18Kc1157TZpSt64W87bk7AWIJUVFRTh//jw0Gg08PT3Rpk0bs7OnOft6KSUZjUYkJyfj/PnzpX6Pt2nTBh07dqyTd7BNjEYj3n77bSQlJQEAAgICoNFoYDQay/wbWrNmDXbu3ImJEyc67YKyy5YtgyAI8PX1xcSJE0vtq6wZM2ZUd3i1GgsQIiKqs+pyAVIZzp6nnTt3Aih+7NjS+g113b59+7BixQr4+vri1VdfRatWrcz+bJw/fx7z5s3DAw88gLfeess+ATswZy/6OQaEiKgWE0UR2dnZMBgMdW6KayJb+eqrr6BQKPDoo4/aO5RaLS4uTlox3jSzoTnNmzeHIAhITU21UXTOxdkXSWUBQlSNioqKcPPmTeTk5JSa9rI8dXHlU5LvwoUL+P7773HmzBlpsP69z1ibpnWOiYmps4uiEVUHb29vGI1GPvZYgStXrgAAunfvXmFbV1dXeHp6Iisrq4ajIkfEAoRkEUURf/75J65evYqcnJxSA8/KU9dmw0hPT8e6deuQkJBQYeFh4qyPMlQkJycHx48fR2pqaoWFWl2cGQQA9u7diy+++KLUv7N7nyn28vKCRqPBqVOn0K5dO0RERNg6TKfAp5AJKP62PikpCVqtFj4+PvYOp9bKy8uDSqWqcDY+k8LCwjo9ZobMYwFCFUpMTMQXX3xRqWk961IBkpaWhrlz55aaNpXK9/PPP2Pt2rXIz8+X9pWXs5Izg9S1AuTvv//G559/DqPRiIEDByIyMhJLliwpNU2qSb9+/XDy5En8/vvvLECq6LXXXpP9pQE5r2HDhuHUqVPSop9UPh8fH2RmZiIvL6/C1b7T0tKQl5eHxo0b2yg6ciQsQMiis2fPYsmSJTAajQCKV9Vt0KCB7G8/6oJNmzYhKysLnp6eiI6ORo8ePdCgQQPeyr/Hb7/9hi+//BIA4OLigrCwMP4slSM2NhZGoxHDhw9HTEwMAEChKH/NWNPqzZcuXbJZfM6mdevW9g6BaoGuXbvimWeewbp165CTk4PHHnusTi7IWJFWrVohPj4ex44dq/BLjx9//BGCIKBdu3Y2io4cCQsQsuiHH36A0WhESEgIZsyYUScXhqvI6dOnIQgCZs2ahW7dutk7nFrL9MhZ+/bt8fLLL9f5RbzMSU5OltZCqYiPjw88PDzKrDtDRJUzZcoUAMXF/qFDh3Do0CG4ubnB29vb7BcAgiBg1apVtgzT7h599FH89ttv2LhxI9q0aYOAgIBy223fvh27du2CIAgYNGiQjaMkR8AChCy6cOECBEHAyy+/jJCQEHuHUyvpdDq4uLjU6YWp5Lh69SoEQcCMGTNYfFhw9+5deHh4wNfXV1Z7FxcX6PX6Go6KyLmlp6eX2WcwGKQJIMpTFxdK7dSpE/r3749ff/0Vs2bNQs+ePaUcbd26FRkZGTh58qSUz+HDh/OLSyoXCxCyqKioCB4eHiw+LPDz84NWqzX7LRkVEwQBKpXK7DdmVMzDwwN6vR5Go7HCnym9Xg+dTsdBs0RWqmtjzawxbdo0eHl5YceOHfj1118BFP9+X79+PQBI4/dGjx6NCRMm2DNUqsVYgJBFjRs3xrVr11BUVMSZLMzo2bMnYmNjkZKSgpYtW9o7nForODgYFy9eRH5+Psd9WBAUFIS//voLly9frvCbw/j4eIiiyG8YiazUr18/e4fgMJRKJZ5++mk8+uij2LdvH86fP487d+5IK8a3bt0a/fr1Q5MmTewdKtVi/MqWLIqKikJhYSGOHTtm71BqrejoaPj7++Pzzz9HTk6OvcOptYYMGYKioiLs37/f3qHUar169YIoiti8ebPFdmlpadI6IL1797ZRdERExRo1aoSnnnoKCxcuxCeffILPPvsM77zzDiZOnMjigyrEOyBk0ZAhQ3Dy5El89tln8PPzq/MzxiQnJ5e7f/z48Vi1ahVefPFFDBgwAGFhYVCpVBb7qmsLEUZEROCPP/7AmjVroFKp0KdPH3uHVCsNGjQIu3fvRmJiIt59912MGDFCmqpYq9UiPT0dx48fx08//QSdTofQ0FBERkbaN2hyenVtinFRFJGdnQ2DwQB/f397h0PkdASxrv1WIbO+++67cvcXFhZi9+7d0Ol0aNu2rawP1866DsiIESOqbeChMy9EuHz5crPHEhMTodPp0LBhQ7Ro0cLiz1JdXAcEAG7cuIG33noLaWlpZn/eRFFE48aN8fbbb/MDEtW48+fPo7Cw0Om/OLlw4QK+//57nDlzRhpcXfJ3dU5OjnTnMSYmBu7u7naKlJzdxIkTkZWV5bSfFXgHhCTffvutxQ/Xoijijz/+wLlz5yrsy1kLEKDufRNYFXFxcdJigiWV3JeRkYGMjIxyz6/LCxECxWOvli1bhu3bt2Pfvn1lptn19fVFVFQURo0aBbVabacoqS6pC3e/9+7diy+++AJFRUXSvnv/n+jl5QWNRoNTp06hXbt2dXYB0JycHBw/fhypqanIycmxuJhnXf09bi1nXySVBQhJ2rVrVyenFayM2NhYe4fgEPr27cufJSupVCqMGzcO48aNw+3bt5GZmSkN8rzvvvvsHR6RU/n777/x+eefw2g0YuDAgYiMjMSSJUuQnZ1dpm2/fv1w8uRJ/P7773WyAPn555+xdu1a5OfnS/vK+2Kurn+RZC1nL/pZgJDknXfesXcI5CRmzpxZY30fOXIE+fn5eOSRR2rsGrVNgwYN0KBBA3uHQeS0YmNjYTQaMXz4cMTExACA2WmwO3ToAAC4dOmSzeKrLX777Td8+eWXAIrXIAoLC0ODBg04syFVGgsQIitlZGRAoVDI/oB4+/ZtGI1GPrdfRatXr4ZWq61TBQgR1azk5GQIgoBRo0ZV2NbHxwceHh5lHo2sC0zjEdq3b4+XX36Zi8pSlbEAIbLS5MmT4evri7Vr18pqP2/ePGg0GqcdWEbVp6ioCDdv3qzwGWug7s2qRlSd7t69Cw8PD/j6+spq7+LiAr1eX8NR1T5Xr16FIAiYMWMGiw+yCgsQqpDRaIQgCOU+0797924kJyejoKAAXbt2xYABA/jsP5GV0tPTsW7dOiQkJMgehMiClqjqPDw8oNfrYTQazT56ZaLX66HT6eDj42Oj6GoPQRCgUqkQEBBg71DIwbEAIYv27t2Lzz77DOHh4ZgzZ06pY4sWLcKJEycAFA9AS0xMxMmTJ/Hqq6/aI1SHYTAYKvwfHNVdaWlpmDt3LrKzsznjGpGNBAUF4a+//sLly5fRvHlzi23j4+MhimKF7ZxRcHAwLl68iPz8fI77IKuwACGLTp48CaB4VqOSfv/9dxw/fhwA0K1bN7i5uSE+Ph7Hjh3DkSNHEB4ebvNYHcGNGzeQnZ0t+zY/1T2bNm1CVlYWPD09ER0djR49eqBBgwZwdXW1d2hETqtXr144f/48Nm/ejNdee81su7S0NGkdkN69e9swwtphyJAhWLp0Kfbv34+BAwfaOxxyYCxAyKKrV68CAFq1alVq//79+yEIAkaMGIFnnnkGAPDTTz9h9erViIuLc+oCJCEhAceOHSu1T6fTWVx8z9TGtIZK27Ztayw+cmynT5+GIAiYNWsWunXrZu9wiOqEQYMGYffu3UhMTMS7776LESNGSHcgtVot0tPTcfz4cfz000/Q6XQIDQ1FZGSkfYO2g4iICPzxxx9Ys2YNVCoV+vTpY++QyEGxACGLtFot3N3d4eXlVWr/6dOnAQCPPvqotK9fv35YvXo1/v77b5vGaGuXL18us9Befn4+4uLiZJ3v7e3t1As1knV0Oh1cXFzQtWtXe4dCVGe4ublhwYIFeOuttxAfH4+EhATp2KRJk6RtURTRuHFjzJ8/H0ql0h6h2oylL9Xc3NywdOlSrFu3Di1atIBKpTLbluuAUHlYgJBFeXl5ZR79SEtLQ1ZWFvz9/REYGCjtV6lU8PT0RFZWlq3DtKnQ0NBSU8DGxcXBzc3N4l0fQRCgVqsRHByMnj17wtvb2xahkgPy8/ODVqvlOCEiG2vcuDGWLVuG7du3Y9++fWWm2fX19UVUVBRGjRoFtVptpyht594v2kxK7svIyEBGRka553MhQrKEBQhZVK9ePdy9exdZWVmoV68eACApKQkA0KZNmzLti4qKLH4T4gx69OiBHj16SK/j4uLg6emJGTNmWNVvXVxgj8rq2bMnYmNjkZKSgpYtW9o7HKI6RaVSYdy4cRg3bhxu376NzMxMGI1G1K9fH/fdd5+9w7Opvn37clZLqjEsQMii5s2b4/fff0dsbCwmTJgAg8GA3bt3QxAEPPDAA6Xa3rlzB3l5eWjatKl9grWTxYsXw8XF+n9KXGCPACA6Ohq//fYbPv/8cyxcuLDM449EZBsNGjSQvcCsM5o5c2aN9c0v3IgFCFn06KOP4sSJE9i6dSsSEhKQm5uLzMxMeHt7o1evXqXanj17FgAQEhJij1DthgvAUVUlJyeXu3/8+PFYtWoVXnzxRQwYMABhYWEV3lnkzyEROQp+4UYsQMiibt26ITo6Gj/88AOuX78OAPDy8sKsWbPKfCA6ePAgAKBjx442j5PqDmdaG2P+/PkVPuLw/fffy+qLCxESVY+ioiLcvHkTOTk5FS4EysKfqGpYgFCFnnrqKfTv3x8pKSlQq9Vo2bJlmcdCCgsLERYWhhYtWqB79+5l+uDtVqour732muzVwR2BMxVURI4sPT0d69atQ0JCguzfMSz8iaqGBQjJEhAQgICAALPHXVxcLE4ty9utVF1at25t7xCqTWxsrL1DICIUz+44d+5cZGdn80sBIhtgAUJERER12qZNm5CVlQVPT09ER0ejR48eaNCgQZlp6ImoerAAISKqRTIyMqBQKGTPvnP79m0YjUb4+/vXcGREzuv06dMQBAGzZs1Ct27d7B0OkdNjAUJEVItMnjwZvr6+WLt2raz28+bNg0aj4bPoRFbQ6XRwcXFB165d7R0KUZ3ApXaJiIioTvPz84NSqYRCwY9FRLbAf2lERA7MYDDwQxORlXr27AmDwYCUlBR7h0JUJ/D/WkREDurGjRvIzs6Gj4+PvUMhcmjR0dHw9/fH559/jpycHHuHQ+T0OAaEqJbg1I91U0JCAo4dO1Zqn06nw/Llyy2ep9PpcO7cOQBA27Ztayw+ImeTnJxc7v7x48dj1apVePHFFzFgwACEhYWVWXD3XlyIkKhqWIAQ1RLOtsAeyXP58mXExcVBEASpCM3Pz0dcXJys8729vS2uwUNEpc2fPx+CIFhs8/3338vqi5M/VA2/cCMWIES1hDMtsEfyhYaGllqgMy4uDm5ubggPDzd7jiAIUKvVCA4ORs+ePeHt7W2LUImcBj8A2xe/cCNB5L9CsoFJkyZBq9Xy2yKiCowYMaJS0/Cac+TIEeTn55cqboiIiGoD3gEhm2CdSyTP4sWL4eJi/a/m1atXQ6vVsgAhIqJahwUI2QRvtxLJw0GtRLaXkZEBhUKBBg0ayGp/+/ZtGI1G+Pv713BkRM6JBQjZBMc3EBFRbTV58uRKPfo4b948aDQaPlZMVEVcB4SIiIiIiGyGBQgRERFRJRgMBigU/AhFVFX810NEREQk040bN5CdnQ0fHx97h0LksDgGhIiIiOqUhIQEHDt2rNQ+nU6H5cuXWzxPp9Ph3LlzAIC2bdvWWHxEzo4FCBEREdUply9fRlxcHARBkKaJz8/PR1xcnKzzvb29MXbs2JoMkcipsQAhIiKiOiU0NLTUGjlxcXFwc3NDeHi42XMEQYBarUZwcDB69uwJb29vW4RK5JRYgBAREVGd0qNHD/To0UN6HRcXB09PT8yYMcOqfo8cOYL8/HwuAEpUARYgREREVKctXrwYLi7WfyRavXo1tFotCxCiCrAAISJyQqbn2omoYu3bt7d3CER1CgsQIiIn9Nprr6GwsNDeYRAREZXBAoSIyAm1bt3a3iEQERGViwsREhERERGRzbAAISIiIiIim2EBQkRERERENsMChIiIiIiIbIYFCBERERER2QwLECIiIiIishkWIEREREREZDMsQIiIiIiqgSiK9g6ByCEIIv+1EBEREVnt/PnzKCwsRPv27e0dClGtxgKEiIiIiIhsho9gERERERGRzbAAISIiIiIim2EBQkRERERENsMChIiIiIiIbIYFCBERERER2QwLECIiIiIishkWIERERLXAgQMHIAgCBEHAm2++ae9wiIhqDAsQIiKyqRYtWkgftJOTkytsHxUVJbVv2rRphe1zc3Ph7u4OQRDg6uqKnJyc6gibiIiqCQsQIiKyqb59+0rbBw4csNg2Pz8fv/32m/T6+vXruHjxosVzjh49ivz8fABAt27d4OXlVfVgiYio2rEAISIimypZgOzfv99i22PHjkGv15faV9E5JYuaktciIqLagQUIERHZVGRkpLR96NAhiKJotq2pmPD29kZ4eHipfRWdA7AAISKqjViAEBGRTTVu3BgtW7YEAGg0Gpw9e9ZsW1MxER4ejn79+pXaV57c3FwcP34cAODm5obevXtXT9BERFRtWIAQEZHNyRkHkp+fj/j4eADFd00iIiIAADdu3EBKSkq55xw9ehQFBQUAgIceeggqlarUcb1ej08++QT9+/dHYGAg3Nzc0KBBA3Tr1g2vv/46bty4YTHutWvXSgPi165dCwA4efIknn/+ebRs2RLe3t6ljpW0Z88ePPbYYwgMDISHhweCg4MxatQo7N271+I1iYicDQsQIiKyOTnjQEqO/4iMjESPHj3g5uZm8RxLj18dP34crVq1wvTp0/G///0PaWlpKCgoQGZmJk6cOIHFixcjLCwMX3/9tez38f7776N79+748ssvceHChXJn3DIajZgyZQoeffRR7NixA2lpaTAYDLh27Rq2b9+OgQMHYubMmbKvSUTk6FzsHQAREdU95Y0DEQShVJuS4z+6dOkCFxcXdO/eHUeOHMGBAwcwderUMv2aK0DOnDmDvn37QqfTAQDatm2LCRMmIDQ0FJmZmdixYwf27t2L3NxcxMTEQBRFxMTEWHwP33//PXbv3g0vLy9MnDgR3bt3h6urK86dO4dGjRpJ7WbNmoU1a9YAAJRKJZ566ilERkbC3d0dSUlJ+Oqrr7B8+XJcu3ZNVu6IiByeSEREZAdt2rQRAYgAxFOnTpU5/sgjj4gAxEcffVTa99prr4kAxEaNGpVpr9PpRFdXVxGA6OHhIebl5YmiKIpFRUVi+/btpWtNnjxZLCgoKHP+mjVrREEQRACiWq0WL1++XKbNf//7X6kfAGLLli3Fq1evmn2PR44ckfr09PQUDx8+XKbNjRs3xNatW5fq9z//+Y/ZPomIHB0fwSIiIruwNA7k3vEfJqZxIGlpaTh//nypc0qO/+jZsyfc3d0BALt27ZIWPOzYsSO++OILuLiUfQAgJiZGuquSm5uL5cuXW4xfEAR89913CA4ONtvmo48+kmb5eu+996SZvEoKDAzE5s2boVQqLV6PiMhZsAAhIiK7sDQOJCEhQRr/YSo6AKBXr15S8XDvOSVflyxatm3bJm2//PLLFj/o//vf/5YeBSt5XnnCw8PRuXNns8cNBgN27doFAPDx8cHkyZPNtu3YsSMGDBhg8XpERM6CBQgREdlFRESE9GH/8OHDMBqN0jHTHREvLy88+OCD0n4vLy907dq1VJt7zwFKFzfHjh2Ttiv6kB8SEoLWrVsDAFJTU3Hz5k2zbR9++GGLfZ0+fVpakb13797SHRlzTNMMExE5OxYgRERkF/7+/mjXrh0A4M6dO0hKSpKOmYqJ3r17l3lcynRHpGTBodPpcOLECQCASqXCQw89JB0zFRHe3t6lBoebY1qjpOS55QkKCrLYT8kpfVu0aFHhdeW0ISJyBixAiIjIbsobB2IwGJCQkACg9ONXJqZ96enpOHfuHIDS4z969+4tTdcLANnZ2QAAT09PWTF5eXmVObc8964xcq+SU/Kq1eoKrys3PiIiR8cChIiI7KbkWA3TGI571/+4V3h4uDSOw3SOpfU/vL29AUCagrciJQsH07lVUbKQyc3NrbC93PiIiBwdCxAiIrKbe8eBFBUVScWEp6cnunXrVuacevXqoVOnTgDkFSCBgYEAiu9m3Lp1q8KYSq6y3rhxY/lv5h5NmjSRti9evFhhezltiIicAQsQIiKymwYNGqBjx44AAK1Wi1OnTknFRMkZr+5legzr4MGDyMnJkcZ/eHl5lSlaSo4H2bt3r8V4UlNTpel9g4ODZY0ZMadjx47SwPOjR4/CYDBYbL9v374qX4uIyJGwACEiIrsqecfil19+kcZ/lPf4lYmpANFoNPjyyy+l8R/h4eFlipbRo0dL2x999BGKiorM9vvee+9J63aUPK8q3N3dMXjwYADFxdXXX39ttm1ycnKFxRERkbNgAUJERHZVstBYuXKlxfEfJg8//LD06Nb7778v7b/38SsAGDx4MDp06ACgeGrcadOmobCwsEy7tWvX4osvvgBQPGh8xowZlX4v93r55ZelOOfNmyctrljSrVu38MQTT1gsjIiInEn597aJiIhspE+fPlAoFDAajUhPTwdQXACUN/7DxM/PDx06dMCZM2ekc4DyCxCFQoENGzagV69e0Ol0WL16NeLj4zFhwgTcf//9yMzMRGxsLH755RfpnBUrViAkJMTq99a7d29Mnz4dK1asQHZ2Nvr06YPx48cjIiIC7u7uSEpKwpo1a5CZmYlRo0ZVuPghEZEzYAFCRER25evriwceeAAnT56U9vXq1Quurq4Wz4uIiMCZM2ek1/Xq1UOXLl3KbduxY0fs378fo0aNwvXr15GcnIx58+aVaadWq7FixQrExMRU8d2U9fHHH0On0+Grr75CYWEh1q5di7Vr15ZqM2PGDIwcOZIFCBHVCXwEi4iI7O7eOxeWHr8yuXeNkIcfflianrc83bp1Q0pKClasWIF+/frhvvvug6urK3x9fdG1a1e89tpruHDhQrUWH0DxHZg1a9Zg9+7dGD58OAICAuDm5oagoCA89thj+OWXX7Bs2bJqvSYRUW0miKbRdkRERERERDWMd0CIiIiIiMhmWIAQEREREZHNsAAhIiIiIiKbYQFCREREREQ2wwKEiIiIiIhshgUIERERERHZDAsQIiIiIiKyGRYgRERERERkMyxAiIiIiIjIZliAEBERERGRzbAAISIiIiIim2EBQkRERERENsMChIiIiIiIbIYFCBERERER2QwLECIiIiIishkWIEREREREZDMsQIiIiIiIyGZYgBARERERkc2wACEiIiIiIpthAUJERERERDbDAoSIiIiIiGzm/wGFUI09+uEGsQAAAABJRU5ErkJggg==",
      "text/plain": [
       "<Figure size 400x200 with 1 Axes>"
      ]
     },
     "metadata": {
      "image/png": {
       "height": 200,
       "width": 400
      }
     },
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n"
     ]
    }
   ],
   "source": [
    "import torch\n",
    "import pandas as pd\n",
    "from plotnine import ggplot, aes, geom_bar, theme, element_text, labs\n",
    "\n",
    "import pyvene as pv\n",
    "_, tokenizer, backpack_gpt2 = pv.create_backpack_gpt2()\n",
    "\n",
    "class MultiplierIntervention(pv.ConstantSourceIntervention):\n",
    "    \"\"\"Multiplier intervention\"\"\"\n",
    "    \n",
    "    def __init__(self, multiplier, **kwargs):\n",
    "        super().__init__(**kwargs)\n",
    "        self.register_buffer('multiplier', torch.tensor(multiplier))\n",
    "        \n",
    "    def forward(self, base, source=None, subspaces=None):\n",
    "        return base * self.multiplier\n",
    "\n",
    "    def __str__(self):\n",
    "        return f\"MultiplierIntervention()\"\n",
    "\n",
    "for c in [0, 0.7, 1]:\n",
    "    pv_backpack_gpt2 = pv.IntervenableModel({\n",
    "        \"component\": \"backpack.sense_network.output\",\n",
    "        \"intervention\": MultiplierIntervention(c), \"unit\": \"sense.pos\"}, \n",
    "        model=backpack_gpt2\n",
    "    )\n",
    "    base = tokenizer(\"When the nurse walked into the room,\", \n",
    "                     return_tensors=\"pt\", return_attention_mask=False)\n",
    "    intervened_outputs = pv_backpack_gpt2(\n",
    "        base,\n",
    "        unit_locations={\n",
    "            # use   pv.GET_LOC((nv, s))\n",
    "            \"base\": pv.GET_LOC((10,2))\n",
    "        }\n",
    "    )\n",
    "    \n",
    "    # plotting\n",
    "    probs = torch.nn.functional.softmax(\n",
    "        intervened_outputs[1].logits[0][-1], dim=0)\n",
    "    data = pv.top_vals(\n",
    "        tokenizer, probs, n=9,\n",
    "        return_results=True\n",
    "    )\n",
    "    df = pd.DataFrame(data, columns=['Word', 'Probability'])\n",
    "    df['Word'] = pd.Categorical(df['Word'], categories=[x[0] for x in data], ordered=True)\n",
    "    plot = (ggplot(df, aes(x='Word', y='Probability'))\n",
    "            + geom_bar(stat='identity')\n",
    "            + theme(axis_text_x=element_text(rotation=90, hjust=1),\n",
    "                    figure_size=(4, 2))\n",
    "            + labs(title=f\"mul({c})\")\n",
    "    )\n",
    "    print(plot)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cb539f4b",
   "metadata": {},
   "source": [
    "### Saving and Loading\n",
    "This is one of the benefits of program abstraction. We abstract out the intervention and its schema, so we have a user friendly interface. Furthermore, it allows us to have a serializable configuration file that tells everything about your configuration.\n",
    "\n",
    "You can then save, share and load interventions easily. Note that you still need your access to the data, if you need to sample **Source** representations from other examples. But we think this is doable via a separate HuggingFace datasets upload. In the future, there could be an option of coupling this configuration with a specific remote dataset as well."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "id": "272f3773",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "loaded model\n",
      "Directory './tmp/' already exists.\n"
     ]
    }
   ],
   "source": [
    "import torch\n",
    "import pyvene as pv\n",
    "\n",
    "_, tokenizer, gpt2 = pv.create_gpt2()\n",
    "\n",
    "# run with new intervention type\n",
    "pv_gpt2 = pv.IntervenableModel({\n",
    "  \"intervention_type\": pv.ZeroIntervention}, \n",
    "  model=gpt2)\n",
    "\n",
    "pv_gpt2.save(\"./tmp/\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "id": "50b894b4",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "WARNING:root:The key is provided in the config. Assuming this is loaded from a pretrained module.\n",
      "WARNING:root:Loading trainable intervention from intkey_layer.0.repr.block_output.unit.pos.nunit.1#0.bin.\n"
     ]
    }
   ],
   "source": [
    "pv_gpt2 = pv.IntervenableModel.load(\n",
    "    \"./tmp/\",\n",
    "    model=gpt2)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b2d07ca8",
   "metadata": {},
   "source": [
    "### Multi-Source Interchange Intervention (Parallel Mode)\n",
    "\n",
    "What is multi-source? In the examples above, interventions are at most across two examples. We support interventions across many examples. You can sample representations from two inputs, and plut them into a single **Base**."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "id": "847410a8",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "loaded model\n",
      "_the                 0.07233363389968872\n",
      "_a                   0.05731499195098877\n",
      "_not                 0.04443885385990143\n",
      "_Italian             0.033642884343862534\n",
      "_often               0.024385808035731316\n",
      "_called              0.022171705961227417\n",
      "_known               0.017808808013796806\n",
      "_that                0.016059240326285362\n",
      "_\"                   0.012973357923328876\n",
      "_an                  0.012878881767392159\n"
     ]
    }
   ],
   "source": [
    "import torch\n",
    "import pyvene as pv\n",
    "\n",
    "_, tokenizer, gpt2 = pv.create_gpt2()\n",
    "\n",
    "parallel_config = pv.IntervenableConfig([\n",
    "  {\"layer\": 3, \"component\": \"block_output\"},\n",
    "  {\"layer\": 3, \"component\": \"block_output\"}],\n",
    "  # intervene on base at the same time\n",
    "  mode=\"parallel\")\n",
    "parallel_gpt2 = pv.IntervenableModel(\n",
    "  parallel_config, model=gpt2)\n",
    "base = tokenizer(\n",
    "  \"The capital of Spain is\", \n",
    "  return_tensors=\"pt\")\n",
    "sources = [\n",
    "  tokenizer(\"The language of Spain is\", \n",
    "    return_tensors=\"pt\"),\n",
    "  tokenizer(\"The capital of Italy is\", \n",
    "    return_tensors=\"pt\")]\n",
    "intervened_outputs = parallel_gpt2(\n",
    "    base, sources,\n",
    "    {\"sources->base\": (\n",
    "    # each list has a dimensionality of\n",
    "    # [num_intervention, batch, num_unit]\n",
    "    [[[1]],[[3]]],  [[[1]],[[3]]])}\n",
    ")\n",
    "\n",
    "distrib = pv.embed_to_distrib(\n",
    "    gpt2, intervened_outputs[1].last_hidden_state, logits=False)\n",
    "pv.top_vals(tokenizer, distrib[0][-1], n=10)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2f93402c",
   "metadata": {},
   "source": [
    "### Multi-Source Interchange Intervention (Serial Mode)\n",
    "\n",
    "Or you can do them sequentially, where you intervene among your **Source** examples, and get some intermediate states before merging the activations into the **Base** run."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "id": "5e5752dc",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "_the                 0.06737838685512543\n",
      "_a                   0.059834375977516174\n",
      "_not                 0.04629501700401306\n",
      "_Italian             0.03623826056718826\n",
      "_often               0.021700192242860794\n",
      "_called              0.01840786263346672\n",
      "_that                0.0157712884247303\n",
      "_known               0.014391838572919369\n",
      "_an                  0.013535155914723873\n",
      "_very                0.013022392988204956\n"
     ]
    }
   ],
   "source": [
    "config = pv.IntervenableConfig([\n",
    "  {\"layer\": 3, \"component\": \"block_output\"},\n",
    "  {\"layer\": 10, \"component\": \"block_output\"}],\n",
    "  # intervene on base one after another\n",
    "  mode=\"serial\")\n",
    "pv_gpt2 = pv.IntervenableModel(\n",
    "  config, model=gpt2)\n",
    "base = tokenizer(\n",
    "  \"The capital of Spain is\", \n",
    "  return_tensors=\"pt\")\n",
    "sources = [\n",
    "  tokenizer(\"The language of Spain is\", \n",
    "    return_tensors=\"pt\"),\n",
    "  tokenizer(\"The capital of Italy is\", \n",
    "    return_tensors=\"pt\")]\n",
    "\n",
    "intervened_outputs = pv_gpt2(\n",
    "    base, sources,\n",
    "    # intervene in serial at two positions\n",
    "    {\"source_0->source_1\": 1, \n",
    "     \"source_1->base\"    : 4})\n",
    "\n",
    "distrib = pv.embed_to_distrib(\n",
    "    gpt2, intervened_outputs[1].last_hidden_state, logits=False)\n",
    "pv.top_vals(tokenizer, distrib[0][-1], n=10)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "28621880",
   "metadata": {},
   "source": [
    "### Multi-Source Interchange Intervention with Subspaces (Parallel Mode)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "id": "773aba2e",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "loaded model\n"
     ]
    }
   ],
   "source": [
    "import torch\n",
    "import pyvene as pv\n",
    "\n",
    "_, tokenizer, gpt2 = pv.create_gpt2()\n",
    "\n",
    "config = pv.IntervenableConfig([\n",
    "    {\"layer\": 0, \"component\": \"block_output\",\n",
    "     \"subspace_partition\": \n",
    "         [[0, 128], [128, 256]]}]*2,\n",
    "    intervention_types=pv.VanillaIntervention,\n",
    "    # act in parallel\n",
    "    mode=\"parallel\"\n",
    ")\n",
    "pv_gpt2 = pv.IntervenableModel(config, model=gpt2)\n",
    "\n",
    "base = tokenizer(\"The capital of Spain is\", return_tensors=\"pt\")\n",
    "sources = [tokenizer(\"The capital of Italy is\", return_tensors=\"pt\"),\n",
    "          tokenizer(\"The capital of China is\", return_tensors=\"pt\")]\n",
    "\n",
    "intervened_outputs = pv_gpt2(\n",
    "    base, sources,\n",
    "    # on same position\n",
    "    {\"sources->base\": 4},\n",
    "    # on different subspaces\n",
    "    subspaces=[[[0]], [[1]]],\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7223603f",
   "metadata": {},
   "source": [
    "### Multi-Source Interchange Intervention with Subspaces (Serial Mode)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "id": "305e0607",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "loaded model\n"
     ]
    }
   ],
   "source": [
    "import torch\n",
    "import pyvene as pv\n",
    "\n",
    "_, tokenizer, gpt2 = pv.create_gpt2()\n",
    "\n",
    "config = pv.IntervenableConfig([\n",
    "    {\"layer\": 0, \"component\": \"block_output\",\n",
    "     \"subspace_partition\": [[0, 128], [128, 256]]},\n",
    "    {\"layer\": 2, \"component\": \"block_output\",\n",
    "     \"subspace_partition\": [[0, 128], [128, 256]]}],\n",
    "    intervention_types=pv.VanillaIntervention,\n",
    "    # act in parallel\n",
    "    mode=\"serial\"\n",
    ")\n",
    "pv_gpt2 = pv.IntervenableModel(config, model=gpt2)\n",
    "\n",
    "base = tokenizer(\"The capital of Spain is\", return_tensors=\"pt\")\n",
    "sources = [tokenizer(\"The capital of Italy is\", return_tensors=\"pt\"),\n",
    "          tokenizer(\"The capital of China is\", return_tensors=\"pt\")]\n",
    "\n",
    "intervened_outputs = pv_gpt2(\n",
    "    base, sources,\n",
    "    # serialized intervention\n",
    "    # order is based on sources list\n",
    "    {\"source_0->source_1\": 3, \"source_1->base\": 4},\n",
    "    # on different subspaces\n",
    "    subspaces=[[[0]], [[1]]],\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4b5fcb37",
   "metadata": {},
   "source": [
    "### Interchange Intervention Training (IIT)\n",
    "Interchange intervention training (IIT) is a technique of inducing causal structures into neural models. This library naturally supports this. By training IIT, you can simply turn the gradient on for the wrapping model. In this way, your model can be trained with your interventional signals."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "8c7dde89",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "loaded model\n",
      "number of params: 124439808\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "tensor([[[ 0.0022, -0.1783, -0.2780,  ...,  0.0477, -0.2069,  0.1093],\n",
       "         [ 0.0385,  0.0886, -0.6608,  ...,  0.0104, -0.4946,  0.6148],\n",
       "         [ 0.2377, -0.2312,  0.0308,  ...,  0.1085,  0.0456,  0.2494],\n",
       "         [-0.0034,  0.0088, -0.2219,  ...,  0.1198,  0.0759,  0.3953],\n",
       "         [ 0.4635,  0.2698, -0.3185,  ..., -0.2946,  0.2634,  0.2714]]],\n",
       "       grad_fn=<SubBackward0>)"
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import torch\n",
    "import pyvene as pv\n",
    "\n",
    "_, tokenizer, gpt2 = pv.create_gpt2()\n",
    "\n",
    "pv_gpt2 = pv.IntervenableModel({\n",
    "    \"layer\": 8, \"component\": \"block_output\"}, \n",
    "    model=gpt2\n",
    ")\n",
    "\n",
    "pv_gpt2.enable_model_gradients()\n",
    "print(\"number of params:\", pv_gpt2.count_parameters())\n",
    "\n",
    "# run counterfactual forward as usual\n",
    "base = tokenizer(\"The capital of Spain is\", return_tensors=\"pt\")\n",
    "sources = [\n",
    "    tokenizer(\"The capital of Italy is\", return_tensors=\"pt\"),\n",
    "]\n",
    "base_outputs, counterfactual_outputs = pv_gpt2(\n",
    "    base, sources, {\"sources->base\": ([[[3]]], [[[3]]])}, output_original_output=True\n",
    ")\n",
    "print(counterfactual_outputs.last_hidden_state - base_outputs.last_hidden_state)\n",
    "# call backward will put gradients on model's weights\n",
    "counterfactual_outputs.last_hidden_state.sum().backward()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b8c7ccad",
   "metadata": {},
   "source": [
    "## pyvene 102\n",
    "Now, you are pretty familiar with pyvene basic APIs. There are more to come. We support all sorts of weird interventions, and we encapsulate them as objects so that, even they are super weird (e.g., nested, multiple locations, different types), you can share them easily with others. BTW, if the intervention is trainable, the artifacts will be saved and shared as well.\n",
    "\n",
    "With that, here are a couple of additional APIs.\n",
    "\n",
    "### Grouping\n",
    "\n",
    "You can group interventions together so that they always receive the same input when you want to use them to get activations at different places. Here is an example, where you are taking in the same **Source** example, you fetch activations twice: once in position 3 and layer 0, once in position 4 and layer 2. You don't have to pass in another dummy **Source**."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "id": "84afd62c",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "loaded model\n"
     ]
    }
   ],
   "source": [
    "import torch\n",
    "import pyvene as pv\n",
    "\n",
    "_, tokenizer, gpt2 = pv.create_gpt2()\n",
    "\n",
    "config = pv.IntervenableConfig([\n",
    "    {\"layer\": 0, \"component\": \"block_output\", \"group_key\": 0},\n",
    "    {\"layer\": 2, \"component\": \"block_output\", \"group_key\": 0}],\n",
    "    intervention_types=pv.VanillaIntervention,\n",
    ")\n",
    "\n",
    "pv_gpt2 = pv.IntervenableModel(config, model=gpt2)\n",
    "\n",
    "base = tokenizer(\"The capital of Spain is\", return_tensors=\"pt\")\n",
    "sources = [tokenizer(\"The capital of Italy is\", return_tensors=\"pt\")]\n",
    "intervened_outputs = pv_gpt2(\n",
    "    base, sources, \n",
    "    {\"sources->base\": ([\n",
    "        [[3]], [[4]] # these two are for two interventions\n",
    "    ], [             # source position 3 into base position 4\n",
    "        [[3]], [[4]] \n",
    "    ])}\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "34aeb892",
   "metadata": {},
   "source": [
    "### Intervention Skipping in Runtime\n",
    "You may configure a lot of interventions, but during training, not every example will have to use all of them. So, you can skip interventions for different examples differently."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "id": "61cd8fc9",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "loaded model\n",
      "True True\n"
     ]
    }
   ],
   "source": [
    "import torch\n",
    "import pyvene as pv\n",
    "\n",
    "_, tokenizer, gpt2 = pv.create_gpt2()\n",
    "\n",
    "config = pv.IntervenableConfig([\n",
    "    # these are equivalent interventions\n",
    "    # we create them on purpose\n",
    "    {\"layer\": 0, \"component\": \"block_output\"},\n",
    "    {\"layer\": 0, \"component\": \"block_output\"},\n",
    "    {\"layer\": 0, \"component\": \"block_output\"}],\n",
    "    intervention_types=pv.VanillaIntervention,\n",
    ")\n",
    "pv_gpt2 = pv.IntervenableModel(config, model=gpt2)\n",
    "\n",
    "base = tokenizer(\"The capital of Spain is\", return_tensors=\"pt\")\n",
    "source = tokenizer(\"The capital of Italy is\", return_tensors=\"pt\")\n",
    "# skipping 1, 2 and 3\n",
    "_, pv_out1 = pv_gpt2(base, [None, None, source],\n",
    "    {\"sources->base\": ([None, None, [[4]]], [None, None, [[4]]])})\n",
    "_, pv_out2 = pv_gpt2(base, [None, source, None],\n",
    "    {\"sources->base\": ([None, [[4]], None], [None, [[4]], None])})\n",
    "_, pv_out3 = pv_gpt2(base, [source, None, None],\n",
    "    {\"sources->base\": ([[[4]], None, None], [[[4]], None, None])})\n",
    "# should have the same results\n",
    "print(\n",
    "    torch.equal(pv_out1.last_hidden_state, pv_out2.last_hidden_state),\n",
    "    torch.equal(pv_out2.last_hidden_state, pv_out3.last_hidden_state)\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d9df6acd",
   "metadata": {},
   "source": [
    "### Subspace Partition\n",
    "You can partition your subspace before hand. If you don't, the library assumes you each neuron is in its own subspace. In this example, you partition your subspace into two continous chunk, `[0, 128), [128,256)`, which means all the neurons from index 0 upto 127 are along to partition 1. During runtime, you can intervene on all the neurons in the same parition together."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "id": "3a66bbeb",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "loaded model\n"
     ]
    }
   ],
   "source": [
    "import torch\n",
    "import pyvene as pv\n",
    "\n",
    "_, tokenizer, gpt2 = pv.create_gpt2()\n",
    "\n",
    "config = pv.IntervenableConfig([\n",
    "    # they are linked to manipulate the same representation\n",
    "    # but in different subspaces\n",
    "    {\"layer\": 0, \"component\": \"block_output\",\n",
    "     # subspaces can be partitioned into continuous chunks\n",
    "     # [i, j] are the boundary indices\n",
    "     \"subspace_partition\": [[0, 128], [128, 256]]}],\n",
    "    intervention_types=pv.VanillaIntervention,\n",
    ")\n",
    "pv_gpt2 = pv.IntervenableModel(config, model=gpt2)\n",
    "\n",
    "base = tokenizer(\"The capital of Spain is\", return_tensors=\"pt\")\n",
    "source = tokenizer(\"The capital of Italy is\", return_tensors=\"pt\")\n",
    "\n",
    "# using intervention skipping for subspace\n",
    "intervened_outputs = pv_gpt2(\n",
    "    base, [source],\n",
    "    {\"sources->base\": 4},\n",
    "    # intervene only only dimensions from 128 to 256\n",
    "    subspaces=1,\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0fdde257",
   "metadata": {},
   "source": [
    "### Intervention Linking\n",
    "Interventions can be linked to share weights and share subspaces. Here is an example of how to link interventions together. If interventions are trainable, then their weights are tied as well.\n",
    "\n",
    "Why this is useful? it is because sometimes, you may want to intervene on different subspaces differently. Say you have a representation in a size of 512, and you hypothesize the first half represents A, and the second half represents B, you can then use the subspace intervention to test it out. With trainable interventions, you can also optimize your interventions on the same representation yet with different subspaces."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "id": "eec19da9",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "loaded model\n",
      "True\n",
      "True\n"
     ]
    }
   ],
   "source": [
    "import torch\n",
    "import pyvene as pv\n",
    "\n",
    "_, tokenizer, gpt2 = pv.create_gpt2()\n",
    "\n",
    "config = pv.IntervenableConfig([\n",
    "    # they are linked to manipulate the same representation\n",
    "    # but in different subspaces\n",
    "    {\"layer\": 0, \"component\": \"block_output\", \n",
    "     \"subspace_partition\": [[0, 128], [128, 256]], \"intervention_link_key\": 0},\n",
    "    {\"layer\": 0, \"component\": \"block_output\",\n",
    "     \"subspace_partition\": [[0, 128], [128, 256]], \"intervention_link_key\": 0}],\n",
    "    intervention_types=pv.VanillaIntervention,\n",
    ")\n",
    "pv_gpt2 = pv.IntervenableModel(config, model=gpt2)\n",
    "\n",
    "base = tokenizer(\"The capital of Spain is\", return_tensors=\"pt\")\n",
    "source = tokenizer(\"The capital of Italy is\", return_tensors=\"pt\")\n",
    "\n",
    "# using intervention skipping for subspace\n",
    "_, pv_out1 = pv_gpt2(\n",
    "    base, [None, source],\n",
    "    # 4 means token position 4\n",
    "    {\"sources->base\": ([None, [[4]]], [None, [[4]]])},\n",
    "    # 1 means the second partition in the config\n",
    "    subspaces=[None, [[1]]],\n",
    ")\n",
    "_, pv_out2 = pv_gpt2(\n",
    "    base,\n",
    "    [source, None],\n",
    "    {\"sources->base\": ([[[4]], None], [[[4]], None])},\n",
    "    subspaces=[[[1]], None],\n",
    ")\n",
    "print(torch.equal(pv_out1.last_hidden_state, pv_out2.last_hidden_state))\n",
    "\n",
    "# subspaces provide a list of index and they can be in any order\n",
    "_, pv_out3 = pv_gpt2(\n",
    "    base,\n",
    "    [source, source],\n",
    "    {\"sources->base\": ([[[4]], [[4]]], [[[4]], [[4]]])},\n",
    "    subspaces=[[[0]], [[1]]],\n",
    ")\n",
    "_, pv_out4 = pv_gpt2(\n",
    "    base,\n",
    "    [source, source],\n",
    "    {\"sources->base\": ([[[4]], [[4]]], [[[4]], [[4]]])},\n",
    "    subspaces=[[[1]], [[0]]],\n",
    ")\n",
    "print(torch.equal(pv_out3.last_hidden_state, pv_out4.last_hidden_state))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "243f146f-1b9a-4574-ba2c-ebf455a96c16",
   "metadata": {},
   "source": [
    "Other than intervention linking, you can also share interventions at the same component across multiple positions via setting a flag in the intervention object. It will have the same effect as creating one intervention per location and linking them all together."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "id": "7c647943-c7e1-4024-8c07-b51062e668ba",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "loaded model\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "tensor([[[0., 0., 0.,  ..., 0., 0., 0.],\n",
       "         [0., 0., 0.,  ..., 0., 0., 0.],\n",
       "         [0., 0., 0.,  ..., 0., 0., 0.],\n",
       "         [0., 0., 0.,  ..., 0., 0., 0.],\n",
       "         [0., 0., 0.,  ..., 0., 0., 0.]]])"
      ]
     },
     "execution_count": 24,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import torch\n",
    "import pyvene as pv\n",
    "\n",
    "_, tokenizer, gpt2 = pv.create_gpt2()\n",
    "\n",
    "config = pv.IntervenableConfig([\n",
    "    # they are linked to manipulate the same representation\n",
    "    # but in different subspaces\n",
    "    {\"layer\": 0, \"component\": \"block_output\", \"intervention_link_key\": 0},\n",
    "    {\"layer\": 0, \"component\": \"block_output\", \"intervention_link_key\": 0}],\n",
    "    intervention_types=pv.VanillaIntervention,\n",
    ")\n",
    "pv_gpt2 = pv.IntervenableModel(config, model=gpt2)\n",
    "\n",
    "base = tokenizer(\"The capital of Spain is\", return_tensors=\"pt\")\n",
    "source = tokenizer(\"The capital of Italy is\", return_tensors=\"pt\")\n",
    "\n",
    "_, pv_out = pv_gpt2(\n",
    "    base,\n",
    "    [source, source],\n",
    "    # swap 3rd and 4th token reprs from the same source to the base\n",
    "    {\"sources->base\": ([[[4]], [[3]]], [[[4]], [[3]]])},\n",
    ")\n",
    "\n",
    "keep_last_dim_config = pv.IntervenableConfig([\n",
    "    # they are linked to manipulate the same representation\n",
    "    # but in different subspaces\n",
    "    {\"layer\": 0, \"component\": \"block_output\", \n",
    "     \"intervention\": pv.VanillaIntervention(keep_last_dim=True)}]\n",
    ")\n",
    "keep_last_dim_pv_gpt2 = pv.IntervenableModel(keep_last_dim_config, model=gpt2)\n",
    "\n",
    "_, keep_last_dim_pv_out = keep_last_dim_pv_gpt2(\n",
    "    base,\n",
    "    [source],\n",
    "    # swap 3rd and 4th token reprs from the same source to the base\n",
    "    {\"sources->base\": ([[[3,4]]], [[[3,4]]])},\n",
    ")\n",
    "keep_last_dim_pv_out.last_hidden_state - pv_out.last_hidden_state"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ef5b7a3e",
   "metadata": {},
   "source": [
    "### Add New Model Type"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "id": "acce6e8f",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "You are using the default legacy behaviour of the <class 'transformers.models.t5.tokenization_t5.T5Tokenizer'>. This is expected, and simply means that the `legacy` (previous) behavior will be used so nothing changes for you. If you want to use the new behaviour, set `legacy=False`. This should only be set if you understand what it means, and thoroughly read the reason why this was added as explained in https://github.com/huggingface/transformers/pull/24565\n",
      "Special tokens have been added in the vocabulary, make sure the associated word embeddings are fine-tuned or trained.\n"
     ]
    }
   ],
   "source": [
    "import torch\n",
    "import pyvene as pv\n",
    "\n",
    "# get a flan-t5 from HuggingFace\n",
    "from transformers import T5ForConditionalGeneration, T5Tokenizer, T5Config\n",
    "config = T5Config.from_pretrained(\"google/flan-t5-small\")\n",
    "tokenizer = T5Tokenizer.from_pretrained(\"google/flan-t5-small\")\n",
    "t5 = T5ForConditionalGeneration.from_pretrained(\n",
    "    \"google/flan-t5-small\", config=config\n",
    ")\n",
    "\n",
    "# config the intervention mapping with pv global vars\n",
    "\"\"\"Only define for the block output here for simplicity\"\"\"\n",
    "pv.type_to_module_mapping[type(t5)] = {\n",
    "    \"mlp_output\": (\"encoder.block[%s].layer[1]\", \n",
    "                   pv.models.constants.CONST_OUTPUT_HOOK),\n",
    "    \"attention_input\": (\"encoder.block[%s].layer[0]\", \n",
    "                        pv.models.constants.CONST_OUTPUT_HOOK),\n",
    "}\n",
    "pv.type_to_dimension_mapping[type(t5)] = {\n",
    "    \"mlp_output\": (\"d_model\",),\n",
    "    \"attention_input\": (\"d_model\",),\n",
    "    \"block_output\": (\"d_model\",),\n",
    "    \"head_attention_value_output\": (\"d_model/num_heads\",),\n",
    "}\n",
    "\n",
    "# wrap as gpt2\n",
    "pv_t5 = pv.IntervenableModel({\n",
    "    \"layer\": 0,\n",
    "    \"component\": \"mlp_output\",\n",
    "    \"source_representation\": torch.zeros(\n",
    "        t5.config.d_model)\n",
    "}, model=t5)\n",
    "\n",
    "# then intervene!\n",
    "base = tokenizer(\"The capital of Spain is\", \n",
    "                 return_tensors=\"pt\")\n",
    "decoder_input_ids = tokenizer(\n",
    "    \"\", return_tensors=\"pt\").input_ids\n",
    "base[\"decoder_input_ids\"] = decoder_input_ids\n",
    "intervened_outputs = pv_t5(\n",
    "    base, \n",
    "    unit_locations={\"base\": 3}\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ba158a92",
   "metadata": {},
   "source": [
    "### Composing Complex Intervention Schema: Path Patching"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "id": "e51cadfe",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "loaded model\n",
      "Directory './tmp/' already exists.\n"
     ]
    }
   ],
   "source": [
    "import pyvene as pv\n",
    "\n",
    "def path_patching_config(\n",
    "    layer, last_layer, \n",
    "    component=\"head_attention_value_output\", unit=\"h.pos\"\n",
    "):\n",
    "    intervening_component = [\n",
    "        {\"layer\": layer, \"component\": component, \"unit\": unit, \"group_key\": 0}]\n",
    "    restoring_components = []\n",
    "    if not component.startswith(\"mlp_\"):\n",
    "        restoring_components += [\n",
    "            {\"layer\": layer, \"component\": \"mlp_output\", \"group_key\": 1}]\n",
    "    for i in range(layer+1, last_layer):\n",
    "        restoring_components += [\n",
    "            {\"layer\": i, \"component\": \"attention_output\", \"group_key\": 1},\n",
    "            {\"layer\": i, \"component\": \"mlp_output\", \"group_key\": 1}\n",
    "        ]\n",
    "    intervenable_config = pv.IntervenableConfig(\n",
    "        intervening_component + restoring_components)\n",
    "    return intervenable_config\n",
    "\n",
    "_, tokenizer, gpt2 = pv.create_gpt2()\n",
    "\n",
    "pv_gpt2 = pv.IntervenableModel(\n",
    "    path_patching_config(4, gpt2.config.n_layer), \n",
    "    model=gpt2\n",
    ")\n",
    "\n",
    "pv_gpt2.save(\n",
    "    save_directory=\"./tmp/\"\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "id": "9074f716",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "WARNING:root:The key is provided in the config. Assuming this is loaded from a pretrained module.\n"
     ]
    }
   ],
   "source": [
    "pv_gpt2 = pv.IntervenableModel.load(\n",
    "    \"./tmp/\",\n",
    "    model=gpt2)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d546e858",
   "metadata": {},
   "source": [
    "### Composing Complex Intervention Schema: Causal Tracing in 15 lines"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "id": "c0b6a70f",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "loaded model\n"
     ]
    }
   ],
   "source": [
    "import pyvene as pv\n",
    "\n",
    "def causal_tracing_config(\n",
    "  l, c=\"mlp_activation\", w=10, tl=48):\n",
    "  s = max(0, l - w // 2)\n",
    "  e = min(tl, l - (-w // 2))\n",
    "  config = pv.IntervenableConfig(\n",
    "    [{\"component\": \"block_input\"}] + \n",
    "    [{\"layer\": l, \"component\": c} \n",
    "      for l in range(s, e)],\n",
    "    [pv.NoiseIntervention] +\n",
    "    [pv.VanillaIntervention]*(e-s))\n",
    "  return config\n",
    "\n",
    "_, tokenizer, gpt2 = pv.create_gpt2()\n",
    "\n",
    "pv_gpt2 = pv.IntervenableModel(\n",
    "    causal_tracing_config(4), \n",
    "    model=gpt2\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5027498b-66b9-428a-9693-94a6b5614bb9",
   "metadata": {},
   "source": [
    "### Inference-time Intervention"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "559bf80a-2a79-46f0-b8a1-8848fd49613b",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "normalizer.cc(51) LOG(INFO) precompiled_charsmap is empty. use identity normalization.\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "09c7100d49c94e1b94f3429440f1aab3",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Loading checkpoint shards:   0%|          | 0/2 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "loaded model\n"
     ]
    }
   ],
   "source": [
    "import time, torch\n",
    "import pyvene as pv\n",
    "config, tokenizer, llama = pv.create_llama(\"meta-llama/Llama-2-7b-chat-hf\")\n",
    "# load cached extracted activations from the ITI repo:\n",
    "# https://github.com/likenneth/honest_llama\n",
    "o_proj_add_activations = torch.load(\n",
    "    \"./tutorials/basic_tutorials/tutorial_data\"\n",
    "    \"/honest_llama2_chat_7B_o_proj_add_activations.bin\"\n",
    ")\n",
    "_ = llama.cuda()\n",
    "\n",
    "pv_llama = pv.IntervenableModel([{\n",
    "        \"component\": f\"model.layers[{i}].self_attn.o_proj.output\",\n",
    "        \"intervention\": pv.AdditionIntervention(\n",
    "            source_representation=o_proj_add_activations[i].to(\"cuda\")\n",
    "        )\n",
    "    } for i in range(config.num_hidden_layers) if torch.count_nonzero(o_proj_add_activations[i])], \n",
    "    model=llama\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "15992a70-288a-43b5-9baf-348e7cb34068",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "vanilla llama-2-chat:\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/sailhome/wuzhengx/.local/lib/python3.8/site-packages/transformers/generation/configuration_utils.py:392: UserWarning: `do_sample` is set to `False`. However, `temperature` is set to `0.6` -- this flag is only used in sample-based generation modes. You should set `do_sample=True` or unset `temperature`.\n",
      "  warnings.warn(\n",
      "/sailhome/wuzhengx/.local/lib/python3.8/site-packages/transformers/generation/configuration_utils.py:397: UserWarning: `do_sample` is set to `False`. However, `top_p` is set to `0.9` -- this flag is only used in sample-based generation modes. You should set `do_sample=True` or unset `top_p`.\n",
      "  warnings.warn(\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "What's a cure for insomnia that always works?\n",
      "\n",
      "Insomnia is a common sleep disorder that can affect anyone, regardless of age or gender. It is characterized by difficulty falling asleep, staying asleep, or getting quality sleep. Insomnia can be caused by a variety of factors, including stress, anxiety, depression, certain\n",
      "Execution time: 2.1844897270202637 seconds\n",
      "\n",
      "honest llama-2-chat:\n",
      "What's a cure for insomnia that always works?\n",
      "There is no single cure for insomnia that works for everyone, and it's important to address the underlying causes of sleep problems. However, some strategies that may help improve sleep quality and duration include:\n",
      "\n",
      "1. Practicing relaxation techniques, such as deep breathing, progressive mus\n",
      "Execution time: 2.6960761547088623 seconds\n"
     ]
    }
   ],
   "source": [
    "q = \"What's a cure for insomnia that always works?\"\n",
    "prompt = tokenizer(q, return_tensors=\"pt\").to(\"cuda\")\n",
    "print(\"vanilla llama-2-chat:\")\n",
    "st = time.time()\n",
    "response = llama.generate(**prompt, max_new_tokens=64, do_sample=False)\n",
    "print(tokenizer.decode(response[0], skip_special_tokens=True))\n",
    "et = time.time()\n",
    "elapsed_time = et - st\n",
    "print('Execution time:', elapsed_time, 'seconds')\n",
    "print()\n",
    "print(\"honest llama-2-chat:\")\n",
    "st = time.time()\n",
    "_, iti_response = pv_llama.generate(prompt, max_new_tokens=64, do_sample=False)\n",
    "print(tokenizer.decode(iti_response[0], skip_special_tokens=True))\n",
    "et = time.time()\n",
    "elapsed_time = et - st\n",
    "print('Execution time:', elapsed_time, 'seconds')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "7ae2ebe8-fa4d-426e-b663-089671323456",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Directory './tmp_llama/' already exists.\n"
     ]
    }
   ],
   "source": [
    "# save to huggingface directly\n",
    "try:\n",
    "    pv_llama.save(\n",
    "        \"./tmp_llama/\",\n",
    "        save_to_hf_hub=True, \n",
    "        hf_repo_name=\"zhengxuanzenwu/intervenable_honest_llama2_chat_7B\"\n",
    "    )\n",
    "except:\n",
    "    print(\"You have to login into huggingface hub before running this.\")\n",
    "    print(\"usage: huggingface-cli login\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1c7b90b0",
   "metadata": {},
   "source": [
    "### IntervenableModel from HuggingFace Directly"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "b5d26852-f86d-4dea-88f1-2db2634e1ba5",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "437f82c9e5ff4a39ae988ccbfd7518df",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Loading checkpoint shards:   0%|          | 0/2 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "c8b98f21c02a4e5da60d67098d99a2c5",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Fetching 17 files:   0%|          | 0/17 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "WARNING:root:The key is provided in the config. Assuming this is loaded from a pretrained module.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "llama-2-chat loaded with interventions:\n",
      "What's a cure for insomnia that always works?\n",
      "There is no single cure for insomnia that works for everyone, and it's important to address the underlying causes of sleep problems. However, some strategies that may help improve sleep quality and duration include:\n",
      "\n",
      "1. Practicing relaxation techniques, such as deep breathing, progressive mus\n"
     ]
    }
   ],
   "source": [
    "# others can download from huggingface and use it directly\n",
    "import torch\n",
    "from transformers import AutoTokenizer, AutoModelForCausalLM\n",
    "import pyvene as pv\n",
    "\n",
    "tokenizer = AutoTokenizer.from_pretrained(\"meta-llama/Llama-2-7b-chat-hf\")\n",
    "model = AutoModelForCausalLM.from_pretrained(\n",
    "    \"meta-llama/Llama-2-7b-chat-hf\",\n",
    "    torch_dtype=torch.bfloat16,\n",
    ").to(\"cuda\")\n",
    "\n",
    "pv_model = pv.IntervenableModel.load(\n",
    "    \"zhengxuanzenwu/intervenable_honest_llama2_chat_7B\", # the activation diff ~0.14MB\n",
    "    model,\n",
    ")\n",
    "\n",
    "print(\"llama-2-chat loaded with interventions:\")\n",
    "q = \"What's a cure for insomnia that always works?\"\n",
    "prompt = tokenizer(q, return_tensors=\"pt\").to(\"cuda\")\n",
    "_, iti_response_shared = pv_model.generate(prompt, max_new_tokens=64, do_sample=False)\n",
    "print(tokenizer.decode(iti_response_shared[0], skip_special_tokens=True))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "956b3b5f",
   "metadata": {},
   "source": [
    "### Path Patching with Trainable Interventions"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "af501960",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "loaded model\n"
     ]
    }
   ],
   "source": [
    "import pyvene as pv\n",
    "\n",
    "def path_patching_with_DAS_config(\n",
    "    layer, last_layer, low_rank_dimension,\n",
    "    component=\"attention_output\", unit=\"pos\"\n",
    "):\n",
    "    intervening_component = [{\n",
    "        \"layer\": layer, \"component\": component, \"group_key\": 0,\n",
    "        \"intervention_type\": pv.LowRankRotatedSpaceIntervention,\n",
    "        \"low_rank_dimension\": low_rank_dimension,\n",
    "    }]\n",
    "    restoring_components = []\n",
    "    if not component.startswith(\"mlp_\"):\n",
    "        restoring_components += [{\n",
    "            \"layer\": layer, \"component\": \"mlp_output\", \"group_key\": 1,\n",
    "            \"intervention_type\": pv.VanillaIntervention,\n",
    "        }]\n",
    "    for i in range(layer+1, last_layer):\n",
    "        restoring_components += [{\n",
    "            \"layer\": i, \"component\": \"attention_output\", \"group_key\": 1, \n",
    "            \"intervention_type\": pv.VanillaIntervention},{\n",
    "            \"layer\": i, \"component\": \"mlp_output\", \"group_key\": 1,\n",
    "            \"intervention_type\": pv.VanillaIntervention\n",
    "        }]\n",
    "    intervenable_config = pv.IntervenableConfig(\n",
    "        intervening_component + restoring_components)\n",
    "    return intervenable_config, len(restoring_components)\n",
    "\n",
    "_, tokenizer, gpt2 = pv.create_gpt2()\n",
    "pv_config, num_restores = path_patching_with_DAS_config(4, 6, 1)\n",
    "pv_gpt2 = pv.IntervenableModel(pv_config, model=gpt2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "be63453e",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor(-0.0694, grad_fn=<SumBackward0>)"
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "base = tokenizer(\"The capital of Spain is\", return_tensors=\"pt\")\n",
    "restore_source = tokenizer(\"The capital of Spain is\", return_tensors=\"pt\")\n",
    "source = tokenizer(\"The capital of Italy is\", return_tensors=\"pt\")\n",
    "\n",
    "# zero-out grads\n",
    "_ = pv_gpt2.model.eval()\n",
    "for k, v in pv_gpt2.interventions.items():\n",
    "    v[0].zero_grad()\n",
    "\n",
    "original_outputs, counterfactual_outputs = pv_gpt2(\n",
    "    base, \n",
    "    sources=[source, restore_source],\n",
    "    unit_locations={\n",
    "        \"sources->base\": 4\n",
    "    }\n",
    ")\n",
    "# put gradients on the trainable intervention only\n",
    "counterfactual_outputs[0].sum().backward()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0907a98c",
   "metadata": {},
   "source": [
    "### Intervene on ResNet with Lambda Functions\n",
    "\n",
    "Huggingface Vision model comes with the support of ResNet. Here, we show how we can use pyvene to intervene on a patch of pixels, like token in transformer, which is like a primitive object in ResNet or ConvNet based NNs.\n",
    "\n",
    "**Caveats:** We go with a pretty much hard-coded way here, but you can customize the hook functions as you want. It does not have to be a lambda function as well."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "bfb48112",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor(0.0005)"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import torch\n",
    "import pyvene as pv\n",
    "from datasets import load_dataset\n",
    "from transformers import AutoFeatureExtractor, AutoModelForImageClassification\n",
    "\n",
    "feature_extractor = AutoFeatureExtractor.from_pretrained(\"microsoft/resnet-18\")\n",
    "resnet = AutoModelForImageClassification.from_pretrained(\"microsoft/resnet-18\")\n",
    "\n",
    "dataset = load_dataset(\"huggingface/cats-image\")\n",
    "base_image = dataset[\"test\"][\"image\"][0]\n",
    "source_image = dataset[\"test\"][\"image\"][0]\n",
    "base_inputs = feature_extractor(base_image, return_tensors=\"pt\")\n",
    "source_inputs = feature_extractor(source_image, return_tensors=\"pt\")\n",
    "source_inputs['pixel_values'] += 0.5*torch.randn(source_inputs['pixel_values'].shape)\n",
    "\n",
    "def create_mask():\n",
    "    _mask = torch.zeros((56, 56))\n",
    "    _mask[56//2:, 56//2:] = 1\n",
    "    return _mask\n",
    "m = create_mask()\n",
    "\n",
    "pv_resnet = pv.IntervenableModel({\n",
    "    \"component\": \"resnet.embedder.pooler.output\", \n",
    "    \"intervention\": lambda b, s: b * (1. - m) + s * m}, \n",
    "    model=resnet\n",
    ")\n",
    "intervened_outputs = pv_resnet(\n",
    "    base_inputs, [source_inputs], return_dict=True, output_original_output=True\n",
    ")\n",
    "(intervened_outputs.intervened_outputs.logits - intervened_outputs.original_outputs.logits).sum()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b4d0aa78",
   "metadata": {},
   "source": [
    "### Intervene on ResNet with Trainable Lambda Functions"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "id": "6d0095f3",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor(0.0068, grad_fn=<SumBackward0>)"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import torch\n",
    "import pyvene as pv\n",
    "from datasets import load_dataset\n",
    "from transformers import AutoFeatureExtractor, AutoModelForImageClassification\n",
    "\n",
    "feature_extractor = AutoFeatureExtractor.from_pretrained(\"microsoft/resnet-18\")\n",
    "resnet = AutoModelForImageClassification.from_pretrained(\"microsoft/resnet-18\")\n",
    "\n",
    "dataset = load_dataset(\"huggingface/cats-image\")\n",
    "base_image = dataset[\"test\"][\"image\"][0]\n",
    "source_image = dataset[\"test\"][\"image\"][0]\n",
    "base_inputs = feature_extractor(base_image, return_tensors=\"pt\")\n",
    "source_inputs = feature_extractor(source_image, return_tensors=\"pt\")\n",
    "source_inputs['pixel_values'] += 0.5*torch.randn(source_inputs['pixel_values'].shape)\n",
    "\n",
    "# trainable DAS directions\n",
    "v = torch.nn.utils.parametrizations.orthogonal(\n",
    "    torch.nn.Linear(56, 10))\n",
    "\n",
    "pv_resnet = pv.IntervenableModel({\n",
    "    \"component\": \"resnet.embedder.pooler.output\", \n",
    "    \"intervention\": lambda b, s: b + ((s @ v.weight.T - b @ v.weight.T) @ v.weight)}, \n",
    "    model=resnet\n",
    ")\n",
    "\n",
    "intervened_outputs = pv_resnet(\n",
    "    base_inputs, [source_inputs], return_dict=True, output_original_output=True\n",
    ")\n",
    "(intervened_outputs.intervened_outputs.logits - intervened_outputs.original_outputs.logits).sum()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "14694aea-934e-47f3-ab27-6843f6b5dc7a",
   "metadata": {},
   "source": [
    "### Run pyvene on [NDIF](https://ndif.us/) backend with `pv.build_intervenable_model(...)`\n",
    "\n",
    "[NDIF](https://ndif.us/) provides APIs for running intervened model inference calls either locally or remotely, enabling Pyvene to run intervened model calls remotely with shared resources. This is especially useful when the intervened model is large (e.g., Llama 400B).\n",
    "\n",
    "Note that setting `remote=True` is still under-construction for remote intervention."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6ba3fc81-8f61-40d2-9e23-8d61c9dc73b6",
   "metadata": {},
   "source": [
    "**Basic activation collection**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "9b289caa-6621-4cc1-883d-c9b91a21617d",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/u/nlp/anaconda/main/anaconda3/envs/wuzhengx-310/lib/python3.10/site-packages/transformers/utils/hub.py:124: FutureWarning: Using `TRANSFORMERS_CACHE` is deprecated and will be removed in v5 of Transformers. Use `HF_HOME` instead.\n",
      "  warnings.warn(\n",
      "WARNING:root:We currently have very limited intervention support for ndif backend.\n",
      "You're using a GPT2TokenizerFast tokenizer. Please note that with a fast tokenizer, using the `__call__` method is faster than using a method to encode the text followed by a call to the `pad` method to get a padded encoding.\n"
     ]
    }
   ],
   "source": [
    "import torch\n",
    "import pyvene as pv\n",
    "from transformers import AutoTokenizer\n",
    "from nnsight import LanguageModel\n",
    "\n",
    "# load any huggingface model as a ndif native model object\n",
    "gpt2_ndif = LanguageModel('openai-community/gpt2', device_map='cpu')\n",
    "tokenizer = AutoTokenizer.from_pretrained('openai-community/gpt2')\n",
    "\n",
    "# pyvene provides pv.build_intervenable_model as the generic model builder\n",
    "pv_gpt2_ndif = pv.build_intervenable_model({\n",
    "    # based on the module printed above, you can access via string, input means the input to the module\n",
    "    \"component\": \"transformer.h[10].attn.attn_dropout.input\",\n",
    "    # you can also initialize the intervention gpt2_ndif\n",
    "    \"intervention\": pv.CollectIntervention()}, model=gpt2_ndif, remote=False)\n",
    "\n",
    "base = \"When John and Mary went to the shops, Mary gave the bag to\"\n",
    "ndif_collected_attn_w = pv_gpt2_ndif(\n",
    "    base = tokenizer(base, return_tensors=\"pt\"\n",
    "    ), unit_locations={\"base\": [h for h in range(12)]}\n",
    ")[0][-1][0]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "a97c01ee-0904-4e01-8dee-2608d0635964",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "loaded model\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "True"
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# gpt2 helper loading model from HuggingFace\n",
    "_, tokenizer, gpt2 = pv.create_gpt2()\n",
    "\n",
    "pv_gpt2 = pv.IntervenableModel({\n",
    "    # based on the module printed above, you can access via string, input means the input to the module\n",
    "    \"component\": \"h[10].attn.attn_dropout.input\",\n",
    "    # you can also initialize the intervention outside\n",
    "    \"intervention\": pv.CollectIntervention()}, model=gpt2)\n",
    "\n",
    "base = \"When John and Mary went to the shops, Mary gave the bag to\"\n",
    "collected_attn_w = pv_gpt2(\n",
    "    base = tokenizer(base, return_tensors=\"pt\"\n",
    "    ), unit_locations={\"base\": [h for h in range(12)]}\n",
    ")[0][-1][0]\n",
    "torch.allclose(ndif_collected_attn_w, collected_attn_w)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e463fc3b-684b-4461-b94e-d69ca310e27e",
   "metadata": {},
   "source": [
    "**Interchange intervention (activation swap between two examples)**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "f4f02279-5601-4db9-9823-2e4d3bd8b56c",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/u/nlp/anaconda/main/anaconda3/envs/wuzhengx-310/lib/python3.10/site-packages/transformers/utils/hub.py:124: FutureWarning: Using `TRANSFORMERS_CACHE` is deprecated and will be removed in v5 of Transformers. Use `HF_HOME` instead.\n",
      "  warnings.warn(\n",
      "WARNING:root:We currently have very limited intervention support for ndif backend.\n",
      "You're using a GPT2TokenizerFast tokenizer. Please note that with a fast tokenizer, using the `__call__` method is faster than using a method to encode the text followed by a call to the `pad` method to get a padded encoding.\n"
     ]
    }
   ],
   "source": [
    "import pyvene as pv\n",
    "from transformers import AutoTokenizer\n",
    "from nnsight import LanguageModel\n",
    "\n",
    "# load any huggingface model as a ndif native model object\n",
    "gpt2_ndif = LanguageModel('openai-community/gpt2', device_map='cpu')\n",
    "tokenizer = AutoTokenizer.from_pretrained('openai-community/gpt2')\n",
    "\n",
    "# create with dict-based config\n",
    "pv_config = pv.IntervenableConfig({\n",
    "  \"component\": \"transformer.h[0].attn.output\",\n",
    "  \"intervention\": pv.VanillaIntervention()}\n",
    ")\n",
    "#initialize model\n",
    "pv_gpt2_ndif = pv.build_intervenable_model(\n",
    "  pv_config, model=gpt2_ndif)\n",
    "# run an interchange intervention \n",
    "intervened_outputs = pv_gpt2_ndif(\n",
    "  # the base input\n",
    "  base=tokenizer(\n",
    "    \"The capital of Spain is\", \n",
    "    return_tensors = \"pt\"), \n",
    "  # the source input\n",
    "  sources=tokenizer(\n",
    "    \"The capital of Italy is\", \n",
    "    return_tensors = \"pt\"), \n",
    "  # the location to intervene at (3rd token)\n",
    "  unit_locations={\"sources->base\": 3},\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "bc6eb49d",
   "metadata": {},
   "source": [
    "### The End\n",
    "Now you are graduating from pyvene entry level course! Feel free to take a look at our tutorials for more challenging interventions."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.9"
  },
  "toc-autonumbering": true,
  "toc-showcode": false,
  "toc-showmarkdowntxt": false,
  "toc-showtags": true
 },
 "nbformat": 4,
 "nbformat_minor": 5
}