notebooks/7_Distributions_in_text.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Distributions in Text"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    ">### Today\n",
    ">\n",
    ">- [Corpora in NLTK](#Corpora-in-NLTK)\n",
    ">\n",
    ">\n",
    ">- [Word Frequencies](#Word-Frequencies)\n",
    ">\n",
    ">\n",
    ">- [The-Zipf's Law](#The-Zipf's-Law)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 55,
   "metadata": {},
   "outputs": [],
   "source": [
    "%matplotlib inline\n",
    "import matplotlib as plt"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Corpora in NLTK\n",
    "\n",
    "The `nltk.corpus` package defines a collection of `corpus reader` classes, which can be used to access the contents of a diverse set of corpora:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import nltk.corpus"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Some of the `corpus reader` classes:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<CategorizedTaggedCorpusReader in '.../corpora/brown' (not loaded yet)>\n",
      "<BracketParseCorpusReader in '.../corpora/treebank/combined' (not loaded yet)>\n",
      "<WordListCorpusReader in '.../corpora/names' (not loaded yet)>\n",
      "<PlaintextCorpusReader in '.../corpora/gutenberg' (not loaded yet)>\n"
     ]
    }
   ],
   "source": [
    "# The Brown corpus\n",
    "print(nltk.corpus.brown)\n",
    "\n",
    "# The Penn Treebank Corpus\n",
    "print(nltk.corpus.treebank)\n",
    "\n",
    "# The Name Genders Corpus\n",
    "print(nltk.corpus.names)\n",
    "\n",
    "# The Gutenberg Corpus\n",
    "print(nltk.corpus.gutenberg)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Corpus Readers\n",
    "\n",
    "Each corpus reader provides a variety of methods implementing a wide range of functionalities, depending on the format of the corpus."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- Want to know more about a corpus? Use the method `readme()` to access the corpus `ReadMe`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "BROWN CORPUS\n",
      "\n",
      "A Standard Corpus of Present-Day Edited American\n",
      "English, for use with Digital Computers.\n",
      "\n",
      "by W. N. Francis and H. Kucera (1964)\n",
      "Department of Linguistics, Brown University\n",
      "Providence, Rhode Island, USA\n",
      "\n",
      "Revised 1971, Revised and Amplified 1979\n",
      "\n",
      "http://www.hit.uib.no/icame/brown/bcm.html\n",
      "\n",
      "Distributed with the permission of the copyright holder,\n",
      "redistribution permitted.\n",
      "\n"
     ]
    }
   ],
   "source": [
    "# Do you remember the Brown corpus?\n",
    "print(nltk.corpus.brown.readme())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- Most plaintext and tagged corpora support methods to read the corpus as raw text, a list of words, a list of sentences, or a list of paragraphs."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[Emma by Jane Austen 1816]\n",
      "\n",
      "VOLUME I\n",
      "\n",
      "CHAPTER I\n",
      "\n",
      "\n",
      "Emma Woodhouse, handsome, clever, and rich, with a comfortable home\n",
      "and happy disposition, seemed to unite some of the best blessings\n",
      "of existence; and had lived nearly twenty-one years in the world\n",
      "with very little to distress or vex her.\n"
     ]
    }
   ],
   "source": [
    "# `nltk.corpus.gutenberg` is a subset of the full Project Gutenberg corpus, starting with Jane Austen's 'Emma'\n",
    "\n",
    "# Accessing corpus as raw text\n",
    "print(nltk.corpus.gutenberg.raw()[:289])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['[', 'Emma', 'by', 'Jane', 'Austen', '1816', ']', 'VOLUME', 'I', 'CHAPTER', 'I', 'Emma', 'Woodhouse', ',', 'handsome', ',', 'clever', ',', 'and', 'rich', ',', 'with', 'a', 'comfortable', 'home', 'and', 'happy', 'disposition', ',', 'seemed', 'to', 'unite', 'some', 'of', 'the', 'best', 'blessings', 'of', 'existence', ';', 'and', 'had', 'lived', 'nearly', 'twenty', '-', 'one', 'years', 'in', 'the', 'world', 'with', 'very', 'little', 'to', 'distress', 'or', 'vex', 'her', '.']\n"
     ]
    }
   ],
   "source": [
    "# list of words\n",
    "print(nltk.corpus.gutenberg.words()[:60])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[['[', 'Emma', 'by', 'Jane', 'Austen', '1816', ']'], ['VOLUME', 'I'], ['CHAPTER', 'I'], ['Emma', 'Woodhouse', ',', 'handsome', ',', 'clever', ',', 'and', 'rich', ',', 'with', 'a', 'comfortable', 'home', 'and', 'happy', 'disposition', ',', 'seemed', 'to', 'unite', 'some', 'of', 'the', 'best', 'blessings', 'of', 'existence', ';', 'and', 'had', 'lived', 'nearly', 'twenty', '-', 'one', 'years', 'in', 'the', 'world', 'with', 'very', 'little', 'to', 'distress', 'or', 'vex', 'her', '.']]\n"
     ]
    }
   ],
   "source": [
    "# list of sentences\n",
    "print(nltk.corpus.gutenberg.sents()[:4])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[[['[', 'Emma', 'by', 'Jane', 'Austen', '1816', ']']], [['VOLUME', 'I']], [['CHAPTER', 'I']], [['Emma', 'Woodhouse', ',', 'handsome', ',', 'clever', ',', 'and', 'rich', ',', 'with', 'a', 'comfortable', 'home', 'and', 'happy', 'disposition', ',', 'seemed', 'to', 'unite', 'some', 'of', 'the', 'best', 'blessings', 'of', 'existence', ';', 'and', 'had', 'lived', 'nearly', 'twenty', '-', 'one', 'years', 'in', 'the', 'world', 'with', 'very', 'little', 'to', 'distress', 'or', 'vex', 'her', '.']]]\n"
     ]
    }
   ],
   "source": [
    "# list of paragraphs\n",
    "print(nltk.corpus.gutenberg.paras()[:4])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- Most corpora are composed by set of files, whose id can be retrieved by using the `fileids()` method"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['austen-emma.txt', 'austen-persuasion.txt', 'austen-sense.txt', 'bible-kjv.txt', 'blake-poems.txt', 'bryant-stories.txt', 'burgess-busterbrown.txt', 'carroll-alice.txt', 'chesterton-ball.txt', 'chesterton-brown.txt', 'chesterton-thursday.txt', 'edgeworth-parents.txt', 'melville-moby_dick.txt', 'milton-paradise.txt', 'shakespeare-caesar.txt', 'shakespeare-hamlet.txt', 'shakespeare-macbeth.txt', 'whitman-leaves.txt']\n"
     ]
    }
   ],
   "source": [
    "print(nltk.corpus.gutenberg.fileids())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- The above methods methods accept a single file name (or a list of file names) to restrict their scope:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[['[', 'Alice', \"'\", 's', 'Adventures', 'in', 'Wonderland', 'by', 'Lewis', 'Carroll', '1865', ']'], ['CHAPTER', 'I', '.'], ['Down', 'the', 'Rabbit', '-', 'Hole'], ['Alice', 'was', 'beginning', 'to', 'get', 'very', 'tired', 'of', 'sitting', 'by', 'her', 'sister', 'on', 'the', 'bank', ',', 'and', 'of', 'having', 'nothing', 'to', 'do', ':', 'once', 'or', 'twice', 'she', 'had', 'peeped', 'into', 'the', 'book', 'her', 'sister', 'was', 'reading', ',', 'but', 'it', 'had', 'no', 'pictures', 'or', 'conversations', 'in', 'it', ',', \"'\", 'and', 'what', 'is', 'the', 'use', 'of', 'a', 'book', \",'\", 'thought', 'Alice', \"'\", 'without', 'pictures', 'or', 'conversation', \"?'\"], ['So', 'she', 'was', 'considering', 'in', 'her', 'own', 'mind', '(', 'as', 'well', 'as', 'she', 'could', ',', 'for', 'the', 'hot', 'day', 'made', 'her', 'feel', 'very', 'sleepy', 'and', 'stupid', '),', 'whether', 'the', 'pleasure', 'of', 'making', 'a', 'daisy', '-', 'chain', 'would', 'be', 'worth', 'the', 'trouble', 'of', 'getting', 'up', 'and', 'picking', 'the', 'daisies', ',', 'when', 'suddenly', 'a', 'White', 'Rabbit', 'with', 'pink', 'eyes', 'ran', 'close', 'by', 'her', '.']]\n"
     ]
    }
   ],
   "source": [
    "# the first 5 sentences of Alice in Wonderland\n",
    "print(nltk.corpus.gutenberg.sents(\"carroll-alice.txt\")[:5])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- Categorized corpora accept the `categories()` methods and the categorize tags can be used to restrict the scope of text accessing methods:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['adventure', 'belles_lettres', 'editorial', 'fiction', 'government', 'hobbies', 'humor', 'learned', 'lore', 'mystery', 'news', 'religion', 'reviews', 'romance', 'science_fiction']\n"
     ]
    }
   ],
   "source": [
    "print(nltk.corpus.brown.categories())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['cn01', 'cn02', 'cn03', 'cn04', 'cn05', 'cn06', 'cn07', 'cn08', 'cn09', 'cn10', 'cn11', 'cn12', 'cn13', 'cn14', 'cn15', 'cn16', 'cn17', 'cn18', 'cn19', 'cn20', 'cn21', 'cn22', 'cn23', 'cn24', 'cn25', 'cn26', 'cn27', 'cn28', 'cn29']\n"
     ]
    }
   ],
   "source": [
    "# fileids of category 'adventure'\n",
    "print(nltk.corpus.brown.fileids('adventure'))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "61604\n",
      "100554\n"
     ]
    }
   ],
   "source": [
    "# the editorial section of the Brown corpus is composed by fewer words than the news one\n",
    "print(len(nltk.corpus.brown.words(categories = \"editorial\")))\n",
    "print(len(nltk.corpus.brown.words(categories = \"news\")))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- Some corpora may have overlapping categories, so that the possibility of interplaying between category names and filenames can be useful:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['acq', 'alum', 'barley', 'bop', 'carcass', 'castor-oil', 'cocoa', 'coconut', 'coconut-oil', 'coffee', 'copper', 'copra-cake', 'corn', 'cotton', 'cotton-oil', 'cpi', 'cpu', 'crude', 'dfl', 'dlr', 'dmk', 'earn', 'fuel', 'gas', 'gnp', 'gold', 'grain', 'groundnut', 'groundnut-oil', 'heat', 'hog', 'housing', 'income', 'instal-debt', 'interest', 'ipi', 'iron-steel', 'jet', 'jobs', 'l-cattle', 'lead', 'lei', 'lin-oil', 'livestock', 'lumber', 'meal-feed', 'money-fx', 'money-supply', 'naphtha', 'nat-gas', 'nickel', 'nkr', 'nzdlr', 'oat', 'oilseed', 'orange', 'palladium', 'palm-oil', 'palmkernel', 'pet-chem', 'platinum', 'potato', 'propane', 'rand', 'rape-oil', 'rapeseed', 'reserves', 'retail', 'rice', 'rubber', 'rye', 'ship', 'silver', 'sorghum', 'soy-meal', 'soy-oil', 'soybean', 'strategic-metal', 'sugar', 'sun-meal', 'sun-oil', 'sunseed', 'tea', 'tin', 'trade', 'veg-oil', 'wheat', 'wpi', 'yen', 'zinc']\n"
     ]
    }
   ],
   "source": [
    "# categories of the Reuters corpus\n",
    "print(nltk.corpus.reuters.categories())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['test/14842', 'test/15411', 'test/15471', 'test/15481', 'test/15803', 'test/15811', 'test/16009', 'test/16149', 'test/16162', 'test/16212', 'test/16248', 'test/16286', 'test/16589', 'test/16604', 'test/17457', 'test/17622', 'test/17632', 'test/17654', 'test/17714', 'test/18013', 'test/18176', 'test/19152', 'test/19764', 'test/19802', 'test/19808', 'test/20096', 'test/20699', 'test/21293', 'test/21314', 'test/21576', 'training/10214', 'training/10216', 'training/10485', 'training/10546', 'training/10662', 'training/1072', 'training/10811', 'training/1082', 'training/10868', 'training/10969', 'training/10994', 'training/11627', 'training/11638', 'training/11830', 'training/11839', 'training/11852', 'training/12442', 'training/12796', 'training/12963', 'training/12987', 'training/13116', 'training/13690', 'training/13757', 'training/13791', 'training/14572', 'training/1607', 'training/1897', 'training/1918', 'training/1942', 'training/2073', 'training/2411', 'training/2559', 'training/2762', 'training/2782', 'training/2785', 'training/2880', 'training/2983', 'training/309', 'training/314', 'training/3322', 'training/3327', 'training/3574', 'training/3625', 'training/3646', 'training/3704', 'training/3793', 'training/3997', 'training/4058', 'training/4066', 'training/4407', 'training/448', 'training/4648', 'training/4708', 'training/4816', 'training/488', 'training/5209', 'training/5255', 'training/5323', 'training/5481', 'training/5526', 'training/5541', 'training/5558', 'training/5564', 'training/5567', 'training/5932', 'training/5958', 'training/6218', 'training/6658', 'training/6872', 'training/6957', 'training/697', 'training/6985', 'training/7023', 'training/7088', 'training/7166', 'training/7629', 'training/7645', 'training/7775', 'training/7823', 'training/7985', 'training/8068', 'training/8331', 'training/8757', 'training/8854', 'training/8857', 'training/8877', 'training/8948', 'training/9104', 'training/9190', 'training/9453', 'training/9489', 'training/9799', 'training/9853', 'training/9866']\n"
     ]
    }
   ],
   "source": [
    "# which files of the Reuters corpus belong to category \"gold\"?\n",
    "print(nltk.corpus.reuters.fileids(\"gold\"))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['cotton', 'gold', 'grain', 'livestock', 'money-fx', 'silver']\n"
     ]
    }
   ],
   "source": [
    "# what are the topics of the file called \"test/16009\"?\n",
    "print(nltk.corpus.reuters.categories(\"test/16009\"))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'U.S. DOLLAR LOSSES PROPEL BROAD COMMODITY GAINS\\n  Commodities from gold to grains to\\n  cotton posted solid gains in a flurry of buying today as losses\\n  in the U.S. dollar and rising interest rates kindled fears of\\n  inflation and economic instability.\\n      Gains were most pronounced on the Commodity Exchange in New\\n  York, where gold jumped 12.40 dlrs and closed at 436.50 dlrs a\\n  troy ounce, and silver 22.5 cents to 6.86 dlrs a troy ounce.\\n      A key factor behind the advance was anticipation that\\n  inflation will be the only way for the major industrial nations\\n  to halt the slide in the value of the U.S. dollar, said Steve\\n  Chronowitz, director of commodity research with Smith Barney,\\n  Harris Upham and Co., in New York.\\n      The dollar tumbled one day after top finance officials from\\n  the seven largest industrial nations reaffirmed their\\n  commitment to support its value, and despite reports of\\n  intervention by the U.S. Federal Reserve Bank, traders said.\\n      Traders said it appears that the industrial nations, known\\n  as the Group of Seven, lack the ability to change the long-term\\n  direction of the currency markets.\\n      \"Maybe they have some ideas or plans,\" said Chronowitz. \"If\\n  they do, it\\'s not evident.\"\\n      \"It looks like there\\'s no cure but to let the free market\\n  take values to where they should be.\\n      \"One way or another, we will force our major trading\\n  partners to stimulate their economies,\" as a measure to correct\\n  the mounting U.S. trade deficit, Chronowitz said.\\n      \"I think the markets believe, and have believed for a long\\n  time, that the only recourse is to reflate at some point. It\\'s\\n  going to be a long and tedious process, but that\\'s what\\'s\\n  happening,\" he said.\\n      The falling value of the dollar makes U.S. commodities\\n  cheaper for foreign buyers, stimulating demand.\\n      At the same time, traders who are holding stocks and bonds\\n  saw the value of their investments falling and many are turning\\n  to commodities such as precious metals as a hedge, said Marty\\n  McNeill, a metals analyst in New York with the trading house of\\n  Dominick and Dominick.\\n      The reaction in the metal markets reverberated throughout\\n  the commodities markets, as grains, livestock, and cotton\\n  posted broad gains.\\n      Traders at the Chicago Board of Trade said attention in the\\n  grain markets has shifted from concern about burdensome\\n  supplies to the outlook that a lower dollar will stimulate\\n  export demand.\\n      After the close of trading, the Agriculture Department\\n  raised its estimate for grain imports by the Soviet Union by\\n  two mln tonnes from the month-earlier report.\\n      Live hogs and frozen pork bellies posted sharp gains on the\\n  Chicago Mercantile Exchange, while live cattle were moderately\\n  higher.\\n      Analysts said several factors boosted hog prices. They said\\n  hogs haven\\'t been making the weight gains that are normal at\\n  this time of year, and farmers have been too busy with field\\n  work to market animals.\\n  \\n\\n'"
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# the raw file\n",
    "nltk.corpus.reuters.raw(\"test/16009\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Loading your Corpus\n",
    "\n",
    "If you want to access a corpus that is not part of the NLTK distribution or to access an existing corpus by using a customized reader (e.g. a customize tokenizer), you may want to create a new corpus reader."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Different corpus readers have different constructor signatures. For instance, the folder `./data/gutenberg-extension` contains a selection of 5 (additional) books from the Gutenberg collection, plus a readme file. We can treat these text files as a nltk corpus by using the `PlaintextCorpusReader()` method to import them."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Arguments of the `PlaintextCorpusReader()` method are the root of the corpus folder plus a list of files (e.g. `[\"austen-pride.txt\", \"doyle-sherlock.txt\"]`) or a pattern matching fileids."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [],
   "source": [
    "# all (txt) files\n",
    "gutenberg_extension = nltk.corpus.PlaintextCorpusReader(\"./data/gutenberg-extension\", '.*.txt')\n",
    "\n",
    "## all files containing 'austen'\n",
    "# gutenberg_extension = nltk.corpus.PlaintextCorpusReader(\"./data/gutenberg-extension\", 'austen.*')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['austen-pride.txt',\n",
       " 'doyle-sherlock.txt',\n",
       " 'joyce-dubliners.txt',\n",
       " 'shelley-frankestein.txt',\n",
       " 'stoker-dracula.txt']"
      ]
     },
     "execution_count": 21,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# note that the README file is not part of the corpus...\n",
    "gutenberg_extension.fileids()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Project Gutenberg Extension\n",
      "http://gutenberg.net/\n",
      "\n",
      "This corpus contains etexts from from Project Gutenberg,\n",
      "by the following authors:\n",
      "\n",
      "* Jane Austen\n",
      "* Arthur Conan Doyle\n",
      "* James Joyce\n",
      "* Mary Wollstonecraft (Godwin) Shelley\n",
      "* Bram Stoker\n",
      "\n",
      "This is just of toy-example for educational purposes. \n",
      "The same legal conditions as the gutenberg corpus in the NLTK\n",
      "distribution apply.\n"
     ]
    }
   ],
   "source": [
    "# ... yet it has been handled in a special way\n",
    "print(gutenberg_extension.readme())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "> For the full list of corpus reader methods see the [Corpus Readers HowTo](http://www.nltk.org/howto/corpus.html) or the official documentation (i.e. `help(nltk.corpus.reader)`)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Text Objects\n",
    "\n",
    "The NLTK `Text` class is a wrapper around a sequence of simple (string) tokens that offers a series of usefull methods supporting the **initial exploration** of text.\n",
    "\n",
    "A `Text` is typically initialized from a given document or corpus:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [],
   "source": [
    "us_inaugural_addresses = nltk.text.Text(nltk.corpus.inaugural.words())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The `concordance(self, word, width=79, lines=25)` method allows you to visually inspect the occurrences of a given \"`word`\", returned in the so-called **KWIC** (Keyword in Context) format. Optional arguments: \n",
    "\n",
    "- \"`lines`\": number of returned occurrences\n",
    "\n",
    "\n",
    "- \"`width`\": width of the context of presentation (i.e. \"line width\")\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Displaying 10 of 56 matches:\n",
      "years under the administration of a citizen who , by a long course of great act\n",
      "ll its branches , under which every citizen may by his merit obtain the highest\n",
      "; a Government which protects every citizen in the full enjoyment of his rights\n",
      "untry , it is the interest of every citizen to maintain it . What are the dange\n",
      " The sentiment in the mind of every citizen is national strength . It ought the\n",
      "ill to perform , the heart of every citizen must expand with joy when he reflec\n",
      "ese promises , made by that eminent citizen at the time of his first induction \n",
      "s the obligation imposed upon every citizen , in his own sphere of action , whe\n",
      "ssessed by each individual American citizen which in his compact with the other\n",
      " . The boasted privilege of a Roman citizen was to him a shield only against a \n"
     ]
    }
   ],
   "source": [
    "us_inaugural_addresses.concordance(\"citizen\", width = 80, lines = 10)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Displaying 10 of 247 matches:\n",
      "Fellow - Citizens of the Senate and of the House of R\n",
      " wisest and most experienced of her citizens a distrustful scrutiny into his qua\n",
      "roof of the confidence of my fellow citizens , and have thence too little consul\n",
      "han my own , nor those of my fellow citizens at large less than either . No peop\n",
      "which can win the affections of its citizens and command the respect of the worl\n",
      "his Government must depend . Fellow citizens , I am again called upon by the voi\n",
      "ffrage , in common with my fellow - citizens , in the adoption or rejection of a\n",
      " the Legislature , are exercised by citizens selected at regular periods by thei\n",
      "rited the gratitude of his fellow - citizens , commanded the highest praises of \n",
      "to be more friendly to us , and our citizens to be more friendly to them ; if an\n"
     ]
    }
   ],
   "source": [
    "# Note: we're matching tokens, so \"citizens\" != \"citizen\"\n",
    "us_inaugural_addresses.concordance(\"citizens\", width = 80, lines = 10)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The `.similar()` method allows to look for tokens that appear in similar contexts:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "people states country nation executive\n"
     ]
    }
   ],
   "source": [
    "# show 5 other words appearing in similar contexts as \"citizen\"\n",
    "us_inaugural_addresses.similar(\"citizen\", 5)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The method `.common_contexts()` can give us an idea of a frequently encountered context of a given word:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "fellow_of her_a fellow_and fellow_at its_and fellow_i fellow_in\n",
      "by_selected fellow_commanded our_to\n"
     ]
    }
   ],
   "source": [
    "us_inaugural_addresses.common_contexts([\"citizens\"], 10)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Or we can use it to find contexts shared by **two** words:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "the_of a_may the_is the_by\n"
     ]
    }
   ],
   "source": [
    "us_inaugural_addresses.common_contexts([\"citizen\", \"president\"])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can easily create a dispersion plot to have a rough idea of where in the corpus our word are used:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAZsAAAEWCAYAAACwtjr+AAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjAsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+17YcXAAAgAElEQVR4nO3deZxkVX338c8XGmgRnGGLgjLTQFRExJFpEAg4jcEFggSfoEA0MgaCGLfREDM+E53GR5NBiIpLBOKjLQmgiJBMUAOKaVEUpIdtACFsg44giwgy7Msvf9xzqTt3qqqru+v04nzfr1e96tY5557zu6du1W/uMtWKCMzMzHLaYKoDMDOz339ONmZmlp2TjZmZZedkY2Zm2TnZmJlZdk42ZmaWnZONrbckfVfSURPsY6GkH0+wj+slDUykj27qxryMY8xBSf82mWPa5HKysRlB0ipJB3Szz4g4MCK+1s0+qyT1SQpJa9LjbkkXSHpdLY6XR8RwrjjGKte8SBqS9ESai/slfU/SzuPop+v7guXnZGOW3+yI2Ax4JfA94HxJC6cqGEk9UzU28Kk0Fy8C7gGGpjAWm0RONjbjSTpY0tWSHpD0E0m7pfKd0r+gd0+vt5N0X3nKStKwpGMq/fyVpJ9LekjSDZX1Fku6tVL+5vHEGRG/johTgEHgREkbpP6f/Ze6pD0ljUj6XToS+nQqL4+SjpV0p6S7JP1NJfYNKnH+RtI5krasrXu0pF8AP5DUK+nfUtsHJF0h6fn1eUn9/r2kOyTdI+kMSbNq/R4l6Rdpbpd0OBePAGcBuzarl3RIOr34QIrnZan8X4E5wH+mI6QPj/V9sKnhZGMzWkoIXwHeBWwFnAYsl7RJRNwK/B1wpqRNga8CQ81OWUl6C0USeAfwPOAQ4Dep+lZgP2AWcALwb5K2nUDY5wF/ALy0Sd0pwCkR8TxgJ+CcWv3+wIuB1wOLK6eT3g8cCiwAtgN+C3yxtu4C4GXAG4Cj0vZsTzFvxwGPNolnYXrsD+wIbAZ8odZm37Qtfwx8rEwM7UjaDHgbcFWTupcAZwOLgG2A71Akl40j4i+AXwBviojNIuJTo41l04OTjc10fwWcFhGXR8TT6VrD48BeABHxL8DNwOXAtkCrf3kfQ3GK54oo3BIRd6Q+vhkRd0bEMxHxjdTfnhOI+c70vGWTuieBP5S0dUSsiYjLavUnRMTDEbGSInkemcrfBSyJiNUR8ThF4jysdspsMK37aBpnK+AP07ytiIjfNYnnbcCnI+K2iFgDfAQ4otbvCRHxaERcA1xDcbqwleMlPQDcQpG4FjZpczjw7Yj4XkQ8CZwMPAfYp02/Ns052dhMNxf4m3S65YH0RbY9xb/uS/9Ccbrm8+mLuJntKY5g1iHpHZXTdA+kvraeQMwvTM/3N6k7GngJcGM6tXVwrf6XleU7aGznXIprQWWMPweeBp7fYt1/BS4Evp5Oy31K0kZN4tkujVMds6fW768ry49QJJFWTo6I2RHxgog4JB19th0zIp5Jsb+wSVubIZxsbKb7JfDJ9AVWPjaNiLPh2dM1nwX+PzBYXsdo0c9O9UJJcymS1XuBrSJiNnAdoAnE/GaKi+M31Ssi4uaIOJLiNNuJwLmSnltpsn1leQ6No6RfAgfW5qE3In5V7b4yzpMRcUJE7EJxxHAwxSnEujspEll1zKeAuzvc1vFYa0xJotjuclv8U/UzkJONzSQbpQvb5aOHIhEcJ+nVKjxX0p9I2jytcwqwIiKOAb4NnNqi7y9TnOKZn/r5w5Ronkvx5XYvgKR30uKi9mgkPV/Se4GlwEfSv9jrbd4uaZtU90AqfrrS5KOSNpX0cuCdwDdS+anAJ1PMSNpG0p+2iWV/Sa+QtCHwO4rTak83aXo28EFJO6TE/Q/ANyLiqbFs+xidA/yJpD9OR1t/Q3Fq9Cep/m6K60c2gzjZ2EzyHYqL2OVjMCJGKK7bfIHiovgtpOsA6cv2jRQXvwE+BOwu6W31jiPim8AnKe6Qegj4d2DLiLgB+CfgpxRfcq8ALh1j3A9IehhYCRwEvCUivtKi7RuB6yWtoUiUR0TEY5X6H6ZtvJjilNRFqfwUYDlwkaSHgMuAV7eJ6QXAuRSJ5uep32b/qfIrFKfcLgFuBx4D3td+cycmIm4C3g58HrgPeBPFDQFPpCb/CPx9OmV4fM5YrHvkP55mNv1J6qP4st8o81GFWRY+sjEzs+ycbMzMLDufRjMzs+x8ZGNmZtlN5Q/yTVtbb7119PX1TXUYZmYzyooVK+6LiG2a1TnZNNHX18fIyMhUh2FmNqNIuqNVnU+jmZlZdk42ZmaWnZONmZll52RjZmbZOdmYmVl2TjZmZpadk42ZmWXnZGNmZtk52ZiZWXZONmZmlp2TjZmZZedkY2Zm2TnZmJlZdk42ZmaWnZONmZll52RjZmbZOdmYmVl2TjZmZpadk42ZmWXnZGNmZtk52ZiZWXZONmZmlp2TjZmZZedkY2Zm2TnZmJlZdk42ZmaWnZONmZll52RjZmbZOdmYmVl2TjZmZpadk42ZmWU3ZclG4jiJd6TlhRLbVeq+LLHLVMVmZmbdNWXJJoJTIzgjvVwIjWQTwTER3DAlgXVBXx8MDhYPaCwPDsLAQGO5bNfXV5SX65bL5brlOtV1m72ujlsvL/sq29RV462O325bmq3brO/SwAD09sLs2cWjbDt7dlFXxlntc2CgsZ1lH/VYquvUn+uxNeurWR919TZlXK3Gqbavzn+5br3v+jr18lbzXn0/O5mPuuqc19s0iwfWjb/V+s1iamaDDZqP3W690fbhqvrnabT+ens7G6uVZuPV97nqvtOs72bvZblOue9VPxu9vY325Xva6n2q7y/ld06797UbFBF5RygHKo5ijgcCuBa4FVgDrAKGgF8BjwJ7A99NbbcDPp66eA6wcQQ7SMwHPg1sBtwHLIzgLolh4HJgf2A2cHQEP5J4OfBVYGOKBPtnEdzcKtb+/v4YGRmZyLY+K2Lt1+1U25ZvS6frjqXvav+l6rhS8/Fb9dFq3bpm2zJan63atpqragyjxd5q3Gbxj2VO2sVVH69Zf83Kq0aLo918NHvf62O2mot2/bTrr75drdZpNsedjtNunXbb1EmbTuZttL7a7ZOttqO6Lc32+2ZGe7/rMTVbfyIkrYiI/mZ1k3Jkk77slwCvjeCVwAfKugjOBUaAt0UwL4JHK3XLU9k84BrgZImNgM8Dh0UwH/gK8MnKcD0R7AksApamsuOAU1I//cDqXNtqZmbr6pmkcV4LnBvBfQAR3D+Wf7FLfBh4NIIvSuwK7Ap8L/WxIXBXpfl56XkF0JeWfwoskXgRcF6zoxpJxwLHAsyZM6fz4MzMbFSTdc1GFKfPxr6i+GPgLRRHJ2Vf15dHPBG8IoLXV1Z5PD0/TUqmEZwFHEJxmu5CidfWx4mI0yOiPyL6t9lmm/GEamZmLUxWsrkYeKvEVgASW9bqHwI2r68kMRf4Z+CtldNrNwHbSOyd2myUTtO1JLEjcFsEnwOWA7tNZGPMzGxsJuU0WgTXS3wS+KHE08BVFDcGlIaAU6VnbxAoLQS2As5Pp8zujOAgicOAz0nMotiGzwLXtwnhcODtEk8Cv6Zx00EWc+fCwoWN10uXNpaHhxt3fQwNFe2Ghhp3kMydu/YdQkuXNtaprlvvq9pf3dBQo69WbaoxLljQ2bY0K2tWX+33sssad/ssWlQ8z5oF8+bBqlXr9jM8XDyX27lgQWO5Plb5uv5cra/OWbWv0eKvb2MZV7s4SnPnrl1eXbfTMVvVV9/Psc5HNbZmY7bqr7p/NIup3etmylPqne5HrepbrVONt5O53mST8Y/Varzq3JWf5bH0Xd3n6vM/MADLlsHixcXr8j1t9T7VY6p+/+Q0aXejzSQTvRvNzGx9NOV3o5mZ2frNycbMzLJzsjEzs+ycbMzMLDsnGzMzy87JxszMsnOyMTOz7JxszMwsOycbMzPLzsnGzMyyc7IxM7PsnGzMzCw7JxszM8vOycbMzLJzsjEzs+ycbMzMLDsnGzMzy87JxszMsnOyMTOz7JxszMwsOycbMzPLzsnGzMyyc7IxM7PsnGzMzCw7JxszM8vOycbMzLJzsjEzs+ycbMzMLLsxJxuJQYnjcwRjk29wsLt9jLW/wcHuxGBTY3AQBga6+x626mtwEHp7G/vM4CBskL7BqjFU96mBgTyxdMvAwLox9vWtvY3NHvXYxvIZnMjndSIUEWNbQQwCayI4OUtEo4/fE8FTOcfo7++PkZGRnENMGxKMcRdo28dY+5OK54nGYFOjfP+ge+9hq32oOlZVxNr7UX15InF14/MxWv+w9hittrOqvm1j+QxO5PM6GkkrIqK/WV1HRzYSSyRukvg+8NJUtpPEf0mskPiRxM6pfEjiSxL/LXGbxAKJr0j8XGKo0ueREislrpM4sVL+RokrJa6RuDiVDUqcLnERcIZEXxrzyvTYp7L+h1O/10gsS3FeWal/scSKMc2gmZlNSM9oDSTmA0cAr0rtrwRWAKcDx0Vws8SrgX8GXptW2yItHwL8J/BHwDHAFRLzgHuAE4H5wG+BiyQOBS4F/gV4TQS3S2xZCWU+sG8Ej0psCrwugsckXgycDfRLHAgcCrw6gkcktozgfokHJeZFcDXwTmgkvcZ26ljgWIA5c+Z0NHlmZtaZUZMNsB9wfgSPAEgsB3qBfYBvVg75Nqms858RhMRK4O4IVqZ1rwf6gLnAcAT3pvIzgdcATwOXRHA7QAT3V/pcHsGjaXkj4AspcT0NvCSVHwB8tYy1sv6XgXdKfAg4HNizvpERcTpFAqW/v98ndczMuqiTZANQ//LdAHgggnkt2j+enp+pLJeve6DlNRc1Gav0cGX5g8DdwCtTLI+Nsv63gKXAD4AVEfymxRhmZpZBJ8nmEmBIYllq/ybgNOB2ibdE8E0JAbtFcE2H414OnCKxNcVptCOBzwM/Bb4osUN5Gq12dFOaBayO4BmJo4ANU/lFwMckzqqeRkun2y4EvgQc3WGM64WlS7vbx1j768b4NnWWLoXh4Ynf9VXvs1X5smWweHGj7OMfL54XLGjEUF1/wYI8sXRLs/jmzoWFC0dft9XnbrSYJ/J5nYiO7kaTWAK8A7gDWA3cQHG08CVgW4rTWl+P4OPpJoALIjhXoi8t75r6qdb9OfARiqOR70Tw4dTmQOAfKI5Y7ongdfU74NJ1mm8BjwD/Dbwvgs1S3eIU6xOp3/+byvdK68yJ4Ol227s+3Y1mZtYt7e5GG/OtzzNV+r9BsyL46GhtnWzMzMauXbLp9JrNjCZxPrATjbvlzMxsEq0XySaCN091DGZm6zP/NpqZmWXnZGNmZtk52ZiZWXZONmZmlp2TjZmZZedkY2Zm2TnZmJlZdk42ZmaWnZONmZll52RjZmbZOdmYmVl2TjZmZpadk42ZmWXnZGNmZtk52ZiZWXZONmZmlp2TjZmZZedkY2Zm2TnZmJlZdk42ZmaWnZONmZll52RjZmbZOdmYmVl2TjZmZpadk42ZmWWXLdlIvF/i5xJndrnfQYnju9nnVOrrg8HBxuuBgcZytbybBgfXfdTruzVO+VzdrlZjVduPJYZW2zHZyvFnz163rF37TsvH0q6vr3m78cRTVd8/6+9ttY/qPDSLo/6ejWffHxhoPMr++voar3t71x5nYKCIq1ynbFu2q8dRLlf3zbK8vr9W56I65uzZa8dWlvf2NmKoqs9Nq89n2Ve1756e4tHbu/Y2NZvncp1mY+SgiMjTsbgRODCC2ytlPRE8NcF+B4E1EZw8wRBb6u/vj5GRkVzdr0Uqnsu3QWq+nGPMquo43Rq37Ke+jc3adNq+1TilTLtzx3GU8XfyHraq63T+O+17ovG067fUyVgTWbddPJ2q7lujtWm2T9afW/U5Wn2z8vo8tYqnHlMn2m1TqzHGS9KKiOhvVpflyEbiVGBHYLnEgxKnS1wEnCGxocRJEldIXCvxrsp6f1spP6FSvkTiJonvAy+tlM+TuCy1P19ii1Q+LPEZiUvS0dUeEudJ3CzxiRzbbGZmrWVJNhEcB9wJ7A98BpgP/GkEfw4cDTwYwR7AHsBfSewg8XrgxcCewDxgvsRrJOYDRwCvAv5PWqd0BvB3EewGrASWVuqeiOA1wKnAfwDvAXYFFkpsVY9Z0rGSRiSN3HvvvV2bCzMzg55JGmd5BI+m5dcDu0kcll7Pokgyr0+Pq1L5Zql8c+D8CB4BkFienmcBsyP4YWr/NeCb1THT80rg+gjuSuvdBmwP/KYaYEScDpwOxWm0iW6wmZk1TFayebiyLOB9EVxYbSDxBuAfIzitVr4IGM+X/+Pp+ZnKcvl6srbbzMyYmi/dC4F3S/wggiclXgL8KpX/P4kzI1gj8ULgSeASYEhiWYr3TcBpETwo8VuJ/SL4EfAX8OxRzowxdy4sXNh4vWBBY3np0nWad8Vo/XZr3LKfpUtheHj0sartxzPOVCvjmDVr3bJ27TstH0u7uXObtxtPPFXN9s/qe1vtozoPncQxnn2/uk55l9XQUOMus2XLYPHitdtffTXMm1e8XrWqaHvZZY121T7L5fq+uWBBY7xqrOVc1Odh0aJGbOXnfZNNYK+9ihiq6tve6nX9u2NoCFavLpZ70jd7uU3VPqrbNDTUfIwcct6NtgroB95L5e4xiQ2AT1AkDQH3Aoem5PEB4JjUxRrg7RHcKrEEeAdwB7AauCGCkyXmUVyT2RS4DXhnBL+VGAaOj2BEYiAtH5zGf7auVeyTeTeamdnvi3Z3o2VLNjOZk42Z2dhN+q3PZmZmVU42ZmaWnZONmZll52RjZmbZOdmYmVl2TjZmZpadk42ZmWXnZGNmZtk52ZiZWXZONmZmlp2TjZmZZedkY2Zm2TnZmJlZdk42ZmaWnZONmZll52RjZmbZOdmYmVl2TjZmZpadk42ZmWXnZGNmZtk52ZiZWXZONmZmlp2TjZmZZedkY2Zm2TnZmJlZdk42ZmaWnZONmZllN+2SjcSgxPFt6udJHFR5fYjE4smJrjODg+3ryvr6M8DAQKNNtS1AX19R32q8ZuM2G6uvr/k6fX2NGKrxjKaTNvV4mpWV49fNnr1uvL29685Ps3jajdeuTbu60fofrY9y/eo2le97O52OW+2/7LvZ69F0sm3jWWc8/XZz/N/Hsavjd7JvTAVFxNSN3oTEILAmgpNb1C8E+iN4b64Y+vv7Y2RkZNzrS9BqWqXiOaLRrtq+rK+q19X7btZPq7pq/83GblU23u3tpG27+Mv6erz17Wg1RrvxOol/tPXHMz+jvSed9NNJzM3e607GaRV3J7q9v4xVzr6n89jV8ce6P3c3Bq2IiP5mddPiyEZiicRNEt8HXprKhiX60/LWEqskNgY+DhwucbXE4RILJb6Q2m0j8S2JK9Ljj1L5gtT+aomrJDafok01M1sv9Ux1ABLzgSOAV1HEcyWwolnbCJ6Q+BiVI5t0pFM6BfhMBD+WmANcCLwMOB54TwSXSmwGPLZuHDoWOBZgzpw5Xdo6MzODaZBsgP2A8yN4BEBi+QT6OgDYpXK64HnpKOZS4NMSZwLnRbC6vmJEnA6cDsVptAnEYGZmNdMh2QA0+3J/isZpvt4O+9kA2DuCR2vlyyS+DRwEXCZxQAQ3ji9UMzMbq+mQbC4BhiSWUcTzJuA0YBUwH/gZcFil/UPQ8prLRcB7gZOguHMtgqsldopgJbBSYm9gZ8iXbJYu7ayuXK6WLVjQ+o6RuXOb37HVrJ9WdUuXwtAQLFy47jpz5zZiqMYzmk7a1ONpVlaOXzdrFixatHbbTTaBxS3uQazG0268dm3a1Y3W/2h9lOtX35Ph4dHvFOp03Hr/w8PNX4+mk20bzzrj6beb4/8+jl0dv5N9YypMi7vRJJYA7wDuAFYDNwAXAOcAa4AfAG+PoE9iS4prMRsB/wg8h3QNR2Jr4IsU12l6gEsiOE7i88D+wNOp74URPN4qnonejWZmtj5qdzfatEg2042TjZnZ2E37W5/NzOz3m5ONmZll52RjZmbZOdmYmVl2TjZmZpadk42ZmWXnZGNmZtk52ZiZWXZONmZmlp2TjZmZZedkY2Zm2TnZmJlZdk42ZmaWnZONmZll52RjZmbZOdmYmVl2TjZmZpadk42ZmWXnZGNmZtk52ZiZWXZONmZmlp2TjZmZZedkY2Zm2TnZmJlZdk42ZmaWnZONmZll52RjZmbZZU02Em+WCImdM/XfL/G5HH2bmVn35D6yORL4MXBEtzuW6IlgJIL3d7vvbhoYgMHBYnlwsHidWzlefbmb/c5Uk70NM2XOZs8eW/uBAejrKx6Dg41HX1+jrmwH687DwMC6n4VqWbt5q9fNnj16/O3GL2NtFUc91oEB2GCDxrb39BTjl3NQtqmOW45R9ld9rj8GBhp9ls9l23J+e3uL8t7eRhwDAyA12vf2NmKv91/2U5aV/fT0FNuWgyIiT8diM+AmYH9geQQ7SwwAJwB3A/OA84CVwAeA5wCHRnCrxDbAqcCc1N2iCC6VGAS2A/qA+4DTgeMjODiN93mgHwjghAi+JfElYI/U/7kRLB0t9v7+/hgZGenCLBRvPkDE2ss5SY0xqsvd7HemmuxtmClzNtY4y325nXKfrz7X129V1i6eTvoayzqj1VW3p1rfSrPtrj7PBOPdZyWtiIj+ZnU9EwloFIcC/xXB/0jcL7F7Kn8l8DLgfuA24MsR7CnxAeB9wCLgFOAzEfxYYg5wYVoHYD6wbwSPpuRV+ijwYASvAJDYIpUvieB+iQ2BiyV2i+DabFttZmbryJlsjgQ+m5a/nl5/G7gigrsAJG4FLkptVlIcBQEcAOxS+VfA8yQ2T8vLI3i0yXgHUDldF8Fv0+JbJY6l2NZtgV1g3WQj6VjgWIA5c+bUq83MbAKyJBuJrYDXArtKBLAhxamt7wCPV5o+U3n9TCWeDYC960klJZ+HWw2bxqi23wE4Htgjgt9KDAG9zVaOiNMpTsvR398/A058mJnNHLluEDgMOCOCuRH0RbA9cDuwb4frXwS8t3whMW8c62wBPI8iOT0o8XzgwA7HNzOzLsp1Gu1IYFmt7FvAu4FbO1j//cAXJa6liPES4LhR1vlEWuc64GmKGwTOk7gKuJ7i+tClnW9CdyxY0LgzZelSGB7OP+bSpc2Xu9nvTDXZ2zBT5mzWrLG1X7AAVq0qlhcubJQPDRV3NZV1CxYUz/V5KMtblbWbt3pdJ7G3G3/u3PZx1D+zCxbAJZfAnDnFtn/iE7DZZrBo0bp9lOOWY5T9DQw0nuuGh+HHPy76XLOmeJ43r2hbzu9llxV3kD32GLzgBUUcw8Pwwx8W8zFvXtFmr72KPqvjDA8X7095RxrAsmVFP6tXwzPPrBtTN2S7G20m6+bdaGZm64t2d6P5FwTMzCw7JxszM8vOycbMzLJzsjEzs+ycbMzMLDsnGzMzy87JxszMsnOyMTOz7JxszMwsOycbMzPLzsnGzMyyc7IxM7PsnGzMzCw7JxszM8vOycbMzLJzsjEzs+ycbMzMLDsnGzMzy87JxszMsnOyMTOz7JxszMwsOycbMzPLzsnGzMyyc7IxM7PsnGzMzCw7JxszM8vOycbMzLJzsjEzs+ycbMzMLDsnGzMzy87JxszMslNETHUM046ke4E7xrn61sB9XQwnB8c4cdM9PnCM3TDd44PpFePciNimWYWTTZdJGomI/qmOox3HOHHTPT5wjN0w3eODmREj+DSamZlNAicbMzPLzsmm+06f6gA64BgnbrrHB46xG6Z7fDAzYvQ1GzMzy89HNmZmlp2TjZmZZedk00WS3ijpJkm3SFqceaztJf23pJ9Lul7SB1L5lpK+J+nm9LxFKpekz6XYrpW0e6Wvo1L7myUdVSmfL2llWudzkjSOODeUdJWkC9LrHSRdnsb6hqSNU/km6fUtqb6v0sdHUvlNkt5QKZ/wfEuaLelcSTemudx7Gs7hB9N7fJ2ksyX1TvU8SvqKpHskXVcpyz5vrcYYQ4wnpff6WknnS5o93vkZz3swWnyVuuMlhaStp3IOuyoi/OjCA9gQuBXYEdgYuAbYJeN42wK7p+XNgf8BdgE+BSxO5YuBE9PyQcB3AQF7AZen8i2B29LzFml5i1T3M2DvtM53gQPHEeeHgLOAC9Lrc4Aj0vKpwLvT8l8Dp6blI4BvpOVd0lxuAuyQ5njDbs038DXgmLS8MTB7Os0h8ELgduA5lflbONXzCLwG2B24rlKWfd5ajTGGGF8P9KTlEysxjnl+xvoedBJfKt8euJDiP5ZvPZVz2NXvrMkYZH14pDf1wsrrjwAfmcTx/wN4HXATsG0q2xa4KS2fBhxZaX9Tqj8SOK1Sfloq2xa4sVK+VrsOY3oRcDHwWuCCtNPfV/mwPztn6cO1d1ruSe1Un8eyXTfmG3gexRe5auXTaQ5fCPwyfZn0pHl8w3SYR6CPtb/Is89bqzE6jbFW92bgzGbbPdr8jGdf7jQ+4FzglcAqGslmyuawWw+fRuue8kuhtDqVZZcO018FXA48PyLuAkjPfzBKfO3KVzcpH4vPAh8GnkmvtwIeiIinmvT5bByp/sHUfqxxj8WOwL3AV1Wc6vuypOcyjeYwIn4FnAz8AriLYl5WML3msTQZ89ZqjPH4S4p/8Y8nxvHsy6OSdAjwq4i4plY1XeewY0423dPsXHz2+8olbQZ8C1gUEb9r17RJWYyjvNO4DgbuiYgVHcQw6fElPRSnMb4UEa8CHqY4rdDKpMeYzqf/KcWpne2A5wIHtul3KuZxNNMuJklLgKeAM8uiMcYynn15tJg2BZYAH2tW3cX4poSTTfespjjXWnoRcGfOASVtRJFozoyI81Lx3ZK2TfXbAveMEl+78hc1Ke/UHwGHSFoFfJ3iVNpngdmSepr0+WwcqX4WcP844h6L1cDqiLg8vT6XIvlMlzkEOAC4PSLujYgngfOAfZhe81iajHlrNUbH0kX0g4G3RTqXNI4Y72Ps78FodqL4R8U16XPzIuBKSS8YR3xZ53BcJuNc3frwoPhX8m0UO0t5IfHlGccTcAbw2Vr5Sax98e9TaflPWPsC489S+ZYU162ruBkAAARkSURBVC22SI/bgS1T3RWpbXmB8aBxxjpA4waBb7L2RdW/TsvvYe2Lquek5Zez9oXb2ygu2nZlvoEfAS9Ny4Np/qbNHAKvBq4HNk19fA1433SYR9a9ZpN93lqNMYYY3wjcAGxTazfm+Rnre9BJfLW6VTSu2UzZHHbrkX2A9elBccfI/1DcvbIk81j7UhwWXwtcnR4HUZwbvhi4OT2XO56AL6bYVgL9lb7+ErglPd5ZKe8HrkvrfIEWFzk7iHWARrLZkeIumVvSh3WTVN6bXt+S6nesrL8kxXATlbu5ujHfwDxgJM3jv6cP7LSaQ+AE4MbUz79SfCFO6TwCZ1NcQ3qS4l/RR0/GvLUaYwwx3kJxjaP8zJw63vkZz3swWny1+lU0ks2UzGE3H/65GjMzy87XbMzMLDsnGzMzy87JxszMsnOyMTOz7JxszMwsOycbs3GS9BlJiyqvL5T05crrf5L0oQn0Pyjp+BZ1x6ZfL75R0s8k7Vup20/Fr0RfLek56ZeOr5d00hjH75P05+ON36zKycZs/H5C8b/5kbQBsDXFfw4s7QNc2klHkjbsdND0U0DvAvaNiJ2B44Cz0v80B3gbcHJEzIuIR1Pb3SPibzsdI+kDnGysK5xszMbvUlKyoUgy1wEPSdpC0ibAy4Cr0t8iOUnF36NZKelwAEkDKv4m0VkU/1EPSUvS3075PvDSFuP+HfC3EXEfQERcSfHLAu+RdAzwVuBjks6UtJzi99Qul3S4pLekOK6RdEkac8MU3xXpb6W8K42zDNgvHSF9sJsTZ+ufntGbmFkzEXGnpKckzaFIOj+l+GXdvSl+6ffaiHhC0p9R/FLBKymOfq4ov+iBPYFdI+J2SfMpft7kVRSfzSspfuG57uVNykeAoyLio+mU2gURcS6ApDURMS8trwTeEBG/UuMPhx0NPBgRe6Qkeamkiyh+yuT4iDh4YjNl5mRjNlHl0c0+wKcpks0+FMnmJ6nNvsDZEfE0xY8g/hDYA/gdxW9c3Z7a7QecHxGPAKSjkk6Jzn7V91JgSNI5FD/qCcUfFNtN0mHp9SzgxcATYxjfrC2fRjObmPK6zSsoTqNdRnFkU71e0+5PQT9ce91JwrgBmF8r2z2VtxURxwF/T/FLwVdL2irF9750jWdeROwQERd1EIdZx5xszCbmUoqfq78/Ip6OiPsp/rT03hSn1QAuAQ5P10a2ofhzwD9r0tclwJvTHWSbA29qMeangBNTokDSPIo/Ff3PowUraaeIuDwiPkbxM/nlnyB+d/qTFUh6Sfojcg9R/MlxswnzaTSziVlJcR3mrFrZZuUFfOB8iuRzDcWRy4cj4teSdq52FBFXSvoGxa8R30Hx5w/WERHLJb0Q+ImkoEgKb4/01xdHcZKkF1MczVycYrqW4s6zKyWJ4q+XHprKn5J0DTAUEZ/poH+zpvyrz2Zmlp1Po5mZWXZONmZmlp2TjZmZZedkY2Zm2TnZmJlZdk42ZmaWnZONmZll97/TY8vA9dXuuwAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Lexical Dispersion Plot for Words in U.S. Presidential Inaugural Addresses\n",
    "us_inaugural_addresses.dispersion_plot([\"citizens\", \"democracy\", \"freedom\", \"duties\", \"America\"])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Word Frequencies\n",
    "\n",
    "> The frequency of words and other linguistic units plays a central role in all branches of corpus linguistics. Indeed, the use of frequency information is what distinguishes corpus-based methodologies from other approaches to language (Baroni, 2009)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- The **ABSOLUTE FREQUENCY** of a *type* $v_i$ word, $f(v_i)$, is its number of occurrences (i.e. *tokens*) in a corpus"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![alt text](images/tokens-types.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "We already know how to compute frequencies by using the `Counter()` method from the `collections` module:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [],
   "source": [
    "from collections import Counter"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {},
   "outputs": [],
   "source": [
    "# case sensitive counts \n",
    "fdist_raw = Counter(nltk.corpus.brown.words())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[('the', 62713),\n",
       " (',', 58334),\n",
       " ('.', 49346),\n",
       " ('of', 36080),\n",
       " ('and', 27915),\n",
       " ('to', 25732),\n",
       " ('a', 21881),\n",
       " ('in', 19536),\n",
       " ('that', 10237),\n",
       " ('is', 10011)]"
      ]
     },
     "execution_count": 34,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# top-10 most frequent words\n",
    "fdist_raw.most_common(10)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {},
   "outputs": [],
   "source": [
    "# let's ignore non-alphanumeric words\n",
    "fdist = Counter([word for word in nltk.corpus.brown.words() if word.isalnum()])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[('the', 62713),\n",
       " ('of', 36080),\n",
       " ('and', 27915),\n",
       " ('to', 25732),\n",
       " ('a', 21881),\n",
       " ('in', 19536),\n",
       " ('that', 10237),\n",
       " ('is', 10011),\n",
       " ('was', 9777),\n",
       " ('for', 8841)]"
      ]
     },
     "execution_count": 36,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "fdist.most_common(10)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {},
   "outputs": [],
   "source": [
    "# If we want the case insensitive counts of the results above...\n",
    "fdist_insensitive = Counter([word.lower() for word in nltk.corpus.brown.words() if word.isalnum()])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[('the', 69971),\n",
       " ('of', 36412),\n",
       " ('and', 28853),\n",
       " ('to', 26158),\n",
       " ('a', 23195),\n",
       " ('in', 21337),\n",
       " ('that', 10594),\n",
       " ('is', 10109),\n",
       " ('was', 9815),\n",
       " ('he', 9548)]"
      ]
     },
     "execution_count": 38,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "fdist_insensitive.most_common(10)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "How about a fancy frequency list that is easy to manipulate? There are many modules in the `pandas` package that can help you doing that."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {},
   "outputs": [],
   "source": [
    "# convert our dictionary into a dataframe\n",
    "import pandas as pd\n",
    "\n",
    "df = pd.DataFrame.from_dict(fdist, orient='index')\n",
    "df.columns = ['fq']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>fq</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>the</th>\n",
       "      <td>62713</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>of</th>\n",
       "      <td>36080</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>and</th>\n",
       "      <td>27915</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>to</th>\n",
       "      <td>25732</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>a</th>\n",
       "      <td>21881</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>in</th>\n",
       "      <td>19536</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>that</th>\n",
       "      <td>10237</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>is</th>\n",
       "      <td>10011</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>was</th>\n",
       "      <td>9777</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>for</th>\n",
       "      <td>8841</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "         fq\n",
       "the   62713\n",
       "of    36080\n",
       "and   27915\n",
       "to    25732\n",
       "a     21881\n",
       "in    19536\n",
       "that  10237\n",
       "is    10011\n",
       "was    9777\n",
       "for    8841"
      ]
     },
     "execution_count": 43,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# let's sort our frequencies in descending order\n",
    "dfs = df.sort_values('fq', ascending = False)\n",
    "dfs.head(10)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "A useful piece of information we want to visualize is the **RANK** of each item, i.e. its position in our sorted frequency list"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>fq</th>\n",
       "      <th>rank</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>the</th>\n",
       "      <td>62713</td>\n",
       "      <td>1.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>of</th>\n",
       "      <td>36080</td>\n",
       "      <td>2.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>and</th>\n",
       "      <td>27915</td>\n",
       "      <td>3.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>to</th>\n",
       "      <td>25732</td>\n",
       "      <td>4.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>a</th>\n",
       "      <td>21881</td>\n",
       "      <td>5.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>in</th>\n",
       "      <td>19536</td>\n",
       "      <td>6.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>that</th>\n",
       "      <td>10237</td>\n",
       "      <td>7.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>is</th>\n",
       "      <td>10011</td>\n",
       "      <td>8.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>was</th>\n",
       "      <td>9777</td>\n",
       "      <td>9.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>for</th>\n",
       "      <td>8841</td>\n",
       "      <td>10.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "         fq  rank\n",
       "the   62713   1.0\n",
       "of    36080   2.0\n",
       "and   27915   3.0\n",
       "to    25732   4.0\n",
       "a     21881   5.0\n",
       "in    19536   6.0\n",
       "that  10237   7.0\n",
       "is    10011   8.0\n",
       "was    9777   9.0\n",
       "for    8841  10.0"
      ]
     },
     "execution_count": 44,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df[\"rank\"] = df['fq'].rank(ascending = False, method = 'first') # add column 'rank' with values of 'fq'\n",
    "df.sort_values('rank', ascending = True, inplace=True)  # sorting our frequencies IN PLACE\n",
    "df.head(10)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- The **CONDITIONAL FREQUENCY** of a *type* $v_i$ word in the condition $X$, $f(v_i|X)$, is its number of occurrences (i.e. *tokens*) in the corpus sections where the target condition is satisfied."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The NLTK `nltk.ConditionalFreqDist()` builds a conditional frequency object by counting the instances in a list of pairs (each pair being a given occurrence in a given condition)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[('adventure', 'Dan'),\n",
       " ('adventure', 'Morgan'),\n",
       " ('adventure', 'told'),\n",
       " ('adventure', 'himself'),\n",
       " ('adventure', 'he'),\n",
       " ('adventure', 'would'),\n",
       " ('adventure', 'forget'),\n",
       " ('adventure', 'Ann'),\n",
       " ('adventure', 'Turner'),\n",
       " ('adventure', '.')]"
      ]
     },
     "execution_count": 45,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# let's build our (condition, word) pairings\n",
    "cond_word_pairs = [(genre, word) for genre in nltk.corpus.brown.categories() for word in nltk.corpus.brown.words(categories = genre)]\n",
    "cond_word_pairs[:10]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[('belles_lettres', 'a'),\n",
       " ('belles_lettres', 'stream'),\n",
       " ('belles_lettres', 'that'),\n",
       " ('belles_lettres', 'turns'),\n",
       " ('belles_lettres', 'a'),\n",
       " ('belles_lettres', 'mill-wheel'),\n",
       " ('belles_lettres', 'there'),\n",
       " ('belles_lettres', 'is'),\n",
       " ('belles_lettres', 'a'),\n",
       " ('belles_lettres', 'lot')]"
      ]
     },
     "execution_count": 46,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "cond_word_pairs[100000:100010]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "metadata": {},
   "outputs": [],
   "source": [
    "# create our conditional frequency object\n",
    "cfd = nltk.ConditionalFreqDist(cond_word_pairs)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['adventure',\n",
       " 'belles_lettres',\n",
       " 'editorial',\n",
       " 'fiction',\n",
       " 'government',\n",
       " 'hobbies',\n",
       " 'humor',\n",
       " 'learned',\n",
       " 'lore',\n",
       " 'mystery',\n",
       " 'news',\n",
       " 'religion',\n",
       " 'reviews',\n",
       " 'romance',\n",
       " 'science_fiction']"
      ]
     },
     "execution_count": 50,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# check the conditions in the conditional frequency object\n",
    "cfd.conditions()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Condition values can be accessed individually, and the return objects are simple frequency distributions"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[('the', 3508),\n",
       " (',', 2766),\n",
       " ('.', 2481),\n",
       " ('of', 1976),\n",
       " ('to', 1554),\n",
       " ('and', 1302),\n",
       " ('a', 1095),\n",
       " ('in', 1001),\n",
       " ('is', 744),\n",
       " ('that', 578)]"
      ]
     },
     "execution_count": 51,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "cfd['editorial'].most_common(10)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The `plot()` and `tabulate()` methods can be used to plot the frequency distributions in the different conditions and to create a contingency table (a.k.a. a two-way table). Optional parameters `conditions` and `samples` can be used to focus on a given set of condition values or samples. This makes possible to load large quantity of data once, and then to focus only on meaningful portions of them."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For instance, let's contrast how frequently modal verbs are used in some of the brown corpus sections."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 52,
   "metadata": {},
   "outputs": [],
   "source": [
    "genres = ['news', 'religion', 'science_fiction', 'romance', 'humor']\n",
    "modals = ['can', 'could', 'may', 'might', 'must', 'will']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 53,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "                  can could   may might  must  will \n",
      "           news    93    86    66    38    50   389 \n",
      "       religion    82    59    78    12    54    71 \n",
      "science_fiction    16    49     4    12     8    16 \n",
      "        romance    74   193    11    51    45    43 \n",
      "          humor    16    30     8     8     9    13 \n"
     ]
    }
   ],
   "source": [
    "cfd.tabulate(conditions = genres, samples = modals)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- the **CORPUS SIZE** is the total number of occurrences (tokens) in the text: \n",
    "\n",
    "\n",
    "$$|C| = f(v_1) + f(v_2) + f(v_3) + ... + f(v_n)$$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 59,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "297 µs ± 8.49 µs per loop (mean ± std. dev. of 7 runs, 1000 loops each)\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "988331"
      ]
     },
     "execution_count": 59,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Recall: keys = words, values = their counts\n",
    "%timeit corpus_size = sum(fdist.values())\n",
    "corpus_size"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 60,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "2.3 s ± 54.2 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "988331"
      ]
     },
     "execution_count": 60,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# equivalent, fully explicit...\n",
    "%timeit corpus_size = len([word for word in nltk.corpus.brown.words() if word.isalnum()])\n",
    "corpus_size"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- The **RELATIVE FREQUENCY** of a *type* $v_i$ word is its absolute frequency divided by the corpus size:\n",
    "\n",
    "\n",
    "$$f_{rel}(v_i) = \\dfrac{f(v_i)}{|C|}$$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 61,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "('the', 62713, 0.063453)\n",
      "('of', 36080, 0.036506)\n",
      "('and', 27915, 0.028245)\n",
      "('to', 25732, 0.026036)\n",
      "('a', 21881, 0.022139)\n",
      "('in', 19536, 0.019767)\n",
      "('that', 10237, 0.010358)\n",
      "('is', 10011, 0.010129)\n",
      "('was', 9777, 0.009892)\n",
      "('for', 8841, 0.008945)\n"
     ]
    }
   ],
   "source": [
    "for word, abs_freq in fdist.most_common(10):\n",
    "    print( (word, abs_freq, round(abs_freq / corpus_size, 6)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "- The **VOCABULARY**, $V_c$, is the total number of *types* instantiated in the corpus (instead of *tokens* above)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 62,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "46969\n"
     ]
    }
   ],
   "source": [
    "vocabulary = fdist.keys()\n",
    "print(len(vocabulary))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- The **FREQUENCY CLASS** $V_i$ is the set of *types* occurring $i$ times"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 63,
   "metadata": {},
   "outputs": [],
   "source": [
    "from collections import defaultdict\n",
    "\n",
    "frequency_classes = defaultdict(set)\n",
    "\n",
    "for atype, freq in fdist.items():    \n",
    "    frequency_classes[freq].add(atype)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 64,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'figures', 'Thomas', 'meant', 'price', 'race', 'rates'}\n",
      "{'the'}\n"
     ]
    }
   ],
   "source": [
    "# a dictionary, where a frequency maps to a set of words occuring that often\n",
    "print(frequency_classes[100])\n",
    "print(frequency_classes[62713])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "$V_1$ is the set of items occurring just one time, they are called **hapax legomena**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 65,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['override', 'Krupa', 'Lyman', 'Kaplan', 'nonsystematic', 'Retail', 'acrobats', 'lopsidedly', 'sighs', 'Dilys', 'ludicrousness', 'Eckart', 'halter', 'Mumford', 'Theodosius', 'unshaved', 'boomed', 'Desperately', 'twittered', 'Salesmanship']\n",
      "19052\n"
     ]
    }
   ],
   "source": [
    "import random\n",
    "\n",
    "print(random.sample(frequency_classes[1], 20))\n",
    "print(len(frequency_classes[1]))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "A **frequency spectrum** reports all the frequency classes of a corpus"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 66,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "19052"
      ]
     },
     "execution_count": 66,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "frequency_spectrum = Counter(fdist.values()) \n",
    "frequency_spectrum[1]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "A frequency spectrum can be visually inspected by plotting the class cardinality as a function of the increasing class frequency"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 67,
   "metadata": {},
   "outputs": [],
   "source": [
    "import matplotlib.pyplot as plt\n",
    "\n",
    "sorted_classes_freqs_tuples = sorted(frequency_spectrum.items())\n",
    "\n",
    "# zip() returns a list of tuples, where each tuple contains the i-th element from each of the argument sequences\n",
    "# the single star * unpacks a sequence/collection into positional arguments\n",
    "x, y = zip(*sorted_classes_freqs_tuples)  # unpack a list of pairs into two tuples"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 70,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAZwAAAEeCAYAAAC+OaPqAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjAsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+17YcXAAAgAElEQVR4nO3df5xddX3n8dd7fpCEJITEDEkkCYk21kliizKrbI2WrBUQfxBdtSRWIk6biiXodt0Kne6CuNllbaMtqcJGQ0nUGaT4K61YQDLATivKBBECAyUgQiRAkJ9GYpLhs3+c7x3OTO78Sibn3sy8n4/Hecw5n/M953y/987cz/2e851zFBGYmZkdbjWVroCZmY0NTjhmZlYIJxwzMyuEE46ZmRXCCcfMzArhhGNmZoVwwjEzs0I44YxRkh6W9KKkX0l6RtL3JM2pdL36UuZ8Sdsk7Za0Q9I/Snpdpes2kD6vb2l6ZaXrVQmSzpR0p6TnJT0l6SZJ8w7zMUPSbx3OY9jwOeGMbe+OiEnALOAJYF1/BSXVFlar3v4O+ARwPjANeA3wHeCdw92RpLqRrdqg3h0Rk3LTY1VQp0KlD/1NwH8FpgDzgS8BL1W4XqP6da9aEeFpDE7Aw8Af5JbPAP49t3wVcDlwHbAb+AOyD4xNwC7g58BfATWp/M+Bk9L8HwEBLEzLfwx8J81fDFyT9vMCcA/Q1E8dFwDdwBsHaMdAdfoI8K/AF4Cngf+Zi60DngPuA942wOtyMfC1ND8e+BrwS+BZ4HZgxlBe31x8XnptmoFHgFtT/GTg39J+fwqckttmPnBLer1uBP4+V6dTgB39HZvsS+UFwIOp3tcA0/rUZWWqy1NAS24/tcBfpm1fALYCc4AvAmv7HPOfgE+Wae/7gTsHeP8uBq4FvpGOcQfwu7n1rwS+md7fnwHnD6F+t6Z27QZ+Bfxh6XUCPg08Dnw1/S509KlPAL+V+xv4EvD9tJ9/BWYCfws8k353Xl/pv+UjaXIPx5B0NNkf5W19Vq0A1gCTgQ6yD+kpwKuA3wfOBs5JZW8h+6MGeCvwUCpTWr4lt9/3AFcDxwKbyT5Ay3kb2Yfpjweo/kB1AnhTqstxqS352HTgIuBbkqYNcIySlelYc4BXAB8DXhzCduX8PtAInCbpeOB7ZAlxGvAp4JuSGlLZVrIP0+nAZ1M9hup8YFk63ivJPii/2KfMEuC3yV7v/yGpMcX/HFhO9mXkGOCjwK+BjcBySTUAkqanbdvKHP8O4LWSviBpqaRJZcqcCfxjansr8B1J9Wn//0SWgI9Px/ikpNMGql9EvDWt/93IepbfSMsz0zFOAFb1/5L18kGyLzHTgd8AP0xtmk6WKD8/xP0YuIczVieyb8G/IvtGvR94DHhdbv1VwKbcci3ZH9zCXOxPgZvTfDOwOc13kfVqrk7LPwfekOYvBn6Q28dC4MV+6tgC3DZAGwar00eAR/ps85HUVuViPwY+nHtd+uvhfJSsF/I7w3x9n+XlHt48sm/Rr8qV/TTw1T7bX0+WWOam92dibl0rQ+/hdNG7BzcL2AfU5eoyu89rcVaavx84s5/2dQFvT/PnAdcN8FqcTNaz2gXsSb9bk3Kv7225sjXATuAtZF8M+r5/FwL/MIT69fRUcq/TXmB8n9+FwXo4X86tWw105ZZfBzxbqb/hI3FyD2dsWxYRxwLjyD40bpE0M7f+0dz8dOAosuRR8nOyb56Q9WDekravJTtF8uZ0cXgKcGduu8dz878GxvdzTv2XZB+Q/RmsTn3bUPKLSJ8YuW2GckH/q2SJ4GpJj0n6nKT6Acovi4hj07Ssz7p8vU4APiDp2dJE1uuYler1TETs7lPfoToB+HZuv11kpyln5Mr0fT9KvZA5ZKerytlIduqU9POr/VUgIm6LiA9GRANZInkr2ZeJkkdzZV8iO/X1ylT3V/Z5Xf4yV/eB6lfOrojYM4zykF3bLHmxzHK5Hpv1wwnHiIjuiPgW2QfRkvyq3PxTZN+MT8jF5gK/SPvYTvZhdT7ZdYkXyD7IVpF9izyYi8Q3AbMlNfWzfsA6lWlDyfGS1Geb0gX93cDRuXU9CTgi9kXEZyJiIfB7wLvITuEdjHy9HiXr4RybmyZGxKVk3/anSprYp74lveqbBnc05NY/Cryjz77HR0T+NerPo8Cr+1n3NeBMSb9LdmrwO0PYHxFxO/AtYHEu3DM6Mp1Gm032fjwK/KxP3SdHxBlDqF/Zw/dZ7vvazcQOKyccKw09PhOYSvYN+AAR0U12WmSNpMmSTiA7h/61XLFbSD2ltHxzn+VhiYgHyC7atkk6RdJRksZLOkvSBUOsUznHAeen6wQfIPvAvC6tuxM4K61rIrvoDUC6BvG69KH+PFmy6z6YtvXxNeDdkk6TVJvaeIqk2RHxc6AT+Exq/xLg3blt/52sh/jO1Nv6K7Iea8kVZK/PCakNDem9HoqvAJ+VtCD9jvyOpFcARMQOskETXwW+GRFlr2VJWiLpTyQdl5ZfS3YNL3+98CRJ70u93E+SnSa9jez03vOSPi1pQnptFkv6D4PVj6wn8qpB2vdTYJGkEyWNJzu9Z4eRE87Y9k+SfkX24bkGWBkR9wxQfjXZt8KHyAYRtAJX5tbfQjbA4NZ+lg/G+WSDCr5Idi3kQeC9ZBeTh1Kncn5ENgLuKbJ2vz8ifpnW/Xeyb83PAJ9J+yuZSXah+HmyxHwLgye3QUXEo2QXzv+S7DrHo8B/4+W/zxVk1zOeJhvksCm37XPAx8k+fH9B9lrsyO3+78gGZtwg6QWyD/I3DbFqnydL6DeQtXkDMCG3fiPZdYx+T6eRvWfvAe5Ov2v/Anwb+FyuzHfJBq08A3wYeF/qTXaTJdcTyUaoPZXaOWUI9bsY2JhOxX2wXMUi4t+BS4AfAA+Q/f7YYaTep7LNRjdJHwH+OCKWDFa2Wkm6mOzC9h8NVvYw1+OtZAl33kGeMq2atlgx3MMxs2FLp+8+AXzlYJONjT2FJRxJcyS1S+qSdI+kT6T4NEk3Snog/Zya4pJ0maTtku6S9Ibcvlam8g9IWpmLnyTp7rTNZX0uDJvZCFD2fzrPko2i+9sKV8eOIIWdUpM0C5gVEXdImkz2j2zLyMbCPx0Rl0q6AJgaEZ+WdAbZ+fkzyM45/11EvEnZP+h1Ak1ko062kv2H+zOSfkz2res2sovAl0XE9wtpoJmZDaiwHk5E7IyIO9L8C2QXXY8nu1i6MRXbSJaESPFNkbkNODYlrdOAGyPi6Yh4huxWH6endcdExA/T/1hsyu3LzMwqrCLXcJT9M+DryUYLzYiInZAlJbIhq5Alo/w/x+1IsYHiO8rEzcysChR+x1Rl91L6JtmN/p4f4DJLuRVxEPFydVhFupfSxIkTT3rta187WLXNzCxn69atT6W7RwxZoQknjWz5JvD19J/tAE9ImhURO9NpsSdTfAe5/0Dm5f8+3sHLN4ksxW9O8dllyh8gItYD6wGampqis7PzEFplZjb2SBrOLZaAYkepiewfs7oiIn+H1c28fPfblWT/BFaKn51Gq50MPJdOuV0PnCppahrRdipwfVr3gqST07HOzu3LzMwqrMgezpvJ/ov4bkmlGzn+JXApcI2k0vNBPpDWXUc2Qq10j65zACLiaUmfJbutBsAlEfF0mj+X7A6vE8ieYeERamZmVWLM32nAp9TMzIZP0taI6O/GumX5TgNmZlYIJ5yD0NbWxuLFi6mtrWXx4sW0tZV70KGZmeUVPiz6SNfW1kZLSwsbNmxgyZIldHR00NzcDMDy5csrXDszs+rlazjDvIazePFi1q1bx9KlS3ti7e3trF69mm3bth2OKpqZVZ2DuYbjhDPMhFNbW8uePXuor3/5ycL79u1j/PjxdHePxLO4zMyqnwcNFKCxsZGOjt7Paero6KCxsbFCNTIzOzI44QxTS0sLzc3NtLe3s2/fPtrb22lubqalpaXSVTMzq2oeNDBMpYEBq1evpquri8bGRtasWeMBA2Zmg/A1HP/jp5nZsPkajpmZVS0nHDMzK4QTjpmZFcIJx8zMCuGEY2ZmhXDCMTOzQjjhmJlZIZxwzMysEIUlHElXSnpS0rZc7BuS7kzTw6VHT0uaJ+nF3LorctucJOluSdslXSZJKT5N0o2SHkg/pxbVNjMzG1yRPZyrgNPzgYj4w4g4MSJOBL4JfCu3+sHSuoj4WC5+ObAKWJCm0j4vAG6KiAXATWnZzMyqRGEJJyJuBZ4uty71Uj4IDPjoTEmzgGMi4oeR3ZNnE7AsrT4T2JjmN+biZmZWBarlGs5bgCci4oFcbL6kn0i6RdJbUux4YEeuzI4UA5gRETsB0s/jDnelzcxs6KrlbtHL6d272QnMjYhfSjoJ+I6kRYDKbDvsu49KWkV2Wo65c+ceRHXNzGy4Kt7DkVQHvA/4RikWEb+JiF+m+a3Ag8BryHo0s3ObzwYeS/NPpFNupVNvT/Z3zIhYHxFNEdHU0NAwks0xM7N+VDzhAH8A3BcRPafKJDVIqk3zryIbHPBQOlX2gqST03Wfs4Hvps02AyvT/Mpc3MzMqkCRw6LbgB8Cvy1ph6TmtOosDhws8FbgLkk/Ba4FPhYRpQEH5wJfAbaT9Xy+n+KXAm+X9ADw9rRsZmZVwg9g8wPYzMyGzQ9gMzOzquWEY2ZmhXDCMTOzQjjhmJlZIZxwzMysEE44ZmZWCCccMzMrhBOOmZkVwgnHzMwK4YRjZmaFcMIxM7NCOOGYmVkhnHDMzKwQTjhmZlYIJxwzMyuEE46ZmRXCCcfMzApR5COmr5T0pKRtudjFkn4h6c40nZFbd6Gk7ZLul3RaLn56im2XdEEuPl/SjyQ9IOkbko4qqm1mZja4Ins4VwGnl4l/ISJOTNN1AJIWAmcBi9I2X5JUK6kW+CLwDmAhsDyVBfg/aV8LgGeA5sPaGjMzG5bCEk5E3Ao8PcTiZwJXR8RvIuJnwHbgjWnaHhEPRcRe4GrgTEkC/hNwbdp+I7BsRBtgZmaHpBqu4Zwn6a50ym1qih0PPJorsyPF+ou/Ang2Ivb3iZclaZWkTkmdu3btGql2mJnZACqdcC4HXg2cCOwE1qa4ypSNg4iXFRHrI6IpIpoaGhqGV2MzMzsodZU8eEQ8UZqX9GXgn9PiDmBOruhs4LE0Xy7+FHCspLrUy8mXNzOzKlDRHo6kWbnF9wKlEWybgbMkjZM0H1gA/Bi4HViQRqQdRTawYHNEBNAOvD9tvxL4bhFtMDOzoSmshyOpDTgFmC5pB3ARcIqkE8lOfz0M/ClARNwj6RrgXmA/8GcR0Z32cx5wPVALXBkR96RDfBq4WtL/BH4CbCioaWZmNgTKOgdjV1NTU3R2dla6GmZmRxRJWyOiaTjbVHrQgJmZjRFOOGZmVggnHDMzK4QTjpmZFcIJx8zMCuGEY2ZmhXDCMTOzQjjhmJlZIZxwzMysEE44ZmZWCCccMzMrhBOOmZkVwgnHzMwK4YRjZmaFcMIxM7NCOOGYmVkhnHDMzKwQhSUcSVdKelLStlzsryXdJ+kuSd+WdGyKz5P0oqQ703RFbpuTJN0tabukyyQpxadJulHSA+nn1KLaZmZmgyuyh3MVcHqf2I3A4oj4HeDfgQtz6x6MiBPT9LFc/HJgFbAgTaV9XgDcFBELgJvSspmZVYnCEk5E3Ao83Sd2Q0TsT4u3AbMH2oekWcAxEfHDiAhgE7AsrT4T2JjmN+biZmZWBarpGs5Hge/nludL+omkWyS9JcWOB3bkyuxIMYAZEbETIP08rr8DSVolqVNS565du0auBWZm1q+qSDiSWoD9wNdTaCcwNyJeD/w50CrpGEBlNo/hHi8i1kdEU0Q0NTQ0HGy1zcxsGOoqXQFJK4F3AW9Lp8mIiN8Av0nzWyU9CLyGrEeTP+02G3gszT8haVZE7Eyn3p4sqg1mZja4ivZwJJ0OfBp4T0T8OhdvkFSb5l9FNjjgoXSq7AVJJ6fRaWcD302bbQZWpvmVubiZmVWBwno4ktqAU4DpknYAF5GNShsH3JhGN9+WRqS9FbhE0n6gG/hYRJQGHJxLNuJtAtk1n9J1n0uBayQ1A48AHyigWWZmNkRKZ7HGrKampujs7Kx0NczMjiiStkZE03C2qYpBA2ZmNvo54ZiZWSGccMzMrBBOOGZmVggnHDMzK4QTjpmZFcIJx8zMCuGEY2ZmhXDCMTOzQjjhmJlZIZxwzMysEE44ZmZWCCccMzMrhBOOmZkVwgnHzMwK4YRjZmaFcMIxM7NCFJpwJF0p6UlJ23KxaZJulPRA+jk1xSXpMknbJd0l6Q25bVam8g9IWpmLnyTp7rTNZUrPrTYzs8oruodzFXB6n9gFwE0RsQC4KS0DvANYkKZVwOWQJSjgIuBNwBuBi0pJKpVZlduu77HMzKxC6gYrIGnuEPf1bEQ8P1CBiLhV0rw+4TOBU9L8RuBm4NMpvikiArhN0rGSZqWyN0bE06l+NwKnS7oZOCYifpjim4BlwPeHWH8zMzuMBk04ZEkggIFOTwVZ72XTQdRhRkTsBIiInZKOS/HjgUdz5Xak2EDxHWXiB5C0iqwnxNy5Q82nZmZ2KAZNOBGxtIiKlFEuwfWX+AaKHxiMWA+sB2hqaipbxszMRtZQeji9SLoEqAXuBO6MiAcOsQ5PSJqVejezgCdTfAcwJ1duNvBYip/SJ35zis8uU97MzKrAoIMGJL02vxwR/wO4DHgB+M+SvnyIddgMlEaarQS+m4ufnUarnQw8l069XQ+cKmlqGixwKnB9WveCpJPT6LSzc/syM7MKG0oP57p0Qf7iiHgEICKeAP4lTUMmqY2sdzJd0g6y0WaXAtdIagYeAT5QOi5wBrAd+DVwTjr205I+C9yeyl1SGkAAnEt2LWkC2WABDxgwM6sSygaBDVBAOgr4U+DPyXoMayJiVwF1K0RTU1N0dnZWuhpmZkcUSVsjomk42wx6Si0i9kbEOqCR7DrJjyRdImnyQdbTzMzGoCH/42dE7ImIvwFeB+wB7pD0qcNWMzMzG1WGnHAkzZN0OvDHwFyyQQP/63BVzMzMRpeh3GngLrIhxo8A9wFdwBbgi8D9h7V2ZmY2agxllNp7gYdisNEFZmZmAxhKwtkHzBnCjZcHvZeamZmNXUO9l9pgDuVeamZmNgZU873UzMxsFPETP83MrBBOOGZmVggnHDMzK4QTjpmZFcIJx8zMCuGEY2ZmhXDCMTOzQjjhmJlZIZxwzMysEBVPOJJ+W9Kduel5SZ+UdLGkX+TiZ+S2uVDSdkn3SzotFz89xbZLuqAyLTIzs3KGci+1wyoi7gdOBJBUC/wC+DZwDvCF9NC3HpIWAmcBi4BXAj+Q9Jq0+ovA28meTHq7pM0RcW8hDTEzswFVPOH08TbgwYj4+QB3pz4TuDoifgP8TNJ24I1p3faIeAhA0tWprBOOmVkVqPgptT7OAtpyy+dJukvSlZKmptjxwKO5MjtSrL/4ASStktQpqXPXrl0jV3szM+tX1SQcSUcB7wH+MYUuB15NdrptJ7C2VLTM5jFA/MBgxPqIaIqIpoaGhkOqt5mZDU01nVJ7B3BHRDwBUPoJIOnLwD+nxR3AnNx2s4HH0nx/cTMzq7Cq6eEAy8mdTpM0K7fuvcC2NL8ZOEvSOEnzgQXAj4HbgQWS5qfe0lmpbFVra2tj8eLF1NbWsnjxYtra2gbfyMzsCFQVPRxJR5ONLvvTXPhzkk4kOy32cGldRNwj6RqywQD7gT+LiO60n/OA64Fa4MqIuKewRhyEtrY2Wlpa2LBhA0uWLKGjo4Pm5mYAli9fXuHamZmNLEWUvcwxZjQ1NUVnZ2dFjr148WLWrVvH0qUvP1S1vb2d1atXs23btgG2NDOrLElbI6JpWNs44VQu4dTW1rJnzx7q6+t7Yvv27WP8+PF0d3dXpE5mZkNxMAmnmq7hjDmNjY10dHT0inV0dNDY2FihGpmZHT5OOBXU0tJCc3Mz7e3t7Nu3j/b2dpqbm2lpaal01czMRlxVDBoYq0oDA1avXk1XVxeNjY2sWbPGAwbMbFTyNZwKXsMxMztS+RqOmZlVLSccMzMrhBOOmZkVwgnHzMwK4YRjZmaFcMIxM7NCOOGYmVkhnHDMzKwQTjhmZlYIJxwzMyuEE46ZmRXCCcfMzApRNQlH0sOS7pZ0p6TOFJsm6UZJD6SfU1Ncki6TtF3SXZLekNvPylT+AUkrK9UeMzPrrWoSTrI0Ik7M3YH0AuCmiFgA3JSWAd4BLEjTKuByyBIUcBHwJuCNwEWlJGVmZpVVbQmnrzOBjWl+I7AsF98UmduAYyXNAk4DboyIpyPiGeBG4PSiK21mZgeqpoQTwA2StkpalWIzImInQPp5XIofDzya23ZHivUX70XSKkmdkjp37do1ws0wM7NyqumJn2+OiMckHQfcKOm+AcqqTCwGiPcORKwH1kP2ALaDqayZmQ1P1fRwIuKx9PNJ4Ntk12CeSKfKSD+fTMV3AHNym88GHhsgbmZmFVYVCUfSREmTS/PAqcA2YDNQGmm2Evhumt8MnJ1Gq50MPJdOuV0PnCppahoscGqKmZlZhVXLKbUZwLclQVan1oj4F0m3A9dIagYeAT6Qyl8HnAFsB34NnAMQEU9L+ixweyp3SUQ8XVwzzMysP4oY25cwmpqaorOzs9LVMDM7okjamvsXliGpilNqZmY2+jnhmJlZIZxwjiBtbW0sXryY2tpaFi9eTFtbW6WrZGY2ZNUyaMAG0dbWRktLCxs2bGDJkiV0dHTQ3NwMwPLlyytcOzOzwXnQwBEyaGDx4sWsW7eOpUuX9sTa29tZvXo127Ztq2DNzGwsOphBA044R0jCqa2tZc+ePdTX1/fE9u3bx/jx4+nu7q5gzcxsLPIotVGssbGRjo6OXrGOjg4aGxsrVCMzs+FxwjlCtLS00NzcTHt7O/v27aO9vZ3m5mZaWloqXTUzsyHxoIEjRGlgwOrVq+nq6qKxsZE1a9Z4wICZHTF8DecIuYZjZlZNfA3HzMyqlhOOmZkVwgnHzMwK4YRjZmaFcMIxM7NCOOGYmVkhnHDMzKwQFU84kuZIapfUJekeSZ9I8Ysl/ULSnWk6I7fNhZK2S7pf0mm5+Okptl3SBZVoTyX58QVmVs2q4U4D+4H/GhF3SJoMbJV0Y1r3hYj4m3xhSQuBs4BFwCuBH0h6TVr9ReDtwA7gdkmbI+LeQlpRYX58gZlVu4r3cCJiZ0TckeZfALqA4wfY5Ezg6oj4TUT8DNgOvDFN2yPioYjYC1ydyo4Ja9asYcOGDSxdupT6+nqWLl3Khg0bWLNmTaWrZmYGVEHCyZM0D3g98KMUOk/SXZKulDQ1xY4HHs1ttiPF+ouXO84qSZ2SOnft2jWCLaicrq4ulixZ0iu2ZMkSurq6KlQjM7PeqibhSJoEfBP4ZEQ8D1wOvBo4EdgJrC0VLbN5DBA/MBixPiKaIqKpoaHhkOteDfz4AjOrdlWRcCTVkyWbr0fEtwAi4omI6I6Il4Avk50yg6znMie3+WzgsQHiY4IfX2Bm1a7igwYkCdgAdEXE53PxWRGxMy2+Fyg9R3kz0Crp82SDBhYAPybr4SyQNB/4BdnAghXFtKLy/PgCM6t21dDDeTPwYeA/9RkC/TlJd0u6C1gK/BeAiLgHuAa4F/gX4M9ST2g/cB5wPdnAg2tS2TFj+fLlbNu2je7ubrZt29aTbDxc2syqQcV7OBHRQfnrL9cNsM0a4IDhVxFx3UDbjUUeLm1m1cIPYBvlD2BbvHgx69atY+nSpT2x9vZ2Vq9ezbZt2wbY0sysfwfzADYnnFGecGpra9mzZw/19fU9sX379jF+/Hi6u7srWDMzO5L5iZ92AA+XNrNq4YQzyvU3XHrp0qUeSGBmhar4oAE7vMoNl37nO9/J9773PQ8kMLNC+RrOKL+GU44HEpjZofI1HBuS/u67du+99/o0m5kdNk44Y1C5gQSf+cxnqKurY926dezZs4d169bR0tLipGNmI8YJZwwqN5Dgc5/7HH/xF3/R83iDxx9/nIhgxYoV7u2Y2YjwoIExqNxAgv3793PRRRcBL9+dYP369Zx22mmsW7fOgwrM7JC5hzNG9b3v2sKFC3tOs5Ue5lZXV8fChQtZunQpK1as4JxzzvH1HTM7aE44BvQ+zdbV1cX+/ft7Hm/Q1tZGa2sre/fu5aqrrmL37t2sWLGCOXPmOPGY2dBFxJieTjrppLBMa2trLFq0KICYN29etLa2RkTEokWLYu3atTF79uyYP39+bNmyJW644YaYPn161NfXh6RYtGhRT3kzG/2Azhjm5617ONajdJqttbUVScycOZN9+/bR1dXFunXrANiwYQNLly7l8ccf56mnnuLSSy+lsbGRZcuWsXLlSiQxfvx4ampqfOrNzHobboYabZN7OOWVejs1NTUxbty4aGlpiZqamti7d29ERMybNy/mzZsXe/fuDUkxf/78+NCHPhT19fWxdu3amD59ejQ0NAQQNTU1Qfa476irq3OPyGwUwD0cGyn5QQX/8A//QGtrK3PnzuXmm2+mvb2dhx9+mEsuuYSOjg6OOuooNmzYwE9+8hP279/PrFmzqK+vJyKYMmUKkjjmmGOYMmUKkydPZtKkSTzyyCOsWLECSdTW1iIJSb4uZDaKOeHYoJYvX86aNWvYv38/p556Kh/96Ec5/vjj2bVrF83Nzezdu5clS5Zw33339TzaeuPGjTz11FMce+yxvPTSS0ybNo2pU6fy8Y9/nN27d/Piiy+yfPlyamqyX8EJEyYwefJkduzYwYoVK6ipqelJQocy1dfX+/SeWbUYbpeo2ifgdOB+YDtwwWDlfUpteEqn2iRFfX19tLS0xMKFC2Pt2rU9yzU1NXHDDTf0nE5buHBh1NTUhKRYuHBhAL3mP/ShD0VNTU3U1dXF8uXLQ1LPKbhx48b1WvY0uqehvNe1tbW9ypZ+t/qWq6urizlz5vSKTZw4MVpbW6O1tTUmTpxYdv+zZ8/uKVM6rTx79uxe5evq6uK88847pL+h0n5nz54dNTU1PaeZ839jpd//vqeg8/sY7unpQ9k2j4M4pVbxBDGSE1ALPAi8CjgK+CmwcKBtnHAOXt/ksyv/pswAAAtiSURBVGzZspg/f37MnDkzZsyYEQ0NDVFXVxctLS0913xK13NKSQiIxsbGng+QUhIqfbDU19fHzJkzo7a2tueDpqam5oDl/AeWE9SRP02YMKHXcv49Lk1ve9vbYvLkyb1+X0rv/fjx43u2qa+vj82bN8eqVat6luvr63u2GzduXK/9Hn300TF58uRoaGiILVu2xKZNm+Loo48OIJYvXx7XXXddTJ06NWpqaoaddFpbW3tGem7atClmzZoVM2fOjE2bNsWWLVuioaEhGhoaoqWlJebPnx9r166NefPm9SyXElJpH3v37o0tW7b0rBvO8Ye7bV844fAfgetzyxcCFw60jRPOyMgnn7q6ugBi+vTpMXny5Jg2bVocc8wxMWXKlKipqYmampqeAQYNDQ09HxL5JDTYh42n6pvyH+L5adKkSWXfy9L7PmnSpF7xUvnJkyf3Shr55FL6dl5fXx+1tbUxderUAGLt2rU9ve3SMerr63t+T88999xe+5g5c2bMmzcv1q5d27OPUkKaN29eRETPcc4999xYtGhRRERs2bIlZsyYEePGjRvW38miRYtiy5Ytvea3bNnSs9/SF7N8udL60s/8upL8PoZ6/OFu2xdOOLwf+Epu+cPA35cptwroBDrnzp077BfaBlcuAeU/cOrq6uI973lPTJkypeeDYfny5VFXVxc1NTUxZcqUYfdwSsueKjMN94vBYOX7rh9ouTS/e/fu2L179wH7Knn22WcP2EdNTU3ZbSRFRPTs+9lnn42ampqIiNi7d+8B+x6K/EjP0vzevXt79ls6PZgvV1pf+plfV5Lfx1CPP9xt+8Kj1FCZWBwQiFgfEU0R0dTQ0FBAtcae0ii3l156iX379vX8wnV3d9Pa2srMmTPZvHkzzz33HJKICNra2pgwYQLjxo3j+eefp7u7m5qaGp544gm6u7vp7u4G4KWXXjpgGehZtsqora0tG580aVLPfGmQCFD68sekSZN6xUvlJ06c2BOvr69n4sSJvfbT2NhIfX09tbW1TJkyBYArrriCK664gvr6eoCegSMlF154Ya99HHfcccydO5crrriCqVOn9hyrvr6eE044AaDnOBdeeGHPo9k7OjqYMWMG48aNG9ZrlL9Te2k+/8j3uXPncsIJJ/QqV1pf+nkoj42v+CPnh5uhqnnCp9SOWK2trTF79uyD/rbsaXRNvobjazhVP5Hd/fohYD4vDxpYNNA2TjijR3+n8TwdOZNHqY3uUWqj7hHTks4A/pZsxNqVEbFmoPJj8RHTZmaH6mAeMT3qnocTEdcB11W6HmZm1ttoGzRgZmZVygnHzMwK4YRjZmaFcMIxM7NCjLpRasMlaRfwLPBcLjxlgOX8/HTgqRGqSt9jHkrZ/taXiw8W6++1GMm291ePgy17KO33e1++vdXQ9qGU93s/tPhItP2EiBjef84Pdxz1aJyA9UNd7jM/7HHoQ63DoZTtb325+GCx/l6LkWx7NbXf732/7a142/3eH/lt9ym1zD8NY7nvusNVh0Mp29/6cvHBYoO9NiOlWtrv937w+ZE03P36vR/++qpp+5g/pXYoJHXGMP/xabQYy22Hsd3+sdx2GNvtP9S2u4dzaNZXugIVNJbbDmO7/WO57TC2239IbXcPx8zMCuEejpmZFcIJx8zMCuGEY2ZmhXDCGUGSJkraKOnLkj5U6foUSdKrJG2QdG2l61IJkpal9/27kk6tdH2KJKlR0hWSrpV0bqXrU7T0d79V0rsqXZeiSTpF0v9L7/8pg5V3whmEpCslPSlpW5/46ZLul7Rd0gUp/D7g2oj4E+A9hVd2hA2n7RHxUEQ0V6amh8cw2/+d9L5/BPjDClR3RA2z7V0R8THgg8ARP1x4mH/zAJ8Grim2lofPMNsfwK+A8cCOQXc+Uv8xO1on4K3AG4BtuVgt8CDwKl5+suhCskdan5jKtFa67kW2Pbf+2krXu8LtXwu8odJ1L7rtZF+w/g1YUem6F9l24A+As8i+aLyr0nWvQPtr0voZwNcH27d7OIOIiFuBp/uE3whsj+xb/V7gauBMsgw/O5U54l/bYbZ91BlO+5X5P8D3I+KOous60ob73kfE5oj4PeCIP5U8zLYvBU4GVgB/ImlM/d1HxEtp/TPAuMH2Peqe+FmQ44FHc8s7gDcBlwF/L+mdHL5bYVRa2bZLegWwBni9pAsj4n9XpHaHX3/v/Wqyb7tTJP1WRFxRicodZv2996eQnU4ex+h92m7ZtkfEeQCSPgI8lfsAHm36e+/fB5wGHAv8/WA7ccI5OCoTi4jYDZxTdGUK1l/bfwl8rOjKVEB/7b+M7AvHaNZf228Gbi62KoUr2/aemYiriqtKRfT33n8L+NZQd3LEd/8qZAcwJ7c8G3isQnUp2lhuO4zt9rvtLxtLbYcRar8TzsG5HVggab6ko8guGm6ucJ2KMpbbDmO7/W772Gw7jFD7nXAGIakN+CHw25J2SGqOiP3AecD1QBdwTUTcU8l6Hg5jue0wttvvto/NtsPhbb9v3mlmZoVwD8fMzArhhGNmZoVwwjEzs0I44ZiZWSGccMzMrBBOOGZmVggnHDMzK4QTjpmZFcIJx0YNSb8n6TN9YhMk3SKpNi3/2wgd61hJHx+JfQ1ynPMldUn6+uE+1uEk6ShJt0ryDYPHMN9pwEY1SX8G1EXE343wfucB/xwRi0dyv2WOcx/wjoj4WZ+4yP5+j5jb4Uu6iOyZKkd08rSD5x6OjRqS/lHSkj7hDwHfzZX5VT/b/pGkH0u6U9L/LfWIBnAp8OpU/q8lfVbSJ3L7W5N6J/Mk3Sdpo6S7JF0r6eihHFPSFWRPWNws6b+kfXVJ+hJwBzCnv31IakmPA/6BpDZJn0rbb8vt/1OSLh7oNcgd88uS7pF0g6QJqfzZqU0/lfTVFCv7OqTF7zAKHtBmh6DSjzP15GmkJrKbCk7JLR8FPN6nzK/KbNdI9sC8+rT8JeDsQY41j96P4J0H3JHma8gex/uKFA/gzWndlcCnhnpM4GFgeu4YLwEnD1Rv4CTgbuBo4Bhgezpm3zp/Crh4kH3NA/bz8qPTrwH+CFgE3J+r27SBXoe0XAvsqvTviafKTT6faqOCpPFkH5bP5cLTgWeHsPnbyD6kb8/OVDEBeHI4x4+IhyX9UtLryZ7v/pOI+KWkycCjEfGvqejXgPOBPQd5zJ9HxG2D1Hsa8O2I+DWApKHcRr6/fd0K/Cwi7kzltpIllanAtRHxVGr/0wO9Dmldt6S9kiZHxAtDqJONMk44NlosAu7tE3sRGD+EbQVsjIgLD7EOXwE+Aswk68mU9L1QGodwzN25+bL7kPTJMseErKeSP42ef23629c84De5UDdZMlI/x4D+XwfIHkO9p5/tbJTzNRwbLV4H3JUPRMQzQG3q/QzkJuD9ko4DkDRN0gmDbPMCMLlP7NvA6cB/IHtuSMlcSf8xzS8HOg7ymEOt963Ae9MIvcnAu1P5J4DjJL1C0jjgXUPY10DH/qCkV5TK59aVfR1S2V0RsW+Y7bRRwgnHRosDEk5yA9B3IAEAkq6T9MqIuBf4K+AGSXcBNwKz8mX6bptOE/2rpG2S/jrF9gLtZA+n6s4V7wJWpn1PAy4f6JhD1d8+IuIO4BvAncA3gf+Xyu8DLgF+BPwzcN9g+xrg2PcAa4BbJP0U+HxuXX+vw1LguuG00UYXD4u2US1dS/jziPhwAceqIRs99oGIeCDF5lHA8OlB6nUx2WCJvynoeAe8Din+LeDCiLi/iHpY9XEPx0a1iPgJ0D6EYc6HRNJCstFgN+U/ZMea/l4HSUcB33GyGdvcwzEzs0K4h2NmZoVwwjEzs0I44ZiZWSGccMzMrBBOOGZmVggnHDMzK4QTjpmZFcIJx8zMCvH/AdjV8qxrdYQsAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.plot(x, y, \"o\", color = \"black\", markerfacecolor='None')\n",
    "\n",
    "plt.xscale('log')  # log tranfor the x axis (but not the x values, note the tick values)\n",
    "\n",
    "plt.ylabel(\"$|V_i|$\")\n",
    "plt.xlabel(\"$i$ (i.e. type frequency)\")\n",
    "\n",
    "plt.title(\"Brown Corpus Frequency Spectrum\")\n",
    "\n",
    "# try this\n",
    "#plt.loglog()\n",
    "\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The sum of the cardinality of each frequency class equals the vocabulary size: \n",
    "$|V_c| = |V_1| + |V_2| + |V_3| + ... + |V_{max(f)}|$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 72,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "46969\n",
      "46969\n"
     ]
    }
   ],
   "source": [
    "print (sum(frequency_spectrum.values()))\n",
    "print (len(vocabulary))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "- When dealing with datatypes that can be **meaningfully** ordered (e.g. age, weekdays, length), the **CUMULATIVE FREQUENCY** for the category $i$ is obtained by summing the absolute frequency of $i$ together with all absolute frequencies of all the events below it:\n",
    "\n",
    "$$f^c(v_i) = f(v_1) + f(v_2) + f(v_3) + ... + f(v_i)$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For instance, let's count how many words of different length are used in some of the translations of the \"Universal Declaration of Human Rights\":"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 75,
   "metadata": {},
   "outputs": [],
   "source": [
    "# let's calculate the frequency of each \n",
    "languages = ['Chickasaw', 'English', 'German_Deutsch', 'Dutch_Nederlands', 'Italian']\n",
    "\n",
    "cfd = nltk.ConditionalFreqDist((lang, len(word)) \n",
    "                               for lang in languages\n",
    "                               for word in nltk.corpus.udhr.words(lang + '-Latin1'))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "It is easy to see how the contingency table reporting the cumulative frequencies gives us a different information than the one reporting the absolute frequencies:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 76,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "                   1   2   3   4   5   6   7   8   9  10  11  12  13  14  15 \n",
      "       Chickasaw 411  99  41  68  91  89  77  70  49  33  16  28  45  10   6 \n",
      "Dutch_Nederlands 173 321 287 141 166 125 118  86  79  56  66  26  20  13   9 \n",
      "         English 185 340 358 114 169 117 157 118  80  63  50  12  11   6   1 \n",
      "  German_Deutsch 171  92 351 103 177 119  97 103  62  58  53  32  27  29  15 \n",
      "         Italian 336 265 142 119 166 129 170 107 109  68  39  33  15  16   9 \n"
     ]
    }
   ],
   "source": [
    "cfd.tabulate(cumulative = False, samples = range(1, 16))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 77,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "                    1    2    3    4    5    6    7    8    9   10   11   12   13   14   15 \n",
      "       Chickasaw  411  510  551  619  710  799  876  946  995 1028 1044 1072 1117 1127 1133 \n",
      "Dutch_Nederlands  173  494  781  922 1088 1213 1331 1417 1496 1552 1618 1644 1664 1677 1686 \n",
      "         English  185  525  883  997 1166 1283 1440 1558 1638 1701 1751 1763 1774 1780 1781 \n",
      "  German_Deutsch  171  263  614  717  894 1013 1110 1213 1275 1333 1386 1418 1445 1474 1489 \n",
      "         Italian  336  601  743  862 1028 1157 1327 1434 1543 1611 1650 1683 1698 1714 1723 \n"
     ]
    }
   ],
   "source": [
    "cfd.tabulate(cumulative = True, samples = range(1, 16))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Zipf's Law\n",
    "\n",
    "To have a look at the frequency distribution in the Brown corpus we can plot word frequencies as a function of word ranks."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 80,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYUAAAEWCAYAAACJ0YulAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjAsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+17YcXAAAgAElEQVR4nO3df5xcdX3v8dc7v2kSIQlBJQm/GsSsSBX3UtpSa9SWgCZQbYGIViUGUQi1xcuNjddqa+5taUvRFBpDSYlWgkiVCxWNVgM0SpUN8iM0IDEI2UDNGiRgMOTX5/5xvrPMDrO7M7t7dubsvJ+Pxzx25nvOnPmcszPnc77f7znfo4jAzMwMYFSjAzAzs+bhpGBmZt2cFMzMrJuTgpmZdXNSMDOzbk4KZmbWzUnBbIhJ+qSkf2l0HK1O0ock/VTSLyRNS3+PS9Oul/TpRsfYjJwUmpSkn0j6Zfoi/1zS1yTNanRclZS5VNImSbsldUr6sqTXNjq2vlRs3/9OO4lJw/j535T0eymB7EtxlB6XD1cczST9D/ambfC0pG9JevUAlzUWuBL4vYiYFBE709+tQxv1yOOk0NzmR8Qk4JXAT4EVvc0oafSwRdXTZ4A/Bi4FpgKvAm4B3lbvgiSNGdrQ+lXavq8DXg98bDg+VNJE4A3AnanoS2mHVXpcUeU9ktQKv9cr0v9kJrADuL7aTDV8V14OTAAeGtLoWkArfMkKLyL2ADcDbaWydFT1j5Jul7QbmCvpUEmfl9Ql6XFJHy/tSNLrN6Tn75YUktrS6w9IuiU9/6Skm9JynpP0kKT2anFJOh64GFgYEd+JiBci4vmI+GJE/FWap6+Y3ifpu5L+XtLTwCfLylZI2iXpYUlvKfvMn0h6a9nr7qYaSRMk/YuknZKekXSPpJfXsH3/G1hHlhxKy32bpB9KelbSNkmfLJt2TNp+75X0hKSfSVrWyzYaK2mtpH+VNC4VvwX4bkS80Fdcku6QtFzSd4HngePS9rxO0lOStkv6dOmAQNJoSX+b4tkq6eIU55j+tl16faqk76Vtd7+kN1XE8pfpf/NcqukcXjb9tLL3bkv/x/+hrPlmTNl875R0X1/rDRARzwM3ACeWxXpz+v8+C7xP0nhJV0l6Mj2uSmWvAh5Ji3pG0nfSMkLS7F629dsl3Zfi/56kk/qLcaRyUigASb8CnAv8Z8WkdwHLgcnABrKaxKHAccDvAH8EvD/NeyfwpvT8jcDWNE/pdemoFWABcCNwGHAr8A+9hPYWoDMiftBH+H3FBPDrKZYj0rqUlx0O/DnwFUlT+/iMkvemz5oFTAMuAn7Z35skzQTOALaUFe9OsR5GVuv5kKSzK956GnAC2Xb4hKQ5Fcs9hKzW9AJwTkTsTZPOBL5Ww/oAvAe4kOx//DiwBtgPzCar3fwe8IE072Lg7am8HfiDGj8DSTNSTJ8mq/F9FPhXSdPLZnsX2f/uCGBcmgdJRwFfJ/tfTydLrvdFxD3ATuB3y5bxbuALNcQzCTgf+GFZ8VlkB0eHAV8ElgGnps/7NeAU4OMR8SPgNek9h0XEm/v5rJOB1cAHyb43nwNulTS+vzhHIieF5naLpGeAZ8l+WH9TMf3/RcR3I+IgsI8scXwsIp6LiJ8Af0e2U4Fsp19KAr8N/N+y179Dz6SwISJuj4gDZD/gX+slvmnAU70Fn45g+4oJ4MmIWBER+yOitAPfAVwVEfsi4ktkR321NEftSzHNjogDEbExIp7tY/5bJD0HbEuf+eelCRFxR0Q8GBEHI+IBYC0vbq+ST0XELyPifuB+em6nlwHfAH4MvD9ty5IzgNvLXp+TjlBLjyPLpl0fEQ9FxH6ynfUZwEciYndE7AD+HjivtByy7bYtIp4m+x/X6t3A7en/fjAivgV0kCWwkn+OiB+l/9NNvFizOh/494hYm/5nOyOiVBtYk5ZNSuynk9UAevPR9J3fAkwC3lc27e6IuCXF98v0uX8RETsiogv4FD2/W7VaDHwuIr6fvjdryBL5qQNYVuE5KTS3syPiMGA8cAlwp6RXlE3fVvb8cLKjt8fLyh4HZqTndwK/nd4/GvgS8FuSjiE7ui6v0v932fPngQmq3oa7k6y/ozf9xVS5DiXbo+dIjY8DR1aZr9IXyJqBbkzNCVco63DszdkRMZmsBvXqFC8Akn5d0nplzV67yGodh1e8v3I7lXdUnwqcBPxV+boo64B/NiLK1/umiDis7PFk2bTy+Y4GxgJPlRII2VHtEWn6kRXzl2/3/hwN/GF5ciKrCZX/f3tb31lkya+afwHmpyP/c4D/iIheDySAv03b4BURsSAiypdb+V05kpd+t2r5nlQ6GrisYt1nDXBZheekUADp6OUrwAGyH2r3pLLnPyM7Uj66rOwoYHtaxhayH/KlwF0R8RzZj/xCsprBwQGE9m1gpnrpc+gvpirrUDJDkireU9pR7gZ+pWxad5JMR6mfiog24DfJmlL+qL+ViIg7yTo0/7as+AayprNZEXEosBLQS9/dq2+SHal/u6Jfo56mI+i5fbaRHcEeXpZAXhYRpaaSp8h2ZiVHVSyr122Xlv2FiuQ0sdQ31I9twK9WDT5iO3A38PtkR/H9Nh31ofK78iQv/W49Sf22Acsr1v1XImLtQAMtMieFAlDmLGAKsLnaPKl54iZguaTJko4G/pTsSK3kTlKNI72+o+J1XSLiUeAaYK2kN0kap6yz9zxJS2uMqZojgEtTJ+0fAnN4sbnlPuC8NK1Hu7mkuZJem5qtniVLSAeozVXA70oqNYlMBp6OiD2STiFrT69LOovoBrLEUKplvI2eTUf1LO8psmTzd5JeJmmUpF+VVGrWuolsu82UNAVYWrGIXrcdLx7Rn66sw3pC+p/OrCG0LwJvlXSOpDHKrgl4Xdn0zwOXA68Fvlr/mvdqLfBxSdPT9v0E/X+3qrkWuCjVDiVporITDSYPYayF4aTQ3G6T9AuyHdxy4L0R0dcpdkvIjga3knU830DWgVZyJ9nO7q5eXg/EpWQd0VcDz5A1I/w+cFuNMVXzfeB4sprGcuAPImJnmva/yY5Kf07WhlzePv0Kso7IZ8mS553UuJNIbdKfT8sH+DDwF6nP4RNkO9y6RcRfknU2/3tKinOA7w1kWckfkTXJ/RfZNriZF5t4riVrPrsfuBf4SsV7e912qTnrLODPgC6yo+f/SQ37iIh4gqwGdBnwNFnyKe9f+SrZEf1XI2J3zWvav0+T9Xs8ADxIts51X5AWER1k/Qr/QLZtttCzL6OlKHyTHWsikt4HfCAiTutv3qKRdA5ZgjtnmD7vGOAxYGzqqG4YST8GPhgR/97IOKx/rimYDZ9nyM4WaimS3knWH/CdRsdi/RvuK0jNWlZEfLPRMQw3SXeQXXT5ngGezGDDzM1HZmbWrWmaj9KZDv8haaXKLq83M7Phk2vzkaTVZOeK74iIE8vK55ENpDYa+Kd0LnQAvyAbxKqzluUffvjhccwxxwx12GZmI9rGjRt/FhHTq03LtflI0hvJdvSfLyWFdA75j8iGbegE7gEWAg9HxMF0oc+VEXF+f8tvb2+Pjo6O3OI3MxuJJG2MiKoXnebafBQRd5Gdt1zuFGBLRGxNA4TdCJxV1gn1c7JhHaqSdKGkDkkdXV1ducRtZtaqGtGnMIOeY5h0kg1r8A5JnyO7DL63UTmJiFUR0R4R7dOnV639mJnZADXilNRq48dEGtun8grM6guQ5gPzZ8+uOjS6mZkNUCNqCp30HLRrJnUOYhURt0XEhYceeuiQBmZm1uoakRTuAY6XdKyyO1GdRzYaZc0kzZe0ateuXbkEaGbWqnJNCpLWkg2be4KyG7ovSmOwXEI2cNdmsrHk67qP6mBqCmvXruXEE09k9OjRnHjiiaxd25Kj45qZVZVrn0JELOyl/HYGOHwwDLxPYe3atSxbtozrrruO0047jQ0bNrBo0SIAFi6sGqqZWUsp9DAX9V6ncOKJJ7JixQrmzp3bXbZ+/XqWLFnCpk2b8gjRzKzp9HWdQkslhdGjR7Nnzx7Gjn3xDo379u1jwoQJHDhQ671YzMyKrWEXr+VloB3Nc+bMYcOGDT3KNmzYwJw5c4YyPDOzwipkUhhoR/OyZctYtGgR69evZ9++faxfv55FixaxbNmynCI1MyuWlrqfQqkzecmSJWzevJk5c+awfPlydzKbmSWF7FMoO/to8aOPPtrocMzMCmXE9Sn4imYzs3wUMimYmVk+Wi4p+IpmM7PeFbKj2Vc0m5nlo5AdzSW+otnMrH6+ojnxFc1mZiPw7KOB8hXNZmZ9K2SfwkAtW7aMc889l4kTJ/LEE09w1FFHsXv3bj7zmc80OjQzs6ZQyJrCYG6ys2fPHrZv387BgwfZvn07e/bsySFCM7NiKmRSGOjFa5dffjmTJk1i3bp17N27l3Xr1jFp0iQuv/zynCI1MyuWQiaFgers7GTNmjXMnTuXsWPHMnfuXNasWUNnZ2ejQzMzawotlRQgOwW1/OK19evXNzokM7Om0VJJYerUqVxxxRVccMEFPPfcc1xwwQVcccUVTJ06tdGhmZk1hZa6TmHWrFns3LmT/fv3s2/fPsaOHcuYMWOYNm0a27ZtyzFSM7Pm4esUku3btzNp0iRmzJiBJGbMmMGkSZPYvn17o0MzM2sKLZUUxo0bx9KlS3nsscc4ePAgjz32GEuXLmXcuHGNDs3MrCkUMikM9DqFvXv3smLFih6341yxYgV79+7NKVIzs2IpZFIY6HUKbW1tnH/++SxZsoQJEyawZMkSzj//fNra2nKK1MysWAqZFAZq2bJl3HDDDaxYsYI9e/awYsUKbrjhBpYtW9bo0MzMmkJLjX1UumfCkiVL2Lx5M3PmzGH58uW+l4KZWdJSp6SamZlPSTUzsxo5KZiZWTcnBTMz6+akYGZm3ZoqKUiaKGmjpLfn9RmlaxQkdV+rYGZmmVyTgqTVknZI2lRRPk/SI5K2SFpaNul/ATflFc+SJUu45pprmDJlCqNGjWLKlClcc801TgxmZkneNYXrgXnlBZJGA1cDZwBtwEJJbZLeCvwX8NO8glm5ciWHHHIIEyZMAGDChAkccsghrFy5Mq+PNDMrlFyTQkTcBTxdUXwKsCUitkbEXuBG4CxgLnAq8C5gsaSqsUm6UFKHpI6urq664tm/fz+TJ09m9erV7Nmzh9WrVzN58mT2799f76qZmY1IjehTmAGU37ygE5gREcsi4iPADcC1EXGw2psjYlVEtEdE+/Tp0+v+8AULFvS4HeeCBQsGsg5mZiNSI4a5UJWy7suqI+L6fhcgzQfmz549u+4Pv/baaznhhBO46KKLWLlyJddee23dyzAzG6kaUVPoBGaVvZ4JPFnPAgY6SurMmTMZNWoUl112GRMnTuSyyy5j1KhRzJw5s67lmJmNVI1ICvcAx0s6VtI44Dzg1noWMND7KbS1tXHgwIEeZQcOHPDQ2WZmSa4D4klaC7wJOJzsrKI/j4jrJJ0JXAWMBlZHxPKBLL/eAfFGjx7NwYMv7aoYNWrUS5KFmdlI1bAB8SJiYUS8MiLGRsTMiLguld8eEa+KiF8dSEIYaE2hlBAWLFhAV1dXdydztURhZtaKWmrobEm8+tWvZvPmzd1lc+bM4eGHH6bI28HMrB4eOrvMww8/zJVXXsnzzz/PlVdeycMPP9zokMzMmkYhawplp6QufvTRR+t5X6/TirgdzMwGYsTVFAZ6SurEiRPrKjczazWFTAoDtXv3bkaN6rnKo0aNYvfu3Q2KyMysubRUUgCYN28e48ePB2D8+PHMmzevn3eYmbWOluxTGDNmDPv37+/+C+5TMLPW4T6FCqUL1XzBmplZT4VMCoNVqhW4dmBm1lMhk8JAr2gGmDZtWnczkiSmTZs21OGZmRVWIZPCYJqPdu7cyaRJkxg1ahSTJk1i586dOURoZlZMhUwKg7V7924OHjzoU1HNzCq0ZFIoDYDngfDMzHpqyaRgZmbVtWRSqDYG0tq1axsQiZlZcylkUhjM2UdQ/VTUxYsXDzYsM7PCK2RSGOzFayVz5szhyCOPBHCns5kZMKbRAQwnSUQEEydOZN++fT1utmNmZgWtKQzUxRdfDGS1gr179zY4GjOz5tNSSWHFihWNDsHMrKm1VFIwM7O+OSmYmVm3QiaFwZ6SWlJ5FzYzs1ZXyL3iUJ2S6mEuzMx6KmRSMDOzfDgpmJlZNycFMzPr5qRgZmbdnBTMzKybk4KZmXVzUjAzs25NkxQkzZG0UtLNkj7U6HjMzFpRrklB0mpJOyRtqiifJ+kRSVskLQWIiM0RcRFwDtCeZ1xmZlZd3jWF64F55QWSRgNXA2cAbcBCSW1p2gJgA/DtnOMyM7Mqck0KEXEX8HRF8SnAlojYGhF7gRuBs9L8t0bEbwLn97ZMSRdK6pDU0dXVlVfoZmYtqRF3XpsBbCt73Qn8uqQ3Ae8AxgO39/bmiFgFrAJob29/6c2WzcxswPpNCpKmRkTl0f5gqEpZRMQdwB01LUCaD8yfPXv2EIZlZma1NB99X9KXJZ0pqdoOvV6dwKyy1zOBJ+tZwFCNkmpmZj3VkhReRdZc8x5gi6T/I+lVg/jMe4DjJR0raRxwHnBrPQsYqvspmJlZT/0mhch8KyIWAh8A3gv8QNKdkn6jr/dKWgvcDZwgqVPSoojYD1wCrAM2AzdFxEP1BO2agplZPmrpU5gGvJuspvBTYAnZkf3rgC8Dx/b23pRIqpXfTh+dyWZm1hi1nH10N/AF4OyI6Cwr75C0Mp+w+uaOZjOzfCii77M6JSn6m6lB2tvbo6Ojo6739NVX3qSraWY2pCRtjIiqI0fU0tH8TUmHlS1siqR1QxbdALij2cwsH7UkhekR8UzpRUT8HDgiv5D6545mM7N81JIUDkg6qvRC0tGA21nMzEagWjqalwEbJN2ZXr8RuDC/kBrnpJNO4oEHHmh0GGZmDdNvUoiIb0g6GTiVbIiKP4mIn+UeWR/yOvvowQcfHNLlmZkVTa2jpI4nG+10F9Am6Y35hdQ/9ymYmeWjlovX/ho4F3gIOJiKA7grx7jMzKwBaulTOBs4ISJeyDuY4TB+/HheeGFErIqZ2ZCrpfloKzA270DqMZjrFPbs2ZNDRGZmI0MtNYXngfskfRvoPsSOiEtzi6ofEXEbcFt7e/viRsVgZjYS1ZIUbqXOoa3NzKyYajkldY2kQ4CjIuKRYYjJzMwapN8+hXRNwH3AN9Lr10lyzcHMbASqpaP5k8ApwDMAEXEffdxDYTjkOSDe0Nxx1MysmGpJCvsjonLv29Cxj3zxmplZPmrpaN4k6V3AaEnHA5cC38s3LDMza4RaagpLgNeQnY66FngW+EieQZmZWWPUcvbR82QjpS7LP5zhMWbMGPbv39/oMMzMmk4tYx+tp0ofQkS8OZeIhsG+ffvcoWxmVkUtfQofLXs+AXgnMKIPsyX5fs1m1pJqaT7aWFH03bIb7jREXvdTMDNrdbVcvDa17HG4pNOBVwxDbL3yKalmZvmopfloI1mfgsiajR4DFuUZlJmZNUYtzUcNvXq5UcaOHcu+ffsaHYaZ2bCq5eyjd/Q1PSK+MnThDJ+I6PMMJJ+yamatqJbmo0XAbwLfSa/nAneQ3a85gEImBTMze6lakkIAbRHxFICkVwJXR8T7c43MzMyGXS3DXBxTSgjJT4FX5RRPU/EFbmbWamqpKdwhaR3ZuEcBnAeszzWqYdJfv4KZWaup5eyjSyT9PvDGVLQqIr6aRzCSzgbeBhxB1kT1zTw+px6+utnMWkktzUcA9wJfi4g/AdZJmlzrB0haLWmHpE0V5fMkPSJpi6SlABFxS0QsBt4HnFvrZ5iZ2dCo5YrmxcDNwOdS0Qzgljo+43pgXsUyRwNXA2cAbcBCSW1ls3w8Tc/dmDG1tKCZmbWGWmoKFwO/RXYfBSLiUbLmnZpExF3A0xXFpwBbImJrROwFbgTOUuavga9HxL21fsZg1HKBmvsdzKxV1JIUXkg7bgAkjWHwt+OcAWwre92ZypYAbwX+QNJF1d4o6UJJHZI6urq6BhmGmZmVq6Xt5E5JfwYcIul3gQ8Dtw3yc6sdekdEfBb4bF9vjIhVkp4C5o8bN+4Ng4yjtMx+awPucDazVlBLTWEp0AU8CHwQuJ2szX8wOoFZZa9nAk/W+maPkmpmlo8+awqpQ3hNRLwbuHYIP/ce4HhJxwLbya59eNcQLr9us2bNYtu2bX3O49qCmY10fdYUIuIAMF3SuIF+gKS1wN3ACZI6JS2KiP3AJcA6YDNwU0Q8VMcy50tatWvXroGG9RJPPPFETfOdfvrpQ/aZZmbNRv0d+Ur6HHAycCuwu1QeEVfmG1r/2tvbo6OjY8iWV+tZRq4tmFmRSdoYEe3VpvVaU5D0hfT0XODf0ryTyx4Nk0dNAWrf2fsUVTMbqfrqU3iDpKOBJ4AVwxRPTSLiNuC29vb2xUO97Ne+9rU8+OCD/c7n/gUzG4n6SgorgW8AxwLlbTQiu07huBzjapgHHnjANQEza1m9Nh9FxGcjYg7wzxFxXNnj2IhoaELIq/moxM1IZtaq+r1OISI+NByB1KOZrlNwYjCzkaTWUVJbTj39BU4MZjZSFDIp5N18VOLEYGatppBJoZmaj8o5MZhZ0RUyKQynek87dWIwsyJzUqiBE4OZtYpCJoXh6lMo58RgZq2gkEmhUX0KTgxmNtIVMik0khODmY1kTgoD4MRgZiOVk8IAOTGY2UhUyKTQiI7mapwYzGykKWRSaKaL15wYzGwkKWRSaDZODGY2UjgpDJGBJAYnBzNrNk4KQ2ggd2JzcjCzZuKkMMQGeotOJwYzawZOCjkYTGJwcjCzRipkUmiWU1L7MtDEAK41mFnjFDIpNNMpqX0ZbGJwcjCz4VbIpFAkg0kM4ORgZsPLSWEYRAQRwSWXXDLgZTgxmNlwcFIYRitWrBh0k9KECROGMCIzs56cFBpgMInhhRdecJOSmeVmTKMDaFWlxDCYnXv5ewfbd2FmBq4pNNxQ7cxdczCzoeCk0ARKHdGD5WYlMxuspkkKko6TdJ2kmxsdS6O41mBmjZZrUpC0WtIOSZsqyudJekTSFklLASJia0QsyjOeIhjKxOCag5nVK++awvXAvPICSaOBq4EzgDZgoaS2nOMolFJzkmsOZjbcck0KEXEX8HRF8SnAllQz2AvcCJyVZxxFNtT9Da49mFlfGtGnMAPYVva6E5ghaZqklcDrJX2stzdLulBSh6SOrq6uvGNtGkNZcwDXHsysukZcp1BtbxQRsRO4qL83R8QqYBVAe3t7y52cHxFDtkOvXI6vdTCzRtQUOoFZZa9nAk/Ws4AiDJ2dp6GuNZS49mBmjUgK9wDHSzpW0jjgPODWehZQlKGz85ZXYqj2MLPWkPcpqWuBu4ETJHVKWhQR+4FLgHXAZuCmiHiozuW2dE2h3FCfqdQbJwaz1qAityO3t7dHR0dHo8NoKnnuvIv8XTGzF0naGBHt1aY1zRXN9XBNoXflNYeh3on31rTkJiazkaOQScF9CrUbzqN7Jwaz4vPQ2S2gMjF4521mvSlkTcHNR4OTZ+2hvyYmNzeZNbdCJgU3Hw1eZd/DcJzBVMmJwaz5FDIpmJlZPgqZFNx8lB+fdmrW2gqZFNx8lK/empYacZqr+yTMhlchk4IZuE/CLA9OClY3NzGZjVyFTAruU2i8/pqYGnVGk5kNTiGTgvsUzMzy4SuardCGo1/BtR1rJYWsKVixFH2n6g5tayWuKdiwGExi8E7ZbPgUsqbgjmYzs3wUMim4o9nMLB+FTApmZpYPJwVreo3uqG7055sNJ3c0WyF4x2w2PJwUzMqMlDOdnERtoNx8ZJaMlIQAI2tdbHg5KZiZWbdCJgVfp2Bmlo9CJgVfp2Bmlo9CJgUzM8uHk4JZMpLO2BlJ62LDy6ekmpXxztRanWsKZmbWzUnBzMy6OSmYmVk3JwUzM+vWNB3NkiYC1wB7gTsi4osNDsmsEDykhQ3lCRK51hQkrZa0Q9KmivJ5kh6RtEXS0lT8DuDmiFgMLMgzLrORwgnBYGi/B3k3H10PzCsvkDQauBo4A2gDFkpqA2YC29JsB3KOy8zMqsg1KUTEXcDTFcWnAFsiYmtE7AVuBM4COskSQ59xSbpQUoekjq6urjzCNjNrWY3oaJ7BizUCyJLBDOArwDsl/SNwW29vjohVEdEeEe3Tp0/PN1IzsxbTiI7mao1fERG7gffXtABpPjB/9uzZQxqYmVmra0RNoROYVfZ6JvBkPQvwKKlmGQ/LYTC034NG1BTuAY6XdCywHTgPeFc9C3BNwexFTgw2lPI+JXUtcDdwgqROSYsiYj9wCbAO2AzcFBEP1bNc1xTMzPKRa00hIhb2Un47cPtAl+uagplZPgo5zIVrCmZm+ShkUjAzs3wUMilImi9p1a5duxodipnZiKIin7kgqQt4fIBvPxz42RCGU0TeBt4G4G3Qiut/dERUvfq30ElhMCR1RER7o+NoJG8DbwPwNmj19a9UyOYjMzPLh5OCmZl1a+WksKrRATQBbwNvA/A2aPX176Fl+xTMzOylWrmmYGZmFZwUzMysW0smhV7uEV1I1e6DLWmqpG9JejT9nZLKJemzab0fkHRy2Xvem+Z/VNJ7y8rfIOnB9J7PqglvCixplqT1kjZLekjSH6fyltkOkiZI+oGk+9M2+FQqP1bS99P6fEnSuFQ+Pr3ekqYfU7asj6XyRySdXlbe9L8bSaMl/VDSv6XXLbX+QyIiWuoBjAZ+DBwHjAPuB9oaHdcg1ueNwMnAprKyK4Cl6flS4K/T8zOBr5Pd6OhU4PupfCqwNf2dkp5PSdN+APxGes/XgTMavc5VtsErgZPT88nAj8ju/90y2yHFNSk9Hwt8P63bTcB5qXwl8KH0/MPAyvT8POBL6Xlb+k2MB45Nv5XRRfndAH8K3AD8W3rdUus/FI9WrCn0do/oQorq98E+C1iTnq8Bzi4r/3xk/hM4TNIrgdOBb0XE0xHxc+BbwLw07WURcXdkv5jPly2raUTEUxFxb3r+HNmQ7DNooe2Q1uUX6eXY9AjgzcDNqbxyG5S2zc3AW1Lt5yzgxoh4ISIeA7GB9qoAAANxSURBVLaQ/Waa/ncjaSbwNuCf0mvRQus/VFoxKfR2j+iR5OUR8RRkO0zgiFTe27r3Vd5ZpbxppWaA15MdKbfUdkhNJ/cBO8gS2o+BZyK7hwn0jLt7XdP0XcA06t82zeQq4HLgYHo9jdZa/yHRikmh6j2ihz2Kxuht3estb0qSJgH/CnwkIp7ta9YqZYXfDhFxICJeR3aL21OAOdVmS39H1DaQ9HZgR0RsLC+uMuuIXP+h1IpJYdD3iC6An6YmD9LfHam8t3Xvq3xmlfKmI2ksWUL4YkR8JRW33HYAiIhngDvI+hQOk1S6mVZ53N3rmqYfStYMWe+2aRa/BSyQ9BOypp03k9UcWmX9h06jOzWG+0F2t7mtZJ1IpQ6j1zQ6rkGu0zH07Gj+G3p2sF6Rnr+Nnh2sP0jlU4HHyDpXp6TnU9O0e9K8pQ7WMxu9vlXWX2Tt/FdVlLfMdgCmA4el54cA/wG8HfgyPTtaP5yeX0zPjtab0vPX0LOjdStZJ2thfjfAm3ixo7nl1n/Q26/RATToS3Mm2RkqPwaWNTqeQa7LWuApYB/Z0cwisrbRbwOPpr+lHZuAq9N6Pwi0ly3nArJOtS3A+8vK24FN6T3/QLoKvpkewGlkVfkHgPvS48xW2g7AScAP0zbYBHwilR9HdubUlrSDHJ/KJ6TXW9L048qWtSyt5yOUnWVVlN9NRVJoufUf7MPDXJiZWbdW7FMwM7NeOCmYmVk3JwUzM+vmpGBmZt2cFMzMrJuTgtkwkvRJSR9tdBxmvXFSMBugNAS3f0M2ovgLbVYHScek+zZcA9wLXCepo/weBmm+n0j6lKR7030YXl1lWYslfV3SIcO5DmZ9cVIwq98JZENvvx64LCLaya4o/h1JJ5XN97OIOBn4R6BHk5GkS4D5wNkR8cthitusX04KZvV7PLL7MACcI+lesiEmXkN2k5aS0sB8G8nGpyp5D3AG8M6IeCHnWM3q4qRgVr/dkN3qkawG8JaIOAn4GtmYOiWlHf4BsgHVSjaRJYnykVfNmoKTgtnAvYwsQeyS9HKyo/9a/BD4IHCrpCPzCs5sIJwUzAYoIu4n28E/BKwGvlvHezeQ1TK+JunwfCI0q59HSTUzs26uKZiZWTcnBTMz6+akYGZm3ZwUzMysm5OCmZl1c1IwM7NuTgpmZtbt/wNt4BtXrNOHegAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "y = sorted(fdist.values(), reverse = True)\n",
    "x = range(1, len(y) + 1)\n",
    "\n",
    "plt.plot(x, y, \"o\", color = \"black\", markerfacecolor='None')\n",
    "plt.yscale('log')  # log tranfor the y axis (but not the x values, note the tick values)\n",
    "\n",
    "plt.ylabel(\"frequency\")\n",
    "plt.xlabel(\"rank\")\n",
    "\n",
    "plt.title(\"Brown Corpus Rank/Frequency Profile\")\n",
    "\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### [Zipf's Law] The FREQUENCY of a word is inversely proportion to its RANK"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "$$f(z) = \\dfrac{C}{z^a} $$\n",
    "\n",
    "\n",
    "- $f(z)$: frequency of the rank-$z$ word\n",
    "\n",
    "\n",
    "- $C$ is the frequency of the top-frequent word (it depends on the corpus length and on the vocabulary size)\n",
    "\n",
    "\n",
    "- $a$ is an index that is inversely proportional to the corpus richness (the richer the corpus, the lower its $a$ value)\n",
    "\n",
    "    - according to Zipf, $a \\approx 1$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The difference $C/n -C/(n-1)$ between the frequencies of the rank $n$ and the rank $n-1$ words decreases **progressively** as a function of the rank\n",
    "\n",
    "\n",
    "- $a = 1 \\ \\Rightarrow \\ f(1) = C \\ \\Rightarrow\\ f(2) = \\dfrac{C}{2}\\ \\Rightarrow\\ f(3) = \\dfrac{C}{3}\\ ...$ \n",
    "\n",
    "\n",
    "- that is: the rank 1 word should occur twice as frequently as the rank 2 word\n",
    "\n",
    "\n",
    "- words in the lower tail tend to have similar frequencies\n",
    "\n",
    "\n",
    "- Zipf’s curve tail is populated by a lot of very rare words: the **hapax legomena**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Scale Invariance (optional)\n",
    "\n",
    "- Zip’s law is a power law distribution with integer values: $y = ax^{-k}$\n",
    "\n",
    "\n",
    "- one attribute of power laws is their scale invariance: \n",
    "\n",
    "    - scaling the argument by a constant factor $c$ causes only a porportionate scaling of the function itself of a $c^{-k}$ factor\n",
    "    \n",
    "    - $f(cx) = a(cx)^{-k} = c^{-k}f(x) \\propto f(x)$\n",
    "    \n",
    "\n",
    "\n",
    "- as a consequence, **a change in corpus size does not affect the shape of the distribution, but only its scale**.\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 87,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAagAAAEYCAYAAAAJeGK1AAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjAsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+17YcXAAAgAElEQVR4nO3de7wdZX33/c937+wkSBASEg7uAAEVmoO5EfJQDxEJ3pWDB/JoqwS1YiMomogV9cbS575JX3eK2taqAaRgIoc2AaSWoqBALYeiiCSIkJgiMQazASEkQEgCYSf79/wxs8La232YvdbMOu3v+/Varz1zrTn8Zu35rWvmmlnXKCIwMzNrNG31DsDMzKw/rqDMzKwhuYIyM7OG5ArKzMwakisoMzNrSK6gzMysIbmCMjNrAJLWSDqh3nE0EldQTULSBkkvStom6VlJN0s6pN5xmTUrSadLuk/SdklPp8OfkqR6xBMR0yPiznqsu1G5gmou74mIccDBwFPAkv4mktRe06jMmoyk84BvAH8HHAQcCHwSeCswuo6hWRlXUE0oIl4CbgCmAUi6UtK3JN0iaTswR9K+kq6WtEnSY5L+WlJbOv1jko5Nhz8sKSSVlvVxSTemwxdKuj5dzgtpE8Ssumy0WU4k7Qv8DfCpiLghIl6IxC8i4kMRsVPSGEl/L+l3kp6SdJmkvdL5T5DUJem89MzrSUkfK1v+YPNOlPQDSc9J2iLpv8rycoOk/5kOHydppaSt6TK+VvtPqv5cQTUhSa8CPgj8rKz4DGAxsA9wD8nZ1b7AEcDbgT8HSkl0F3BCOnw8sD6dpjR+V9ly3wtcC+wH3ARcnOvGmNXem4ExwL8PMs1XgCOBo4HXAZ3A/y57/yCS/OoE5gOXSBqfYd7zgC5gEslZ218B/fU39w3gGxHxauC1wPXD2sIW4Qqqudwo6TlgK/AnJM0TJf8eET+JiB6gm6QC+1J6dLgB+AfgI+m0d/FKhfQ24KKy8bfTu4K6JyJuiYjdwDXA/8h/s8xqaiLwTETsKhVI+ml6VvOipLcDZwF/GRFbIuIF4G+B08uW0Q38TUR0R8QtwDbgqPT61WDzdpM00R+Wzvtf0X+HqN3A6yRNjIhtEfGzfqZpea6gmsvciNiP5OhvAXCXpIPS9zaWTTeRpB39sbKyx0iO5CCpgN6WztsOXAe8VdIUkqPCB8vm+33Z8A5grKRRuWyNWX1sBiaW78cR8ZY0tzaTnNm8CliVVlrPAT8iOevZs4zyCo4kN8al0ww2798B64DbJK2XdP4AMc4nOQv7b0n3S3p3ldvclFxBNaGI2B0R3wN2A7NLxWWTPENyBHZYWdmhwOPp/OtIEuozwN3pUd7vgbNJzph6it0Cs7q6F9gJnDbA+88ALwLTI2K/9LVveoPSUAadN23ROC8ijgDeA3xO0jv6LiQiHo2IecABJE2GN0jae9hb2uRcQTUhJU4DxgNr+76fNsddDyyWtI+kw4DPAf9cNtldpGdh6fidfcbNWlJEPAcsAi6V9KeSxklqk3Q0sDfQA1wB/KOkAwAkdUo6KcOyB51X0rslvS5tCtxKcpC5u+9y0puXJqXLey4t/oPpWp0rqObyfUnbSHbsxcBHI2LNANMuBLaT3ABxD7AcWFb2/l0kN1TcPcC4WcuKiK+SHLR9EXia5Gcb/wT8L+Cn6d91wM8kbQX+Azgq4+IHm/f16fg2kjO5Swf47dPJwJo0378BnJ7evTuiyA8sNDOzRuQzKDMza0iuoMzMrCE1TAWV/jr7v9JfXZ9Q73jMzKy+Cq2gJC1LuwJZ3af8ZEmPSFpX9juAILlwOJbkl9ZmZjaCFXqThKTjSSqdqyNiRlrWDvyapCeELuB+YB7w3xHRI+lA4GsR8aGhlj9x4sSYMmVKUeGb7bFq1apnImLS0FO2DueX1cqA+RURhb6AKcDqsvE3A7eWjX+JpEue0vho4IZBlnc2sBJYeeihh0Zfy5cvj+nTp0dbW1tMnz49li9f/gfTmA0XsDIKzpVGex177LF/8Dk4v6wIA+VXPbqs6aR3tzxdwB9Leh9wEkmnpAN2SBoRlwOXA8yaNavX6d+KFSu44IILWLp0KbNnz+aee+5h/vz5AMybNy/frTAbYVasWMG5557L3nsnHRps376dc889F3B+WTHqcZNEfw8Di4j4XkR8IiI+GBU+tGvx4sUsXbqUOXPm0NHRwZw5c1i6dCmLFy+uLmIz44tf/CKjRo1i2bJlvPTSSyxbtoxRo0bxxS9+sd6hWYuqRwXVBZQ/CXYy8EQeC167di2zZ8/uVTZ79mzWrv2D3oDMbJi6uro488wzWbhwIWPHjmXhwoWceeaZdHX5niYrRj0qqPuB10s6XNJokm7ob8pjwVOnTmXRokXMmDGD9vZ2ZsyYwaJFi5g6dWoeizcb8S699FK2b98OJE18l156aZ0jslZW9G3mK0j6mzoqfQLl/Ei6qF8A3ErS0en1MXB/csMyZ84cLrroIjZv3gzA5s2bueiii5gzZ04eizcb0dra2njhhRdYuHBhr79tbQ3zc0prMU3dF9+sWbNi5cqVe8YPOeQQtmzZQnd3N93d3XR0dNDR0cGECRPYuHHjIEsyG5ykVRExoh533ze/JNHR0UF3d/eestJ4M3+PWP0NlF8tdejT1dVFR0cHnZ2dtLW10dnZSUdHh9vIzXLS3d1Ne3s7AO3t7b0qK7O8tVQFBbBr165Bx82sOsmjjF75a1aUlqugXnzxxV5t5C+++GK9QzJrKaWDPh/8WdFaroIaO3YsS5YsYdy4cSxZsoSxY8fWOyQzM6tAy1VQo0aNGnTczMyaQ0t9e0+ePJknn3ySrVu3ArBhwwba29uZPHlynSMzM7PhaqkzqPHjx7N79+49v8toa2tj9+7djB8/vs6RmdWXpKnps9ZukHRONcsqzy+zIrXUHvbwww/T0dHR6zbYjo4OHn744TpHZpa/4TxvLSLWRsQngQ8AVf2eq6enp9dfs6K0VAUF7PmRbt9hsxZ0JXByeUH6vLVLgFOAacA8SdPS994L3AP8uLZhmlWm5SoogClTprBu3Tr8sDVrZRFxN7ClT/FxwLqIWB8RLwPXAqel098UEW8BBnwYqKSzJa2UtHLTpk1FhW6WyZA3SUiaEBF9k6Chbdiwgde97nX1DsNsSAXk10DPWzsBeB8wBrhloJkHe96aWa1luYvvPkkPAt8BfhjudMssT3nn10DPW7sTuLPKZZvVVJYmviNJjqg+AqyT9LeSjiw2LLMRI+/8Kux5a2a1NmQFlT4y/vaImAd8HPgo8HNJd0l6c+ERmrWwAvKrsOetmdValmtQ+wMfJjnCewpYSLLDHw18Fzi8yADNWlk1+ZU+b+0EYKKkLuD/RMRSSaXnrbUDy/J63ppZrWW5BnUvcA0wNyLKn1uxUtJlxYRlNmJUnF/pWVd/5bcwyI0QZs0iSwV11EAXbiPiKznHYzbSOL/MBpDlJonbJO1XGpE0XtKtBcZkNpI4v8wGkKWCmhQRz5VGIuJZ4IDiQjIbUVoiv2bOnFnvEKwFZamgdks6tDQi6TDAv4Uyy0dL5Jf7u7QiZLkGdQFwj6S70vHjgbPzDkTSVOBcYCLw44j4Vt7rMGtANckvs2Y0ZAUVET+SdAzwJpJfqf9lRDyTZeGSlgHvBp6OiBll5ScD3yC5DfbbEfHliFgLfFJSG3DF8DfFrPlUk19mrS5rZ7FjSDqlfB6YJun4jPNdiXtbNhtKpfll1tKy/FD3K8AHgTVA6QEwAdw91LwRcbekKX2K9/S2nC6/1NvyryLiJuAmSTcDyzNug1nTqia/ai0ikPrr6s+sGFmuQc0l+a3GzpzWWVVvy5LOJm2jP/TQQweazKxZ5J1fZi0jSwW1HugA8kqgqnpb9uMArMXknV9109HR4QeEWq6yVFA7gAcl/ZiyJIqIz1S4zrr0tiwJPynEGlDe+VU3u3btqncI1mKyVFA3kW9vyHt6WwYeJ+lt+Ywcl2/WTPLOL7OWkeU286sk7QUcGhGPDGfh7m3ZbHDV5JdZqxvyNnNJ7wEeBH6Ujh8tKdMRX0TMi4iDI6IjIiZHxNK0/JaIODIiXhsRi6vZALNmVk1+NaK2tqy/XDEbWpa96UKSW8OfA4iIB2nQZ0C9853vrHcIZsN1IU2SX8CQ13F9ndfylKWC2hURz/cpa8i98NZb3Qm0NZ2myS+zWstyk8RqSWcA7ZJeD3wG+GmxYZmNGC2XX75j1vKS5QxqITCd5BbYFcBW4LNFBlUU/wreGlDL5JdZ3rLcxbeDpMflC4oPx2xkacb8ytLlkc+iLA9Z+uK7g37axCPixEIiMhtBnF9mA8tyDerzZcNjgfcDTfuTcR/ZWYNpqfwq197ezu7du+sdhjWxLE18q/oU/aTs4WpmVoVmza8szXw9PT2Dvm82lCxNfBPKRtuAY4GDCouoSm4ft2bSbPk1XM41q0aWJr5VJG3kIml6+C0wv8igasGJYw2iafPLz4eyomVp4mvYX7VXy5WU1Vsr51eJ88wqlaWJ732DvR8R38svnHwM58jOF3Ktnpoxv8plzTVXUlaJLE1884G3AP+Zjs8hebDg8yRNEw2dQEPp6elx8lg9tXR+lXOe2XBlqaACmBYRTwJIOhi4JCI+VmhkVRpu+7iTx+qkKfOr3HByzXlmw5Glq6MppeRJPQUcWVA8uRpuIviCr9VB0+ZXpZxnllWWM6g7Jd1K0k9YkDwB945Co8pRJWdSpfnMaqCp86vELRZWhCHPoCJiAXAZ8D+Ao4HLI2Jh0YHlacGCBcOex0d5Vgu1yi9JR0haKumGvJddUkmLRell1p8sZ1AADwAvRMR/SHqVpH0i4oUiA8vTkiVLALj44ouHNZ/PpqxGKsovScuAdwNPR8SMsvKTgW8A7cC3I+LLEbEemF9kBQWV/zbKuWb9yfLI97OAG4B/Sos6gRuLDKoIS5YsqXjn91GeFaXK/LoSOLnP8tqBS4BTgGnAPEnTcgk2o2oqGeealctyk8SngbeSPKeGiHgUOKDIoIqUR/I4gSxHFedXRNwNbOlTfBywLiLWR8TLwLXAaVmDkXS2pJWSVm7atCnrbP3FVvG8aRxVzW+tIUsFtTPd0QGQNIomfyR1Hs0IrqgsJ3nnVyewsWy8C+iUtL+ky4A3SvrSQDNHxOURMSsiZk2aNKmKMFxJWfWyXIO6S9JfAXtJ+hPgU8D38w5E0hEkD23bNyL+NO/l95VXP2Lly3D7uVUg7/zqb6eOiNgMfLKK5Vak2jzrO69zbGTJcgZ1PrAJeBj4BHAL8NdZFi5pmaSnJa3uU36ypEckrZN0PkDaJFHTTjIjItcd3mdVVoGK82sAXcAhZeOTgSeqWF7VIoJRo7LejzU459fIMuhek15wvSoiPgxcUcHyrwQuBq7us8xLgD8hSab7Jd0UEb+qYPm5yLtXZt+RZFnkkF/9uR94vaTDgcdJfld1Rk7Lrlh3d/ee4Wpzza0WI8egZ1ARsRuYJGl0JQtv5Iu4/cSa+85eflOFj/ysr2rzS9IK4F7gKEldkuZHxC5gAXArsBa4PiLW5BZ0DvLMtb455nxrLVnOuzeQPOXzJmB7qTAivlbhOvu7iPvHkvYHFpNexI2Ii/qbOSIuBy4HmDVrVu6HT6XEKWIHd3u69WMDFeZXRMwboPwWkqbChlb086TcW0XzG/AMStI16eAHgR+k0+5T9qrUgBdxI+KTEfHagSqnWiod5RW5g/uob+QqML+aiisQG8xgZ1DHSjoM+B2wJMd1NtxF3KEUeVbV11DrcEK3jKLyq+nUstViqBissQxWQV0G/Ag4HFhZVi6S32kcUeE6G/IibhblO3G9znbcbNEyisqvplXP/HJeNaYBm/gi4psRMRX4TkQcUfY6PCIyJU+zXsTNoujmv8EMdmHYTYbNIY/8amWuLAwy3CQREedUuvBmv4ibRSmRxo0bx/bt24eYuvZ8ZNjYqsmvVtd3vy36gKuo5Tv/KpfPr+eMbdu29Rr32YtZvgb6om/0XPNBYuVcQRWk1kd/g8l73U42M6sFV1A10t+XeqMf+Q3ER4TWSIr+PZXVjyuoOhrsS94JZ5Zd1gMm51VzydJZrNWBz1DMbKTzGVQDG04l5SNDs6HVoznQB5uVcwVlFWn1CtFfKq2r6O7LspSNdFn/B27iaxH+Qs2Xv1RsuLzPZJf1s/IZVAsp4nEhZmb14jMoMzNrSK6gzMysIbmCMjOzhqRmvrguaRPw2ABvTwSeqWE49VbU9h5bwDKbwao+44dFxKS6RFInIzi/qtm2kZovw5Upv5q6ghqMpJURMavecdSKt9dqqZU//1bdtmbcLjfxmZlZQ3IFZWZmDamVK6jL6x1AjXl7rZZa+fNv1W1ruu1q2WtQZmbW3Fr5DMrMzJqYKygzM2tILVdBSTpZ0iOS1kk6v97xDIekQyTdIWmtpDWSzk3LJ0i6XdKj6d/xabkkfTPd1ockHVO2rI+m0z8q6aNl5cdKejid55tqgA73JLVL+oWkH6Tjh0u6L439Okmj0/Ix6fi69P0pZcv4Ulr+iKSTysqbdn9oRM3yeUpaJulpSavLyloij0bU90REtMwLaAd+AxwBjAZ+CUyrd1zDiP9g4Jh0eB/g18A04KvA+Wn5+cBX0uFTgR8CAt4E3JeWTwDWp3/Hp8Pj0/d+Drw5neeHwCkNsN2fA5YDP0jHrwdOT4cvA85Jhz8FXJYOnw5clw5PS//XY4DD032gvdn3h0Z7NdPnCRwPHAOsLitriTwaSd8TrXYGdRywLiLWR8TLwLXAaXWOKbOIeDIiHkiHXwDWAp0k23BVOtlVwNx0+DTg6kj8DNhP0sHAScDtEbElIp4FbgdOTt97dUTcG8leeHXZsupC0mTgXcC303EBJwI3pJP03d7S53AD8I50+tOAayNiZ0T8FlhHsi809f7QgJrm84yIu4EtfYpbIo9G0vdEq1VQncDGsvGutKzppM1XbwTuAw6MiCch2TmBA9LJBtrewcq7+imvp68DXwR60vH9geciYlc6Xh7jnu1K338+nX64n4NVptk/z5bLo1b/nmi1Cqq/dtKmu49e0jjgX4HPRsTWwSbtpywqKK8LSe8Gno6I8n65Bouxqbe3BbTq59mU+9VI+J5otQqqCzikbHwy8ESdYqmIpA6Sne5fIuJ7afFT6Wk36d+n0/KBtnew8sn9lNfLW4H3StpA0lx0IskZ1X6SSg/TLI9xz3al7+9L0owz3M/BKtPsn2fL5NGI+Z6ox4Wvol4kTwheT3KhvHQRd3q94xpG/CJp7/16n/K/o/fFz6+mw++i98XPn6flE4Dfklz4HJ8OT0jfuz+dtnTx89R6b3ca1wm8cpPEd+l9k8Sn0uFP0/smievT4en0vkliPckF/abeHxrt1WyfJzCF3jdJtEQejaTvibrvRAX8804luavlN8AF9Y5nmLHPJjmVfgh4MH2dSnKd5cfAo+nf0k4k4JJ0Wx8GZpUt6y9IbhZYB3ysrHwWsDqd52LS3kTq/epTQR1BchfROpLKakxaPjYdX5e+f0TZ/Bek2/QIZXccNfP+0IivZvk8gRXAk0A3yRnB/FbJo5H0PeGujszMrCG12jUoMzNrEa6gzMysIbmCMjOzhuQKyszMGpIrKDMza0iuoAwASRdK+ny94zBrRc6vyriCakFp9/r+35oVwPlVO/6QW4SkKenzYS4FHgCWSlqZPi9mUdl0GyQtkvRA+ryXP+pnWWdJ+qGkvWq5DWaNyvlVH66gWstRJN3qvxE4LyJmATOBt0uaWTbdMxFxDPAtoFezg6QFwHuAuRHxYo3iNmsGzq8acwXVWh6L5HkvAB+Q9ADwC5K+6qaVTVfqXHIVSX9lJR8BTgHeHxE7C47VrNk4v2rMFVRr2Q7JI9NJjtzeEREzgZtJ+rErKSXHbpIOQEtWkyRUeU/GZpZwftWYK6jW9GqSZHpe0oEkR21Z/AL4BHCTpNcUFZxZk3N+1YgrqBYUEb8kSYY1wDLgJ8OY9x6So8ObJU0sJkKz5uX8qh33Zm5mZg3JZ1BmZtaQXEGZmVlDcgVlZmYNyRWUmZk1JFdQZmbWkFxBmZlZQ3IFZWZmDckVlJmZNSRXUGZm1pBcQZmZWUNyBWVmZg3JFZSZmTUkV1BWiPTR1/+z3nGYDUXSX0n6doXzDrifSzpBUld10VVO0pWS/m+91p8HV1ANwF/mZpWTNFvSTyU9L2mLpJ9I+n+yzh8RfxsRH0+XNUVSSBo11HxWPP8TbNgkjYqIXfWOw0zSq4EfAOcA1wOjgbfxylNtW14r56PPoBqIpDMl3SPp7yU9K+m3kk4pe/9jktZKekHSekmf6Dtvn+WFpNelw/tL+r6krZLul/R/S9P3d9Qo6U5JHy9b9k8k/aOkLcCFkl4r6T8lbZb0jKR/kbRfwR+RWV9HAkTEiojYHREvRsRtEfGQpMckHQsg6cPpPj4tHf+4pBvT4Qsl/XO6vLvTv89J2ibpzek0Z5Xl3q8kHVMWw9GSHkrP4K6TVP7491IT4jNpS8mHysr3lXS1pE1prH8tqS19b9D8Spf1vyQ9BGyXNErSGyU9kMZ4Hb0fQ9+UXEE1nj8GHgEmAl8FlkpS+t7TwLtJHjn9MeAf+yTKYC4heUz1QcBH09dw41oPHAAsBgRcBLwGmAocAlw4zGWaVevXwG5JV0k6RdL4svfuAk5Ih48n2X/fXjZ+Vz/LOz79u19EjIuIeyX9Gcm+/eckufdeYHPZPB8ATgYOB2YCZ5a9dxBJLneS5Nzlko5K31sC7Asckcb15yR5Ddnyax7wLmA/ku/yG4FrgAnAd4H397N9TcUVVON5LCKuiIjdwFXAwcCBABFxc0T8JhJ3AbeRNGcMSlI7yc76fyJiR0T8Kl32cDwREUsiYld6lLouIm6PiJ0RsQn4Gq8kv1lNRMRWYDYQwBXAJkk3STqQpAIq7ZNvI/nCL42/nf4rqP58HPhqRNyf5t66iHis7P1vRsQTEbEF+D5wdJ/5/780T+4CbgY+kObkB4EvRcQLEbEB+AfgI+l2Zcmvb0bExoh4EXgT0AF8PSK6I+IG4P6M29ewXEE1nt+XBiJiRzo4DiA9QvxZeiH4OeBUkqOzoUwiud64saxs4wDTDqTX9JIOkHStpMclbQX+OWMsZrmKiLURcWZETAZmkJx1fJ2kAnqbpIOAduA64K2SppCcuTyYcRWHAL8Z5P3flw3vIM3X1LMRsb1s/LE0vokk18se6/NeJ2TOr/KcfA3weEREn+U1NVdQTULSGOBfgb8HDoyI/YBbSJoCIGm+e1XZ9AeVzb4J2AVMLis7pGy4lECvKisrnx+SI9RyF6VlMyPi1cCHy2Ixq4uI+G/gSmBGRKwjqTA+A9wdES+QVCZnA/dERE9/i+inbCPw2gpDGi9p77LxQ4EngGeAbuCwPu89ng5nya/yWJ8EOssuB5SW19RcQTWP0cAY0somvXninWXv/xKYLuno9CLthaU30ubC75Hc3PAqSX9E0t5den8TSWJ8WFK7pL9g6ITcB9hGcjG5E/hCtRtoNlyS/kjSeZImp+OHkFyb+Vk6yV3AAl5pzruzz3hfm4AekutCJd8GPi/pWCVeJ+mw/mfv1yJJoyW9jeQa8nfTnLweWCxpn3R5nyM5U4Lh59e9JAehn0lvmHgfcNwwYmxIrqCaRHr09xmSnfpZ4AzgprL3fw38DfAfwKPAPX0WsYCkWeP3JBdSV9D7VtyzSJJgMzAd+OkQIS0CjgGeJ2lX/14Fm2VWrRdIbuC5T9J2koppNXBe+v5dJF/2dw8w3kvarL4Y+Imk5yS9KSK+m5YtT9d3I8mNCFn8niRfnwD+BfhkepYHsJCk9WI9Sb4uB5al7w0rvyLiZeB9JDdoPEtyfavpc1K9myxtpJD0FeCgiBju3XxmZjXhM6gRIm0KmZk2URwHzAf+rd5xmZkNxD1JjBz7kDTrvYbk91T/APx7XSMyMxuEm/jMzKwhuYnPzMwaUlM38U2cODGmTJlS7zBsBFi1atUzETGp3nHUkvPLamXA/IqIpn0de+yx0dfy5ctj+vTp0dbWFtOnT4/ly5f/wTRmwwWsjAbY52v5cn5ZrQyUX019BtXXihUruOCCC1i6dCmzZ8/mnnvuYf78+QDMmzevztGZNTfnl9VaU98kMWvWrFi5cuWe8RkzZjB37lxuvPFG1q5dy9SpU/eMr169uo6RWrOTtCoiZtU7jlrqL7+WLFnCnDlz9pTdcccdLFy40PllVRkov1rqDOpXv/oVO3bs+IMjvA0bNtQ7NLOmt3btWmbPnt2rbPbs2axdu7ZOEVmra6m7+EaPHs2CBQuYM2cOHR0dzJkzhwULFjB69Oh6h2bW9KZOncqiRYuYMWMG7e3tzJgxg0WLFjF16tR6h2YtqqXOoF5++WUuuugilixZwu9+9zsOPfRQtm3bxssvv1zv0Mya3pw5c1i8ePGe8TVr1rBmzRoWLFhQx6islbXUGVRnZyc7duzg8ccfp6enh8cff5wdO3bQ2dlZ79DMmt53vvOdYZWbVaulKqgdO3awc+dOvvzlL7N9+3a+/OUvs3PnTnbs2DH0zGY2qO3bk8eGHXTQQbS1tXHQQQf1KjfLW0tVUFu2bOELX/gCy5YtY5999mHZsmV84QtfYMuWLfUOzawljBkzhrFjxwIwduxYxowZU+eIrJW1VAUFcOKJJ7J69Wp2797N6tWrOfHEE+sdklnL2LlzJ11dXfT09NDV1cXOnTuHnsmsQi11k8TkyZP5sz/7M8aPH7/nJolnn32WyZMnDz2zWQuTNBU4F5gI/DgivlXpskaNGsWuXbv2/DUrSkudQc2dO5etW7eyceNGenp62LhxI1u3bmXu3Ln1Ds0sd5KWSXpa0uo+5SdLekTSOknnA0TE2oj4JPABoKofHL/00ku9/poVpaUqqBtvvJG99tqLtrZks9ra2thrr7248cYb6xyZWSGuBE4uL5DUDlwCnAJMA+ZJmpa+916SR4v/uLZhmlWmpSqorq4uOjo66OzspK2tjc7OTjo6Oujq6qp3aGa5i4i7gb53AB0HrIuI9RHxMnAtcFo6/U0R8RbgQwMtU9LZklZKWha2MIwAABGGSURBVLlp06Z+pxk3blyvv2ZFaakKCqC9vZ1ly5bx0ksvsWzZMtrb2+sdklktdQIby8a7gE5JJ0j6pqR/Am4ZaOaIuDwiZkXErEmT+n+6yIsvvtjrr1lRWuomCUh+k3HSSSfR3d1NR0fHnuY+sxFC/ZRFRNwJ3Fntwvfaay927drF7t27aWtrY/To0a6orDBDfntLmlCLQPKyc+dOdu/eDcDu3bt9G6w1tALyqws4pGx8MvBEXgt/6aWX6O7uBqC7u9s3Slihspxe3Cfpu5JOldTf0VkuJE2VdJmkGySdU82yenp6ev01a2B559f9wOslHS5pNHA6cFMOywWSB5wONm6WpywV1JHA5cBHgHWS/lbSkVkWXq/bYM2aSDX5tQK4FzhKUpek+RGxC1gA3AqsBa6PiDUFxW5WqCErqPSJvLdHxDzg48BHgZ9LukvSm4eY/Up8G6zZgKrJr4iYFxEHR0RHREyOiKVp+S0RcWREvDYiFg+2jEp0dHTQ0dHRq2zFihV5r8Ys0zWo/SWdK2kl8HlgIcmv0c8Dlg82bxG3wZq1kmryqx4k0d3dvec6VIkfuWFFyHIX373ANcDciCj/QdFKSZdVsM7+boP9Y0knAO8DxjDIbbCSzgbOBjj00EMrWL1ZQ8k7vwpVuuYkqdf1J3fIbEXIUkEdFQNcCY2Ir1Swzqpug42Iy0na7Jk1a5av0Fqzyzu/CtPe3r7nDlnfHGG1kOUmidsk7VcakTRe0q1VrLPQ22DNmkze+VWYc86p6uZas2HLUkFNiojnSiMR8SxwQBXrLPQ2WLMmk3d+FWbJkiX1DsFGmCwV1G5Jey72SDoMyHR+30i3wS5cuLDoVZhVouL8Mmt1WSqoC4B7JF0j6RrgbuBLWRZer9tg+3PppZfWYjVmw1VxfjUaHwRa3rL8DupHwDHAdcD1wLER0ZBt5INxrxLWiFolvwAuvvjieodgLSZrZ7FjSH7PNAqYlt5iendxYVXGT/i0JtUU+WVWa0NWUJK+AnwQWAOUTkOCpCmioVx99dWcccYZ9Q7DLLNmyi+zWstyBjWX5LcaDd8t+Lx581xBWbNpmvwCeMMb3sDDDz884Pv7778/mzdvrmFE1sqy3CSxHugYciozq0RT5ddDDz006PvuUcLylOUMagfwoKQfA3uO8iLiM4VFZTZyOL/MBpClgroJ/5DWrCjOL7MBZLnN/CqS219/FhFXlV7Fh5Y/PxLAGk0z5tdQPZc7zywvWR638R7gQeBH6fjRkpryiM83UFijacb8GqrLow99yE/LsXxkuUniQpJnOD0HEBEPAocXGJPZSHIhLZZf7unc8pKlgtoVEc/3KfMeaJYP55fZALJUUKslnQG0S3q9pCXATwuOq2KjRg1+38dJJ51Uo0jMMmmq/Cppaxv8q0Pq77FvZsOTpYJaCEwnuQV2BbAV+GyRQVWj76Oo+7rttttqFIlZJk2VXyWlBxcOxjdLWLXUzO3Fs2bNipUrV/5B+VBHb828zVYfklZFxKx6x1FLA+VXSZazJOeaZTFQfmXpi+8O+mkTj4gTc4qt5tLOOOsdhllL5lc5d31k1cjyQ93Plw2PBd4PuMtws3w0bX69853vHLLJ3F0fWTWGrKAiYlWfop9IuqugeHIREUM2P/gsyhpBM+ZXya233pqpmc+5ZpXK0sQ3oWy0DTgWOKiwiMxGkGbPrywHg+BKyiqTpYlvFUkbuUiaHn4LzC8yqFpx0lgDaPr8yvqg0JkzZw7ZG7pZuSxNfE35q3Yf2VkzaNb8Ktfd3Z0p1wZ7jpRZf7I08b1vsPcj4nv5hVMfrqSsXlolv/bee2+2b98+5HTONRuOLE1884G3AP+Zjs8B7gSeJ2maaNgEynoWBU4cq5umza9y27Ztc65Z7rJUUAFMi4gnASQdDFwSER8rNLKcDPWI6nKlBHPyWA01dX6V8wGh5S1LV0dTSsmTego4sqB4clfJRVn3I2Y11NT51ddwKh1JQ/bpZyNbljOoOyXdStJPWACnA3cUGlXOhnNkV+KzKauRps+vapRy03lm/clyF98CSf8vcHxadHlE/FuxYeWvkkoK3BRhxapVfkk6ArgA2Dci/jTv5Zer9IDQeWZ9ZT2/fgC4OSL+ErhV0j4FxlSYShNAkpv9rEgV5ZekZZKelrS6T/nJkh6RtE7S+QARsT4iavb7qkpyTRLt7e0FRGPNKssj388CbgD+KS3qBG7MOxBJR0haKumGvJddrpqjNFdUlrcq8+tK4OQ+y2sHLgFOAaYB8yRNyyXYYaok13p6epDE2LFjC4jImk2WM6hPA28leU4NEfEocECWhTfqEV61TQmlisoVluWg4vyKiLuBvr2xHgesS/PpZeBa4LSswUg6W9JKSSs3bdqUdbbBYqxovp07dzq/LFMFtTPd0QGQNIrsj6S+kgY9wouI3Nq8XVlZFarJr/50AhvLxruATkn7S7oMeKOkLw00c0RcHhGzImLWpEmTqgij1zKrmt95NXJluYvvLkl/Bewl6U+ATwHfz7LwiLhb0pQ+xXuO8AAklY7wfpU16DxVevPEQHyx14ap4vwaQH87c0TEZuCTVSy3KqWcqDTXSvNJoqenJ7e4rLFlOYM6H9gEPAx8ArgF+Osq1lnVEV7eTRCQ79kU+FqVDUve+dUFHFI2Phl4oorl5araPCsdUJa/xo0bl1N01mgGPYNKm+OuiogPA1fktM6qjvAi4nLgckgeSZ1TTKVl53421d86zKCw/LofeL2kw4HHSX5XdUZOy85FRNDR0ZGpB/Qstm/fzrhx49i2bVsuy7PGMegZVETsBiZJGp3jOhv+CK/ISqTv0Z/PtEauavNL0grgXuAoSV2S5kfELmABcCuwFrg+ItbkFnROuru7c82z7du395tbCxcuzG0dVntZrkFtIHnK503Anu6KI+JrFa6z4Y/woPo28+HwdasRbQMV5ldEzBug/BaSpsKGV3SeXXzxxQAsWbKkkOVbsQY8g5J0TTr4QeAH6bT7lL2G1MxHeCWlM6p6nFn5DKt15ZFfrSQieMMb3lDIsq+4Iq/WU6u1wc6gjpV0GPA7oKLDj1Y4wuurlmdWJT7DaklV51ereeihh5g5c2buDzYs/aYqC+dZYxmsgroM+BFwOLCyrFwkv9M4osC4Gl7eN1QMZTjrcpI1BedXP/o+faDWLQg+GGwsAzbxRcQ3I2Iq8J2IOKLsdXhEjMjk6au86W/ChAn1DmcPNws2PudXNrVqYrfGlKU383NqEUiz27x58x+UuaKwoTi/hqfWLRdWX1nu4rMK9T3ic2KZVW+gM6m88quoPPUZ4PC5gqqhWh795b0eJ5dZdXx9a/hcQdXYYDtoI59hObms0bn5r/W4gmogTjCz6mQ9iHKeNQdXUA1mOGcpTjIza2VZH/luZmZWUz6DssxG0hmbr7e1tno1p4+kHBrIggULMveN6AqqifmaVXF8U0jrK+r/65wc3HA68HUTX5Pr+0v7al9mZkXL2oGvKygzM6upnTt3ZprOFZSZmdXUmDFjMk3nCsrMzGrqrLPOyjSdKyjrxdehEv4crFLedwY3nLv41MwfpqRNwGMDvD0ReKaG4QylkeJppFigseIZKJbDImJSrYOppwbOr3rvL972/PWbX01dQQ1G0sqImFXvOEoaKZ5GigUaK55GiqWR1fNzqvf/yNteu/W7ic/MzBqSKygzM2tIrVxBXV7vAPpopHgaKRZorHgaKZZGVs/Pqd7/I297jbTsNSgzM2turXwGZWZmTcwVlJmZNaSWq6AknSzpEUnrJJ1f4HoOkXSHpLWS1kg6Ny2/UNLjkh5MX6eWzfOlNK5HJJ2UZ8ySNkh6OF3nyrRsgqTbJT2a/h2flkvSN9P1PSTpmLLlfDSd/lFJH60wlqPKtv9BSVslfbaWn42kZZKelrS6rCy3z0PSsennvS6dd0R0YV1UfhX9/xpi3QPlcuHrlzRW0s8l/TJd96K0/HBJ96XLuU7S6LR8TDq+Ln1/Stmy+s2hDDG0S/qFpB/Uet1Dyrs37Hq+gHbgN8ARwGjgl8C0gtZ1MHBMOrwP8GtgGnAh8Pl+pp+WxjMGODyNsz2vmIENwMQ+ZV8Fzk+Hzwe+kg6fCvwQEPAm4L60fAKwPv07Ph0en8P/5PfAYbX8bIDjgWOA1UV8HsDPgTen8/wQOKXe+3/RryLzq+j/1xDrHiiXC19/uoxx6XAHcF+6zOuB09Pyy4Bz0uFPAZelw6cD1w2WQxk/+88By4EfpOM1W/dQr1Y7gzoOWBcR6yPiZeBa4LQiVhQRT0bEA+nwC8BaoHOQWU4Dro2InRHxW2BdGm+RMZ8GXJUOXwXMLSu/OhI/A/aTdDBwEnB7RGyJiGeB24GTq4zhHcBvImKgHglK8eT62UTE3cCWftZT9eeRvvfqiLg3kgy9umxZraywfbXI/1eGdQ+Uy4WvP13GtnS0I30FcCJwwwDrLsV0A/CO9Ox9oBwalKTJwLuAb6fjqtW6s2i1CqoT2Fg23sXglUYu0lPdN5Ic/QAsSE/9l5WaBQaJLa+YA7hN0ipJZ6dlB0bEk5AkIXBAjWIpdzqwomy8Hp9NSV6fR2c6nFdczaLW+VXz/bdPLtdk/WkT24PA0ySV2m+A5yJiVz/L2bOO9P3ngf0rXTfwdeCLQE86vn8N1z2kVqug+rsOUOh99JLGAf8KfDYitgLfAl4LHA08CfzDELHlFfNbI+IY4BTg05KOHyzsgmNJVpK0Xb8X+G5aVK/PZijDXX/N97MG0SjbXcj/pZ9crsn6I2J3RBwNTCY585g6yHJyW7ekdwNPR8Sq8uJarDurVquguoBDysYnA08UtTJJHSQ79L9ExPcAIuKpdIfrAa7glVPdgWLLJeaIeCL9+zTwb+l6n0qbHkj/Pl2LWMqcAjwQEU+lsdXlsymT1+fRlQ7nFVezqGl+UcP9t79cruX6ASLiOeBOkmtQ+0ka1c9y9qwjfX9fkqbRStb9VuC9kjaQNNeeSHJGVYt1Z5PHhaxGeQGjSC5MHs4rF3GnF7QukVx7+Hqf8oPLhv+SpG0WYDq9LySuJ7noXHXMwN7APmXDPyVp+/47el/k/Wo6/C56X+T9eVo+AfgtyQXe8enwhCo+o2uBj9XrswGm0Puie26fB3B/Om3pJolT673/F/0qOr+K/H8Nsd6Bcrnw9QOTgP3S4b2A/wLeTdLqUH6jwqfS4U/T+0aF6wfLoWF89ifwyk0SNV33oHHVe6fP+0Vyh82vSdpxLyhwPbNJTmMfAh5MX6cC1wAPp+U30ftL+YI0rkcou+ur2phJ7qr6ZfpaU1oGSfvwj4FH07+lL1cBl6TrexiYVbasvyC5yLmOssqlgpheBWwG9i0rq9lnQ3Ld60mgm+QIb36enwcwC1idznMxaa8srf4qKr+K/n8Nse6Bcrnw9QMzgV+k614N/O+0/AiSO0XXkVQYY9Lysen4uvT9I4bKoYyfwQm8UkHVdN2DvdzVkZmZNaRWuwZlZmYtwhWUmZk1JFdQZmbWkFxBmZlZQ3IFZWZmDckVlAF7emH/fL3jMGtFzq/KuIJqQenjAPy/NSuA86t2/CG3CElT0ufZXAo8ACyVtLL8GTPpdBskLZL0gJLnGf1RP8s6S9IPJe1Vy20wa1TOr/pwBdVajiJ5DMAbgfMiYhbJL9XfLmlm2XTPRNKx7LeAXs0OkhYA7wHmRsSLNYrbrBk4v2rMFVRreSyS59MAfEDSAyTdqEwneahYSakzzFUk/Z+VfISkg9f3R8TOgmM1azbOrxpzBdVatkPyyGaSI7d3RMRM4GaSfrRKSsmxm6QD0JLVJAlV3lO3mSWcXzXmCqo1vZokmZ6XdCDJUVsWvwA+Adwk6TVFBWfW5JxfNeIKqgVFxC9JkmENsAz4yTDmvYfk6PBmSROLidCseTm/ase9mZuZWUPyGZSZmTUkV1BmZtaQXEGZmVlDcgVlZmYNyRWUmZk1JFdQZmbWkFxBmZlZQ/r/AXh3TFUFREiqAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 4 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "corpora = [nltk.corpus.brown, nltk.corpus.genesis, nltk.corpus.inaugural, nltk.corpus.switchboard]\n",
    "corpora_titles = [\"Brown\",\"Genesis\",\"Inaugural\",\"Switchboard\"]\n",
    "\n",
    "for i, corpus in enumerate(corpora):\n",
    "    plt.subplot(2, 2, i+1)\n",
    "    \n",
    "    y = sorted(Counter([word for word in corpus.words() if word.isalnum()]).values(), reverse = True)\n",
    "    x = range(1, len(y) + 1)\n",
    "    \n",
    "    plt.plot(x, y, \"o\", color = \"black\", markerfacecolor='None')\n",
    "    plt.yscale('log')\n",
    "    \n",
    "    plt.ylabel(\"frequency\")\n",
    "    plt.xlabel(\"rank\")\n",
    "    \n",
    "    plt.title(corpora_titles[i])\n",
    "\n",
    "plt.tight_layout()  # adjust spacing between subplots to minimize the overlaps.\n",
    "plt.show()   "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Log-log Space (optional)\n",
    "\n",
    "If we plot our data so that both axes are on a logarithmic scale, Zipf’s law can be reformulated as a **straight line**:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "$$log(f(z)) = log(C) - a\\ log(z)$$\n",
    "\n",
    "\n",
    "\n",
    "- intercept if the log-frequency of the most frequent word\n",
    "\n",
    "\n",
    "- slope is a function of the corpus richness (i.e. $a$)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 88,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXAAAAD4CAYAAAD1jb0+AAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjAsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+17YcXAAAYZ0lEQVR4nO3df2zc9X3H8df77uyYOKVpEpNCnF+TWmryY7BGU7dGU1M26Mq0uhKrei4/NFlCrYph6yQIzR+VKrmCaEWrzmgUETrS4axV1nqo0CVVk2mLNNGFH01SnLbTACf8CCahBCexfT/e+8M+y3bsxHff7933vnfPh2Q59/X37t4HySuffH6auwsAED+JqAsAAJSHAAeAmCLAASCmCHAAiCkCHABiKlXNN1uxYoWvW7eumm8JALH3/PPPv+PubbOvVzXA161bp8OHD1fzLQEg9szstbmu04UCADFFgANATBHgABBTBDgAxBQBDgAxVfMBvmfPHm3cuFHJZFIbN27Unj17oi4JAGpCVacRlmrPnj3asWOHdu3apa1bt+rQoUPq7u6WJKXT6YirA4BoWTW3k92yZYuXMg9848aNymQy2rZt29S1gwcPqqenR8eOHatEiQBQc8zseXffctH1Wg7wZDKp0dFRNTU1TV3LZrNqaWlRPp+vRIkAUHPmC/Ca7gPv6OjQoUOHZlw7dOiQOjo6IqoIAGpHTQf4jh071N3drYMHDyqbzergwYPq7u7Wjh07oi4NACJX04OYxYHKnp4eDQ4OqqOjQ729vQxgAoBqvA8cABDTPnAAwPwIcACIqYYLcFZ2AqgXNT2IGTZWdgKoJw01iMnKTgBxFMuVmGFjZSeAOGIWiljZCaC+NFSAl7qykwFPADXN3av29fGPf9yj1t/f7xs2bPBEIuEbNmzw/v7+ee9bv369HzhwwMfHx/3AgQO+fv36Gfcv9LUAIAhJh32OTG24AF+oDRs2+IEDB2ZcO3DggG/YsMHdFxbwABCG+QK8oQYxS3G5AU9mtAColrIHMc3sCTN728yOTbu2zMx+Zma/nfz+obALjtrlBjwHBwe1devWGT/funWrBgcHJdF/DqDyFjKI+U+SPjPr2nZJP3f3j0j6+eTjunK5Ac9LBXxxwVAmk9Ho6KgymYx27NhBiAMI11z9KrO/JK2TdGza419Lunry11dL+vVCXidOfeDulx6kvFQf+Hz95+3t7Qx6AiiZggxizhHgv5v183cv8dy7JB2WdHjNmjVV+8DVMF/AJxIJHx8fn3Hv7t27XRKDngBKNl+AL2gQ08zWSfqJu2+cfPw7d1867efvuvtl+8HjNIgZxFwDnOvXr5ckvfLKK1PXGPQEsBBhr8Q8ZWZXT77w1ZLeDlJcvZmr//zVV1/VN7/5zRn3TR/0BIBSlbsb4dOS7pT04OT3fwutojow11Fw7e3tam9vn3Efy/gBBLGQaYR7JP23pGvN7KSZdWsiuP/MzH4r6c8mH2OadDqtY8eOKZ/P69ixY9q5cycHNAMI1WVb4O4+30bZN4ZcS13jgGYAYWMlJgDUOLaTBYA6Q4ADQEwR4AAQUwQ4AMQUAR6h2TsW9vT0sIMhgAUjwCMye8fCzs5OPfroo+rs7Jx6fOeddyqRSBDmAOZEgEekt7dXu3bt0rZt29TU1KSBgQE99NBDGhgY0N69e9Xf368HH3xQHR0dhDmAuc21w1WlvuK2nWwlzd6xMJFI+Llz56Z2NizuWmhmvn79ev/2t7/t1113HbsYAg1I8+xGSAs8IrMPhOjo6NCjjz6qjo6OqdN+Dh06pObmZu3atUs9PT06fvy4tm3bpl27dqm3tzfC6gHUAgI8IrN3LOzs7NT999+vzs5OfexjH1Mmk1F3d7fGx8enwry48RW7GAKQCPDIpNNp9fb2qqenRy0tLRoYGNCXv/xlDQwMaHBwUNu3b1dXV5c6Ojqmwrx4LNtHP/pRFQoFtbS00C8ONDD2QqlRe/bsUW9vr15++WWlUindd999uvbaa3X//ffr/Pnzam5u1vbt25XJZHT99dfrmWeeUTab1aJFizQ+Pq7rrrtOO3bsYLMsoA6wF0rMFLejLRQKevLJJzUwMKA77rhDixYt0gc+8AH94Ac/0Ne+9jV96Utf0jPPPKMvfOELampq0re+9S0tX75cQ0ND6urqkpmpqalJZkaLHagzBHgMFMM8kUjoN7/5jd544w1t3bpVkvTjH/9YuVxOL774onK5nK6++mrlcjlduHBB6XRaqVRKixYtUiKR0K233qq1a9eqs7NzqjsGQHwR4DFSnLkyfQbL8ePH1dHRMfW9t7dXH/zgB5XP5/XLX/5S+XxebW1tamtr00svvaQnnnhCAwMDzGQB6gABHiPFmSudnZ3q7u7Www8/rGQyqRtuuEHJZFKf//znNTg4qBMnTswI9aGhIZ06dWpqeuL07wDiiwCPkeLMlYGBAb366qv6+te/rmw2qx/+8Ie65ZZb1N/fr6uuukpmNiPU16xZo5UrV+qaa66ZmsFyxRVXqFAoyMy0evVqulOAGGIWSh2YPmMlmUwql8splUrps5/9rH7xi1/o/fff1/nz59Xa2qpEIqFsNquxsTEtWbJEn/rUp/Tss88ql8upvb1dO3fuZOYKUGOYhVLHps9YyWaz6u/v14c//GE9/fTTeuuttzQ2NiZ319jYmM6ePatcLqcHHnhAPT09+ulPf6qHHnpIa9euVSqVYnATiJHLHmqM+Emn0xe1opPJpM6dO6eWlhbl83l94xvf0PXXX69cLqe2tja99tprU/fecccdU68DoHbRAm8Q02ewrFmzRocOHdLx48d1zTXX6J577lEymdSKFSu0YsUK5XI5dXV1afny5bTGgRpGgDeI6TNYzp07p1tvvVVmppGREZ09e1bNzc0aGxtToVDQlVdeKTPTmTNnCHKghtGF0iCK3SG9vb165513lEwmlc/n9d5770mSli1bpnw+rwsXLkxtVSlJV111lU6fPq177713xusAiB6zUBrc6tWrderUKWWzWUnSqlWrlM/n5e5KpVJ6/fXXp+5NpVLavXs3IQ5UGbNQMKedO3dq6dKlSqVSWrlypd544w299dZbOn/+vN59910lk0ktX75ckpTL5XT77berp6cn4qoBSAR4w0un0/rOd76jK6+8UqdOnZo45SORUCKR0OjoqJqbm5XP57Vy5UqtXbtWK1as0He/+136xIEaECjAzexvzexXZnbMzPaYWUtYhaF60um0Tp8+rf7+fi1btkyFQkHvvfeeCoWCWltbZWY6f/68zp49O9XdUtzpkFWcQHTKDnAzWyXpHklb3H2jpKSkL4ZVGKpvepCnUhPj2++8887Uz8fHx2VmMrOpaydPnmSmChCRoF0oKUlXmFlK0mJJbwQvCVFLp9PavXu32tralEql1NzcrMWLF+vChQuSpObmZpmZksmkmpubJWlqyuGSJUsIcqBKyg5wd39d0t9LGpL0pqT33H3/7PvM7C4zO2xmh4eHh8uvFFU1u2/81KlTKhQKcnflcjktX75c+Xxe4+PjM5537tw5dXV16eabb46ocqBxBOlC+ZCkz0laL+kaSa1mdtvs+9z9MXff4u5b2trayq8UVVfsUmlvb1dTU9PU9UKhoDNnzkiSEomErrjiihnPSyQS2r9/P7NVgAoL0oXyp5Jecfdhd89K+pGkPw6nLNSS4lTDpUuXKpFIzOgDLxQKunDhghYvXqwVK1YolUqpUChIkvr6+uhOASooSIAPSfqEmS22iT/RN0rihIA6VOxOWbJkiQqFwtSXpKkwLxQKOn369Ixwl6TbbruNljhQIUH6wJ+TtFfSC5KOTr7WYyHVhRqTTqd14sQJubv6+/vV3t4uSVNL7sfGxtTc3KxsNisz06pVqyRNBHtfX58SiQRBDoSMpfQIpKenR4888shUkBfD++TJk3Pev2nTJh05cqSaJQKxx1J6VEQmk1GhUNCyZcskTbTIT548KTPTxo0bL7r/6NGjU3PJzUypVIqWOVAmAhyh6OvrUzKZVDKZlDQR5MeOHZt6PJ98Pq++vj6mHQJlIMARinQ6ra985StTg5tF+Xxeki4a3Jxt//79hDhQIgIcoclkMnrqqafU2tp60c8WMtbC3HGgNAQ4QpVOpzUyMqK77767rOc/8sgjIVcE1C8CHBWRyWTk7tq0aVNJz3N3rVmzpkJVAfWFAEdFHTlyRHfffbcSiUv/Vps+2HnixAmZGX3iwGUQ4Ki4TCYzdUxbcSHQ9EHNVCo1Ndg53f79+5VKpViOD8yDAEfVpdNpffWrX516nMvl5r03n8+rq6tLmzdvrkZpQKwQ4IhEJpPRTTfdtOD7jx49St84MAsBjsjs27dPq1evnvNn07evLSr2jTPVEJhAgCNSQ0NDc4Z4Npud9zl9fX1qamqibxwNjwBH5IaGhkqeN57L5Tj5Bw2PAEdNKM4bn69LZT779+/X8uXLK1QVUNsIcNSUclrjZ86cUUtLS4UqAmoXAY6ak8lk1N/ff9mdDKcbGxuTmamlpYW+cTQMAhw1KZ1O6/vf/75SqVRJzxsbG1NXVxdTDtEQCHDUrHQ6rWw2W9bGWEw5RCPgSDXExubNm3X06NGSn7ds2TKdPn26AhUB1cGRaoi9I0eOqL+/v+TnnTlzhs2xUJcIcMRKOp2Wu5fcNy5NTDmkWwX1hABHLGWz2ZLnjBf19fUxyIm6QIAjtoaGhso6NEKaGORk7jjijgBH7JXbNz42NkZLHLFGgKMuFPvGS22NnzhxgqX4iC0CHHWlnNZ4cZYKEDeBAtzMlprZXjM7bmaDZvZHYRUGlKvYGi/lwAhJMjNmqSBWAi3kMbMnJf2Xuz9uZs2SFrv77+a7n4U8iEJLS4vGxsZKek5ra6tGRkYqVBFQmtAX8pjZlZL+RNIuSXL38UuFNxCV0dHRkqccnjt3bqpFzuZYqFVBulB+T9KwpO+Z2Ytm9riZtYZUFxCqoaEhtbaW99uzq6tLZsbByqg5QQI8JekPJP2ju98g6Zyk7bNvMrO7zOywmR0eHh4O8HZAMCMjI2WHuDRxsDKDnaglQQL8pKST7v7c5OO9mgj0Gdz9MXff4u5b2traArwdENzIyEhZc8anM7M5D10Gqq3sAHf3tySdMLNrJy/dKOnlUKoCKqg4S2XRokVlv0Yul6M1jsgFnQfeI+kpMzsi6XpJ3wpeElAdo6OjZW+MVcQuh4hSoAB395cmu0c2u3unu78bVmFAtWSz2ZLnjE9X3OWQIEe1sRITkLRv3z65e6CulWKQM+0Q1UKAA7MUu1bKDfKuri4GOVEVBDgwj9HR0bLO45QY5ER1EODAJWQymbL2VSkyMy1ZsiTkqoAJBDiwAPv27St7/nhxWT6bZCFsnEoPlKipqUm5XK7s57NRFkrFqfRASLLZrNxdiUR5f3yKLXIgKAIcKFM+n1fA7ZgJcgRCgAMBuXugICbIUS4CHAhBoVAoec/x2dgkC6UiwIGQDA0NBd5bpTh/nIOWsRAEOBCy4iBnEBy0jIUgwIEKKe6tEgT947gUAhyoMHfXpk2bAr0GQY65EOBAFRw5coQgR+gIcKCKikEexowVghwEOBCB4oyVoAjyxkaAAxEKY6BTEiHeoAhwoAa4e9l7jxfRGm88BDhQI4p7jwdZCCQR5I2EAAdqTHEhUBhBjvpGgAM1KowVnbTG6xsBDtS44kBna2tr2a9BkNcnAhyIiZGRkVBa5KgfBDgQM0GnHhZb4zfffHOIVSEKBDgQU0EPkti/fz8t8pgjwIEYKxQKDHQ2sMABbmZJM3vRzH4SRkEASufu6u/vD/QahHj8hNECv1fSYAivAyCAdDodeEUnrfF4CRTgZtYu6RZJj4dTDoCgiis6gyDE4yFoC/wfJN0nqTDfDWZ2l5kdNrPDw8PDAd8OwEKFNVsFtavsADezv5D0trs/f6n73P0xd9/i7lva2trKfTsAZQojyJuamkKsCGEJ0gL/pKS/NLNXJf2LpE+b2T+HUhWA0AUJ8VwuR4u8BpUd4O7+gLu3u/s6SV+UdMDdbwutMgChC+ugZdQG5oEDDcjdddNNN5X9fFrjtSHYfpWT3P0/JP1HGK8FoDr27dsnKViL2sxCOVEI5aEFDjS4MLpUNm/eHFI1KAUBDiBw3/jRo0fpUokAAQ5gShhTDlE9BDiAiwQJckK8eghwAPMixGsbAQ7gksptjTPVsPIIcAALQmu89hDgABYsSGsc4SPAAZSMLpXaQIADKAtdKtEjwAGUjRCPFgEOIBBCPDoEOIDAmGoYDQIcQGhojVcXAQ4gVEFCnCAvDQEOIHTspVIdBDiAiuGwh8oiwAFUFIOblUOAA6g4d1d/f3/JzyPIL40AB1AV6XSafvGQEeAAqop+8fAQ4ACqjh0Nw0GAA4gEIR4cAQ4gMoR4MAQ4gEiVs+iH2SkTCHAANYHWeOkIcAA1gxAvTdkBbmarzeygmQ2a2a/M7N4wCwOAhWrUEE8FeG5O0t+5+wtm9gFJz5vZz9z95ZBqA9CA3L2sQC4+p5HmmZfdAnf3N939hclfvy9pUNKqsAoD0LiChHAjtcZD6QM3s3WSbpD03Bw/u8vMDpvZ4eHh4TDeDkADaKSWdLkCB7iZLZH0r5L+xt3Pzv65uz/m7lvcfUtbW1vQtwPQQDiq7dICBbiZNWkivJ9y9x+FUxIAzMQmWHMLMgvFJO2SNOjuD4dXEgBcjC6ViwVpgX9S0u2SPm1mL01+fTakugDgIkG6U+qxNV72NEJ3PySp/v6LAKhpxRAvd6phPbXkWYkJADFFgAOIpXpqSZcryEpMAIhUOd0ps++N818EtMABNLQ4D24S4ABiL86t6CDoQgFQF6aHeJxb1aWgBQ4AMUULHEDDm6vFHoduGVrgAOpOGOEbh24YWuAA6tLsEI9DIJeKFjgAxBQBDgAxRYADQEwR4AAaQqkDm3GYhcIgJoCGEYdQLgUBDqDhlTNDpRb+MqALBUBDK3d6YS1MSyTAASCmCHAAiCkCHABiigAHgJgiwAE0tHJnk9TCLBSmEQJoeLUQxuWgBQ4AMUWAA0BMEeAAEFMEOADEVKAAN7PPmNmvzex/zWx7WEUBQC0xs9C+wlR2gJtZUtIjkv5c0nWS0mZ2XViFAUAtCDt0w3y9IC3wP5T0v+7+f+4+LulfJH0unLIAAJcTJMBXSTox7fHJyWszmNldZnbYzA4PDw8HeDsAwHRBAnyufwdcNBve3R9z9y3uvqWtrS3A2wEApgsS4CclrZ72uF3SG8HKAQAsVJAA/x9JHzGz9WbWLOmLkp4OpywAqA1hL7MP8/XK3gvF3XNmdrekfZKSkp5w91+FVhkA1Iha3Ssl0GZW7v6spGdDqgUAUAJWYgJATBHgABBTBDgAxBQBDgAxZdUcXTWzYUmvlfn0FZLeCbGcWlTvn7HeP59U/5+x3j+fVJufca27X7QSsqoBHoSZHXb3LVHXUUn1/hnr/fNJ9f8Z6/3zSfH6jHShAEBMEeAAEFNxCvDHoi6gCur9M9b755Pq/zPW++eTYvQZY9MHDgCYKU4tcADANAQ4AMRULAK8ng9PNrPVZnbQzAbN7Fdmdm/UNVWKmSXN7EUz+0nUtYTNzJaa2V4zOz75//KPoq4pbGb2t5O/R4+Z2R4za4m6pqDM7Akze9vMjk27tszMfmZmv538/qEoa7yUmg/wBjg8OSfp79y9Q9InJH21zj7fdPdKGoy6iAr5jqR/d/ePSfp91dnnNLNVku6RtMXdN2piC+kvRltVKP5J0mdmXdsu6efu/hFJP598XJNqPsBV54cnu/ub7v7C5K/f18Qf/IvOFo07M2uXdIukx6OuJWxmdqWkP5G0S5LcfdzdfxdtVRWRknSFmaUkLVYdnMDl7v8p6cysy5+T9OTkr5+U1FnVokoQhwBf0OHJ9cDM1km6QdJz0VZSEf8g6T5JhagLqYDfkzQs6XuTXUSPm1lr1EWFyd1fl/T3koYkvSnpPXffH21VFbPS3d+UJhpYkq6KuJ55xSHAF3R4ctyZ2RJJ/yrpb9z9bNT1hMnM/kLS2+7+fNS1VEhK0h9I+kd3v0HSOdXwP7vLMdkP/DlJ6yVdI6nVzG6LtirEIcDr/vBkM2vSRHg/5e4/irqeCvikpL80s1c10QX2aTP752hLCtVJSSfdvfgvp72aCPR68qeSXnH3YXfPSvqRpD+OuKZKOWVmV0vS5Pe3I65nXnEI8Lo+PNnMTBN9p4Pu/nDU9VSCuz/g7u3uvk4T//8OuHvdtN7c/S1JJ8zs2slLN0p6OcKSKmFI0ifMbPHk79kbVWcDtdM8LenOyV/fKenfIqzlkgKdiVkNDXB48icl3S7pqJm9NHnt65PnjSI+eiQ9NdnI+D9Jfx1xPaFy9+fMbK+kFzQxc+pFxWjJ+XzMbI+kT0laYWYnJX1D0oOSfmhm3Zr4i+uvoqvw0lhKDwAxFYcuFADAHAhwAIgpAhwAYooAB4CYIsABIKYIcACIKQIcAGLq/wG99L4JY/xbygAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "import numpy as np\n",
    "\n",
    "log_y = np.log(sorted(fdist.values(), reverse = True))\n",
    "log_x = np.log(range(1, len(fdist.values()) + 1))\n",
    "\n",
    "plt.plot(log_x, log_y, \"o\", color = \"black\", markerfacecolor='None');"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 95,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAX4AAAEWCAYAAABhffzLAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjAsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+17YcXAAAgAElEQVR4nO3debxc8/3H8dcnm6yEbPZErQlBIhpKW0EtrZLar52QtkiUliJatA0trdJIldpqSSyx1C4/JCEoEktW+xJBJAiRkPV+fn98zuTOvZl779xlZu6deT8fj3nMzDlnznzP3Lmf853v93s+X3N3RESkdLQodAFERCS/FPhFREqMAr+ISIlR4BcRKTEK/CIiJUaBX0SkxCjwi+SAmfUyMzezVoUui9SNmU00s5MLXY5cUuCvJzN738y+NbPFZrbQzB42s00KXa6qLAw3sxlmtsTM5prZ3WbWt9Blq0ny+e6d4/e4yMxWJH/DL83sOTPbNZfvWeX9jzKzMfk6SSQBbWlyvKlb3o63Kany//upmd1kZh3ruI9me3JX4G+Yn7p7R2AD4FNgVHUbmlnLvJWqsquAM4DhwHrAVsD9wE/quqPm+AXPwp3J37ArMAG4O4/v/WPgkTy+H8Dp7t4x7fZ81Q2K9O+cSer/tz+wM3BBgcuTNwr8jcDdlwLjgD6pZWZ2s5ldY2aPmNkSYJCZrWNmt5jZAjP7wMwuMLMWyfYfmNlOyeNjkppEn+T5yWZ2f/L4IjO7K9nP12Y208wGZCqXmW0JnAaUuftT7r7M3b9x99vd/c/JNjWV6QQze9bM/m5mXwAXpS0bZWZfmdnrZrZX2ntWqqkn5b0tedzWzG4zs8+TGvZLZtajrp+3mZ1iZm+b2Rdm9oCZbZi2bh8zeyMp2z/NbFI2P9vdfSVwO7CRmXVL9rWumT2UfDYLk8cbp73XRDP7Y/J5fG1m482sazVlPiT5bLZLnrcAfgQ8VsuxrmVmV5rZx8ntSjNbK239OWb2SbLu5OR7s0Vtx5vhfdzMTjOzt4C3kmXbmNn/JZ/zG2Z2eNr2XZLPfpGZvZh8DpOTdWvUhK1K84mZnWRms5PP9XEz61mlLL8ws7eS9aPNzNLWn5K89mszm2Vm/c3sbDO7p8oxjTKzK2s7dnf/CHgU2C7D59Ii+Z/4wMzmJ/8r6ySrn07uv7Rm9utJgb8RmFl74Ajgf1VWHQWMBDoBk4lfBOsA3wF+CBwHnJhsOwnYI3n8A+DdZJvU80lp+z0QuAPoDDwAXF1N0fYC5rr7izUUv6YyAQxMytI9OZb0ZV2BC4F7zWy9Gt4j5fjkvTYBugC/AL7N4nWrmdmewKXA4cQvrQ+Iz4Ik6I4Dzkv2/wbwvSz324Y49s+BhcniFsBNQE9g06SsVT/ro4jPqzvQBvhNhn2fCPwF2NvdZySLvwu86+6f1VK0EcAuwI7ADsnrLkj2ux9wFrA3sAUV35f6Gkz8bfuYWQfg/4AxxLGVAf80s22TbUcDS4m/wUnJLStmNhg4HzgY6AY8A4ytstkBRC18B+JvvW/y2sOAi4i/1drE/8LnwG3AfmbWOdmuFfE/eWsW5dmE+PX1SobVJyS3QcT/SEcqvgM/SO47V/frqclyd93qcQPeBxYDXwIrgY+BvmnrbwZuSXveElgG9Elb9nNgYvJ4CPBA8ng2cDJwR/L8A6B/8vgi4Im0ffQBvq2mjCOA/9VwDLWV6QRgTpXXnJAcq6UtexE4Nu1z2Ttt3UXAbcnjk4DngO2z/Hz3zrD8BuCytOcdgRVALyIYPJ+2zoAPgZOreY+LgOXJ33AVEUD2qKFMOwIL055PBC5Ie34q8FjyuBfgxIlgFrBxlX39EfhdlW1bZXjPd4Afpz3fF3g/eXwjcGnaui2S/WxRTfknAt8kx/sl8HLaOgf2THt+BPBMlddfS5zoWyaf+TZp6y4BJld3PMl7n5w8fhQYkrauRVKunmll2T1t/V3Aucnjx4Ezqjm+R4FTkscHALOy/P/9APgn0C5DWZ8ETk173dbJsbeq6e/W1G+q8TfMYHfvDKwFnA5MMrP109Z/mPa4K1Ej/CBt2QfARsnjScD3k9e3BO4EdjOzXkQt+dW0181Le/wN0NYyt8t+TtTIqlNbmaoeQ8pHnvwXpL1mwwzbVXUr8Y97R9I0cZmZtc7idek2TC+vuy8mjnOjZN2HaescmFvL/u5K/oY9gBnATqkVZtbezK5NfuYvIn7ad7bK/TVV/xZVOwjPBka7e9VyZNu+X+l4qfxZVzpeMv+tqhru7p2TW/8q69Jf3xMYmDTJfWlmXwJHA+sTtfRWVbZPL2NtegJXpe33C+Iknf69q+5z3YQ4GWbyH+CY5PEx1F7bH5x8Dj3d/VR3z/TrM9Pn34r4vjRbCvyNwN1Xufu9RK1x9/RVaY8/I2oKPdOWbQp8lOzjbeILPhx42t2/Jr78Q4maVHk9ivYksLFV0wdQW5kyHEPKRultrslrPk4eLwHap61bfSJ09xXufrG79yGaYA4gaul18XF6eZMmiS5JmT8B0tvgLf15TTyaXH5O9GOkTpa/Jmp4A919bSp+2luGXVRnH+ACMzskrVzrEyfkl7N4faXjpfJnXel4iaDYEOl/6w+BSWkniVRzxi+BBcSv3PT32zTt8ZLkPuP3INn3z6vsu527P5dFGT8ENq9m3f3A9kk/ygFEn01DZfr8VxKDOZptamMF/kZg4SBgXaKZZg3uvor4yTrSzDolnVlnEW2TKZNIfjkkzydWeV4n7v4W8RN2rJntYWZtLDpYjzSzc7MsUybdgeFm1jppc+1NRe31VeDIZN0A4NDUi8xskJn1TWrMi4iTzqoa3qd1Ut7UrRXR5nyime2YdHJeArzg7u8DDwN9zWxwsu1pVA44tX1erxO/SM5JFnUi2vW/TPowLsx2X2lmAvsBo83swGTZj4kmoaqBY60qx9uCaPu+wMy6JX0Yv6fi73MX8Vn0TvqZfl+P8lXnIWArMzs2+Vu2NrOdzax38r25lzhJtrcYhHB86oXuvoA4ER9jZi3N7CQqB+t/Aeel+gssBhgclmW5rgd+Y2Y7Jf93WyTfW7xikMUY4EV3n9OwjwCIz/9MM9vMYrjnJcRIsJXECbCcaPtvVhT4G+ZBM1tMBLGRwPHuPrOG7YcRtaF3ic7eMUQ7bcokItg8Xc3z+hhOdEaNJtoz3wF+BjyYZZkyeQHYkvjFMBI41N0/T9b9jvgnXwhcnOwvZX3iH3MRcYKcRM0nmUeIwJu6XeTuTybvcQ9R490cOBJW19oPAy4jmn/6AFOIfoxsXQ4MNbPuwJVAu+Q4/0ctI3Cq4+6vETXQf5vZ/lTfzLOYyse7J/Cn5BimAdOJXwl/Svb7KPAPYhjq20Cqc7Eux1tdmb8mfq0cSdR65xEd1KkRRacTzS/ziP6sm6rs4hSimetzYFuibye17/uSfd2RNKHNAPbPslx3E9+5McDXRC0/fWDBf4C+ZNGpm6Ubk309DbxHdGgPS8ryTVKWZ5Nmq10a6T1zztasdIhUz8xOIDq+dq9t20JLasxzgaPdfUKhywOrR5vMAzZ3968aed+9iSC6VlIjzZum8r0ws02B14H13X1RIcvSlKnGL0XFzPY1s85JM9D5RHt81WG2hbQeMZqnUYK+mf0sacJbl6hFP5jvoN9UJCf6s4jRcAr6NVDgl2KzK9Gc9RnwU2LkRp2uFcgld5/v7tc04i5/TrQ1v0P0l/yyEffdbCSd/IuIi+Lq0xdTUtTUIyJSYlTjFxEpMc0iGVPXrl29V69ehS6GiEizMnXq1M/cvVvV5c0i8Pfq1YspU6YUuhgiIs2KmWW8olpNPSIiJUaBX0SkxCjwi4iUGAV+EZESo8AvIlJiijbwjx0L220HLVvG/diq8/uIiJSoZjGcs67GjoURI+CGG2D33WHyZBgyJNaVlRW2bCIihVaUNf6RIyPoDxoErd+axaA9nBtuiOUiIqWuKAP/7NlR0+edd6BfP/jJT9h90znMzjhFiohIaSnKwN+7dzTv0KsXXHYZTJqE9d2WP3S/GsrrM4OhiEjxKMrAP2JEtOlPeLolK049g+evn8lzfI8R84ZF+8/KkkxXLiICFGnnbqoDd9iwaPbp3bsXI65/DFbeCh98AK2Swy4vhxZFee4TEalWs8jHP2DAAG/0JG0TJ8IZZ8C//w3f/W7j7ltEpAkws6nuPqDq8tKt7q5cCZ9/DrvuCmedBUuWFLpEIiJ5UbqBf++9YeZMGDoU/v536NsXnnyy0KUSEcm50g38AOusA9dcE80+rVrFiaAauhJYRIpFUXbu1tkPfwivvQZt2sTzcePi/pBDwExXAotIUSntGn+6du2iOg/R4XvYYXDwwfDxx5WvBG4d97oSWESaKwX+TB5+OC78euwx6NOH3Wb9m913qzz6affd0ZXAItIsKfBn0qoVnH02TJ8O/ftzrQ9lxj+frrTJ5MlxhbCISHOjwF+TLbaAJ59kwnnjOeQfP2TCBFj5zPNMfGIlQ4ZEu38m6ggWkaZMgb82Zgy65EeMHAkjf/Ehq36wB91+OpDRp7yasWM31RE8ahQsXRr3I0ZUDv46MYhIQbl7k7/ttNNO3iSUl7uPG+feo4d7y5bu553n/u23lTbZdlv3p56q/LKnnorl7u5jxrhvtlksW7487jfbLJaLiDQmYIpniKmlm7KhIRYuhN/8Bm68MRr6X34Z2rYFoha/dGmM/klZsSJWr1oVNfxRo2JkUMqECZFXaMaMPB+HiBS1vKdsMLMbzWy+mc1IW7aemf2fmb2V3K+bq/fPqXXXjfGcTzwBJ520OuizbFlFSug06R3Bq+cKSJM+QkjNQCKSa7ls478Z2K/KsnOBJ919S+DJ5HnztddeUfOHqLZvsQX/3O+BSAk9IWr6EyZQqSO4phNDNv0DIiINlqn9p7FuQC9gRtrzN4ANkscbAG9ks58m08Zfk6lT3fv2dQd/f5fD/Qdbz/MWLaJtP739vqY2/ur6BzbeONZl2p+ISHWopo0/34H/yyrrF9bw2qHAFGDKpptumsOPphEtW+b+xz+6t2njvt561UboVJCvGshbtIiTQbpbbom/kjqDRaSuqgv8Oe3cNbNewEPuvl3y/Et375y2fqG719rO3+Q6d2szezaccgoceSScfnrWL8vU8bvZZnH/3nsVy9QZLCLZqK5zN99J2j41sw3c/RMz2wCYn+f3z4/eveHptCt9x4yBBQviJJDKB5RBasrI9GRw778Pt9xSeTulixCRhsj3BVwPAMcnj48H/pvn98+fFi0qpnV89FH41a9gt91qTP1cVhaJ34YNi4FCw4bBxhvHLZ3SRYhIQ+RyOOdY4HlgazOba2ZDgD8DPzKzt4AfJc+L3y23wG23wdtvQ79+cOGFsGxZxk3LyqIJZ9WquL/sMmocJSQiUle6gCufFiyAM8+E22+PzJ/77pvVy8aOjV8CMXF8BH3NAyAitamujV+BvxBefhn694/Hjz8eTUAdOxa2TCJSdDTZelOSCvrz58PgwTGc5/HHC1smESkZCvyF1L17pH1o1w722w+OOw4+/7zQpRKRIqfAX2i77QavvAIXXBCN+X37wuLFhS6ViBQxTbbeFLRtC3/8Y8zz++yzFe39X30F66xT2LKJSNFRjb8p2X57+OUv4/FTT8Gmm8K//gXl5ZU2q5rBc9gwZfQUkewp8DdVvXrBgAFxIhg0CN54A1gzg+fgwXFuGDy44vnxx8e1YzoJiEgmCvxN1Xe+Ex2/N9wA06bBDjvAlVcycmQsGjQoJnu5/374y1/ifty4yA7x5z/HeH+dBEQkE43jbw4++STacwYNouXw0yrN8NWyJXz9NXTqFMF+1KjI5bPWWvGj4fTT40Rx9dVxxe/Ikbr4S6RU6AKuYuDOdn2Nu356K31WTYeLLmK777bnpJNiFsjZs6O5Z/Jk2H//SBG0++4V0z4qq6dIadEFXMXAjBEj4JlrpsPll+Pbb8+ZO07gt7+NZp1ttoka/5AhsHx5RYbPVEI3ZfUUEVDgb3bKymDtay7jxF5P8c47MOT2PZm41VCevOdLZs+Gc8+Fo46qaPZJJXQbOxa22ioGCLVtq3Z/kVKmwN8MlZXBTe8NYosl0+Dss9nt9Rt4/q/PUl4O//lPdPSmnwRWroRf/xoWLoRu3eCSS6BnT9hyy+j8NdPJQKSUKPA3Z+3bR97mN9+En/wEgLI29zDjiXmVTgLHHRedvZ06wZ13wllnwdFHw8MPw+GHR0fxJZdAly4wZ06cLMxiuU4KIsVHgb8YbL553H/1FZx0EvTpAzfdRNmRzowZEbTffBM+/jja+QHuuy9+CbzyStxvsEHcf/tt/KJo1SpOFi1awKGHxi+EwYMrmo1EpPlS4C8m66wDL74YVfOTToJ99oF336V374pO3smTY9PXX4/nqfuRI+Plq1bBa6/FfbducXv11Rg1dP/9MTR05MjCHqaINIwCf7HZemuYOBGuuQZeeAF23JGLz/iCIUOixj5kCFxxRYz/79cv7n/2s+gT+PDDyieDOXPg009jXWpEkEYGiTR/CvzFqEUL+MUvYNYsGD2aQ05Zj5Ej4bm7P+L99+H882Max7vuiq6BMWMiQ7RZ5ZPBpptCjx6w4YYVI4LatYt7M9hkEzX7iDRHCvzFbOON4dhjASjr8RRPvbcZ5SN+x9KvluEenb9vvQXvvw+ffRbNO3feGRd/3XBDzBQ5fz58+SV88UUEfXdYe2048ECYNy86gnUCEGleFPhLxQ47RK/tn/4U1frnnls9sXt5efwCGDMG1l8fHngggvqyZRHoly2DRYui8/e88+Lq30cfjRxBPXtGR7A6fUWaDwX+UtGlS1TxH3sMvvkmGuvPO6/SJmVl0c7vHrcVK6LVaMmSuF+1Ci68sGJEULdu8MEH8Yvhvfdi2KiCv0jTp8BfavbdN6r5w4dHdb0W6SOCNt00Hr/+erT7Dx8e/QFdu8Zt5cpo+unSRScAkaZMgb8UdewIV14ZHcAAN90UV3QtWLDGpiNGsHpE0JIlMabfLGaHXLQI2rSJpqDy8mj7N4v+AJ0ARJouBX6JSH333XHh1+23RztPoqwsxu3ff390AC9aFE0+X30VwX699aBDh1iWaiKCGCX01VdwxhkK/iJNjQK/RCKfV16JK4CPOQYOOCAG8SeqdgK7x4Ch1q3ho4+iI7hjxzgB9OgBG20Uo4FWrYofEWr7F2laFPglbLttTPR+5ZUwaRJMn17j5pddBp07x4ieHj0iHcS8edFvvHBhtP136RLbrlwZo0qHDcvDcYhIrRT4pULLltE28957q5O+ccstGS/VLSuDq66Kdv1PP41fAS1axG3p0mj7X7UqTgo9e0bn77XXquYv0hQUJPCb2ZlmNtPMZpjZWDNrW4hySDW6dYv7b76B3/4WdtwR/vjHmN0lTVkZfP55jP9fb71oCkq1/XfoEB2933wT/QKffhrNRKnMn7roS6Rw8h74zWwjYDgwwN23A1oCR+a7HJKF9u0jQ9vPfga//z0MGBBJ4KpIPwG0ahXLPvusYv3y5RHszSqWzZ2rkT8ihVKopp5WQDszawW0Bz4uUDmkNj16wB13wH//G6N/vv/9mPw9g7KyaBnq1i1OAG3axLnj229jfZs2EfxbtozHUDH0s2NHnQBE8iXvgd/dPwL+CswBPgG+cvfxVbczs6FmNsXMpizIML5c8uzAA2HmzEjms8EGsWzWrDU2q9r2/+mn0fTjHp28XbpE23+VViOWLIkTwL775uFYREpcIZp61gUOAjYDNgQ6mNkxVbdz9+vcfYC7D+iWanOWwlpnnbiSCyL187bbRt7/hQsrbZZq+kkN+UwpL48aPkQncLt2lXffogWMH6/RPyK5Voimnr2B99x9gbuvAO4FvleAckhDDBwYk/reckvkc7jnnjU2SQ357Nw5gnp6G395eTQBtW8fI35atYplAFdfrWYfkVwqROCfA+xiZu3NzIC9AE3t0dy0aweXXgovvRRXbB16KJxwQqVNUs0+HTtGUE/doOIkUF4evw7STwoQ15Gp5i+SG4Vo438BGAe8DExPynBdvsshjaRfv5jp6y9/iY5fqJS7IT3j55gx0fyT2gQiz0+bNjHU0yzOIRAnhKuvjl8KOgGINC7ztLwsTdWAAQN8ypQphS6GZOvGG+HWW+Hf/4Yttsi4ybBhMHp0xQkgFfTnzs28y759Ydq0HJVXpEiZ2VR3H1B1ua7clcbXpg28/HJE68sui+E8VYwaVZHkDeIEMHdunAC2227NXU6fXnEtgFn0CeiXgEj9KPBL4zvmmEjzsN9+ceXvwIHw2msZN7366hjX37JlPHePhHCp59VZtSpeq+GfInWnwC+5seGGcO+9ke75o4+qbcMpK4Nf/rKi0zdl1aq4r9rpW9X48Qr+InWlwC+5Yxajfd55pyLp2zXXwDPPVNps1KiYBqBDhzV3kU0XlMb+i9SNAr/kXiqiL18e4zt/8IOo5i9atHqTsrKY1ev00+v3FqNHN0I5RUpErYHfzE5PrrYVaZg2bWDqVDjzTLjuurjy96GHKm0yalTU8vv2rduu3WNOYBGpXTY1/vWBl8zsLjPbL7noSqR+OnSAK66A55+PS3oHD4b3319js2nTovbfopZvaHon8IcfRuuS2vxFalZr4Hf3C4AtgRuAE4C3zOwSM9s8x2WTYvbd70btf/x46NUrlj37bKVG/VGjKs/lO2ZM5c7eVq0qOoHTjR8f65T2QSSzrNr4Pa7ympfcVgLrAuPM7LIclk2KXZs2sOee8fi552D33WH//TP+AoDoBzjttIrnGS4PWG3Vqsj2uf32jVdckWKRTRv/cDObClwGPAv0dfdfAjsBh+S4fFIqdtklqvjPPhtXcF11Vcbq/KhRsM8+2e92+nS1/YtUlU2NvytwsLvv6+53Jxk1cfdy4ICclk5KR4sW0ag/c2aM+vnVr2IIaIbxnI8/HlM3ZpKeBjol1favIZ8iIZvA/wjwReqJmXUys4EA7q6smtK4Nt0UHn44BvanJuh1X2PmljlzMgf/FSuq3/XVV8eJQW3/UuqyCfzXAIvTni9JlonkhlkE/eOOi+c33gj9+0cW0DRz5tR93P/KlZrpSySbwG+elsIzaeJplbsiiVSx8cZxsdeuu8Y1AEuWrF6VGvdfXdNPdcaPj2kgRUpRNoH/3aSDt3VyOwN4N9cFE1lt332j7f/UU+HKK6Pzd8KESpvUp/b/xRfQtm0jllOkmcgm8P+CmBrxI2AuMBAYmstCiayhU6dopJ88OaL10qVrbDJqVIz1ry2zZ7ply6JlqW1btf1L6dBELNL8rFwZV2gB/P3vkQn08MNXX901dmx0D9Q0zr86m2wSvx5EikG9J2Ixs25mdr6ZXWdmN6ZuuSmmSBZSQX/Vqpjk/cgj4aCDVqd+LiuL0T31SfimoZ9SCrJp6vkvsA7wBPBw2k2ksFq2hEmT4G9/gyeegD594F//Wp3cv74J3yBaldT5K8Uqm8Df3t1/6+53ufs9qVvOSyaSjZYt4ayzYtqunXeOav6bb1baZNq0aPuvqy++UNI3KU7ZBP6HzOzHOS+JSEN85ztR63/hBdhmm1j24IOrr+gqK4vaf6t6DEQeP17NP1Jcsgn8ZxDBf6mZLTKzr81sUa2vEsk3M9hpp3j8yitw4IEVWUATK1bUfcx/ytVXK++PFIds0jJ3cvcW7t7W3ddOnq+dj8KJ1Fu/fjHn77x5Mdn7OefAN98AMWqnvm3/H36osf/S/GUzqsfM7Bgz+13yfBMz+27uiybSQD/7GcyeDSeeCJdfHsnf0mZ1r2/b/7JlqvlL85ZNU88/gV2Bo5LniwHNcCrNQ+fO8O9/w5NPwtlnRxZQ99Xz/aba/uta+//wQ436keYrm8A/0N1PA5YCuPtCoE1OSyXS2PbcE444Ih7fcEN0AN9//+rV9an9p0b9iDQ32QT+FWbWEnCIC7qA8ppfUjMz62xm48zsdTObbWa7NmR/InXSrx907x5NQYcdFv0AVNT+6zLRC0Tw16gfaU6yCfz/AO4DupvZSGAycEkD3/cq4DF33wbYAVBef8mfnXaCl16CSy6JIZ99+sC4catXP/54nADWWqtuu736aujYsZHLKpID2YzquR04B7gU+AQY7O531/cNzWxt4AfE5O24+3J3/7K++xOpl9at4bzz4LXXIttnhw5rbLJ0ad2Hfi5ZUvELQEnfpKmqNUmbmWUcv+Du9UplZWY7AtcBs4ja/lTgDHdfUt1rlKRNcsq9orH+kktivOYZZ6xO89mxY6UpAOqsb9/oQxDJt3onaSPy8jyU3D9J5OJ/tAFlaQX0B65x937EjF7nZijwUDObYmZTFixY0IC3E6lFKui7w5Qp8Otfx6Qv06cDsHhxxh8EWZs+XZ3A0rRk09TT1923T+63BL5LtPPX11xgrrun5tEbR5wIqr7vde4+wN0HdOvWrQFvJ5Ils8j2OXYsvP9+TPf4u9/BsmUsXly/Mf9Vd59pMniRfMumxl+Ju78M7FzfN3T3ecCHZrZ1smgvotlHpPDMIs3zrFkxzOfPf4a33gIqRv3UtdM33cqVqv1L4dWassrMzkp72oKonTe07WUYcLuZtSGajk5s4P5EGlfXrnDLLXDhhbD55rHs1lth8GCWLu0ERO29PpO9QAT/ffaJEUQi+ZZNjb9T2m0toq3/oIa8qbu/mjTjbO/ug5OLwkSanlTQf+MNOOEE2HZbeDS6uFasqPuY/3SprJ9K+yz5pqkXRbL1/PMwZEjk/zn66Jj4vWvX1avbto08PvU1Zkw0J4k0lupG9WQznPOBmta7+4ENLFutFPilyVi2LIZ8Xnop9OwZJ4EqSf4bcgJo1Wr1FAIiDVZd4M9mWor3gPWB25LnZcD7gFonpfSstRZcfHGkenjnnYjU7pH2YYMNgLjwa9iwuJK3rlKdv83gh7g0Y9m08fdz9yPc/cHkdhSwu7tPcvdJuS6gSJO03XYxwTtE0rettoLRo9eY77e+fQBmSv8guZNN4O9mZt9JPTGzzQANrBdJ2WuvuODr9NMj5//rr69e9fjj9R//n0r/oORv0tiyCfxnAhPNbKKZTQQmAL/KaalEmjO16uwAABPQSURBVJPNNosIf/PNMf5/hx3gX/9avboh8/1CNBnpF4A0plq/iu7+mJltCSQzWPO6uzdg7IJIETKD44+H/faD4cPjZFBFqtO2ZctKE4FlLfULQO3/0lDZXMDVHjgL6Onup5jZlma2tbs/lPviiTQzPXrAnXdWPL/ookj284c/QPv2AKxaFavqewVvemohkfrIpqnnJmA5Mf0iRK6dP+WsRCLFwh0+/xz+9rdI0fnkk2usbkj6hlT6Z5G6yibwb+7ulwErANz9W0BfN5HamMXwnokTo31n773jArCFFReql5fXPed/prdR8jepi2wC/3Iza0fF1IubA2rjF8nWD38YE76ce24M8ZlTeSqLOXMa1vkLFeP/NQG8ZCObwH8h8BiwiZndTuTkPyenpRIpNu3axdW+H3wQo34ghut88snqTVasaHi7vSaAl2zUGPjNzIDXgYOBE4CxwAB3n5jzkokUo+7d437OHDj7bOjdG66/vlLEd2/4CUDt/1KTGgO/RyKf+939c3d/2N0fcvfP8lQ2keK16aYxH+OOO8Ipp8RFYG+/XWkT9+gTbgidACSTbJp6/mdm9Z54RUSqseWW8NRTcO21MHVqXPVbJbvbtGk6AUjjyybwDwKeN7N3zGyamU03M00dLdIYWrSAoUPjit///CeSwJWXV0r7ABUngMYYAaQTgFQ7jsDMNnP394D981gekdK00UZxg0j9MHQonHMO/P73kec5kRoQ1NDgrYvASltNNf5xyf2N7v5B1Vs+CidSkgYPhmOPjVFAO+wATz+9xiaN0QEMqv2XqpoCfwszuxDYyszOqnrLVwFFSs5668FNN8XcjMuXx3UAf/hDxk3dIyloQ6j5p/TUFPiPBJYSzUGdMtxEJJd+9COYMQPOPBMGDoxlGbK7pXL/N+QCMNAJoJRU+1Vx9zeAv5jZNHd/NI9lEpGUDh3giisqnv/ud/Duu3DVVRXXBCRS2T9bt44reetLGUCLX62jehT0RZqQjh3hnnviwq9bb80YoRvjCmDV/otbNsM5RaSpOO88ePVV2HprOO442H//NXL/pKQ6gDt0qP/b6QRQnBT4RZqbPn3gmWfgH/+AF1+Er76qcfPFixvnF4AUj1oDv5kdnOG2l5l1r+21IpIjLVvGZLxz5lRc1nvJJXEhWDUaOgQ0Vfvfd9/670Oahmxq/EOA64Gjk9u/iRm5njWzY3NYNhGpTWoi3k8/jU7gHXeEiy+OYaDVaOgEMOPH6xdAc5dN4C8Herv7Ie5+CNCHyMc/EPhtLgsnIlnq0SNq+4ceGtM99u8P//tftZuXl6sDuJRlE/h7ufunac/nA1u5+xcks3LVh5m1NLNXzExz94o0hu7dY6KXBx+Mdv+f/CQa+GvgHi9pCAX/5iebwP+MmT1kZseb2fHAA8DTZtYB+LIB730GMLsBrxeRTA44AGbOhAceiKag8nJ4/vlqNy8ra/gVwKr9Ny/ZBP7TiAnXdwT6Af8BTnP3Je4+qD5vamYbAz8h+g5EpLGtvTbstls8vv12+N734IQTYoquaqSuAG4IBf/mIZsLuByYDDwFPAE8nSxriCuJ6RvXvP48YWZDzWyKmU1ZsGBBA99OpIQddhicf36cAHr3hrvuqjHCN9boH2m6shnOeTjwInAocDjwgpkdWt83NLMDgPnuPrWm7dz9Oncf4O4DunXrVt+3E5G2bWHkSJgyJRL6H3FEDAWtRWOcAFq3rv/rJXeySes0AtjZ3ecDmFk3ouY/rsZXVW834EAz+zHQFljbzG5z92PquT8RycYOO8RInyuvrJjwfcWKuCagRfV1wIYM/1y5Urn/m6Js2vhbpIJ+4vMsX5eRu5/n7hu7ey8iA+hTCvoiedKqFfzmN5H5E+DCC2HPPeHNN2t8WWNNAC9NQzYB/DEze9zMTjCzE4CHgUdyWywRyYuttorcP9tvD3/+c0WKz2q4wz771P/t1P7fNFg2/bRmdgjRRGNE5+59uS5YugEDBviUKVPy+ZYipeOTT2Is5733xpW/t9yS1ezuDQ3gavrJPTOb6u4Dqi7PqsnG3e9x97Pc/cx8B30RybENNohUz+PGxXDPLCN6YzT9bL99w/Yh9VNt4Dezr81sUYbb12a2KJ+FFJE8OOQQePtt2G67eH7++TBxYo0vaWjb//TpavophGoDv7t3cve1M9w6ufva+SykiORJavzll1/GeP9Bg+DnP6819XNjDP2U/FE+fhFZU+fOMG1ajAC6/vqYA+C//631ZQ05ASj4548Cv4hk1r49XH45vPACdO0aKR8WLszqpQr+TZsCv4jUbMCAuOp3wgRYd91I+vbgg7VG9/rW/jXkM/cU+EWkdq1bx1BPiGGfBx4YU3G9916tL1Xtv+lR4BeRujn4YBg9OlI9b7cd/P3vsGpVjS9pSO1fGp8Cv4jUTYsWcOqpMePXoEFw1llw1FFZvVRNP01DNknaRETWtMkm0dZ/xx2w/vqxbOnSiNJrrVXty+qb9M1MV/s2FtX4RaT+zGIKr0HJnEwXXwz9+sFzz9X4MrX7F5YCv4g0nj32gCVLYPfdI+f/119Xu6mCf+Eo8ItI49l3X5gxI5K+jR4N224LkydXu7mGfBaGAr+INK5OneAf/4Bnn4UuXWLsfy1U+88vBX4RyY1dd4WXX45aP8Dw4TB2bLVRviHBXyeAulHgF5HcSUXkJUsi9cNRR8FPfwoffphxc+X6yQ8FfhHJvQ4dYqTPFVdE6odtt4V//jPSP2SgYZu5pcAvIvnRsiWceWZ0/g4cCBdcAJ99Vu3m6vTNHQV+EcmvzTaD8ePhpZege/eo9d94Y8b5ft1hzJi6v4VOADVT4BeR/DODzTePx+PHw5AhFVlAqygrU7t/Y1PgF5HC2m8/uP/+aPYZODAmf/nmmzU2U7t/41HgF5HCO+igSPp28snwt7/F8wyU4bNxKEmbiDQN66wD114bQz5bJHXSb7+NxG9pF4HVJ8mbErxVphq/iDQtP/whfP/78fjii2O+33vvrbSJav4No8AvIk3X4YdHyudDDonbJ5+sXlWfi7002ico8ItI09W/P7z4Ilx6KTz8cNT+H3qo0iaq/dedAr+ING2tW8O558K0abDzznEdQBUK/nWT98BvZpuY2QQzm21mM83sjHyXQUSaoa22ijH/qaRvJ58Mf/0rrFxZ712WavAvRI1/JfBrd+8N7AKcZmZ9ClAOEWmuli+Pcf9nnw277AKvvabsnnWQ98Dv7p+4+8vJ46+B2cBG+S6HiDRjbdrAfffBnXdGps8BA2DECPzbpfXeZSkF/4K28ZtZL6Af8EKGdUPNbIqZTVmwYEG+iyYiTZ1ZjPqZNQuOPhquuQa+/FLj9bNQsMBvZh2Be4Bfufuiquvd/Tp3H+DuA7p165b/AopI89ClC9x8M8yeHUM/y8vxv/4N/2qNsFKrUmn2KUjgN7PWRNC/3d3vrW17EZFa9egR988/D+ecA9tuiz/0cL12VezBvxCjegy4AZjt7lfk+/1FpMjttltM+tK5MxxwAF52FF1Rc3G6QtT4dwOOBfY0s1eT248LUA4RKVYDB8LUqZHyYdw4Fgz8KVC3xv9Us08x1v7znqTN3ScDRfhRikiT0qYN/P73cOih8PXX+ECDb7+lZ/v5zKFnnXZVbEnedOWuiBS3Pn3iFwDAn/7ETLbldEbRglWFLVcBKfCLSOkYOpSO+32fUQznGb5Pb2YVukQFocAvIqWjZ0945BG49Va+1+VNXqEfx3JLVi9Nb/Nv7u3+CvwiUlrM4JhjYNYs7uQIpjAgFlNe5900Vwr8IlKaunfnOL+F2USqsFs5lis4k/YsKXDBck+BX0RKmjv4ylUcfWpnzuRKZrAdP2J8oYuVUwr8IiItW8Lo0Xyfp1lOG8azLzdxAp1ZWOiS5YQCv4hIYjLfZwdeYyTnsx+P0YblNW5ftcO3ubT7K/CLiCTcYRltuYCRbMHbzKcHRjl/YgQb8lFW+2gOwV+BX0QkTWoS98XeEXfYnmmcyd+ZRR+Gcm2dR/80RQr8IiI1eI0d6ct0prIT1/ILnmJPtuTNQherQRT4RURq8S6bsxdPMoTr2ZFXuZeDqWvSt6ZEgV9EJCvGjQyhN7M5llsBox3fsAOvFrpgdabALyJSg6pZOeexAa/SD4Df8hemMIBLOZe2fJtx+6ZIgV9EpBapDt+qtwu/+BWtTjqec/kL3265Az5xUqGLmhUFfhGRerL11sVuvIG9eIJ33loFe+zBmfb3jOP7m9I4fwV+EZF6SA/iT7EXfZnO5fyGx9gPgNbVXPzVFIJ/3mfgEhEpRt/SnnO4fPXzWzkWwxnGKObTo4AlW5Nq/CIijc6ZxvYcxH+ZTW+O52aa0vBPBX4RkUZnXMIIduA1ZtGHmzmRx9mXjZhb6IIBCvwiIjnzBtvwA57mVEazFW+ygtaFLhKgwC8iUi/Zjtd3WnANp7IlbzGfHvjKVXDqqTBjRm4LWAMFfhGReqpufH+m2wpvHSeLd96Bu++G/v3hwgth2bK8l1uBX0Qkn7baCmbPhsMPhz/8Afr1g+efz2sRFPhFRPKta1e47TZ45BFYvBhOOAFWrcrb2yvwi4gUyv77w8yZcN99Mf3jN9/AE0/k/G0V+EVECqlTJ+jTJx6PGgU/+hEceyx89lnO3rIggd/M9jOzN8zsbTM7txBlEBHJtZpy9mS6rXXur/gDv2PFbXewoFtvDrF7cpLjJ++B38xaAqOB/YE+QJmZ9cl3OUREcqk+wXo5a3Ehf6A/L/Mu36ELnzdof9UpRK6e7wJvu/u7AGZ2B3AQMKsAZRERaXJm0Jfv8RxObjK6FaKpZyPgw7Tnc5NllZjZUDObYmZTFixYkLfCiYg0BeW0xHMUogsR+DOdwta4Bs7dr3P3Ae4+oFu3bnkolohIaShE4J8LbJL2fGPg4wKUQ0SkJBUi8L8EbGlmm5lZG+BI4IEClENEJGcae+7dxtxf3jt33X2lmZ0OPA60BG5095n5LoeISK411YnXCzIDl7s/AjxSiPcWESl1unJXRKTEKPCLiJQYBX4RkRKjwC8iUmLMm2q3cxozWwB8UM+XdwVyl+auaSj2Yyz244PiP8ZiPz5omsfY093XuAK2WQT+hjCzKe4+oNDlyKViP8ZiPz4o/mMs9uOD5nWMauoRESkxCvwiIiWmFAL/dYUuQB4U+zEW+/FB8R9jsR8fNKNjLPo2fhERqawUavwiIpJGgV9EpMQUdeAv5kndzWwTM5tgZrPNbKaZnVHoMuWKmbU0s1fM7KFCl6WxmVlnMxtnZq8nf8tdC12mxmZmZybf0RlmNtbM2ha6TA1lZjea2Xwzm5G2bD0z+z8zeyu5X7eQZaxJ0Qb+EpjUfSXwa3fvDewCnFZkx5fuDGB2oQuRI1cBj7n7NsAOFNlxmtlGwHBggLtvR6RiP7KwpWoUNwP7VVl2LvCku28JPJk8b5KKNvCTNqm7uy8HUpO6FwV3/8TdX04ef00EjDXmLm7uzGxj4CfA9YUuS2Mzs7WBHwA3ALj7cnf/srClyolWQDszawW0pwhm3HP3p4Evqiw+CPhP8vg/wOC8FqoOijnwZzWpezEws15AP+CFwpYkJ64EzgHKC12QHPgOsAC4KWnKut7MOhS6UI3J3T8C/grMAT4BvnL38YUtVc70cPdPICpmQPcCl6daxRz4s5rUvbkzs47APcCv3H1RocvTmMzsAGC+u08tdFlypBXQH7jG3fsBS2jCzQP1kbRzHwRsBmwIdDCzYwpbKinmwF/0k7qbWWsi6N/u7vcWujw5sBtwoJm9TzTV7WlmtxW2SI1qLjDX3VO/1MYRJ4JisjfwnrsvcPcVwL3A9wpcplz51Mw2AEju5xe4PNUq5sBf1JO6m5kRbcOz3f2KQpcnF9z9PHff2N17EX+/p9y9aGqL7j4P+NDMtk4W7QXMKmCRcmEOsIuZtU++s3tRZB3YaR4Ajk8eHw/8t4BlqVFB5tzNhxKY1H034Fhgupm9miw7P5nPWJqPYcDtSeXkXeDEApenUbn7C2Y2DniZGIn2Cs0otUF1zGwssAfQ1czmAhcCfwbuMrMhxAnvsMKVsGZK2SAiUmKKualHREQyUOAXESkxCvwiIiVGgV9EpMQo8IuIlBgFfik5Zra40GWASLWRnt1RJF8U+EUaQZKATKRZUOCXkmXh8iRP/HQzOyJZ3sLM/pnkkH/IzB4xs0MzvH6imV1iZpOAM8zsp2b2QpJw7Qkz65Fsd1GSv32imb1rZsMz7Os7yet2zvmBS8lTLUVK2cHAjkQe/K7AS2b2NHFVdC+gL5FhcTZwYzX76OzuP4TVCcl2cXc3s5OJrKK/TrbbBhgEdALeMLNrUjtIUjbcAZzo7q8ikmMK/FLKdgfGuvsqIsHWJGDnZPnd7l4OzDOzCTXs4860xxsDdyYJutoA76Wte9jdlwHLzGw+0CNZ3o3I6XJIkaUUkSZMTT1SyjKl7q5peSZL0h6PAq52977Az4H0KQaXpT1eRUWl6yti3ojd6vCeIg2iwC+l7GngiGRO327EbFgvApOBQ5K2/h5EMq5srAN8lDw+vqYN0ywnZmo6zsyOyrrkIg2gph4pZfcBuwKvEZP0nOPu88zsHiJ98AzgTWJms6+y2N9FwN1m9hHwP2LykVq5+5Jk0pn/M7Ml7t5k0/lKcVB2TpEMzKyjuy82sy7Er4Ddkvz5Is2eavwimT1kZp2JTto/KuhLMVGNX0SkxKhzV0SkxCjwi4iUGAV+EZESo8AvIlJiFPhFRErM/wOdrOEzXF1bnQAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "# we use the least squares method to estimate the slope (i.e. Zipf’s law’s parameters a)\n",
    "import scipy.optimize \n",
    "\n",
    "c = np.log(max(fdist.values()))  # intercept if the log-frequency of the most frequent word\n",
    "\n",
    "def func(x, a):\n",
    "    return c - a * x\n",
    "\n",
    "a = scipy.optimize.curve_fit(func, log_x, log_y)[0]  # estimated Zipf's law's a coefficient\n",
    "\n",
    "zipf_fit = [func(x, a) for x in log_x]\n",
    "\n",
    "plt.plot(log_x, log_y, \"o\", color = \"blue\", markerfacecolor='None');\n",
    "plt.plot(log_x, zipf_fit, linestyle = \"--\", color = \"red\")\n",
    "plt.ylabel(\"log frequency\")\n",
    "plt.xlabel(\"log rank\")\n",
    "\n",
    "plt.title(\"Brown Corpus Log Rank/Log Frequency Plot\")\n",
    "\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Zipf’s law only **approximates** the relation between rank and frequency in an actual corpus:\n",
    "\n",
    "- the approximation works better for **middle range data**\n",
    "\n",
    "\n",
    "- **low fit** for lowest and higher ranks\n",
    "\n",
    "\n",
    "- the highest ranks of the actual counts are composed by increasingly wider frequency ranges\n",
    "\n",
    "\n",
    "- However it is **universal**: the word distribution of each text (in every language, approximately) follows the Zipf’s law"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Zipf’s Law(s?)\n",
    "\n",
    "The frequency of a word is correlated with other properties:\n",
    "\n",
    "\n",
    "- the most frequent words are shorter ($l$ = length in characters): $f_v \\propto \\frac{1}{l_v}$\n",
    "\n",
    "\n",
    "- the most frequent words are more ambiguos ($s$ = number of senses): $s_v \\propto \\sqrt{f_v}$\n",
    "\n",
    "\n",
    "- many other phenomena follow the Zipf’s law: number of website accesses; world city populations; social networks; income distribution\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### OK, but Why?\n",
    "\n",
    "- According to Zipf, word distributions follows this law due to a principle of **economy of communication**\n",
    "\n",
    "\n",
    "- Both the speaker and the listener try to minimize their efforts:\n",
    "\n",
    "    - the speaker’s effort is minimized in using a **small vocabulary** of very common words\n",
    "    - the listener’s effort is minimized in using a **big vocabulary** of rare (and unambiguos)  words\n",
    "\n",
    "\n",
    "- Zipf’s law is the best (i.e. the most economic) compromize between these two conflicting trends"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### OK, So What? (aka \"Practical Lessons from Zipf's law\")\n",
    "\n",
    "\n",
    "- Every (collections of) text uses a **small group of very frequent words**…\n",
    "\n",
    "    - usually “closed class” words (propositions, conjunctions, determinants…)\n",
    "\n",
    "\n",
    "\n",
    "- … and large groups of rare words and hapaxes (**Large Number of Rare Events**):\n",
    "\n",
    "    - the so-called “open class” words (i.e. nouns, adjectives, verbs and adverbs)\n",
    "    - they are informative of the semantic content of a document\n",
    "\n",
    "\n",
    "- novel topics introduce novel words\n",
    "\n",
    "    - the vocabulary is **open**\n",
    "\n",
    "\n",
    "- even large corpora are not sampling the full vocabulary of types\n",
    "\n",
    "    - Lingustic data are **sparse!**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Exercise\n",
    "\n",
    "We have seen **Naive Bayes** in class last time, as a method to do classification."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "At the core of the method, there is the estimation of two probabilities:\n",
    "    \n",
    "* Given $N_c$ as the number of documents from class $c$ and $N$ as the total number of documents: $\\hat{P}(c) = \\frac{N_c}{N}$\n",
    "\n",
    "* Define $f(w_{i},c)$ as the count of the occurrences of word $i$ in documents of class $c$: $\\hat{P}(w_i|c) = \\frac{f(w_{i},c)}{\\sum_{j \\in V}f(w_{j},c)}$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "1. Consider any corpus you like from `nltk.corpus`, provided it has categories, e.g. Reuters.\n",
    "\n",
    "\n",
    "2. Prepare a dataset using the provided train and test splits. Do any pre-processing you deem necessary (and motivate your choices).\n",
    "\n",
    "\n",
    "3. Use the categories as classes and train a Naive Bayes model in order to predict the classes of new documents.\n",
    "\n",
    "\n",
    "4. Test it on your test data, and inspect the most discriminative features per class (hint: those are the highest probability words given a certain class)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}