notebooks/Generate data.ipynb

{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "from tqdm import tqdm_notebook as tqdm\n",
    "from presidio_evaluator.data_generator.main import generate,read_synth_dataset\n",
    "\n",
    "import datetime\n",
    "import json"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Generate fake PII data using Presidio's data generator"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Presidio's data generator allows you to generate a synthetic dataset with two preriquisites:\n",
    "1. A fake PII csv (We used https://www.fakenamegenerator.com/)\n",
    "2. A text file with template sentences or paragraphs. In this file, each PII entity placeholder is written in brackets. The name of the PII entity should be one of the columns in the fake PII csv file.\n",
    "\n",
    "The generator creates fake sentences based on the provided fake PII csv AND a list of [extension functions](../presidio_evaluator/data_generator/extensions.py) and a few additional 3rd party libraries like `Faker`, and `haikunator`.\n",
    "\n",
    "\n",
    "For example:\n",
    "1. **A fake PII csv**:\n",
    "\n",
    "| FIRST_NAME  |  LAST_NAME  |  EMAIL |\n",
    "|-------------|-------------|-----------|\n",
    "| David       |  Brown      |  david.brown@jobhop.com |\n",
    "| Mel         |  Brown      |  melb@hobjob.com |\n",
    "\n",
    "\n",
    "2. **Templates**:\n",
    "\n",
    "My name is [FIRST_NAME]\n",
    "\n",
    "You can email me at [EMAIL]. Thanks, [FIRST_NAME]\n",
    "\n",
    "What's your last name? It's [LAST_NAME]\n",
    "\n",
    "Every time I see you falling I get down on my knees and pray\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Generate files\n",
    "Based on these two prerequisites, a requested number of examples and an output file name:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "EXAMPLES = 100\n",
    "SPAN_TO_TAG = True #Whether to create tokens + token labels (tags)\n",
    "TEMPLATES_FILE = '../presidio_evaluator/data_generator/' \\\n",
    "                 'raw_data/templates.txt'\n",
    "KEEP_ONLY_TAGGED = False\n",
    "LOWER_CASE_RATIO = 0.1\n",
    "IGNORE_TYPES = {\"IP_ADDRESS\", 'US_SSN', 'URL'}\n",
    "\n",
    "cur_time = datetime.date.today().strftime(\"%B_%d_%Y\")\n",
    "\n",
    "OUTPUT = \"../data/generated_size_{}_date_{}.json\".format(EXAMPLES, cur_time)\n",
    "\n",
    "fake_pii_csv = '../presidio_evaluator/data_generator/' \\\n",
    "               'raw_data/FakeNameGenerator.com_3000.csv'\n",
    "utterances_file = TEMPLATES_FILE\n",
    "dictionary_path = None\n",
    "\n",
    "examples = generate(fake_pii_csv=fake_pii_csv,\n",
    "                        utterances_file=utterances_file,\n",
    "                        dictionary_path=dictionary_path,\n",
    "                        output_file=OUTPUT,\n",
    "                        lower_case_ratio=LOWER_CASE_RATIO,\n",
    "                        num_of_examples=EXAMPLES,\n",
    "                        ignore_types=IGNORE_TYPES,\n",
    "                        keep_only_tagged=KEEP_ONLY_TAGGED,\n",
    "                        span_to_tag=SPAN_TO_TAG)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To read a dataset file into the InputSample format, use `read_synth_dataset`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "input_samples = read_synth_dataset(OUTPUT)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "input_samples[0]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The full structure of each input_sample is the following. It includes different feature values per token as calculated by Spacy"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "input_samples[0].to_dict()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Verify randomness of dataset"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "from collections import Counter\n",
    "count_per_template_id = Counter([sample.metadata['Template#'] for sample in input_samples])\n",
    "for key in sorted(count_per_template_id):\n",
    "    print(\"{}: {}\".format(key,count_per_template_id[key]))\n",
    "    \n",
    "print(sum(count_per_template_id.values()))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Transform to the CONLL structure:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "from presidio_evaluator import InputSample\n",
    "\n",
    "conll = InputSample.create_conll_dataset(input_samples)\n",
    "conll.head(5)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Copyright notice:\n",
    "\n",
    "\n",
    "Data generated for evaluation was created using Fake Name Generator.\n",
    "\n",
    "Fake Name Generator identities by the [Fake Name Generator](https://www.fakenamegenerator.com/) \n",
    "are licensed under a [Creative Commons Attribution-Share Alike 3.0 United States License](http://creativecommons.org/licenses/by-sa/3.0/us/). Fake Name Generator and the Fake Name Generator logo are trademarks of Corban Works, LLC.\n"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.4"
  },
  "pycharm": {
   "stem_cell": {
    "cell_type": "raw",
    "source": [],
    "metadata": {
     "collapsed": false
    }
   }
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}