sequence_labeling_dataset_examples.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "provenance": [],
      "authorship_tag": "ABX9TyNgzFbnrZa8dAcZ77a7v67U",
      "include_colab_link": true
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    },
    "language_info": {
      "name": "python"
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/TurkuNLP/textual-data-analysis-course/blob/main/sequence_labeling_dataset_examples.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "# Sequence labeling dataset examples\n",
        "\n",
        "Let's have a look at some sequence labling datasets from the Hugging Face datasets repository (https://huggingface.co/datasets).\n",
        "\n",
        "(You can find a tutorial to `datasets` here: https://huggingface.co/docs/datasets/tutorial)\n",
        "\n",
        "First, install the `datasets` Python package:"
      ],
      "metadata": {
        "id": "PoamCQ7gRSOX"
      }
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "HTtSPJiFRNQm"
      },
      "outputs": [],
      "source": [
        "!pip install --quiet datasets"
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "Make loading a bit less verbose. (This only affects what shows on screen when loading.)"
      ],
      "metadata": {
        "id": "kGtnpTPKTNMj"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "from datasets import disable_progress_bar\n",
        "\n",
        "disable_progress_bar()"
      ],
      "metadata": {
        "id": "tGyI08cPTIkO"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "We'll again mainly use the `load_dataset` to download data by data name. For available datasets, see https://huggingface.co/datasets with the filter \"token classification\"."
      ],
      "metadata": {
        "id": "BBPfBoivR2Sx"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Example: CoNLL 2003\n",
        "\n",
        "A classic reference dataset for Named Entity Recognition. Still relevant for comparing methods, although mostly superceded by more recent resources for practical purposes."
      ],
      "metadata": {
        "id": "olVO4LerbCXD"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "from datasets import load_dataset, load_dataset_builder\n",
        "from pprint import pprint\n",
        "\n",
        "builder = load_dataset_builder('conll2003')\n",
        "\n",
        "print(builder.info.description)"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "0tfL47DWRp5M",
        "outputId": "0d071b25-08e2-461d-a829-89ab62fef913"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "The shared task of CoNLL-2003 concerns language-independent named entity recognition. We will concentrate on\n",
            "four types of named entities: persons, locations, organizations and names of miscellaneous entities that do\n",
            "not belong to the previous three groups.\n",
            "\n",
            "The CoNLL-2003 shared task data files contain four columns separated by a single space. Each word has been put on\n",
            "a separate line and there is an empty line after each sentence. The first item on each line is a word, the second\n",
            "a part-of-speech (POS) tag, the third a syntactic chunk tag and the fourth the named entity tag. The chunk tags\n",
            "and the named entity tags have the format I-TYPE which means that the word is inside a phrase of type TYPE. Only\n",
            "if two phrases of the same type immediately follow each other, the first word of the second phrase will have tag\n",
            "B-TYPE to show that it starts a new phrase. A word with tag O is not part of a phrase. Note the dataset uses IOB2\n",
            "tagging scheme, whereas the original dataset uses IOB1.\n",
            "\n",
            "For more details see https://www.clips.uantwerpen.be/conll2003/ner/ and https://www.aclweb.org/anthology/W03-0419\n",
            "\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "conll2003 = load_dataset('conll2003')"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "rAI4H4zgdNM2",
        "outputId": "0a193aa0-615c-4d93-c4b2-97d020ce06d9"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "WARNING:datasets.builder:Found cached dataset conll2003 (/root/.cache/huggingface/datasets/conll2003/conll2003/1.0.0/9a4d16a94f8674ba3466315300359b0acd891b68b6c8743ddf60b9c702adce98)\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "Let's see a summary of the contents:"
      ],
      "metadata": {
        "id": "h095kek3YGXR"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "print(conll2003)"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "ZyKi5Cf_YCGb",
        "outputId": "2149796f-d229-4842-c7f0-7ad4dd0f54ff"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "DatasetDict({\n",
            "    train: Dataset({\n",
            "        features: ['id', 'tokens', 'pos_tags', 'chunk_tags', 'ner_tags'],\n",
            "        num_rows: 14041\n",
            "    })\n",
            "    validation: Dataset({\n",
            "        features: ['id', 'tokens', 'pos_tags', 'chunk_tags', 'ner_tags'],\n",
            "        num_rows: 3250\n",
            "    })\n",
            "    test: Dataset({\n",
            "        features: ['id', 'tokens', 'pos_tags', 'chunk_tags', 'ner_tags'],\n",
            "        num_rows: 3453\n",
            "    })\n",
            "})\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "We have the standard three-way split:\n",
        "\n",
        "* `train`: 14,041 rows\n",
        "* `validation`: 3,250 rows\n",
        "* `test`: 3,453 rows\n",
        "\n",
        "Each row of this dataset has an `id`, `tokens` and three sets of \"labels\": `pos_tags`, `chunk_tags` and `ner_tags`. The POS and chunk tags are included as part of the input for machine learning approaches with manually engineered features, but are rarely used in deep learning approaches.\n",
        "\n",
        "Let's look at an example:"
      ],
      "metadata": {
        "id": "v1h7JPqjYQMD"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "pprint(conll2003['train'][0]['tokens'])\n",
        "print('\\nLabel:', conll2003['train'][0]['ner_tags'])"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "HRHTr8CpST21",
        "outputId": "cf71bbd5-1821-48ca-d4b3-57c9b9500294"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "['EU', 'rejects', 'German', 'call', 'to', 'boycott', 'British', 'lamb', '.']\n",
            "\n",
            "Label: [3, 0, 7, 0, 0, 0, 7, 0, 0]\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "By contrast to text classification datasets:\n",
        "\n",
        "* The text is pre-split into tokens to facilitate pairing labels (here, NER tags) with tokens. (Note that these will in general _not_ align fully with the tokenization created by a tokenizer for a deep learning model!)\n",
        "* Each element of the dataset typically has multiple tokens and thus multiple labels, so the number of labeled pieces of data (\"signal\" for training) is many times larger than the number of rows in the dataset.\n",
        "\n",
        "As before, to interpret the label IDs, we can look at `features`:"
      ],
      "metadata": {
        "id": "vKuYsCshZvx9"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "pprint(conll2003['train'].features)"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "L7mYglnzYON_",
        "outputId": "24aa55c0-aeae-49e3-91ee-d255c8f369c6"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "{'chunk_tags': Sequence(feature=ClassLabel(names=['O', 'B-ADJP', 'I-ADJP', 'B-ADVP', 'I-ADVP', 'B-CONJP', 'I-CONJP', 'B-INTJ', 'I-INTJ', 'B-LST', 'I-LST', 'B-NP', 'I-NP', 'B-PP', 'I-PP', 'B-PRT', 'I-PRT', 'B-SBAR', 'I-SBAR', 'B-UCP', 'I-UCP', 'B-VP', 'I-VP'], id=None), length=-1, id=None),\n",
            " 'id': Value(dtype='string', id=None),\n",
            " 'ner_tags': Sequence(feature=ClassLabel(names=['O', 'B-PER', 'I-PER', 'B-ORG', 'I-ORG', 'B-LOC', 'I-LOC', 'B-MISC', 'I-MISC'], id=None), length=-1, id=None),\n",
            " 'pos_tags': Sequence(feature=ClassLabel(names=['\"', \"''\", '#', '$', '(', ')', ',', '.', ':', '``', 'CC', 'CD', 'DT', 'EX', 'FW', 'IN', 'JJ', 'JJR', 'JJS', 'LS', 'MD', 'NN', 'NNP', 'NNPS', 'NNS', 'NN|SYM', 'PDT', 'POS', 'PRP', 'PRP$', 'RB', 'RBR', 'RBS', 'RP', 'SYM', 'TO', 'UH', 'VB', 'VBD', 'VBG', 'VBN', 'VBP', 'VBZ', 'WDT', 'WP', 'WP$', 'WRB'], id=None), length=-1, id=None),\n",
            " 'tokens': Sequence(feature=Value(dtype='string', id=None), length=-1, id=None)}\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "print(conll2003['train'].features['ner_tags'])"
      ],
      "metadata": {
        "id": "l2sffyxXWYvd",
        "outputId": "2f6873b1-7399-4b69-b837-49396fa80172",
        "colab": {
          "base_uri": "https://localhost:8080/"
        }
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Sequence(feature=ClassLabel(names=['O', 'B-PER', 'I-PER', 'B-ORG', 'I-ORG', 'B-LOC', 'I-LOC', 'B-MISC', 'I-MISC'], id=None), length=-1, id=None)\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "Label `0` corresponds to `O` (\"out\"), label 1 to `B-PER` (\"begin person name\"), etc."
      ],
      "metadata": {
        "id": "qebipkGCaJ9f"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "---\n",
        "\n",
        "## Example: OntoNotes NER\n",
        "\n",
        "Following the same process as above:"
      ],
      "metadata": {
        "id": "GIgp17W6a_sa"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "builder = load_dataset_builder('conll2012_ontonotesv5', 'english_v12')\n",
        "\n",
        "print(builder.info.description)"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "dpEbxnHtdtVm",
        "outputId": "f941226a-adbe-479c-ef0d-03bc8f541eb1"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "OntoNotes v5.0 is the final version of OntoNotes corpus, and is a large-scale, multi-genre,\n",
            "multilingual corpus manually annotated with syntactic, semantic and discourse information.\n",
            "\n",
            "This dataset is the version of OntoNotes v5.0 extended and is used in the CoNLL-2012 shared task.\n",
            "It includes v4 train/dev and v9 test data for English/Chinese/Arabic and corrected version v12 train/dev/test data (English only).\n",
            "\n",
            "The source of data is the Mendeley Data repo [ontonotes-conll2012](https://data.mendeley.com/datasets/zmycy7t9h9), which seems to be as the same as the official data, but users should use this dataset on their own responsibility.\n",
            "\n",
            "See also summaries from paperwithcode, [OntoNotes 5.0](https://paperswithcode.com/dataset/ontonotes-5-0) and [CoNLL-2012](https://paperswithcode.com/dataset/conll-2012-1)\n",
            "\n",
            "For more detailed info of the dataset like annotation, tag set, etc., you can refer to the documents in the Mendeley repo mentioned above.\n",
            "\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "ontov5eng = load_dataset('conll2012_ontonotesv5', 'english_v12')"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "ZngtHHu4aH0-",
        "outputId": "aa31f1f3-7ea9-42be-9002-25979d785978"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Downloading and preparing dataset conll2012_ontonotesv5/english_v12 to /root/.cache/huggingface/datasets/conll2012_ontonotesv5/english_v12/1.0.0/c541e760a5983b07e403e77ccf1f10864a6ae3e3dc0b994112eff9f217198c65...\n",
            "Dataset conll2012_ontonotesv5 downloaded and prepared to /root/.cache/huggingface/datasets/conll2012_ontonotesv5/english_v12/1.0.0/c541e760a5983b07e403e77ccf1f10864a6ae3e3dc0b994112eff9f217198c65. Subsequent calls will reuse this data.\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "print(ontov5eng)"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "d1ztQcntbT8j",
        "outputId": "1180d0eb-fe33-4700-dc7e-bc1402ac5730"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "DatasetDict({\n",
            "    train: Dataset({\n",
            "        features: ['document_id', 'sentences'],\n",
            "        num_rows: 10539\n",
            "    })\n",
            "    validation: Dataset({\n",
            "        features: ['document_id', 'sentences'],\n",
            "        num_rows: 1370\n",
            "    })\n",
            "    test: Dataset({\n",
            "        features: ['document_id', 'sentences'],\n",
            "        num_rows: 1200\n",
            "    })\n",
            "})\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "Here we have the conventional split into train, validation and test sets, but this time on the document level:\n",
        "\n",
        "* `train`: 10,539 documents\n",
        "* `validation`: 1,370 documents\n",
        "* `test`: 1,200 documents\n",
        "\n",
        "Let's see what's in one of those documents:"
      ],
      "metadata": {
        "id": "Cex-0nf5aXam"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "print(type(ontov5eng['train'][0]['sentences']))"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "rpWN-pltagTP",
        "outputId": "5b322bfb-a116-4aaa-8ce1-9f8546c70996"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "<class 'list'>\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "pprint(ontov5eng['train'][0]['sentences'][0])"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "icUkyU3ObFG3",
        "outputId": "7e27db9a-347c-4fea-fed4-abcf6d74216a"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "{'coref_spans': [],\n",
            " 'named_entities': [0, 0, 0, 0, 0],\n",
            " 'parse_tree': '(TOP(SBARQ(WHNP(WHNP (WP What)  (NN kind) )(PP (IN of) (NP (NN '\n",
            "               'memory) ))) (. ?) ))',\n",
            " 'part_id': 0,\n",
            " 'pos_tags': [48, 25, 18, 25, 8],\n",
            " 'predicate_framenet_ids': [None, None, None, None, None],\n",
            " 'predicate_lemmas': [None, None, None, 'memory', None],\n",
            " 'speaker': 'Speaker#1',\n",
            " 'srl_frames': [],\n",
            " 'word_senses': [None, None, None, 1.0, None],\n",
            " 'words': ['What', 'kind', 'of', 'memory', '?']}\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "This is quite a lot of annotation! For NER, we would only use `words` (tokens) and `named_entities`.\n",
        "\n",
        "Again, we can find the textual labels for numerical IDs in `features`:"
      ],
      "metadata": {
        "id": "YaAoFX6JbLw3"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "pprint(ontov5eng['train'].features)"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "HUPP9i6JaMlt",
        "outputId": "16de3f1b-51fa-4d15-fc97-5c2a0cc0f9b1"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "{'document_id': Value(dtype='string', id=None),\n",
            " 'sentences': [{'coref_spans': Sequence(feature=Sequence(feature=Value(dtype='int32', id=None), length=3, id=None), length=-1, id=None),\n",
            "                'named_entities': Sequence(feature=ClassLabel(names=['O', 'B-PERSON', 'I-PERSON', 'B-NORP', 'I-NORP', 'B-FAC', 'I-FAC', 'B-ORG', 'I-ORG', 'B-GPE', 'I-GPE', 'B-LOC', 'I-LOC', 'B-PRODUCT', 'I-PRODUCT', 'B-DATE', 'I-DATE', 'B-TIME', 'I-TIME', 'B-PERCENT', 'I-PERCENT', 'B-MONEY', 'I-MONEY', 'B-QUANTITY', 'I-QUANTITY', 'B-ORDINAL', 'I-ORDINAL', 'B-CARDINAL', 'I-CARDINAL', 'B-EVENT', 'I-EVENT', 'B-WORK_OF_ART', 'I-WORK_OF_ART', 'B-LAW', 'I-LAW', 'B-LANGUAGE', 'I-LANGUAGE'], id=None), length=-1, id=None),\n",
            "                'parse_tree': Value(dtype='string', id=None),\n",
            "                'part_id': Value(dtype='int32', id=None),\n",
            "                'pos_tags': Sequence(feature=ClassLabel(names=['XX', '``', '$', \"''\", '*', ',', '-LRB-', '-RRB-', '.', ':', 'ADD', 'AFX', 'CC', 'CD', 'DT', 'EX', 'FW', 'HYPH', 'IN', 'JJ', 'JJR', 'JJS', 'LS', 'MD', 'NFP', 'NN', 'NNP', 'NNPS', 'NNS', 'PDT', 'POS', 'PRP', 'PRP$', 'RB', 'RBR', 'RBS', 'RP', 'SYM', 'TO', 'UH', 'VB', 'VBD', 'VBG', 'VBN', 'VBP', 'VBZ', 'VERB', 'WDT', 'WP', 'WP$', 'WRB'], id=None), length=-1, id=None),\n",
            "                'predicate_framenet_ids': Sequence(feature=Value(dtype='string', id=None), length=-1, id=None),\n",
            "                'predicate_lemmas': Sequence(feature=Value(dtype='string', id=None), length=-1, id=None),\n",
            "                'speaker': Value(dtype='string', id=None),\n",
            "                'srl_frames': [{'frames': Sequence(feature=Value(dtype='string', id=None), length=-1, id=None),\n",
            "                                'verb': Value(dtype='string', id=None)}],\n",
            "                'word_senses': Sequence(feature=Value(dtype='float32', id=None), length=-1, id=None),\n",
            "                'words': Sequence(feature=Value(dtype='string', id=None), length=-1, id=None)}]}\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "print(ontov5eng['train'].features['sentences'][0]['named_entities'])"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "8JOUYN8-bvR_",
        "outputId": "c4437c65-9d17-437e-901a-68c7fd234a78"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Sequence(feature=ClassLabel(names=['O', 'B-PERSON', 'I-PERSON', 'B-NORP', 'I-NORP', 'B-FAC', 'I-FAC', 'B-ORG', 'I-ORG', 'B-GPE', 'I-GPE', 'B-LOC', 'I-LOC', 'B-PRODUCT', 'I-PRODUCT', 'B-DATE', 'I-DATE', 'B-TIME', 'I-TIME', 'B-PERCENT', 'I-PERCENT', 'B-MONEY', 'I-MONEY', 'B-QUANTITY', 'I-QUANTITY', 'B-ORDINAL', 'I-ORDINAL', 'B-CARDINAL', 'I-CARDINAL', 'B-EVENT', 'I-EVENT', 'B-WORK_OF_ART', 'I-WORK_OF_ART', 'B-LAW', 'I-LAW', 'B-LANGUAGE', 'I-LANGUAGE'], id=None), length=-1, id=None)\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "---\n",
        "\n",
        "## Example: `wikiann`\n",
        "\n",
        "[`wikiann`](https://huggingface.co/datasets/wikiann/viewer/fi/validation) is a massively multilingual automatically annotated NER dataset based on Wikipedia. Let's have a look at the Finnish (`fi`) subset."
      ],
      "metadata": {
        "id": "klzDi1TEfaX0"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "builder = load_dataset_builder('wikiann', 'fi')\n",
        "\n",
        "pprint(builder.info.description)"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "MpmpGbpefebf",
        "outputId": "f3c8d845-9961-44e4-a3a8-3c6b0183f5cf"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "('WikiANN (sometimes called PAN-X) is a multilingual named entity recognition '\n",
            " 'dataset consisting of Wikipedia articles annotated with LOC (location), PER '\n",
            " '(person), and ORG (organisation) tags in the IOB2 format. This version '\n",
            " 'corresponds to the balanced train, dev, and test splits of Rahimi et al. '\n",
            " '(2019), which supports 176 of the 282 languages from the original WikiANN '\n",
            " 'corpus.')\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "wikiannfi = load_dataset('wikiann', 'fi')"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "Dl78T8HNfiSq",
        "outputId": "bafd5f5e-9524-4dd0-b15a-cb7be4c056bb"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Downloading and preparing dataset wikiann/fi to /root/.cache/huggingface/datasets/wikiann/fi/1.1.0/4bfd4fe4468ab78bb6e096968f61fab7a888f44f9d3371c2f3fea7e74a5a354e...\n",
            "Dataset wikiann downloaded and prepared to /root/.cache/huggingface/datasets/wikiann/fi/1.1.0/4bfd4fe4468ab78bb6e096968f61fab7a888f44f9d3371c2f3fea7e74a5a354e. Subsequent calls will reuse this data.\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "print(wikiannfi)"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "XkPKTbHIgHLY",
        "outputId": "0b4096e1-1656-4237-c9d2-0cb80035c777"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "DatasetDict({\n",
            "    validation: Dataset({\n",
            "        features: ['tokens', 'ner_tags', 'langs', 'spans'],\n",
            "        num_rows: 10000\n",
            "    })\n",
            "    test: Dataset({\n",
            "        features: ['tokens', 'ner_tags', 'langs', 'spans'],\n",
            "        num_rows: 10000\n",
            "    })\n",
            "    train: Dataset({\n",
            "        features: ['tokens', 'ner_tags', 'langs', 'spans'],\n",
            "        num_rows: 20000\n",
            "    })\n",
            "})\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "The usual split:\n",
        "\n",
        "* `train`: 20,000 rows\n",
        "* `validation`: 10,000 rows\n",
        "* `test`: 10,000 rows"
      ],
      "metadata": {
        "id": "CaySYAmBgYuX"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "pprint(wikiannfi['train'][0]['tokens'])\n",
        "print('\\nLabel:', wikiannfi['train'][0]['ner_tags'])"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "IvEqAfYzdAS0",
        "outputId": "99852e13-4771-4242-a23f-726f29f833f8"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "['Se',\n",
            " 'sijaitsee',\n",
            " 'Borneon',\n",
            " 'saarella',\n",
            " 'ja',\n",
            " 'on',\n",
            " 'Etelä-Kalimantanin',\n",
            " 'provinssin',\n",
            " 'pääkaupunki',\n",
            " '.']\n",
            "\n",
            "Label: [0, 0, 5, 0, 0, 0, 5, 0, 0, 0]\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "print(wikiannfi['train'].features['ner_tags'])"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "NvEcpPvrdDUZ",
        "outputId": "0af747de-363e-4fb6-c7dd-b05548519c5a"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Sequence(feature=ClassLabel(names=['O', 'B-PER', 'I-PER', 'B-ORG', 'I-ORG', 'B-LOC', 'I-LOC'], id=None), length=-1, id=None)\n"
          ]
        }
      ]
    }
  ]
}