Gemma2B_Colab_Prototyp.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "4a3dc99b",
   "metadata": {},
   "source": [
    "# Gemma-2B Modelltraining in Google Colab Pro"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8b9f0e47",
   "metadata": {},
   "source": [
    "\n",
    "In diesem Notebook werden wir ein leichtes Sprachmodell, Gemma-2B, in Google Colab Pro laden, es mit einem speziellen Datensatz trainieren und anschließend das trainierte Modell speichern. Dieses Notebook ist für die Arbeit mit beschränkten Ressourcen optimiert und nutzt Modelle, die effizient auf Google Colab ausgeführt werden können.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "31b75620",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Requirement already satisfied: transformers in c:\\users\\alexa\\appdata\\local\\programs\\python\\python311\\lib\\site-packages (4.35.0)\n",
      "Collecting datasets\n",
      "  Downloading datasets-2.21.0-py3-none-any.whl.metadata (21 kB)\n",
      "Requirement already satisfied: bitsandbytes in c:\\users\\alexa\\appdata\\local\\programs\\python\\python311\\lib\\site-packages (0.41.2.post2)\n",
      "Requirement already satisfied: huggingface_hub in c:\\users\\alexa\\appdata\\local\\programs\\python\\python311\\lib\\site-packages (0.17.3)\n",
      "Requirement already satisfied: filelock in c:\\users\\alexa\\appdata\\local\\programs\\python\\python311\\lib\\site-packages (from transformers) (3.12.4)\n",
      "Requirement already satisfied: numpy>=1.17 in c:\\users\\alexa\\appdata\\local\\programs\\python\\python311\\lib\\site-packages (from transformers) (1.26.2)\n",
      "Requirement already satisfied: packaging>=20.0 in c:\\users\\alexa\\appdata\\local\\programs\\python\\python311\\lib\\site-packages (from transformers) (23.2)\n",
      "Requirement already satisfied: pyyaml>=5.1 in c:\\users\\alexa\\appdata\\local\\programs\\python\\python311\\lib\\site-packages (from transformers) (6.0.1)\n",
      "Requirement already satisfied: regex!=2019.12.17 in c:\\users\\alexa\\appdata\\local\\programs\\python\\python311\\lib\\site-packages (from transformers) (2023.10.3)\n",
      "Requirement already satisfied: requests in c:\\users\\alexa\\appdata\\local\\programs\\python\\python311\\lib\\site-packages (from transformers) (2.28.1)\n",
      "Requirement already satisfied: tokenizers<0.15,>=0.14 in c:\\users\\alexa\\appdata\\local\\programs\\python\\python311\\lib\\site-packages (from transformers) (0.14.1)\n",
      "Requirement already satisfied: safetensors>=0.3.1 in c:\\users\\alexa\\appdata\\local\\programs\\python\\python311\\lib\\site-packages (from transformers) (0.4.0)\n",
      "Requirement already satisfied: tqdm>=4.27 in c:\\users\\alexa\\appdata\\local\\programs\\python\\python311\\lib\\site-packages (from transformers) (4.66.1)\n",
      "Requirement already satisfied: pyarrow>=15.0.0 in c:\\users\\alexa\\appdata\\local\\programs\\python\\python311\\lib\\site-packages (from datasets) (17.0.0)\n",
      "Collecting dill<0.3.9,>=0.3.0 (from datasets)\n",
      "  Downloading dill-0.3.8-py3-none-any.whl.metadata (10 kB)\n",
      "Requirement already satisfied: pandas in c:\\users\\alexa\\appdata\\local\\programs\\python\\python311\\lib\\site-packages (from datasets) (2.2.2)\n",
      "Collecting requests (from transformers)\n",
      "  Using cached requests-2.32.3-py3-none-any.whl.metadata (4.6 kB)\n",
      "Collecting tqdm>=4.27 (from transformers)\n",
      "  Using cached tqdm-4.66.5-py3-none-any.whl.metadata (57 kB)\n",
      "Collecting xxhash (from datasets)\n",
      "  Downloading xxhash-3.5.0-cp311-cp311-win_amd64.whl.metadata (13 kB)\n",
      "Collecting multiprocess (from datasets)\n",
      "  Downloading multiprocess-0.70.16-py311-none-any.whl.metadata (7.2 kB)\n",
      "Requirement already satisfied: fsspec<=2024.6.1,>=2023.1.0 in c:\\users\\alexa\\appdata\\local\\programs\\python\\python311\\lib\\site-packages (from fsspec[http]<=2024.6.1,>=2023.1.0->datasets) (2023.10.0)\n",
      "Requirement already satisfied: aiohttp in c:\\users\\alexa\\appdata\\local\\programs\\python\\python311\\lib\\site-packages (from datasets) (3.9.5)\n",
      "Collecting huggingface_hub\n",
      "  Downloading huggingface_hub-0.24.6-py3-none-any.whl.metadata (13 kB)\n",
      "Requirement already satisfied: typing-extensions>=3.7.4.3 in c:\\users\\alexa\\appdata\\local\\programs\\python\\python311\\lib\\site-packages (from huggingface_hub) (4.12.2)\n",
      "Requirement already satisfied: aiosignal>=1.1.2 in c:\\users\\alexa\\appdata\\local\\programs\\python\\python311\\lib\\site-packages (from aiohttp->datasets) (1.3.1)\n",
      "Requirement already satisfied: attrs>=17.3.0 in c:\\users\\alexa\\appdata\\local\\programs\\python\\python311\\lib\\site-packages (from aiohttp->datasets) (23.1.0)\n",
      "Requirement already satisfied: frozenlist>=1.1.1 in c:\\users\\alexa\\appdata\\local\\programs\\python\\python311\\lib\\site-packages (from aiohttp->datasets) (1.4.0)\n",
      "Requirement already satisfied: multidict<7.0,>=4.5 in c:\\users\\alexa\\appdata\\local\\programs\\python\\python311\\lib\\site-packages (from aiohttp->datasets) (6.0.4)\n",
      "Requirement already satisfied: yarl<2.0,>=1.0 in c:\\users\\alexa\\appdata\\local\\programs\\python\\python311\\lib\\site-packages (from aiohttp->datasets) (1.9.2)\n",
      "Requirement already satisfied: charset-normalizer<4,>=2 in c:\\users\\alexa\\appdata\\local\\programs\\python\\python311\\lib\\site-packages (from requests->transformers) (2.1.1)\n",
      "Requirement already satisfied: idna<4,>=2.5 in c:\\users\\alexa\\appdata\\local\\programs\\python\\python311\\lib\\site-packages (from requests->transformers) (3.7)\n",
      "Requirement already satisfied: urllib3<3,>=1.21.1 in c:\\users\\alexa\\appdata\\local\\programs\\python\\python311\\lib\\site-packages (from requests->transformers) (1.26.18)\n",
      "Requirement already satisfied: certifi>=2017.4.17 in c:\\users\\alexa\\appdata\\local\\programs\\python\\python311\\lib\\site-packages (from requests->transformers) (2023.7.22)\n",
      "INFO: pip is looking at multiple versions of tokenizers to determine which version is compatible with other requirements. This could take a while.\n",
      "Collecting tokenizers<0.15,>=0.14 (from transformers)\n",
      "  Downloading tokenizers-0.14.0-cp311-none-win_amd64.whl.metadata (6.8 kB)\n",
      "Collecting datasets\n",
      "  Downloading datasets-2.20.0-py3-none-any.whl.metadata (19 kB)\n",
      "Collecting pyarrow-hotfix (from datasets)\n",
      "  Downloading pyarrow_hotfix-0.6-py3-none-any.whl.metadata (3.6 kB)\n",
      "Collecting datasets\n",
      "  Downloading datasets-2.19.2-py3-none-any.whl.metadata (19 kB)\n",
      "  Downloading datasets-2.19.1-py3-none-any.whl.metadata (19 kB)\n",
      "INFO: pip is still looking at multiple versions of tokenizers to determine which version is compatible with other requirements. This could take a while.\n",
      "  Downloading datasets-2.19.0-py3-none-any.whl.metadata (19 kB)\n",
      "  Downloading datasets-2.18.0-py3-none-any.whl.metadata (20 kB)\n",
      "  Downloading datasets-2.17.1-py3-none-any.whl.metadata (20 kB)\n",
      "INFO: This is taking longer than usual. You might need to provide the dependency resolver with stricter constraints to reduce runtime. See https://pip.pypa.io/warnings/backtracking for guidance. If you want to abort this run, press Ctrl + C.\n",
      "  Downloading datasets-2.17.0-py3-none-any.whl.metadata (20 kB)\n",
      "  Downloading datasets-2.16.1-py3-none-any.whl.metadata (20 kB)\n",
      "Collecting dill<0.3.8,>=0.3.0 (from datasets)\n",
      "  Downloading dill-0.3.7-py3-none-any.whl.metadata (9.9 kB)\n",
      "Collecting datasets\n",
      "  Downloading datasets-2.16.0-py3-none-any.whl.metadata (20 kB)\n",
      "  Downloading datasets-2.15.0-py3-none-any.whl.metadata (20 kB)\n",
      "  Downloading datasets-2.14.7-py3-none-any.whl.metadata (19 kB)\n",
      "Requirement already satisfied: colorama in c:\\users\\alexa\\appdata\\local\\programs\\python\\python311\\lib\\site-packages (from tqdm>=4.27->transformers) (0.4.6)\n",
      "INFO: pip is looking at multiple versions of multiprocess to determine which version is compatible with other requirements. This could take a while.\n",
      "Collecting multiprocess (from datasets)\n",
      "  Downloading multiprocess-0.70.15-py311-none-any.whl.metadata (7.2 kB)\n",
      "Requirement already satisfied: python-dateutil>=2.8.2 in c:\\users\\alexa\\appdata\\roaming\\python\\python311\\site-packages (from pandas->datasets) (2.8.2)\n",
      "Requirement already satisfied: pytz>=2020.1 in c:\\users\\alexa\\appdata\\local\\programs\\python\\python311\\lib\\site-packages (from pandas->datasets) (2023.3.post1)\n",
      "Requirement already satisfied: tzdata>=2022.7 in c:\\users\\alexa\\appdata\\local\\programs\\python\\python311\\lib\\site-packages (from pandas->datasets) (2023.3)\n",
      "Requirement already satisfied: six>=1.5 in c:\\users\\alexa\\appdata\\roaming\\python\\python311\\site-packages (from python-dateutil>=2.8.2->pandas->datasets) (1.16.0)\n",
      "Downloading datasets-2.14.7-py3-none-any.whl (520 kB)\n",
      "Downloading dill-0.3.7-py3-none-any.whl (115 kB)\n",
      "Downloading multiprocess-0.70.15-py311-none-any.whl (135 kB)\n",
      "Downloading pyarrow_hotfix-0.6-py3-none-any.whl (7.9 kB)\n",
      "Downloading xxhash-3.5.0-cp311-cp311-win_amd64.whl (30 kB)\n",
      "Installing collected packages: xxhash, pyarrow-hotfix, dill, multiprocess, datasets\n",
      "Successfully installed datasets-2.14.7 dill-0.3.7 multiprocess-0.70.15 pyarrow-hotfix-0.6 xxhash-3.5.0\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "ERROR: Invalid requirement: '#': Expected package name at the start of dependency specifier\n",
      "    #\n",
      "    ^\n"
     ]
    }
   ],
   "source": [
    "\n",
    "# Install necessary libraries\n",
    "!pip install transformers datasets bitsandbytes huggingface_hub\n",
    "!pip install ctransformers  # Für die Arbeit mit GGUF-Modellen\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "45fa8d10",
   "metadata": {},
   "source": [
    "## Modell von Hugging Face laden"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "93b63d2e",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "639c17994b4b40709abc504309a7f9de",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Fetching 12 files:   0%|          | 0/12 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "40f191c0bea743ad82bb19afc786a750",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Downloading README.md:   0%|          | 0.00/21.5k [00:00<?, ?B/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "ename": "GatedRepoError",
     "evalue": "401 Client Error. (Request ID: Root=1-66d06ab8-07588401657b07a3538d9179;e8c91b55-8ae2-4d6b-8108-2ca0c00a97a3)\n\nCannot access gated repo for url https://huggingface.co/google/gemma-2b/resolve/68e273d91b1d6ea57c9e6024c4f887832f7b43fa/.gitattributes.\nAccess to model google/gemma-2b is restricted. You must be authenticated to access it.",
     "output_type": "error",
     "traceback": [
      "\u001b[1;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[1;31mHTTPError\u001b[0m                                 Traceback (most recent call last)",
      "File \u001b[1;32mc:\\Users\\alexa\\AppData\\Local\\Programs\\Python\\Python311\\Lib\\site-packages\\huggingface_hub\\utils\\_errors.py:261\u001b[0m, in \u001b[0;36mhf_raise_for_status\u001b[1;34m(response, endpoint_name)\u001b[0m\n\u001b[0;32m    260\u001b[0m \u001b[38;5;28;01mtry\u001b[39;00m:\n\u001b[1;32m--> 261\u001b[0m     \u001b[43mresponse\u001b[49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43mraise_for_status\u001b[49m\u001b[43m(\u001b[49m\u001b[43m)\u001b[49m\n\u001b[0;32m    262\u001b[0m \u001b[38;5;28;01mexcept\u001b[39;00m HTTPError \u001b[38;5;28;01mas\u001b[39;00m e:\n",
      "File \u001b[1;32mc:\\Users\\alexa\\AppData\\Local\\Programs\\Python\\Python311\\Lib\\site-packages\\requests\\models.py:1021\u001b[0m, in \u001b[0;36mResponse.raise_for_status\u001b[1;34m(self)\u001b[0m\n\u001b[0;32m   1020\u001b[0m \u001b[38;5;28;01mif\u001b[39;00m http_error_msg:\n\u001b[1;32m-> 1021\u001b[0m     \u001b[38;5;28;01mraise\u001b[39;00m HTTPError(http_error_msg, response\u001b[38;5;241m=\u001b[39m\u001b[38;5;28mself\u001b[39m)\n",
      "\u001b[1;31mHTTPError\u001b[0m: 401 Client Error: Unauthorized for url: https://huggingface.co/google/gemma-2b/resolve/68e273d91b1d6ea57c9e6024c4f887832f7b43fa/.gitattributes",
      "\nThe above exception was the direct cause of the following exception:\n",
      "\u001b[1;31mGatedRepoError\u001b[0m                            Traceback (most recent call last)",
      "Cell \u001b[1;32mIn[2], line 5\u001b[0m\n\u001b[0;32m      3\u001b[0m \u001b[38;5;66;03m# Lade das Gemma-2B Modell herunter\u001b[39;00m\n\u001b[0;32m      4\u001b[0m model_id \u001b[38;5;241m=\u001b[39m \u001b[38;5;124m\"\u001b[39m\u001b[38;5;124mgoogle/gemma-2b\u001b[39m\u001b[38;5;124m\"\u001b[39m\n\u001b[1;32m----> 5\u001b[0m \u001b[43msnapshot_download\u001b[49m\u001b[43m(\u001b[49m\u001b[43mrepo_id\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[43mmodel_id\u001b[49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[43mlocal_dir\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[38;5;124;43m\"\u001b[39;49m\u001b[38;5;124;43mgemma-2b\u001b[39;49m\u001b[38;5;124;43m\"\u001b[39;49m\u001b[43m)\u001b[49m\n",
      "File \u001b[1;32mc:\\Users\\alexa\\AppData\\Local\\Programs\\Python\\Python311\\Lib\\site-packages\\huggingface_hub\\utils\\_validators.py:118\u001b[0m, in \u001b[0;36mvalidate_hf_hub_args.<locals>._inner_fn\u001b[1;34m(*args, **kwargs)\u001b[0m\n\u001b[0;32m    115\u001b[0m \u001b[38;5;28;01mif\u001b[39;00m check_use_auth_token:\n\u001b[0;32m    116\u001b[0m     kwargs \u001b[38;5;241m=\u001b[39m smoothly_deprecate_use_auth_token(fn_name\u001b[38;5;241m=\u001b[39mfn\u001b[38;5;241m.\u001b[39m\u001b[38;5;18m__name__\u001b[39m, has_token\u001b[38;5;241m=\u001b[39mhas_token, kwargs\u001b[38;5;241m=\u001b[39mkwargs)\n\u001b[1;32m--> 118\u001b[0m \u001b[38;5;28;01mreturn\u001b[39;00m \u001b[43mfn\u001b[49m\u001b[43m(\u001b[49m\u001b[38;5;241;43m*\u001b[39;49m\u001b[43margs\u001b[49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[38;5;241;43m*\u001b[39;49m\u001b[38;5;241;43m*\u001b[39;49m\u001b[43mkwargs\u001b[49m\u001b[43m)\u001b[49m\n",
      "File \u001b[1;32mc:\\Users\\alexa\\AppData\\Local\\Programs\\Python\\Python311\\Lib\\site-packages\\huggingface_hub\\_snapshot_download.py:239\u001b[0m, in \u001b[0;36msnapshot_download\u001b[1;34m(repo_id, repo_type, revision, endpoint, cache_dir, local_dir, local_dir_use_symlinks, library_name, library_version, user_agent, proxies, etag_timeout, resume_download, force_download, token, local_files_only, allow_patterns, ignore_patterns, max_workers, tqdm_class)\u001b[0m\n\u001b[0;32m    237\u001b[0m         _inner_hf_hub_download(file)\n\u001b[0;32m    238\u001b[0m \u001b[38;5;28;01melse\u001b[39;00m:\n\u001b[1;32m--> 239\u001b[0m     \u001b[43mthread_map\u001b[49m\u001b[43m(\u001b[49m\n\u001b[0;32m    240\u001b[0m \u001b[43m        \u001b[49m\u001b[43m_inner_hf_hub_download\u001b[49m\u001b[43m,\u001b[49m\n\u001b[0;32m    241\u001b[0m \u001b[43m        \u001b[49m\u001b[43mfiltered_repo_files\u001b[49m\u001b[43m,\u001b[49m\n\u001b[0;32m    242\u001b[0m \u001b[43m        \u001b[49m\u001b[43mdesc\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[38;5;124;43mf\u001b[39;49m\u001b[38;5;124;43m\"\u001b[39;49m\u001b[38;5;124;43mFetching \u001b[39;49m\u001b[38;5;132;43;01m{\u001b[39;49;00m\u001b[38;5;28;43mlen\u001b[39;49m\u001b[43m(\u001b[49m\u001b[43mfiltered_repo_files\u001b[49m\u001b[43m)\u001b[49m\u001b[38;5;132;43;01m}\u001b[39;49;00m\u001b[38;5;124;43m files\u001b[39;49m\u001b[38;5;124;43m\"\u001b[39;49m\u001b[43m,\u001b[49m\n\u001b[0;32m    243\u001b[0m \u001b[43m        \u001b[49m\u001b[43mmax_workers\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[43mmax_workers\u001b[49m\u001b[43m,\u001b[49m\n\u001b[0;32m    244\u001b[0m \u001b[43m        \u001b[49m\u001b[38;5;66;43;03m# User can use its own tqdm class or the default one from `huggingface_hub.utils`\u001b[39;49;00m\n\u001b[0;32m    245\u001b[0m \u001b[43m        \u001b[49m\u001b[43mtqdm_class\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[43mtqdm_class\u001b[49m\u001b[43m \u001b[49m\u001b[38;5;129;43;01mor\u001b[39;49;00m\u001b[43m \u001b[49m\u001b[43mhf_tqdm\u001b[49m\u001b[43m,\u001b[49m\n\u001b[0;32m    246\u001b[0m \u001b[43m    \u001b[49m\u001b[43m)\u001b[49m\n\u001b[0;32m    248\u001b[0m \u001b[38;5;28;01mif\u001b[39;00m local_dir \u001b[38;5;129;01mis\u001b[39;00m \u001b[38;5;129;01mnot\u001b[39;00m \u001b[38;5;28;01mNone\u001b[39;00m:\n\u001b[0;32m    249\u001b[0m     \u001b[38;5;28;01mreturn\u001b[39;00m \u001b[38;5;28mstr\u001b[39m(os\u001b[38;5;241m.\u001b[39mpath\u001b[38;5;241m.\u001b[39mrealpath(local_dir))\n",
      "File \u001b[1;32mc:\\Users\\alexa\\AppData\\Local\\Programs\\Python\\Python311\\Lib\\site-packages\\tqdm\\contrib\\concurrent.py:69\u001b[0m, in \u001b[0;36mthread_map\u001b[1;34m(fn, *iterables, **tqdm_kwargs)\u001b[0m\n\u001b[0;32m     55\u001b[0m \u001b[38;5;250m\u001b[39m\u001b[38;5;124;03m\"\"\"\u001b[39;00m\n\u001b[0;32m     56\u001b[0m \u001b[38;5;124;03mEquivalent of `list(map(fn, *iterables))`\u001b[39;00m\n\u001b[0;32m     57\u001b[0m \u001b[38;5;124;03mdriven by `concurrent.futures.ThreadPoolExecutor`.\u001b[39;00m\n\u001b[1;32m   (...)\u001b[0m\n\u001b[0;32m     66\u001b[0m \u001b[38;5;124;03m    [default: max(32, cpu_count() + 4)].\u001b[39;00m\n\u001b[0;32m     67\u001b[0m \u001b[38;5;124;03m\"\"\"\u001b[39;00m\n\u001b[0;32m     68\u001b[0m \u001b[38;5;28;01mfrom\u001b[39;00m \u001b[38;5;21;01mconcurrent\u001b[39;00m\u001b[38;5;21;01m.\u001b[39;00m\u001b[38;5;21;01mfutures\u001b[39;00m \u001b[38;5;28;01mimport\u001b[39;00m ThreadPoolExecutor\n\u001b[1;32m---> 69\u001b[0m \u001b[38;5;28;01mreturn\u001b[39;00m \u001b[43m_executor_map\u001b[49m\u001b[43m(\u001b[49m\u001b[43mThreadPoolExecutor\u001b[49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[43mfn\u001b[49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[38;5;241;43m*\u001b[39;49m\u001b[43miterables\u001b[49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[38;5;241;43m*\u001b[39;49m\u001b[38;5;241;43m*\u001b[39;49m\u001b[43mtqdm_kwargs\u001b[49m\u001b[43m)\u001b[49m\n",
      "File \u001b[1;32mc:\\Users\\alexa\\AppData\\Local\\Programs\\Python\\Python311\\Lib\\site-packages\\tqdm\\contrib\\concurrent.py:51\u001b[0m, in \u001b[0;36m_executor_map\u001b[1;34m(PoolExecutor, fn, *iterables, **tqdm_kwargs)\u001b[0m\n\u001b[0;32m     47\u001b[0m \u001b[38;5;28;01mwith\u001b[39;00m ensure_lock(tqdm_class, lock_name\u001b[38;5;241m=\u001b[39mlock_name) \u001b[38;5;28;01mas\u001b[39;00m lk:\n\u001b[0;32m     48\u001b[0m     \u001b[38;5;66;03m# share lock in case workers are already using `tqdm`\u001b[39;00m\n\u001b[0;32m     49\u001b[0m     \u001b[38;5;28;01mwith\u001b[39;00m PoolExecutor(max_workers\u001b[38;5;241m=\u001b[39mmax_workers, initializer\u001b[38;5;241m=\u001b[39mtqdm_class\u001b[38;5;241m.\u001b[39mset_lock,\n\u001b[0;32m     50\u001b[0m                       initargs\u001b[38;5;241m=\u001b[39m(lk,)) \u001b[38;5;28;01mas\u001b[39;00m ex:\n\u001b[1;32m---> 51\u001b[0m         \u001b[38;5;28;01mreturn\u001b[39;00m \u001b[38;5;28;43mlist\u001b[39;49m\u001b[43m(\u001b[49m\u001b[43mtqdm_class\u001b[49m\u001b[43m(\u001b[49m\u001b[43mex\u001b[49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43mmap\u001b[49m\u001b[43m(\u001b[49m\u001b[43mfn\u001b[49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[38;5;241;43m*\u001b[39;49m\u001b[43miterables\u001b[49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[43mchunksize\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[43mchunksize\u001b[49m\u001b[43m)\u001b[49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[38;5;241;43m*\u001b[39;49m\u001b[38;5;241;43m*\u001b[39;49m\u001b[43mkwargs\u001b[49m\u001b[43m)\u001b[49m\u001b[43m)\u001b[49m\n",
      "File \u001b[1;32mc:\\Users\\alexa\\AppData\\Local\\Programs\\Python\\Python311\\Lib\\site-packages\\tqdm\\notebook.py:249\u001b[0m, in \u001b[0;36mtqdm_notebook.__iter__\u001b[1;34m(self)\u001b[0m\n\u001b[0;32m    247\u001b[0m \u001b[38;5;28;01mtry\u001b[39;00m:\n\u001b[0;32m    248\u001b[0m     it \u001b[38;5;241m=\u001b[39m \u001b[38;5;28msuper\u001b[39m(tqdm_notebook, \u001b[38;5;28mself\u001b[39m)\u001b[38;5;241m.\u001b[39m\u001b[38;5;21m__iter__\u001b[39m()\n\u001b[1;32m--> 249\u001b[0m \u001b[43m    \u001b[49m\u001b[38;5;28;43;01mfor\u001b[39;49;00m\u001b[43m \u001b[49m\u001b[43mobj\u001b[49m\u001b[43m \u001b[49m\u001b[38;5;129;43;01min\u001b[39;49;00m\u001b[43m \u001b[49m\u001b[43mit\u001b[49m\u001b[43m:\u001b[49m\n\u001b[0;32m    250\u001b[0m \u001b[43m        \u001b[49m\u001b[38;5;66;43;03m# return super(tqdm...) will not catch exception\u001b[39;49;00m\n\u001b[0;32m    251\u001b[0m \u001b[43m        \u001b[49m\u001b[38;5;28;43;01myield\u001b[39;49;00m\u001b[43m \u001b[49m\u001b[43mobj\u001b[49m\n\u001b[0;32m    252\u001b[0m \u001b[38;5;66;03m# NB: except ... [ as ...] breaks IPython async KeyboardInterrupt\u001b[39;00m\n",
      "File \u001b[1;32mc:\\Users\\alexa\\AppData\\Local\\Programs\\Python\\Python311\\Lib\\site-packages\\tqdm\\std.py:1182\u001b[0m, in \u001b[0;36mtqdm.__iter__\u001b[1;34m(self)\u001b[0m\n\u001b[0;32m   1179\u001b[0m time \u001b[38;5;241m=\u001b[39m \u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39m_time\n\u001b[0;32m   1181\u001b[0m \u001b[38;5;28;01mtry\u001b[39;00m:\n\u001b[1;32m-> 1182\u001b[0m \u001b[43m    \u001b[49m\u001b[38;5;28;43;01mfor\u001b[39;49;00m\u001b[43m \u001b[49m\u001b[43mobj\u001b[49m\u001b[43m \u001b[49m\u001b[38;5;129;43;01min\u001b[39;49;00m\u001b[43m \u001b[49m\u001b[43miterable\u001b[49m\u001b[43m:\u001b[49m\n\u001b[0;32m   1183\u001b[0m \u001b[43m        \u001b[49m\u001b[38;5;28;43;01myield\u001b[39;49;00m\u001b[43m \u001b[49m\u001b[43mobj\u001b[49m\n\u001b[0;32m   1184\u001b[0m \u001b[43m        \u001b[49m\u001b[38;5;66;43;03m# Update and possibly print the progressbar.\u001b[39;49;00m\n\u001b[0;32m   1185\u001b[0m \u001b[43m        \u001b[49m\u001b[38;5;66;43;03m# Note: does not call self.update(1) for speed optimisation.\u001b[39;49;00m\n",
      "File \u001b[1;32mc:\\Users\\alexa\\AppData\\Local\\Programs\\Python\\Python311\\Lib\\concurrent\\futures\\_base.py:619\u001b[0m, in \u001b[0;36mExecutor.map.<locals>.result_iterator\u001b[1;34m()\u001b[0m\n\u001b[0;32m    616\u001b[0m \u001b[38;5;28;01mwhile\u001b[39;00m fs:\n\u001b[0;32m    617\u001b[0m     \u001b[38;5;66;03m# Careful not to keep a reference to the popped future\u001b[39;00m\n\u001b[0;32m    618\u001b[0m     \u001b[38;5;28;01mif\u001b[39;00m timeout \u001b[38;5;129;01mis\u001b[39;00m \u001b[38;5;28;01mNone\u001b[39;00m:\n\u001b[1;32m--> 619\u001b[0m         \u001b[38;5;28;01myield\u001b[39;00m \u001b[43m_result_or_cancel\u001b[49m\u001b[43m(\u001b[49m\u001b[43mfs\u001b[49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43mpop\u001b[49m\u001b[43m(\u001b[49m\u001b[43m)\u001b[49m\u001b[43m)\u001b[49m\n\u001b[0;32m    620\u001b[0m     \u001b[38;5;28;01melse\u001b[39;00m:\n\u001b[0;32m    621\u001b[0m         \u001b[38;5;28;01myield\u001b[39;00m _result_or_cancel(fs\u001b[38;5;241m.\u001b[39mpop(), end_time \u001b[38;5;241m-\u001b[39m time\u001b[38;5;241m.\u001b[39mmonotonic())\n",
      "File \u001b[1;32mc:\\Users\\alexa\\AppData\\Local\\Programs\\Python\\Python311\\Lib\\concurrent\\futures\\_base.py:317\u001b[0m, in \u001b[0;36m_result_or_cancel\u001b[1;34m(***failed resolving arguments***)\u001b[0m\n\u001b[0;32m    315\u001b[0m \u001b[38;5;28;01mtry\u001b[39;00m:\n\u001b[0;32m    316\u001b[0m     \u001b[38;5;28;01mtry\u001b[39;00m:\n\u001b[1;32m--> 317\u001b[0m         \u001b[38;5;28;01mreturn\u001b[39;00m \u001b[43mfut\u001b[49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43mresult\u001b[49m\u001b[43m(\u001b[49m\u001b[43mtimeout\u001b[49m\u001b[43m)\u001b[49m\n\u001b[0;32m    318\u001b[0m     \u001b[38;5;28;01mfinally\u001b[39;00m:\n\u001b[0;32m    319\u001b[0m         fut\u001b[38;5;241m.\u001b[39mcancel()\n",
      "File \u001b[1;32mc:\\Users\\alexa\\AppData\\Local\\Programs\\Python\\Python311\\Lib\\concurrent\\futures\\_base.py:456\u001b[0m, in \u001b[0;36mFuture.result\u001b[1;34m(self, timeout)\u001b[0m\n\u001b[0;32m    454\u001b[0m     \u001b[38;5;28;01mraise\u001b[39;00m CancelledError()\n\u001b[0;32m    455\u001b[0m \u001b[38;5;28;01melif\u001b[39;00m \u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39m_state \u001b[38;5;241m==\u001b[39m FINISHED:\n\u001b[1;32m--> 456\u001b[0m     \u001b[38;5;28;01mreturn\u001b[39;00m \u001b[38;5;28;43mself\u001b[39;49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43m__get_result\u001b[49m\u001b[43m(\u001b[49m\u001b[43m)\u001b[49m\n\u001b[0;32m    457\u001b[0m \u001b[38;5;28;01melse\u001b[39;00m:\n\u001b[0;32m    458\u001b[0m     \u001b[38;5;28;01mraise\u001b[39;00m \u001b[38;5;167;01mTimeoutError\u001b[39;00m()\n",
      "File \u001b[1;32mc:\\Users\\alexa\\AppData\\Local\\Programs\\Python\\Python311\\Lib\\concurrent\\futures\\_base.py:401\u001b[0m, in \u001b[0;36mFuture.__get_result\u001b[1;34m(self)\u001b[0m\n\u001b[0;32m    399\u001b[0m \u001b[38;5;28;01mif\u001b[39;00m \u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39m_exception:\n\u001b[0;32m    400\u001b[0m     \u001b[38;5;28;01mtry\u001b[39;00m:\n\u001b[1;32m--> 401\u001b[0m         \u001b[38;5;28;01mraise\u001b[39;00m \u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39m_exception\n\u001b[0;32m    402\u001b[0m     \u001b[38;5;28;01mfinally\u001b[39;00m:\n\u001b[0;32m    403\u001b[0m         \u001b[38;5;66;03m# Break a reference cycle with the exception in self._exception\u001b[39;00m\n\u001b[0;32m    404\u001b[0m         \u001b[38;5;28mself\u001b[39m \u001b[38;5;241m=\u001b[39m \u001b[38;5;28;01mNone\u001b[39;00m\n",
      "File \u001b[1;32mc:\\Users\\alexa\\AppData\\Local\\Programs\\Python\\Python311\\Lib\\concurrent\\futures\\thread.py:58\u001b[0m, in \u001b[0;36m_WorkItem.run\u001b[1;34m(self)\u001b[0m\n\u001b[0;32m     55\u001b[0m     \u001b[38;5;28;01mreturn\u001b[39;00m\n\u001b[0;32m     57\u001b[0m \u001b[38;5;28;01mtry\u001b[39;00m:\n\u001b[1;32m---> 58\u001b[0m     result \u001b[38;5;241m=\u001b[39m \u001b[38;5;28;43mself\u001b[39;49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43mfn\u001b[49m\u001b[43m(\u001b[49m\u001b[38;5;241;43m*\u001b[39;49m\u001b[38;5;28;43mself\u001b[39;49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43margs\u001b[49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[38;5;241;43m*\u001b[39;49m\u001b[38;5;241;43m*\u001b[39;49m\u001b[38;5;28;43mself\u001b[39;49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43mkwargs\u001b[49m\u001b[43m)\u001b[49m\n\u001b[0;32m     59\u001b[0m \u001b[38;5;28;01mexcept\u001b[39;00m \u001b[38;5;167;01mBaseException\u001b[39;00m \u001b[38;5;28;01mas\u001b[39;00m exc:\n\u001b[0;32m     60\u001b[0m     \u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39mfuture\u001b[38;5;241m.\u001b[39mset_exception(exc)\n",
      "File \u001b[1;32mc:\\Users\\alexa\\AppData\\Local\\Programs\\Python\\Python311\\Lib\\site-packages\\huggingface_hub\\_snapshot_download.py:214\u001b[0m, in \u001b[0;36msnapshot_download.<locals>._inner_hf_hub_download\u001b[1;34m(repo_file)\u001b[0m\n\u001b[0;32m    213\u001b[0m \u001b[38;5;28;01mdef\u001b[39;00m \u001b[38;5;21m_inner_hf_hub_download\u001b[39m(repo_file: \u001b[38;5;28mstr\u001b[39m):\n\u001b[1;32m--> 214\u001b[0m     \u001b[38;5;28;01mreturn\u001b[39;00m \u001b[43mhf_hub_download\u001b[49m\u001b[43m(\u001b[49m\n\u001b[0;32m    215\u001b[0m \u001b[43m        \u001b[49m\u001b[43mrepo_id\u001b[49m\u001b[43m,\u001b[49m\n\u001b[0;32m    216\u001b[0m \u001b[43m        \u001b[49m\u001b[43mfilename\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[43mrepo_file\u001b[49m\u001b[43m,\u001b[49m\n\u001b[0;32m    217\u001b[0m \u001b[43m        \u001b[49m\u001b[43mrepo_type\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[43mrepo_type\u001b[49m\u001b[43m,\u001b[49m\n\u001b[0;32m    218\u001b[0m \u001b[43m        \u001b[49m\u001b[43mrevision\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[43mcommit_hash\u001b[49m\u001b[43m,\u001b[49m\n\u001b[0;32m    219\u001b[0m \u001b[43m        \u001b[49m\u001b[43mendpoint\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[43mendpoint\u001b[49m\u001b[43m,\u001b[49m\n\u001b[0;32m    220\u001b[0m \u001b[43m        \u001b[49m\u001b[43mcache_dir\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[43mcache_dir\u001b[49m\u001b[43m,\u001b[49m\n\u001b[0;32m    221\u001b[0m \u001b[43m        \u001b[49m\u001b[43mlocal_dir\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[43mlocal_dir\u001b[49m\u001b[43m,\u001b[49m\n\u001b[0;32m    222\u001b[0m \u001b[43m        \u001b[49m\u001b[43mlocal_dir_use_symlinks\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[43mlocal_dir_use_symlinks\u001b[49m\u001b[43m,\u001b[49m\n\u001b[0;32m    223\u001b[0m \u001b[43m        \u001b[49m\u001b[43mlibrary_name\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[43mlibrary_name\u001b[49m\u001b[43m,\u001b[49m\n\u001b[0;32m    224\u001b[0m \u001b[43m        \u001b[49m\u001b[43mlibrary_version\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[43mlibrary_version\u001b[49m\u001b[43m,\u001b[49m\n\u001b[0;32m    225\u001b[0m \u001b[43m        \u001b[49m\u001b[43muser_agent\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[43muser_agent\u001b[49m\u001b[43m,\u001b[49m\n\u001b[0;32m    226\u001b[0m \u001b[43m        \u001b[49m\u001b[43mproxies\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[43mproxies\u001b[49m\u001b[43m,\u001b[49m\n\u001b[0;32m    227\u001b[0m \u001b[43m        \u001b[49m\u001b[43metag_timeout\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[43metag_timeout\u001b[49m\u001b[43m,\u001b[49m\n\u001b[0;32m    228\u001b[0m \u001b[43m        \u001b[49m\u001b[43mresume_download\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[43mresume_download\u001b[49m\u001b[43m,\u001b[49m\n\u001b[0;32m    229\u001b[0m \u001b[43m        \u001b[49m\u001b[43mforce_download\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[43mforce_download\u001b[49m\u001b[43m,\u001b[49m\n\u001b[0;32m    230\u001b[0m \u001b[43m        \u001b[49m\u001b[43mtoken\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[43mtoken\u001b[49m\u001b[43m,\u001b[49m\n\u001b[0;32m    231\u001b[0m \u001b[43m    \u001b[49m\u001b[43m)\u001b[49m\n",
      "File \u001b[1;32mc:\\Users\\alexa\\AppData\\Local\\Programs\\Python\\Python311\\Lib\\site-packages\\huggingface_hub\\utils\\_validators.py:118\u001b[0m, in \u001b[0;36mvalidate_hf_hub_args.<locals>._inner_fn\u001b[1;34m(*args, **kwargs)\u001b[0m\n\u001b[0;32m    115\u001b[0m \u001b[38;5;28;01mif\u001b[39;00m check_use_auth_token:\n\u001b[0;32m    116\u001b[0m     kwargs \u001b[38;5;241m=\u001b[39m smoothly_deprecate_use_auth_token(fn_name\u001b[38;5;241m=\u001b[39mfn\u001b[38;5;241m.\u001b[39m\u001b[38;5;18m__name__\u001b[39m, has_token\u001b[38;5;241m=\u001b[39mhas_token, kwargs\u001b[38;5;241m=\u001b[39mkwargs)\n\u001b[1;32m--> 118\u001b[0m \u001b[38;5;28;01mreturn\u001b[39;00m \u001b[43mfn\u001b[49m\u001b[43m(\u001b[49m\u001b[38;5;241;43m*\u001b[39;49m\u001b[43margs\u001b[49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[38;5;241;43m*\u001b[39;49m\u001b[38;5;241;43m*\u001b[39;49m\u001b[43mkwargs\u001b[49m\u001b[43m)\u001b[49m\n",
      "File \u001b[1;32mc:\\Users\\alexa\\AppData\\Local\\Programs\\Python\\Python311\\Lib\\site-packages\\huggingface_hub\\file_download.py:1346\u001b[0m, in \u001b[0;36mhf_hub_download\u001b[1;34m(repo_id, filename, subfolder, repo_type, revision, endpoint, library_name, library_version, cache_dir, local_dir, local_dir_use_symlinks, user_agent, force_download, force_filename, proxies, etag_timeout, resume_download, token, local_files_only, legacy_cache_layout)\u001b[0m\n\u001b[0;32m   1340\u001b[0m     \u001b[38;5;28;01mraise\u001b[39;00m LocalEntryNotFoundError(\n\u001b[0;32m   1341\u001b[0m         \u001b[38;5;124m\"\u001b[39m\u001b[38;5;124mCannot find the requested files in the disk cache and outgoing traffic has been disabled. To enable\u001b[39m\u001b[38;5;124m\"\u001b[39m\n\u001b[0;32m   1342\u001b[0m         \u001b[38;5;124m\"\u001b[39m\u001b[38;5;124m hf.co look-ups and downloads online, set \u001b[39m\u001b[38;5;124m'\u001b[39m\u001b[38;5;124mlocal_files_only\u001b[39m\u001b[38;5;124m'\u001b[39m\u001b[38;5;124m to False.\u001b[39m\u001b[38;5;124m\"\u001b[39m\n\u001b[0;32m   1343\u001b[0m     )\n\u001b[0;32m   1344\u001b[0m \u001b[38;5;28;01melif\u001b[39;00m \u001b[38;5;28misinstance\u001b[39m(head_call_error, RepositoryNotFoundError) \u001b[38;5;129;01mor\u001b[39;00m \u001b[38;5;28misinstance\u001b[39m(head_call_error, GatedRepoError):\n\u001b[0;32m   1345\u001b[0m     \u001b[38;5;66;03m# Repo not found => let's raise the actual error\u001b[39;00m\n\u001b[1;32m-> 1346\u001b[0m     \u001b[38;5;28;01mraise\u001b[39;00m head_call_error\n\u001b[0;32m   1347\u001b[0m \u001b[38;5;28;01melse\u001b[39;00m:\n\u001b[0;32m   1348\u001b[0m     \u001b[38;5;66;03m# Otherwise: most likely a connection issue or Hub downtime => let's warn the user\u001b[39;00m\n\u001b[0;32m   1349\u001b[0m     \u001b[38;5;28;01mraise\u001b[39;00m LocalEntryNotFoundError(\n\u001b[0;32m   1350\u001b[0m         \u001b[38;5;124m\"\u001b[39m\u001b[38;5;124mAn error happened while trying to locate the file on the Hub and we cannot find the requested files\u001b[39m\u001b[38;5;124m\"\u001b[39m\n\u001b[0;32m   1351\u001b[0m         \u001b[38;5;124m\"\u001b[39m\u001b[38;5;124m in the local cache. Please check your connection and try again or make sure your Internet connection\u001b[39m\u001b[38;5;124m\"\u001b[39m\n\u001b[0;32m   1352\u001b[0m         \u001b[38;5;124m\"\u001b[39m\u001b[38;5;124m is on.\u001b[39m\u001b[38;5;124m\"\u001b[39m\n\u001b[0;32m   1353\u001b[0m     ) \u001b[38;5;28;01mfrom\u001b[39;00m \u001b[38;5;21;01mhead_call_error\u001b[39;00m\n",
      "File \u001b[1;32mc:\\Users\\alexa\\AppData\\Local\\Programs\\Python\\Python311\\Lib\\site-packages\\huggingface_hub\\file_download.py:1232\u001b[0m, in \u001b[0;36mhf_hub_download\u001b[1;34m(repo_id, filename, subfolder, repo_type, revision, endpoint, library_name, library_version, cache_dir, local_dir, local_dir_use_symlinks, user_agent, force_download, force_filename, proxies, etag_timeout, resume_download, token, local_files_only, legacy_cache_layout)\u001b[0m\n\u001b[0;32m   1230\u001b[0m \u001b[38;5;28;01mtry\u001b[39;00m:\n\u001b[0;32m   1231\u001b[0m     \u001b[38;5;28;01mtry\u001b[39;00m:\n\u001b[1;32m-> 1232\u001b[0m         metadata \u001b[38;5;241m=\u001b[39m \u001b[43mget_hf_file_metadata\u001b[49m\u001b[43m(\u001b[49m\n\u001b[0;32m   1233\u001b[0m \u001b[43m            \u001b[49m\u001b[43murl\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[43murl\u001b[49m\u001b[43m,\u001b[49m\n\u001b[0;32m   1234\u001b[0m \u001b[43m            \u001b[49m\u001b[43mtoken\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[43mtoken\u001b[49m\u001b[43m,\u001b[49m\n\u001b[0;32m   1235\u001b[0m \u001b[43m            \u001b[49m\u001b[43mproxies\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[43mproxies\u001b[49m\u001b[43m,\u001b[49m\n\u001b[0;32m   1236\u001b[0m \u001b[43m            \u001b[49m\u001b[43mtimeout\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[43metag_timeout\u001b[49m\u001b[43m,\u001b[49m\n\u001b[0;32m   1237\u001b[0m \u001b[43m        \u001b[49m\u001b[43m)\u001b[49m\n\u001b[0;32m   1238\u001b[0m     \u001b[38;5;28;01mexcept\u001b[39;00m EntryNotFoundError \u001b[38;5;28;01mas\u001b[39;00m http_error:\n\u001b[0;32m   1239\u001b[0m         \u001b[38;5;66;03m# Cache the non-existence of the file and raise\u001b[39;00m\n\u001b[0;32m   1240\u001b[0m         commit_hash \u001b[38;5;241m=\u001b[39m http_error\u001b[38;5;241m.\u001b[39mresponse\u001b[38;5;241m.\u001b[39mheaders\u001b[38;5;241m.\u001b[39mget(HUGGINGFACE_HEADER_X_REPO_COMMIT)\n",
      "File \u001b[1;32mc:\\Users\\alexa\\AppData\\Local\\Programs\\Python\\Python311\\Lib\\site-packages\\huggingface_hub\\utils\\_validators.py:118\u001b[0m, in \u001b[0;36mvalidate_hf_hub_args.<locals>._inner_fn\u001b[1;34m(*args, **kwargs)\u001b[0m\n\u001b[0;32m    115\u001b[0m \u001b[38;5;28;01mif\u001b[39;00m check_use_auth_token:\n\u001b[0;32m    116\u001b[0m     kwargs \u001b[38;5;241m=\u001b[39m smoothly_deprecate_use_auth_token(fn_name\u001b[38;5;241m=\u001b[39mfn\u001b[38;5;241m.\u001b[39m\u001b[38;5;18m__name__\u001b[39m, has_token\u001b[38;5;241m=\u001b[39mhas_token, kwargs\u001b[38;5;241m=\u001b[39mkwargs)\n\u001b[1;32m--> 118\u001b[0m \u001b[38;5;28;01mreturn\u001b[39;00m \u001b[43mfn\u001b[49m\u001b[43m(\u001b[49m\u001b[38;5;241;43m*\u001b[39;49m\u001b[43margs\u001b[49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[38;5;241;43m*\u001b[39;49m\u001b[38;5;241;43m*\u001b[39;49m\u001b[43mkwargs\u001b[49m\u001b[43m)\u001b[49m\n",
      "File \u001b[1;32mc:\\Users\\alexa\\AppData\\Local\\Programs\\Python\\Python311\\Lib\\site-packages\\huggingface_hub\\file_download.py:1608\u001b[0m, in \u001b[0;36mget_hf_file_metadata\u001b[1;34m(url, token, proxies, timeout)\u001b[0m\n\u001b[0;32m   1598\u001b[0m \u001b[38;5;66;03m# Retrieve metadata\u001b[39;00m\n\u001b[0;32m   1599\u001b[0m r \u001b[38;5;241m=\u001b[39m _request_wrapper(\n\u001b[0;32m   1600\u001b[0m     method\u001b[38;5;241m=\u001b[39m\u001b[38;5;124m\"\u001b[39m\u001b[38;5;124mHEAD\u001b[39m\u001b[38;5;124m\"\u001b[39m,\n\u001b[0;32m   1601\u001b[0m     url\u001b[38;5;241m=\u001b[39murl,\n\u001b[1;32m   (...)\u001b[0m\n\u001b[0;32m   1606\u001b[0m     timeout\u001b[38;5;241m=\u001b[39mtimeout,\n\u001b[0;32m   1607\u001b[0m )\n\u001b[1;32m-> 1608\u001b[0m \u001b[43mhf_raise_for_status\u001b[49m\u001b[43m(\u001b[49m\u001b[43mr\u001b[49m\u001b[43m)\u001b[49m\n\u001b[0;32m   1610\u001b[0m \u001b[38;5;66;03m# Return\u001b[39;00m\n\u001b[0;32m   1611\u001b[0m \u001b[38;5;28;01mreturn\u001b[39;00m HfFileMetadata(\n\u001b[0;32m   1612\u001b[0m     commit_hash\u001b[38;5;241m=\u001b[39mr\u001b[38;5;241m.\u001b[39mheaders\u001b[38;5;241m.\u001b[39mget(HUGGINGFACE_HEADER_X_REPO_COMMIT),\n\u001b[0;32m   1613\u001b[0m     etag\u001b[38;5;241m=\u001b[39m_normalize_etag(\n\u001b[1;32m   (...)\u001b[0m\n\u001b[0;32m   1623\u001b[0m     size\u001b[38;5;241m=\u001b[39m_int_or_none(r\u001b[38;5;241m.\u001b[39mheaders\u001b[38;5;241m.\u001b[39mget(HUGGINGFACE_HEADER_X_LINKED_SIZE) \u001b[38;5;129;01mor\u001b[39;00m r\u001b[38;5;241m.\u001b[39mheaders\u001b[38;5;241m.\u001b[39mget(\u001b[38;5;124m\"\u001b[39m\u001b[38;5;124mContent-Length\u001b[39m\u001b[38;5;124m\"\u001b[39m)),\n\u001b[0;32m   1624\u001b[0m )\n",
      "File \u001b[1;32mc:\\Users\\alexa\\AppData\\Local\\Programs\\Python\\Python311\\Lib\\site-packages\\huggingface_hub\\utils\\_errors.py:277\u001b[0m, in \u001b[0;36mhf_raise_for_status\u001b[1;34m(response, endpoint_name)\u001b[0m\n\u001b[0;32m    273\u001b[0m \u001b[38;5;28;01melif\u001b[39;00m error_code \u001b[38;5;241m==\u001b[39m \u001b[38;5;124m\"\u001b[39m\u001b[38;5;124mGatedRepo\u001b[39m\u001b[38;5;124m\"\u001b[39m:\n\u001b[0;32m    274\u001b[0m     message \u001b[38;5;241m=\u001b[39m (\n\u001b[0;32m    275\u001b[0m         \u001b[38;5;124mf\u001b[39m\u001b[38;5;124m\"\u001b[39m\u001b[38;5;132;01m{\u001b[39;00mresponse\u001b[38;5;241m.\u001b[39mstatus_code\u001b[38;5;132;01m}\u001b[39;00m\u001b[38;5;124m Client Error.\u001b[39m\u001b[38;5;124m\"\u001b[39m \u001b[38;5;241m+\u001b[39m \u001b[38;5;124m\"\u001b[39m\u001b[38;5;130;01m\\n\u001b[39;00m\u001b[38;5;130;01m\\n\u001b[39;00m\u001b[38;5;124m\"\u001b[39m \u001b[38;5;241m+\u001b[39m \u001b[38;5;124mf\u001b[39m\u001b[38;5;124m\"\u001b[39m\u001b[38;5;124mCannot access gated repo for url \u001b[39m\u001b[38;5;132;01m{\u001b[39;00mresponse\u001b[38;5;241m.\u001b[39murl\u001b[38;5;132;01m}\u001b[39;00m\u001b[38;5;124m.\u001b[39m\u001b[38;5;124m\"\u001b[39m\n\u001b[0;32m    276\u001b[0m     )\n\u001b[1;32m--> 277\u001b[0m     \u001b[38;5;28;01mraise\u001b[39;00m GatedRepoError(message, response) \u001b[38;5;28;01mfrom\u001b[39;00m \u001b[38;5;21;01me\u001b[39;00m\n\u001b[0;32m    279\u001b[0m \u001b[38;5;28;01melif\u001b[39;00m error_code \u001b[38;5;241m==\u001b[39m \u001b[38;5;124m\"\u001b[39m\u001b[38;5;124mRepoNotFound\u001b[39m\u001b[38;5;124m\"\u001b[39m \u001b[38;5;129;01mor\u001b[39;00m response\u001b[38;5;241m.\u001b[39mstatus_code \u001b[38;5;241m==\u001b[39m \u001b[38;5;241m401\u001b[39m:\n\u001b[0;32m    280\u001b[0m     \u001b[38;5;66;03m# 401 is misleading as it is returned for:\u001b[39;00m\n\u001b[0;32m    281\u001b[0m     \u001b[38;5;66;03m#    - private and gated repos if user is not authenticated\u001b[39;00m\n\u001b[0;32m    282\u001b[0m     \u001b[38;5;66;03m#    - missing repos\u001b[39;00m\n\u001b[0;32m    283\u001b[0m     \u001b[38;5;66;03m# => for now, we process them as `RepoNotFound` anyway.\u001b[39;00m\n\u001b[0;32m    284\u001b[0m     \u001b[38;5;66;03m# See https://gist.github.com/Wauplin/46c27ad266b15998ce56a6603796f0b9\u001b[39;00m\n\u001b[0;32m    285\u001b[0m     message \u001b[38;5;241m=\u001b[39m (\n\u001b[0;32m    286\u001b[0m         \u001b[38;5;124mf\u001b[39m\u001b[38;5;124m\"\u001b[39m\u001b[38;5;132;01m{\u001b[39;00mresponse\u001b[38;5;241m.\u001b[39mstatus_code\u001b[38;5;132;01m}\u001b[39;00m\u001b[38;5;124m Client Error.\u001b[39m\u001b[38;5;124m\"\u001b[39m\n\u001b[0;32m    287\u001b[0m         \u001b[38;5;241m+\u001b[39m \u001b[38;5;124m\"\u001b[39m\u001b[38;5;130;01m\\n\u001b[39;00m\u001b[38;5;130;01m\\n\u001b[39;00m\u001b[38;5;124m\"\u001b[39m\n\u001b[1;32m   (...)\u001b[0m\n\u001b[0;32m    291\u001b[0m         \u001b[38;5;124m\"\u001b[39m\u001b[38;5;124m make sure you are authenticated.\u001b[39m\u001b[38;5;124m\"\u001b[39m\n\u001b[0;32m    292\u001b[0m     )\n",
      "\u001b[1;31mGatedRepoError\u001b[0m: 401 Client Error. (Request ID: Root=1-66d06ab8-07588401657b07a3538d9179;e8c91b55-8ae2-4d6b-8108-2ca0c00a97a3)\n\nCannot access gated repo for url https://huggingface.co/google/gemma-2b/resolve/68e273d91b1d6ea57c9e6024c4f887832f7b43fa/.gitattributes.\nAccess to model google/gemma-2b is restricted. You must be authenticated to access it."
     ]
    }
   ],
   "source": [
    "\n",
    "from huggingface_hub import snapshot_download\n",
    "\n",
    "# Lade das Gemma-2B Modell herunter\n",
    "model_id = \"google/gemma-2b\"\n",
    "snapshot_download(repo_id=model_id, local_dir=\"gemma-2b\")\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "56bbfed9",
   "metadata": {},
   "source": [
    "## Konvertierung des Modells in GGUF-Format"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "bc85c892",
   "metadata": {},
   "outputs": [],
   "source": [
    "\n",
    "# Klonen des llama.cpp Repositories\n",
    "!git clone https://github.com/ggerganov/llama.cpp.git\n",
    "\n",
    "# Konvertiere das Modell in das GGUF-Format\n",
    "!python llama.cpp/convert.py gemma-2b --outfile gemma-2b.gguf --outtype q4_K_M\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d1fcea7d",
   "metadata": {},
   "source": [
    "## Datensatz vorbereiten und das Modell trainieren"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "708f3706",
   "metadata": {},
   "outputs": [],
   "source": [
    "\n",
    "from transformers import AutoModelForCausalLM, AutoTokenizer, Trainer, TrainingArguments\n",
    "from datasets import load_dataset\n",
    "\n",
    "# Lade das Tokenizer und Modell\n",
    "tokenizer = AutoTokenizer.from_pretrained(\"google/gemma-2b\")\n",
    "model = AutoModelForCausalLM.from_pretrained(\"google/gemma-2b\")\n",
    "\n",
    "# Lade und bereite den Datensatz vor\n",
    "dataset = load_dataset(\"Abirate/english_quotes\")\n",
    "dataset = dataset.map(lambda samples: tokenizer(samples[\"quote\"]), batched=True)\n",
    "\n",
    "# Trainingsargumente definieren\n",
    "training_args = TrainingArguments(\n",
    "    output_dir=\"./results\",\n",
    "    per_device_train_batch_size=2,\n",
    "    num_train_epochs=3,\n",
    "    logging_dir='./logs',\n",
    ")\n",
    "\n",
    "# Trainer konfigurieren\n",
    "trainer = Trainer(\n",
    "    model=model,\n",
    "    args=training_args,\n",
    "    train_dataset=dataset[\"train\"],\n",
    ")\n",
    "\n",
    "# Training starten\n",
    "trainer.train()\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "99eb82a1",
   "metadata": {},
   "source": [
    "## Modell speichern"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "bc6d277f",
   "metadata": {},
   "outputs": [],
   "source": [
    "\n",
    "from google.colab import drive\n",
    "drive.mount('/content/drive')\n",
    "\n",
    "# Speichere das trainierte Modell in Google Drive\n",
    "model.save_pretrained(\"/content/drive/MyDrive/trained_model\")\n",
    "tokenizer.save_pretrained(\"/content/drive/MyDrive/trained_model\")\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5f87f166",
   "metadata": {},
   "source": [
    "## Modell auf Hugging Face hochladen"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "1eeffaee",
   "metadata": {},
   "outputs": [],
   "source": [
    "\n",
    "from huggingface_hub import HfApi\n",
    "\n",
    "# Lade das Modell auf Hugging Face hoch\n",
    "api = HfApi()\n",
    "api.upload_file(\n",
    "    path_or_fileobj=\"gemma-2b.gguf\",\n",
    "    path_in_repo=\"gemma-2b.gguf\",\n",
    "    repo_id=\"your-username/gemma-2b\"\n",
    ")\n"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}