Experiments/NLP/notebooks/optimization-approaches-for-transformers.ipynb

{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"name":"python","version":"3.7.12","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"<h1>Table of contents</h1>\n\n<ul>\n    <li style=\"font-size: 15px;\">Introduction</li>\n    <li style=\"font-size: 15px;\">Pre-tokenization / pre-encoding</li>\n    <li style=\"font-size: 15px;\">Turn Dropout off</li>\n    <li style=\"font-size: 15px;\">TorchScript</li>\n    <li style=\"font-size: 15px;\">DeepSpeed</li>\n    <li style=\"font-size: 15px;\">Layers Fusing</li>\n    <li style=\"font-size: 15px;\">Conclusion</li>\n    <li style=\"font-size: 15px;\">Feedback</li>\n    <li style=\"font-size: 15px;\">References</li>\n    <li style=\"font-size: 15px;\">Releases</li>\n</ul>","metadata":{}},{"cell_type":"markdown","source":"<h1>Introduction</h1>\n\n<p style=\"font-size: 15px;\">\nThis article is the continuation of <a href=\"https://www.kaggle.com/code/vad13irt/optimization-approaches-for-transformers\">Optimization approaches for Transformers</a>, where the authors described some optimization approaches, which can significantly reduce memory utilization and time for training or inference of the Deep Learning models, in particular language models (LMs) such as <a href=\"https://arxiv.org/abs/1810.04805\">Transformers</a> in the Natural Language Understanding (NLP) tasks.<br><br>\nIn the article, we propose to study less known and less used approaches for optimization of language models, but the further described approaches can potentially reduce memory footprint and time by a lot too. Worth noting, that some of the proposed methods can be also successfully used not only with language models, and some others even don't directly belong to the model's optimization.<br><br>\nSometimes, the authors use quotes from other sources to provide more contextual information and not forget to note something important.\n</p>","metadata":{}},{"cell_type":"code","source":"!pip uninstall -qq -y transformers","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2022-09-10T16:09:51.050841Z","iopub.execute_input":"2022-09-10T16:09:51.051738Z","iopub.status.idle":"2022-09-10T16:09:55.657295Z","shell.execute_reply.started":"2022-09-10T16:09:51.051645Z","shell.execute_reply":"2022-09-10T16:09:55.656154Z"},"trusted":true},"execution_count":1,"outputs":[]},{"cell_type":"code","source":"import sys\nsys.path.append(\"../input/transformers/src/\")\nimport transformers\nimport pandas as pd\nimport numpy as np\nimport warnings\nimport os\n\n\nos.environ[\"TOKENIZERS_PARALLELISM\"] = \"true\"\n\nwarnings.simplefilter(\"ignore\")\ntransformers.logging.set_verbosity_error()","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2022-09-10T16:09:55.659431Z","iopub.execute_input":"2022-09-10T16:09:55.659743Z","iopub.status.idle":"2022-09-10T16:10:02.378339Z","shell.execute_reply.started":"2022-09-10T16:09:55.659713Z","shell.execute_reply":"2022-09-10T16:10:02.377347Z"},"trusted":true},"execution_count":2,"outputs":[]},{"cell_type":"code","source":"texts_path = \"../input/feedback-prize-english-language-learning/train.csv\"\ntexts = pd.read_csv(texts_path)[\"full_text\"].values","metadata":{"execution":{"iopub.status.busy":"2022-09-10T16:10:02.382661Z","iopub.execute_input":"2022-09-10T16:10:02.383400Z","iopub.status.idle":"2022-09-10T16:10:02.628753Z","shell.execute_reply.started":"2022-09-10T16:10:02.383362Z","shell.execute_reply":"2022-09-10T16:10:02.627943Z"},"_kg_hide-input":true,"trusted":true},"execution_count":3,"outputs":[]},{"cell_type":"markdown","source":"<h1>Pre-tokenization / pre-encoding</h1>\n\n<p style=\"font-size: 15px;\">\nThe easiest to implement and at least powerful method for optimization training and inference of language models is pre-tokenization / pre-encoding. The idea behind the pre-tokenization is to tokenize the sequences beforehand neither do it on the fly, i.e during batching in training or inference respectively. Mathematically, such approach can potentially reduce data pre-processing time for training from $ O(n*e) $ to $ O(n) $, where $ n $ - number of sequences, $ e $ - number of epochs. In case of Cross-Validation from $ O(n*e*k) $ to $ O(n*k) $, where $ k $ - number of folds, and for inference from $ O(n*k) $ to $ O(n) $ respectively.<br><br>\n    Note, that different tokenizers have different tokenization rules (<a href=\"https://arxiv.org/abs/1609.08144v2\">Word Piece</a>, <a href=\"https://arxiv.org/abs/1808.06226\">Sentence Piece</a>, <a href=\"https://arxiv.org/abs/1907.11692\">Byte Pair Encoding</a>, etc.), so authors recommend checking how the tokenizer works before doing pre-tokenization, for example, Sentence Piece tokenization realizes <a href=\"https://github.com/google/sentencepiece#subword-regularization-and-bpe-dropout\">\"Subword regularization and BPE-dropout\"</a> mechanisms, which add small augmentations during training or inference (similar to Test Time Augmentations), so doing tokenization on the fly with such tokenizers might be helpful to improve accuracy as well as the robustness of the language models.\n</p>\n<pre style=\"font-size: 15px; white-space: pre-wrap; background: #F6F8FA; padding: 20px; width: 100%;\">\n>>> import sentencepiece as spm\n>>> s = spm.SentencePieceProcessor(model_file='spm.model')\n>>> for n in range(5):\n...     s.encode('New York', out_type=str, enable_sampling=True, alpha=0.1, nbest_size=-1)\n...\n['▁', 'N', 'e', 'w', '▁York']\n['▁', 'New', '▁York']\n['▁', 'New', '▁Y', 'o', 'r', 'k']\n['▁', 'New', '▁York']\n['▁', 'New', '▁York']\n</pre>\n\n<p style=\"font-size: 15px;\">\nSource: <a href=\"https://github.com/google/sentencepiece#subword-regularization-and-bpe-dropout\">Subword regularization and BPE-dropout</a>\n</p>","metadata":{}},{"cell_type":"markdown","source":"<h3>Implementation</h3>","metadata":{}},{"cell_type":"code","source":"from torch.utils.data import Dataset\nfrom typing import List, Optional, Dict, Any\nfrom transformers import PreTrainedTokenizer\n\n\nclass TextDataset(Dataset):\n    def __init__(\n        self, \n        texts: List[str], \n        tokenizer: PreTrainedTokenizer, \n        max_length: Optional[int] = None, \n        texts_pair: Optional[List[str]] = None,\n        pre_tokenize: bool = False,\n    ) -> None:\n        super().__init__()\n        self.texts = texts\n        self.tokenizer = tokenizer\n        self.max_length = max_length\n        self.texts_pair = texts_pair\n        self.pre_tokenize = pre_tokenize\n        \n        # pre-tokenization\n        if self.pre_tokenize:\n            if self.texts_pair is not None:\n                self.all_tokenized = [self.tokenize(text=text, text_pair=text_pair) for text, text_pair in zip(self.texts, self.texts_pair)]\n            else:\n                self.all_tokenized = [self.tokenize(text=text) for text in self.texts]\n        \n    def __len__(self) -> int:\n        return len(self.texts)\n    \n    def tokenize(self, text: str, text_pair: Optional[str] = None) -> Dict[str, Any]:\n        tokenized = self.tokenizer(\n            text=text, \n            text_pair=text_pair,\n            max_length=self.max_length,\n            truncation=True,\n            padding=False,\n            return_attention_mask=True,\n            add_special_tokens=True,\n            return_special_tokens_mask=True,\n            return_token_type_ids=False,\n            return_offsets_mapping=False,\n            return_tensors=None,\n        )\n        \n        return tokenized\n    \n    def __getitem__(self, index: int) -> Dict[str, Any]:\n        if not self.pre_tokenize:\n            text = self.texts[index]\n            text_pair = None\n\n            if self.texts_pair is not None:\n                text_pair = self.texts_pair[index]\n\n            tokenized = self.tokenize(text=text, text_pair=text_pair)\n        else:\n            tokenized = self.all_tokenized[index]\n            \n        return tokenized","metadata":{"_kg_hide-input":false,"execution":{"iopub.status.busy":"2022-09-10T16:10:02.634565Z","iopub.execute_input":"2022-09-10T16:10:02.634894Z","iopub.status.idle":"2022-09-10T16:10:02.783833Z","shell.execute_reply.started":"2022-09-10T16:10:02.634863Z","shell.execute_reply":"2022-09-10T16:10:02.783048Z"},"trusted":true},"execution_count":4,"outputs":[]},{"cell_type":"code","source":"import time\nfrom transformers import AutoTokenizer\nimport gc\n\n# extensions\ngc.enable()\n\n# config\nepochs = 5\nmodel_path = \"distilbert-base-uncased\"\nuse_fast_tokenizer = False\ntime_decimals = 2\n\n# tokenization on fly\ntokenizer = AutoTokenizer.from_pretrained(model_path, use_fast=use_fast_tokenizer)\n\nstart_time = time.time()\ndataset = TextDataset(\n    texts=texts, \n    texts_pair=None, \n    max_length=None, \n    tokenizer=tokenizer,\n    pre_tokenize=False,\n)\n\nfor epoch in range(epochs):\n    for index in range(len(dataset)):\n        sample = dataset[index]\n        \nend_time = time.time()\n\nfly_difference_time = end_time - start_time\nprint(f\"Tokenization on fly: {fly_difference_time:.{time_decimals}f} seconds.\")\n\n# memory clearing\ndel dataset, tokenizer\ngc.collect()\n\n# pre-tokenization\ntokenizer = AutoTokenizer.from_pretrained(model_path, use_fast=use_fast_tokenizer)\n\nstart_time = time.time()\ndataset = TextDataset(\n    texts=texts, \n    texts_pair=None, \n    max_length=None, \n    tokenizer=tokenizer,\n    pre_tokenize=True,\n)\n\nfor epoch in range(epochs):\n    for index in range(len(dataset)):\n        sample = dataset[index]\n        \nend_time = time.time()\n\npre_difference_time = end_time - start_time\nprint(f\"Pre-tokenization: {pre_difference_time:.{time_decimals}f} seconds.\")\n\n# memory clearing\ndel dataset, tokenizer\ngc.collect()\n\n# computing time difference\ndifference_time = (fly_difference_time - pre_difference_time)\npercentage_difference_time = (fly_difference_time / pre_difference_time) * 100\nprint(f\"Difference: {difference_time:.{time_decimals}f} seconds. Percentage difference: {percentage_difference_time:.{time_decimals}f}%.\")","metadata":{"execution":{"iopub.status.busy":"2022-09-10T16:10:02.787657Z","iopub.execute_input":"2022-09-10T16:10:02.789968Z","iopub.status.idle":"2022-09-10T16:14:35.326923Z","shell.execute_reply.started":"2022-09-10T16:10:02.789930Z","shell.execute_reply":"2022-09-10T16:14:35.325921Z"},"trusted":true},"execution_count":5,"outputs":[{"output_type":"display_data","data":{"text/plain":"Downloading tokenizer_config.json:   0%|          | 0.00/28.0 [00:00<?, ?B/s]","application/vnd.jupyter.widget-view+json":{"version_major":2,"version_minor":0,"model_id":"af61552b54fc4b8abc9640b7da04de77"}},"metadata":{}},{"output_type":"display_data","data":{"text/plain":"Downloading config.json:   0%|          | 0.00/483 [00:00<?, ?B/s]","application/vnd.jupyter.widget-view+json":{"version_major":2,"version_minor":0,"model_id":"d1f404dec3d24e91867b65f9d533d3d4"}},"metadata":{}},{"output_type":"display_data","data":{"text/plain":"Downloading vocab.txt:   0%|          | 0.00/226k [00:00<?, ?B/s]","application/vnd.jupyter.widget-view+json":{"version_major":2,"version_minor":0,"model_id":"9587ada7a3044cc1a531f8289c3d4fd0"}},"metadata":{}},{"name":"stdout","text":"Tokenization on fly: 220.79 seconds.\nPre-tokenization: 44.04 seconds.\nDifference: 176.74 seconds. Percentage difference: 501.28%.\n","output_type":"stream"}]},{"cell_type":"markdown","source":"<h1>Turn Dropout off</h1>\n<p style=\"font-size: 15px;\">\n<i>Dropout is a regularization technique for neural networks that drops a unit (along with connections) at training time with a specified probability $ p $ (a common value is 0.5). At test time, all units are present, but with weights scaled by $ p $ (i.e.  becomes $ w*p $).<br><br>\nThe idea is to prevent co-adaptation, where the neural network becomes too reliant on particular connections, as this could be symptomatic of overfitting. Intuitively, dropout can be thought of as creating an implicit ensemble of neural networks.</i><br>\nSource: <a href=\"https://paperswithcode.com/method/dropout\">Dropout</a>\n</p>\n<center>\n    <img src=\"https://production-media.paperswithcode.com/methods/Screen_Shot_2020-05-23_at_6.19.24_PM.png\" alt=\"Dropout Visualization\"><br>\n    <span style=\"font-size: 15px;\">Visualization of how Dropout works</span>\n</center><br><br>\n<p style=\"font-size: 15px;\">\nIn spite of the easy implementation of Dropout regularization, it can be a serious bottleneck in large-scale models, because the Dropout implementation is based on masking (choose ~ $ p*100 $ % of neurons) and dot product (multiply selected neurons by zero) operations, hence turn Dropout off (set $ p $  to  $ 0.0 $) may lead to quite significant increasing of model's performance during training, however, authors recommend turn Dropout off when there is no overfitting during training and the dataset is relatively large (one of the techniques to prevent overfitting).\n</p>","metadata":{}},{"cell_type":"markdown","source":"<h3>Implementation</h3>\n<p style=\"font-size: 15px;\">\nThere are a lot of kinds of Dropout: <a href=\"https://arxiv.org/abs/1506.02142v6\">Monte-Carlo Dropout</a>, <a href=\"https://theaisummer.com/regularization/\">DropConnect, Gaussian Dropout</a>, and even <a href=\"https://github.com/huggingface/transformers/blob/main/src/transformers/models/deberta/modeling_deberta.py#L231\">StableDropout</a> from the HuggingFace Transformers library (used in the <a href=\"https://arxiv.org/abs/2006.03654\">DeBERTa</a> model), etc. However, our provided implementation for turning Dropout off only works for <a href=\"https://pytorch.org/docs/stable/generated/torch.nn.Dropout.html\">Dropout from the PyTorch framework</a>, despite this, the implementation can be easily re-implemented for another specific kind.\n</p>","metadata":{}},{"cell_type":"code","source":"import torch\nfrom torch import nn\nfrom transformers import AutoModel\nimport gc\n\n\n# extensions\ngc.enable()\n\n# utilities\n@torch.no_grad()\ndef turn_off_dropout(module: nn.Module) -> None:\n    if isinstance(module, nn.Dropout):\n        module.p = 0.0\n        \n# initializing model\nmodel_path = \"distilbert-base-uncased\"\nmodel = AutoModel.from_pretrained(model_path)\n\n# turning Dropout off\nmodel.apply(turn_off_dropout)\nprint(model)\n\n# memory clearing\ndel model\ngc.collect()","metadata":{"execution":{"iopub.status.busy":"2022-09-10T16:14:35.328504Z","iopub.execute_input":"2022-09-10T16:14:35.328882Z","iopub.status.idle":"2022-09-10T16:14:50.370958Z","shell.execute_reply.started":"2022-09-10T16:14:35.328845Z","shell.execute_reply":"2022-09-10T16:14:50.370003Z"},"trusted":true},"execution_count":6,"outputs":[{"output_type":"display_data","data":{"text/plain":"Downloading pytorch_model.bin:   0%|          | 0.00/256M [00:00<?, ?B/s]","application/vnd.jupyter.widget-view+json":{"version_major":2,"version_minor":0,"model_id":"7b52a05c686f4e09a7838ff6cb785172"}},"metadata":{}},{"name":"stdout","text":"DistilBertModel(\n  (embeddings): Embeddings(\n    (word_embeddings): Embedding(30522, 768, padding_idx=0)\n    (position_embeddings): Embedding(512, 768)\n    (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n    (dropout): Dropout(p=0.0, inplace=False)\n  )\n  (transformer): Transformer(\n    (layer): ModuleList(\n      (0): TransformerBlock(\n        (attention): MultiHeadSelfAttention(\n          (dropout): Dropout(p=0.0, inplace=False)\n          (q_lin): Linear(in_features=768, out_features=768, bias=True)\n          (k_lin): Linear(in_features=768, out_features=768, bias=True)\n          (v_lin): Linear(in_features=768, out_features=768, bias=True)\n          (out_lin): Linear(in_features=768, out_features=768, bias=True)\n        )\n        (sa_layer_norm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n        (ffn): FFN(\n          (dropout): Dropout(p=0.0, inplace=False)\n          (lin1): Linear(in_features=768, out_features=3072, bias=True)\n          (lin2): Linear(in_features=3072, out_features=768, bias=True)\n          (activation): GELUActivation()\n        )\n        (output_layer_norm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n      )\n      (1): TransformerBlock(\n        (attention): MultiHeadSelfAttention(\n          (dropout): Dropout(p=0.0, inplace=False)\n          (q_lin): Linear(in_features=768, out_features=768, bias=True)\n          (k_lin): Linear(in_features=768, out_features=768, bias=True)\n          (v_lin): Linear(in_features=768, out_features=768, bias=True)\n          (out_lin): Linear(in_features=768, out_features=768, bias=True)\n        )\n        (sa_layer_norm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n        (ffn): FFN(\n          (dropout): Dropout(p=0.0, inplace=False)\n          (lin1): Linear(in_features=768, out_features=3072, bias=True)\n          (lin2): Linear(in_features=3072, out_features=768, bias=True)\n          (activation): GELUActivation()\n        )\n        (output_layer_norm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n      )\n      (2): TransformerBlock(\n        (attention): MultiHeadSelfAttention(\n          (dropout): Dropout(p=0.0, inplace=False)\n          (q_lin): Linear(in_features=768, out_features=768, bias=True)\n          (k_lin): Linear(in_features=768, out_features=768, bias=True)\n          (v_lin): Linear(in_features=768, out_features=768, bias=True)\n          (out_lin): Linear(in_features=768, out_features=768, bias=True)\n        )\n        (sa_layer_norm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n        (ffn): FFN(\n          (dropout): Dropout(p=0.0, inplace=False)\n          (lin1): Linear(in_features=768, out_features=3072, bias=True)\n          (lin2): Linear(in_features=3072, out_features=768, bias=True)\n          (activation): GELUActivation()\n        )\n        (output_layer_norm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n      )\n      (3): TransformerBlock(\n        (attention): MultiHeadSelfAttention(\n          (dropout): Dropout(p=0.0, inplace=False)\n          (q_lin): Linear(in_features=768, out_features=768, bias=True)\n          (k_lin): Linear(in_features=768, out_features=768, bias=True)\n          (v_lin): Linear(in_features=768, out_features=768, bias=True)\n          (out_lin): Linear(in_features=768, out_features=768, bias=True)\n        )\n        (sa_layer_norm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n        (ffn): FFN(\n          (dropout): Dropout(p=0.0, inplace=False)\n          (lin1): Linear(in_features=768, out_features=3072, bias=True)\n          (lin2): Linear(in_features=3072, out_features=768, bias=True)\n          (activation): GELUActivation()\n        )\n        (output_layer_norm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n      )\n      (4): TransformerBlock(\n        (attention): MultiHeadSelfAttention(\n          (dropout): Dropout(p=0.0, inplace=False)\n          (q_lin): Linear(in_features=768, out_features=768, bias=True)\n          (k_lin): Linear(in_features=768, out_features=768, bias=True)\n          (v_lin): Linear(in_features=768, out_features=768, bias=True)\n          (out_lin): Linear(in_features=768, out_features=768, bias=True)\n        )\n        (sa_layer_norm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n        (ffn): FFN(\n          (dropout): Dropout(p=0.0, inplace=False)\n          (lin1): Linear(in_features=768, out_features=3072, bias=True)\n          (lin2): Linear(in_features=3072, out_features=768, bias=True)\n          (activation): GELUActivation()\n        )\n        (output_layer_norm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n      )\n      (5): TransformerBlock(\n        (attention): MultiHeadSelfAttention(\n          (dropout): Dropout(p=0.0, inplace=False)\n          (q_lin): Linear(in_features=768, out_features=768, bias=True)\n          (k_lin): Linear(in_features=768, out_features=768, bias=True)\n          (v_lin): Linear(in_features=768, out_features=768, bias=True)\n          (out_lin): Linear(in_features=768, out_features=768, bias=True)\n        )\n        (sa_layer_norm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n        (ffn): FFN(\n          (dropout): Dropout(p=0.0, inplace=False)\n          (lin1): Linear(in_features=768, out_features=3072, bias=True)\n          (lin2): Linear(in_features=3072, out_features=768, bias=True)\n          (activation): GELUActivation()\n        )\n        (output_layer_norm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n      )\n    )\n  )\n)\n","output_type":"stream"},{"execution_count":6,"output_type":"execute_result","data":{"text/plain":"223"},"metadata":{}}]},{"cell_type":"markdown","source":"<h1>TorchScript</h1>\n\n\n<i style=\"font-size: 15px;\"><a href=\"https://pytorch.org/docs/stable/jit.html\">TorchScript</a> is a way to create serializable and optimizable models from PyTorch code. Any TorchScript program can be saved from a Python process and loaded in a process where there is no Python dependency.<br><br>\nWe provide tools to incrementally transition a model from a pure Python program to a TorchScript program that can be run independently from Python, such as in a standalone C++ program. This makes it possible to train models in PyTorch using familiar tools in Python and then export the model via TorchScript to a production environment where Python programs may be disadvantageous for performance and multi-threading reasons.\n</i>\n\n\n    \n<p style=\"font-size: 15px;\">\n    Source: <a href=\"https://pytorch.org/docs/stable/jit.html\">TorchScript</a>\n</p>","metadata":{}},{"cell_type":"markdown","source":"<h3>Implementation</h3>","metadata":{}},{"cell_type":"code","source":"import torch\nfrom transformers import AutoModel, AutoTokenizer\nimport gc\n\n\n# extensions\ngc.enable()\n        \n# config\nmodel_path = \"distilbert-base-uncased\"\n\n# initializing tokenizer\ntokenizer = AutoTokenizer.from_pretrained(model_path)\n\n# example inputs\ntext = \"It is simple example text!\"\ntokenized = tokenizer(\n    text=text, \n    add_special_tokens=True, \n    return_attention_mask=True, \n    return_tensors=\"pt\",\n)\n\ninput_ids = tokenized[\"input_ids\"]\nattention_mask = tokenized[\"attention_mask\"]\nexample_inputs = [input_ids, attention_mask]\n\n# initializing model\nmodel = AutoModel.from_pretrained(model_path, torchscript=True)\n\n# model must be in evaluation mode\nmodel.eval()\n\n# wrapping and saving model via TorchScript\ntraced_model_path = \"./traced_model.pt\"\ntraced_model = torch.jit.trace(model, example_inputs=example_inputs)\ntorch.jit.save(traced_model, traced_model_path)\n\n# loading TorchScript model\nmodel = torch.jit.load(traced_model_path)\nmodel.eval()\n\n# inference...\n\n# memory clearing\ndel model, traced_model, example_inputs\ndel tokenized, input_ids, attention_mask, tokenizer\ngc.collect()","metadata":{"execution":{"iopub.status.busy":"2022-09-10T16:14:50.372484Z","iopub.execute_input":"2022-09-10T16:14:50.372937Z","iopub.status.idle":"2022-09-10T16:14:57.165803Z","shell.execute_reply.started":"2022-09-10T16:14:50.372900Z","shell.execute_reply":"2022-09-10T16:14:57.164871Z"},"trusted":true},"execution_count":7,"outputs":[{"output_type":"display_data","data":{"text/plain":"Downloading tokenizer.json:   0%|          | 0.00/455k [00:00<?, ?B/s]","application/vnd.jupyter.widget-view+json":{"version_major":2,"version_minor":0,"model_id":"cf90ee79e73844dab7a970f4f77b326f"}},"metadata":{}},{"execution_count":7,"output_type":"execute_result","data":{"text/plain":"1066"},"metadata":{}}]},{"cell_type":"markdown","source":"<h1>DeepSpeed</h1>\n<br>\n<p style=\"font-size: 15px;\"><i>\nDeepSpeed is an open source deep learning optimization library for PyTorch. The library is designed to reduce computing power and memory use and to train large distributed models with better parallelism on existing computer hardware. DeepSpeed is optimized for low latency, high throughput training. It includes the Zero Redundancy Optimizer (ZeRO) for training models with 1 trillion or more parameters. Features include mixed precision training, single-GPU, multi-GPU, and multi-node training as well as custom model parallelism. The DeepSpeed source code is licensed under MIT License and available on <a href=\"https://github.com/microsoft/DeepSpeed\">GitHub</a>.<br><br>\nThe team claimed to achieve up to a 6.2x throughput improvement, 2.8x faster convergence, and 4.6x less communication.\n</i><br>\nSource: <a href=\"https://en.wikipedia.org/wiki/DeepSpeed\">Wikipedia</a>\n<p style=\"font-size: 15px;\">\nAlso, DeepSpeed functionality is integrated in the popular libraries: <a href=\"https://pytorch-lightning.readthedocs.io/en/stable/advanced/model_parallel.html#deepspeed\">PyTorch Lightning</a>, <a href=\"https://huggingface.co/docs/transformers/main_classes/deepspeed\">HuggingFace Transformers</a>, HuggingFace Accelerator, etc.\n</p>\n    <br><br>\n<center>\n    <iframe width=\"100%\" height=\"500px\" src=\"https://www.youtube-nocookie.com/embed/ovQC7FqXHXk?start=1\" title=\"YouTube video player\" frameborder=\"0\" allow=\"accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture\" allowfullscreen></iframe><br>\n    <span style=\"font-size: 15px;\">DeepSpeed | PyTorch Developer Day 2020</span>\n</center>\n<br>\n<p style=\"font-size: 15px;\"><i>\"I have tried DeepSpeed and I am able to fit a Longformer Large with BS 6 on my RTX 6000 24GB card rented from Jarvislabs . The best part is integrating DeepSpeed with huggingface trainer is super easy and although there is not much to it , I thought it would really help if I share a code example for it.\"</i> - <a href=\"https://www.kaggle.com/tanulsingh077\">Tanul Singh</a>'s comment on <a href=\"https://www.kaggle.com/competitions/feedback-prize-2021/discussion/304706\">the forum</a> under <a href=\"https://www.kaggle.com/competitions/feedback-prize-2021/overview\">Feedback Prize - Evaluating Student Writing</a> competition.</p>","metadata":{}},{"cell_type":"markdown","source":"<p style=\"font-size: 15px;\">\n    <a href=\"\">DeepSeed Compression</a> is a library based on DeepSpeed. DeepSpeed Compression provides easy-to-use, fast, and powerful functionality for compression large-scale language models such as <a href=\"https://huggingface.co/docs/transformers/model_doc/bloom\">BLOOM</a> (~176 billion parameters), <a href=\"https://github.com/yandex/YaLM-100B\">YaLM</a> (~100 billion parameters), <a href=\"https://s3-us-west-2.amazonaws.com/openai-assets/research-covers/language-unsupervised/language_understanding_paper.pdf\">GPT</a>, etc. DeepSpeed Compression uses modern compression techniques, e.g Quantization, Pruning, Distillation, etc, due to them the accuracy of the compressed model doesn't significantly differ from the original model. A detailed description of the DeepSpeed Compression library can be found in the DeepSpeed Compression developers' blog post - <a href=\"https://www.microsoft.com/en-us/research/blog/deepspeed-compression-a-composable-library-for-extreme-compression-and-zero-cost-quantization/\">DeepSpeed Compression: A composable library for extreme compression and zero-cost quantization</a>.\n<center>\n    <img src=\"https://www.microsoft.com/en-us/research/uploads/prod/2022/07/1400x788_Deepspeed_blog_hero_no_logo_V2-1920x1080.jpg\">\n    <br>\n    <span style=\"font-size: 15px;\">Results of compression by DeepSpeed Compression</span>\n</center>\n</p>","metadata":{}},{"cell_type":"markdown","source":"<h3>Implementation</h3>\n\n<p style=\"font-size: 15px;\">\nThe below implementation doesn't show all functionality of the DeepSpeed library, if you are interested in deeply learning DeepSpeed's API, please refer to the official documentation - <a href=\"https://deepspeed.readthedocs.io/en/latest/\">DeepSpeed's Documentation</a>. DeepSpeed automatically does Gradient Clipping, Gradient Accumulation, Automatic Mixed Precision, and other approaches (if they are configured in <a href=\"https://www.deepspeed.ai/docs/config-json/\">DeepSpeed Configuration</a>), so DeepSpeed library's users do not need to write and integrate them by themselves.\n</p>","metadata":{}},{"cell_type":"code","source":"!conda install -qq -y mpi4py \n!pip install -qq deepspeed","metadata":{"execution":{"iopub.status.busy":"2022-09-10T16:14:57.167053Z","iopub.execute_input":"2022-09-10T16:14:57.167492Z","iopub.status.idle":"2022-09-10T16:16:20.626308Z","shell.execute_reply.started":"2022-09-10T16:14:57.167456Z","shell.execute_reply":"2022-09-10T16:16:20.625094Z"},"trusted":true},"execution_count":8,"outputs":[{"name":"stdout","text":"Collecting package metadata (current_repodata.json): ...working... done\nSolving environment: ...working... done\n\n## Package Plan ##\n\n  environment location: /opt/conda\n\n  added / updated specs:\n    - mpi4py\n\n\nThe following packages will be downloaded:\n\n    package                    |            build\n    ---------------------------|-----------------\n    ca-certificates-2022.6.15.1|       ha878542_0         150 KB  conda-forge\n    certifi-2022.6.15.1        |     pyhd8ed1ab_0         155 KB  conda-forge\n    conda-4.14.0               |   py37h89c1867_0        1010 KB  conda-forge\n    mpi-1.0                    |            mpich           4 KB  conda-forge\n    mpi4py-3.1.3               |   py37h52370cb_2         602 KB  conda-forge\n    mpich-4.0.2                |     h846660c_100         6.0 MB  conda-forge\n    openssl-1.1.1q             |       h166bdaf_0         2.1 MB  conda-forge\n    ------------------------------------------------------------\n                                           Total:        10.0 MB\n\nThe following NEW packages will be INSTALLED:\n\n  mpi                conda-forge/linux-64::mpi-1.0-mpich\n  mpi4py             conda-forge/linux-64::mpi4py-3.1.3-py37h52370cb_2\n  mpich              conda-forge/linux-64::mpich-4.0.2-h846660c_100\n\nThe following packages will be UPDATED:\n\n  ca-certificates                    2022.5.18.1-ha878542_0 --> 2022.6.15.1-ha878542_0\n  certifi            conda-forge/linux-64::certifi-2022.5.~ --> conda-forge/noarch::certifi-2022.6.15.1-pyhd8ed1ab_0\n  conda                               4.12.0-py37h89c1867_0 --> 4.14.0-py37h89c1867_0\n  openssl                                 1.1.1o-h166bdaf_0 --> 1.1.1q-h166bdaf_0\n\n\nPreparing transaction: ...working... done\nVerifying transaction: ...working... done\nExecuting transaction: ...working... done\n","output_type":"stream"}]},{"cell_type":"code","source":"from torch import nn\nfrom torch.optim import AdamW\nfrom transformers import AutoModel\nimport deepspeed\nfrom typing import Iterator\nimport gc\nimport os\n\n\n# extensions\ngc.enable()\n\n# utilities\nno_decay_parameters = (\"bias\", \"LayerNorm.bias\", \"LayerNorm.weight\")\n\ndef get_decay_module_parameters(\n    module: nn.Module, \n    no_decay_parameters: Iterator[str] = no_decay_parameters,\n    recurse: bool = True,\n) -> Iterator[nn.Parameter]:\n                                \n    for name, parameter in list(module.named_parameters(recurse=recurse)):\n        if name not in no_decay_parameters:\n            yield parameter\n\n            \ndef get_no_decay_module_parameters(\n    module: nn.Module, \n    no_decay_parameters: Iterator[str] = no_decay_parameters,\n    recurse: bool = True,\n) -> Iterator[nn.Parameter]:\n\n    for name, parameter in list(module.named_parameters(recurse=recurse)):\n        if name in no_decay_parameters:\n            yield parameter\n\n# config\nmodel_path = \"distilbert-base-uncased\"\nweight_decay = 0.01\nlr = 1e-5\ntrain_batch_size = 1\ngradient_accumulation_steps = 1\ndevice = \"cpu\"\n\n# initializing model\nmodel = AutoModel.from_pretrained(model_path)\n\n# initializing optimizer\nmodel_parameters = [\n    {\"params\": get_decay_module_parameters(model), \"weight_decay\": weight_decay},\n    {\"params\": get_no_decay_module_parameters(model), \"weight_decay\": 0.0},\n]\n\noptimizer = AdamW(params=model_parameters, lr=lr, weight_decay=weight_decay)\n\n# initializing learning rate scheduler\nscheduler = None\n\n# DeepSpeed config\ndeepspeed_arguments = {}\ndeepspeed_config = {\n    \"zero_optimization\": {\n        \"offload_param\": {\n            \"device\": device,\n        }\n    },\n    \"train_batch_size\": train_batch_size,\n    \"gradient_accumulation_steps\": gradient_accumulation_steps,\n}\n\n# wrapping model via DeepSpeed\ndeepspeed.init_distributed(dist_backend=\"gloo\")\nmodel_engine, optimizer, training_dataloader, scheduler = deepspeed.initialize(\n    args=deepspeed_arguments,\n    model=model.requires_grad_(False), # WARNING: here is the issue I need to figure out.\n    optimizer=optimizer,\n    model_parameters=None,\n    lr_scheduler=scheduler,\n    config=deepspeed_config,\n    dist_init_required=False,\n)","metadata":{"execution":{"iopub.status.busy":"2022-09-10T16:16:20.628518Z","iopub.execute_input":"2022-09-10T16:16:20.628917Z","iopub.status.idle":"2022-09-10T16:16:52.909709Z","shell.execute_reply.started":"2022-09-10T16:16:20.628877Z","shell.execute_reply":"2022-09-10T16:16:52.907500Z"},"trusted":true},"execution_count":9,"outputs":[{"name":"stdout","text":"[2022-09-10 16:16:27,460] [INFO] [comm.py:618:init_distributed] Not using the DeepSpeed or dist launchers, attempting to detect MPI environment...\n[2022-09-10 16:16:27,505] [INFO] [comm.py:675:mpi_discovery] Discovered MPI settings of world_rank=0, local_rank=0, world_size=1, master_addr=172.19.2.2, master_port=29500\n[2022-09-10 16:16:27,506] [INFO] [comm.py:635:init_distributed] Initializing TorchBackend in DeepSpeed with backend gloo\n[2022-09-10 16:16:27,514] [INFO] [logging.py:68:log_dist] [Rank 0] DeepSpeed info: version=0.7.2, git-hash=unknown, git-branch=unknown\n[2022-09-10 16:16:32,973] [INFO] [logging.py:68:log_dist] [Rank 0] DeepSpeed Flops Profiler Enabled: False\n[2022-09-10 16:16:32,975] [INFO] [logging.py:68:log_dist] [Rank 0] Removing param_group that has no 'params' in the client Optimizer\n[2022-09-10 16:16:32,976] [INFO] [logging.py:68:log_dist] [Rank 0] Using client Optimizer as basic optimizer\n[2022-09-10 16:16:32,980] [INFO] [logging.py:68:log_dist] [Rank 0] DeepSpeed Basic Optimizer = {basic_optimizer.__class__.__name__}\n[2022-09-10 16:16:32,981] [INFO] [logging.py:68:log_dist] [Rank 0] DeepSpeed Final Optimizer = AdamW\n[2022-09-10 16:16:32,982] [INFO] [logging.py:68:log_dist] [Rank 0] DeepSpeed using client LR scheduler\n[2022-09-10 16:16:32,982] [INFO] [logging.py:68:log_dist] [Rank 0] DeepSpeed LR Scheduler = None\n[2022-09-10 16:16:32,983] [INFO] [logging.py:68:log_dist] [Rank 0] step=0, skipped=0, lr=[1e-05], mom=[(0.9, 0.999)]\n[2022-09-10 16:16:32,985] [INFO] [config.py:987:print] DeepSpeedEngine configuration:\n[2022-09-10 16:16:32,987] [INFO] [config.py:991:print]   activation_checkpointing_config  {\n    \"partition_activations\": false, \n    \"contiguous_memory_optimization\": false, \n    \"cpu_checkpointing\": false, \n    \"number_checkpoints\": null, \n    \"synchronize_checkpoint_boundary\": false, \n    \"profile\": false\n}\n[2022-09-10 16:16:32,988] [INFO] [config.py:991:print]   aio_config ................... {'block_size': 1048576, 'queue_depth': 8, 'thread_count': 1, 'single_submit': False, 'overlap_events': True}\n[2022-09-10 16:16:32,989] [INFO] [config.py:991:print]   amp_enabled .................. False\n[2022-09-10 16:16:32,990] [INFO] [config.py:991:print]   amp_params ................... False\n[2022-09-10 16:16:32,991] [INFO] [config.py:991:print]   autotuning_config ............ {\n    \"enabled\": false, \n    \"start_step\": null, \n    \"end_step\": null, \n    \"metric_path\": null, \n    \"arg_mappings\": null, \n    \"metric\": \"throughput\", \n    \"model_info\": null, \n    \"results_dir\": null, \n    \"exps_dir\": null, \n    \"overwrite\": true, \n    \"fast\": true, \n    \"start_profile_step\": 3, \n    \"end_profile_step\": 5, \n    \"tuner_type\": \"gridsearch\", \n    \"tuner_early_stopping\": 5, \n    \"tuner_num_trials\": 50, \n    \"model_info_path\": null, \n    \"mp_size\": 1, \n    \"max_train_batch_size\": null, \n    \"min_train_batch_size\": 1, \n    \"max_train_micro_batch_size_per_gpu\": 1.024000e+03, \n    \"min_train_micro_batch_size_per_gpu\": 1, \n    \"num_tuning_micro_batch_sizes\": 3\n}\n[2022-09-10 16:16:32,992] [INFO] [config.py:991:print]   bfloat16_enabled ............. False\n[2022-09-10 16:16:32,993] [INFO] [config.py:991:print]   checkpoint_tag_validation_enabled  True\n[2022-09-10 16:16:32,994] [INFO] [config.py:991:print]   checkpoint_tag_validation_fail  False\n[2022-09-10 16:16:32,995] [INFO] [config.py:991:print]   comms_config ................. <deepspeed.comm.config.DeepSpeedCommsConfig object at 0x7effa4017a10>\n[2022-09-10 16:16:32,996] [INFO] [config.py:991:print]   communication_data_type ...... None\n[2022-09-10 16:16:32,996] [INFO] [config.py:991:print]   compression_config ........... {'weight_quantization': {'shared_parameters': {'enabled': False, 'quantizer_kernel': False, 'schedule_offset': 0, 'quantize_groups': 1, 'quantize_verbose': False, 'quantization_type': 'symmetric', 'quantize_weight_in_forward': False, 'rounding': 'nearest', 'fp16_mixed_quantize': False, 'quantize_change_ratio': 0.001}, 'different_groups': {}}, 'activation_quantization': {'shared_parameters': {'enabled': False, 'quantization_type': 'symmetric', 'range_calibration': 'dynamic', 'schedule_offset': 1000}, 'different_groups': {}}, 'sparse_pruning': {'shared_parameters': {'enabled': False, 'method': 'l1', 'schedule_offset': 1000}, 'different_groups': {}}, 'row_pruning': {'shared_parameters': {'enabled': False, 'method': 'l1', 'schedule_offset': 1000}, 'different_groups': {}}, 'head_pruning': {'shared_parameters': {'enabled': False, 'method': 'topk', 'schedule_offset': 1000}, 'different_groups': {}}, 'channel_pruning': {'shared_parameters': {'enabled': False, 'method': 'l1', 'schedule_offset': 1000}, 'different_groups': {}}, 'layer_reduction': {'enabled': False}}\n[2022-09-10 16:16:32,997] [INFO] [config.py:991:print]   curriculum_enabled ........... False\n[2022-09-10 16:16:32,998] [INFO] [config.py:991:print]   curriculum_params ............ False\n[2022-09-10 16:16:32,999] [INFO] [config.py:991:print]   dataloader_drop_last ......... False\n[2022-09-10 16:16:32,999] [INFO] [config.py:991:print]   disable_allgather ............ False\n[2022-09-10 16:16:33,000] [INFO] [config.py:991:print]   dump_state ................... False\n[2022-09-10 16:16:33,001] [INFO] [config.py:991:print]   dynamic_loss_scale_args ...... None\n[2022-09-10 16:16:33,002] [INFO] [config.py:991:print]   eigenvalue_enabled ........... False\n[2022-09-10 16:16:33,003] [INFO] [config.py:991:print]   eigenvalue_gas_boundary_resolution  1\n[2022-09-10 16:16:33,003] [INFO] [config.py:991:print]   eigenvalue_layer_name ........ bert.encoder.layer\n[2022-09-10 16:16:33,004] [INFO] [config.py:991:print]   eigenvalue_layer_num ......... 0\n[2022-09-10 16:16:33,005] [INFO] [config.py:991:print]   eigenvalue_max_iter .......... 100\n[2022-09-10 16:16:33,005] [INFO] [config.py:991:print]   eigenvalue_stability ......... 1e-06\n[2022-09-10 16:16:33,006] [INFO] [config.py:991:print]   eigenvalue_tol ............... 0.01\n[2022-09-10 16:16:33,007] [INFO] [config.py:991:print]   eigenvalue_verbose ........... False\n[2022-09-10 16:16:33,007] [INFO] [config.py:991:print]   elasticity_enabled ........... False\n[2022-09-10 16:16:33,008] [INFO] [config.py:991:print]   flops_profiler_config ........ {\n    \"enabled\": false, \n    \"profile_step\": 1, \n    \"module_depth\": -1, \n    \"top_modules\": 1, \n    \"detailed\": true, \n    \"output_file\": null\n}\n[2022-09-10 16:16:33,009] [INFO] [config.py:991:print]   fp16_auto_cast ............... None\n[2022-09-10 16:16:33,010] [INFO] [config.py:991:print]   fp16_enabled ................. False\n[2022-09-10 16:16:33,010] [INFO] [config.py:991:print]   fp16_master_weights_and_gradients  False\n[2022-09-10 16:16:33,011] [INFO] [config.py:991:print]   global_rank .................. 0\n[2022-09-10 16:16:33,012] [INFO] [config.py:991:print]   gradient_accumulation_steps .. 1\n[2022-09-10 16:16:33,012] [INFO] [config.py:991:print]   gradient_clipping ............ 0.0\n[2022-09-10 16:16:33,013] [INFO] [config.py:991:print]   gradient_predivide_factor .... 1.0\n[2022-09-10 16:16:33,014] [INFO] [config.py:991:print]   initial_dynamic_scale ........ 4294967296\n[2022-09-10 16:16:33,014] [INFO] [config.py:991:print]   load_universal_checkpoint .... False\n[2022-09-10 16:16:33,015] [INFO] [config.py:991:print]   loss_scale ................... 0\n[2022-09-10 16:16:33,016] [INFO] [config.py:991:print]   memory_breakdown ............. False\n[2022-09-10 16:16:33,017] [INFO] [config.py:991:print]   monitor_config ............... <deepspeed.monitor.config.DeepSpeedMonitorConfig object at 0x7effa4017910>\n[2022-09-10 16:16:33,018] [INFO] [config.py:991:print]   nebula_config ................ {\n    \"enabled\": false, \n    \"persistent_storage_path\": null, \n    \"persistent_time_interval\": 100, \n    \"num_of_version_in_retention\": 2, \n    \"enable_nebula_load\": true, \n    \"load_path\": null\n}\n[2022-09-10 16:16:33,018] [INFO] [config.py:991:print]   optimizer_legacy_fusion ...... False\n[2022-09-10 16:16:33,019] [INFO] [config.py:991:print]   optimizer_name ............... None\n[2022-09-10 16:16:33,020] [INFO] [config.py:991:print]   optimizer_params ............. None\n[2022-09-10 16:16:33,020] [INFO] [config.py:991:print]   pipeline ..................... {'stages': 'auto', 'partition': 'best', 'seed_layers': False, 'activation_checkpoint_interval': 0}\n[2022-09-10 16:16:33,021] [INFO] [config.py:991:print]   pld_enabled .................. False\n[2022-09-10 16:16:33,022] [INFO] [config.py:991:print]   pld_params ................... False\n[2022-09-10 16:16:33,022] [INFO] [config.py:991:print]   prescale_gradients ........... False\n[2022-09-10 16:16:33,023] [INFO] [config.py:991:print]   scheduler_name ............... None\n[2022-09-10 16:16:33,024] [INFO] [config.py:991:print]   scheduler_params ............. None\n[2022-09-10 16:16:33,024] [INFO] [config.py:991:print]   sparse_attention ............. None\n[2022-09-10 16:16:33,025] [INFO] [config.py:991:print]   sparse_gradients_enabled ..... False\n[2022-09-10 16:16:33,026] [INFO] [config.py:991:print]   steps_per_print .............. 10\n[2022-09-10 16:16:33,026] [INFO] [config.py:991:print]   train_batch_size ............. 1\n[2022-09-10 16:16:33,027] [INFO] [config.py:991:print]   train_micro_batch_size_per_gpu  1\n[2022-09-10 16:16:33,027] [INFO] [config.py:991:print]   wall_clock_breakdown ......... False\n[2022-09-10 16:16:33,028] [INFO] [config.py:991:print]   world_size ................... 1\n[2022-09-10 16:16:33,029] [INFO] [config.py:991:print]   zero_allow_untested_optimizer  False\n[2022-09-10 16:16:33,029] [INFO] [config.py:991:print]   zero_config .................. stage=0 contiguous_gradients=True reduce_scatter=True reduce_bucket_size=500000000 allgather_partitions=True allgather_bucket_size=500000000 overlap_comm=False load_from_fp32_weights=True elastic_checkpoint=False offload_param=DeepSpeedZeroOffloadParamConfig(device='cpu', nvme_path=None, buffer_count=5, buffer_size=100000000, max_in_cpu=1000000000, pin_memory=False) offload_optimizer=None sub_group_size=1000000000 cpu_offload_param=None cpu_offload_use_pin_memory=None cpu_offload=None prefetch_bucket_size=50000000 param_persistence_threshold=100000 model_persistence_threshold=9223372036854775807 max_live_parameters=1000000000 max_reuse_distance=1000000000 gather_16bit_weights_on_model_save=False stage3_gather_fp16_weights_on_model_save=False ignore_unused_parameters=True legacy_stage1=False round_robin_gradients=False\n[2022-09-10 16:16:33,030] [INFO] [config.py:991:print]   zero_enabled ................. False\n[2022-09-10 16:16:33,031] [INFO] [config.py:991:print]   zero_optimization_stage ...... 0\n[2022-09-10 16:16:33,031] [INFO] [config.py:982:print_user_config]   json = {\n    \"zero_optimization\": {\n        \"offload_param\": {\n            \"device\": \"cpu\"\n        }\n    }, \n    \"train_batch_size\": 1, \n    \"gradient_accumulation_steps\": 1\n}\nUsing /root/.cache/torch_extensions/py37_cu110 as PyTorch extensions root...\nCreating extension directory /root/.cache/torch_extensions/py37_cu110/utils...\nEmitting ninja build file /root/.cache/torch_extensions/py37_cu110/utils/build.ninja...\nBuilding extension module utils...\nAllowing ninja to set a default number of workers... (overridable by setting the environment variable MAX_JOBS=N)\n[1/2] c++ -MMD -MF flatten_unflatten.o.d -DTORCH_EXTENSION_NAME=utils -DTORCH_API_INCLUDE_EXTENSION_H -DPYBIND11_COMPILER_TYPE=\\\"_gcc\\\" -DPYBIND11_STDLIB=\\\"_libstdcpp\\\" -DPYBIND11_BUILD_ABI=\\\"_cxxabi1013\\\" -isystem /opt/conda/lib/python3.7/site-packages/torch/include -isystem /opt/conda/lib/python3.7/site-packages/torch/include/torch/csrc/api/include -isystem /opt/conda/lib/python3.7/site-packages/torch/include/TH -isystem /opt/conda/lib/python3.7/site-packages/torch/include/THC -isystem /opt/conda/include/python3.7m -D_GLIBCXX_USE_CXX11_ABI=1 -fPIC -std=c++14 -c /opt/conda/lib/python3.7/site-packages/deepspeed/ops/csrc/utils/flatten_unflatten.cpp -o flatten_unflatten.o \n[2/2] c++ flatten_unflatten.o -shared -L/opt/conda/lib/python3.7/site-packages/torch/lib -lc10 -ltorch_cpu -ltorch -ltorch_python -o utils.so\nLoading extension module utils...\nTime to load utils op: 18.740045070648193 seconds\n","output_type":"stream"}]},{"cell_type":"code","source":"%%script false --no-raise-error\n\n# training\nfor step, batch in enumerate(dataloader, 1):\n    \n    # prepare inputs and targets for the model and loss function respectively.\n    \n    # forward pass\n    outputs = model_engine(inputs)\n    \n    # computing loss\n    loss = loss_fn(outputs, targets)\n    \n    # backward pass\n    model_engine.backward()\n    \n    # optimization step\n    model_engine.step()","metadata":{"execution":{"iopub.status.busy":"2022-09-10T16:16:52.929285Z","iopub.execute_input":"2022-09-10T16:16:52.935221Z","iopub.status.idle":"2022-09-10T16:16:53.094655Z","shell.execute_reply.started":"2022-09-10T16:16:52.935175Z","shell.execute_reply":"2022-09-10T16:16:53.093245Z"},"trusted":true},"execution_count":10,"outputs":[]},{"cell_type":"code","source":"# memory clearing\ndel model, model_engine, optimizer, scheduler, model_parameters\ngc.collect()","metadata":{"execution":{"iopub.status.busy":"2022-09-10T16:16:53.096203Z","iopub.execute_input":"2022-09-10T16:16:53.097001Z","iopub.status.idle":"2022-09-10T16:16:53.584676Z","shell.execute_reply.started":"2022-09-10T16:16:53.096942Z","shell.execute_reply":"2022-09-10T16:16:53.583811Z"},"trusted":true},"execution_count":11,"outputs":[{"execution_count":11,"output_type":"execute_result","data":{"text/plain":"46"},"metadata":{}}]},{"cell_type":"markdown","source":"<h1>Layers Fusing</h1>\n\n<p style=\"font-size: 15px;\">\nIn progress...\n</p>","metadata":{}},{"cell_type":"markdown","source":"<h1>Conclusion</h1>\n\n<p style=\"font-size: 15px;\">\n    In this paper, the authors described and implemented additional approaches for further optimizing large-scale models, especially Transformers-based models in the NLP tasks. Authors showed that even primitive and simple methods such as Pre-tokenization and Turning Dropout off can significantly reduce training and inference time, then authors investigated more powerful approaches such as TorchScript and DeepSpeed for faster inference and training of large-scale models respectively. Authors will extend this work by the Layers Fusing approach as soon as possible, and also authors want to discover more approaches for optimization in the future.\n</p>","metadata":{}},{"cell_type":"markdown","source":"<h1>Feedback</h1>\n\n<p style=\"font-size: 15px;\">\nIf you run into problems with running implementations, want to share personal results with some of the described approaches, or offer your methods for optimization, you can put comments under this article or write a personal message to one of the authors' social networks.\n<ul>\n    <li style=\"font-size: 15px;\">Twitter - <a href=\"https://twitter.com/vad13irt\">vad13irt</a></li>\n    <li style=\"font-size: 15px;\">Telegram - <a href=\"https://t.me/vad13irt\">Vadim Irtlach</a></li>\n    <li style=\"font-size: 15px;\">Discord - vad13irt#0534</li>\n    <li style=\"font-size: 15px;\">Google Mail (gmail) - <a href=\"vadimirtlach@gmail.com\">vadimirtlach@gmail.com</a></li>\n</ul>\n</p>","metadata":{}},{"cell_type":"markdown","source":"<h1>References</h1>\n<p style=\"font-size: 15px;\">\nDuring writing this article, authors referred to this literature:\n</p>\n<ul>\n    <li><a style=\"font-size: 15px;\" href=\"https://huggingface.co/blog/hf-bitsandbytes-integration\">A Gentle Introduction to 8-bit Matrix Multiplication for transformers at scale using Hugging Face Transformers, Accelerate and bitsandbytes</a></li>\n    <li><a style=\"font-size: 15px;\" href=\"https://habr.com/ru/company/yandex/blog/672396/\">Яндекс выложил YaLM 100B — сейчас это крупнейшая GPT-подобная нейросеть в свободном доступе. Вот как удалось её обучить</a></li>\n    <li><a style=\"font-size: 15px;\" href=\"https://paperswithcode.com/method/dropout\">Dropout</a></li>\n    <li><a style=\"font-size: 15px;\" href=\"https://huggingface.co/docs/transformers/main/en/serialization\">Export 🤗 Transformers Models</a></li>\n    <li><a style=\"font-size: 15px;\" href=\"https://www.kaggle.com/competitions/bms-molecular-translation/discussion/230498#1262312\">Tips for Transformer inference</a></li>\n    <li><a style=\"font-size: 15px;\" href=\"https://www.microsoft.com/en-us/research/blog/deepspeed-compression-a-composable-library-for-extreme-compression-and-zero-cost-quantization/\">DeepSpeed Compression: A composable library for extreme compression and zero-cost quantization</a></li>\n    <li><a style=\"font-size: 15px;\" href=\"https://jarvislabs.ai/blogs/deepspeed\">Training Large NLP Models Efficiently with DeepSpeed Hugging Face</a>\n    <li><a style=\"font-size: 15px;\" href=\"https://www.kaggle.com/competitions/feedback-prize-effectiveness/discussion/339147\">DeepSpeed Compression</a></li>\n    <li><a style=\"font-size: 15px;\" href=\"https://www.kaggle.com/code/tanulsingh077/longformer-training-with-deepspeed-and-hf-trainer/notebook?scriptVersionId=86784214\">LongFormer Training with DeepSpeed and HF-Trainer</a></li>\n    <li><a style=\"font-size: 15px;\" href=\"https://www.kaggle.com/competitions/feedback-prize-effectiveness/discussion/347537\">Team Hydrogen: Efficiency Prize 1st Place</a></li>\n</ul>","metadata":{}},{"cell_type":"markdown","source":"<h1>Releases</h1>\n<ul>\n    <li style=\"font-size: 15px;\"><b>12.09.2022</b> - initial release.</li>\n</ul>","metadata":{}}]}