byom documentation completed

README.md

@@ -1,6 +1,11 @@
 <div align="center">
 
-# reliability-score
+<h1> reliability-score 🎯 </h1>
+
+<p align="center">
+  <a href="http://149.169.30.58:8000/">[reliability-score documentation]</a>
+  <br> <br>
+</p>
 
 <a href="https://pytorch.org/get-started/locally/"><img alt="PyTorch" src="https://img.shields.io/badge/PyTorch-ee4c2c?logo=pytorch&logoColor=white"></a>
 <a href="https://pytorchlightning.ai/"><img alt="Lightning" src="https://img.shields.io/badge/-Lightning-792ee5?logo=pytorchlightning&logoColor=white"></a>
@@ -11,7 +16,7 @@
 
 ## Description
 
-What it does
+A tool for performing the reliability tests on NLP models.
 
 ## How to run
 
@@ -23,8 +28,8 @@ git clone [email protected]:Maitreyapatel/reliability-score.git
 cd reliability-score
 
 # [OPTIONAL] create conda environment
-conda create -n myenv python=3.8
-conda activate myenv
+conda create -n venv python=3.8
+conda activate venv
 
 # install pytorch according to instructions
 # https://pytorch.org/get-started/
@@ -37,7 +42,7 @@ Evaluate example model/data with default configuration
 
 ```bash
 # train on CPU
-python src/eval.py trainer=cpu
+python src/eval.py
 
 # train on GPU
 python src/eval.py trainer=gpu
@@ -46,15 +51,19 @@ python src/eval.py trainer=gpu
 Evaluate model with chosen dataset-specific experiment configuration from [configs/experiment/](configs/experiment/)
 
 ```bash
-python src/eval.py experiment=experiment_name.yaml
+python src/eval.py experiment=<experiment_name>
 ```
 
 Specify the custom model_name as shown in following MNLI example
 
 ```bash
 # if model_name is used for tokenizer as well.
-python src/eval.py experiment=mnli custom_model.model_name="ishan/bert-base-uncased-mnli"
+python src/eval.py experiment=mnli custom_model="bert-base-uncased-mnli"
 
 # if model_name is different for tokenizer then
-python src/eval.py experiment=mnli custom_model.model_name="ishan/bert-base-uncased-mnli" custom_model.tokenizer.model_name="ishan/bert-base-uncased-mnli"
+python src/eval.py experiment=mnli custom_model="bert-base-uncased-mnli" custom_model.tokenizer.model_name="ishan/bert-base-uncased-mnli"
 ```
+
+## Documentation:
+
+The locally hosted documentation can be found at: [LINK](http://149.169.30.58:8000/)

docs/source/byom.rst

@@ -1,5 +1,13 @@
 .. _byom:
 
+.. role:: python(code)
+  :language: python
+  :class: highlight
+
+.. role:: yaml(code)
+  :language: yaml
+  :class: highlight
+
 BYoM (Bring Your own Model)
 ===========================
 
@@ -46,30 +54,154 @@ Following example shows how the standard template looks-like:
 .. code-block:: yaml
 
     model_name: "roberta-large-mnli"
-    model_type: "discriminative" 
+    model_type: "discriminative"
     decoder_model_name: null
-    model_path: null 
+    model_path: null
     tie_embeddings: false
     label: null
     tie_encoder_decoder: false
     pipeline: null
 
-    additional_model_inputs: null 
+    additional_model_inputs: null
 
     tokenizer:
         model_name: ${..model_name} ## modify this only if tokenizer is a different then the model
-        label2id: 
+        label2id:
             contradiction: 0
             neutral: 1
             entailment: 2
         args:
             truncation: true
 
     data_processing:
-        header: null 
-        footer: null 
-        separator: " [SEP] " 
+        header: null
+        footer: null
+        separator: " [SEP] "
         columns:
             null
 
 
+As shown above yaml files for custom_model contains various parameters which allows different level of flexibility without touching the source-code.
+Below we explain each of the parameters in details.
+
+Level-1 set of parameters:
+^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+* :yaml:`model_name`: str: give model name from huggingface spaces
+    * example: `navteca/bart-large-mnli <https://huggingface.co/navteca/bart-large-mnli>`_
+* :yaml:`model_type`: str: provide the type of the model
+    * choices: :python:`["encode-decode","decoder-only","bart","discriminative","shared","hybrid","t5"]`
+    * BERT/RoBERTa are 'discriminative' models, while MT5 is T5 based model which works as discriminative model for MNLI dataset.
+    * Similarly, :yaml:`pipeline=zero-shot-classification` is discriminative type even if the base :yaml:`model_name` contains generative model (given that transformers supports this).
+* :yaml:`decoder_model_name`: str: provide the decoder model name if it is different than the :yaml:`mode_name`
+    * default: keep default to :yaml:`null`
+* :yaml:`model_path`: str: provide the path to the custom-trained model on local.
+    * default: keep default to :yaml:`null`
+* :yaml:`tie_embeddings`: bool: feature in progress
+    * default: keep default to :yaml:`false`
+* :yaml:`label`: feature in progress
+    * default: keep default to :yaml:`null`
+* :yaml:`tie_encoder_decoder`: bool: feature in progress
+    * default: keep default to :yaml:`false`
+* :yaml:`pipeline`: support of different huggingface pipelines
+    * choices: :python:`["zero-shot-classification"]`
+    * default: keep default to :yaml:`null`
+* :yaml:`additional_model_inputs`: dict: define the additional fixed inputs used while inference
+    * default: :yaml:`null`
+    * example: generative model uses different inputs such as :python:`num_beams=1`
+    * this is a level-2 type parameter
+* :yaml:`tokenizer`: dict: define tokenizer specific arguments
+    * this is a level-2 type parameter
+* :yaml:`data_processing`: dict: define the custom data pre-processing steps.
+    * you can use this for prompt/instruction enginerring
+    * this is a level-2 type parameter
+
+
+Level-2 set of parameters:
+^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+**additional_model_inputs**:
+
+This is a great example of unrestricted additional input arguments. Model like BERT/RoBERTa do not require any extra arguments apart from the :python:`**inputs` which is direct output from the tokenizer.
+However, models like T5 will require the extra input arguments and that can be defined as:
+
+.. code-block:: yaml
+
+    additional_model_inputs:
+        output_scores: true
+        return_dict_in_generate: true
+        num_beams: 1
+
+Similarly, if you are using :yaml:`pipeline` then it also takes additional arguments such as:
+
+.. code-block:: yaml
+
+    additional_model_inputs:
+        multi_label: false
+
+
+**tokenizer**:
+
+Tokenization can vary a lot based on the selected model or even the data.
+It is important to define the proper mapping between your trained version vs the reliability-score requirements.
+:yaml:`tokenizer` parameter contains the several reuqired parameters and again some unrestricted set of parameters:
+
+* :yaml:`model_name`: str: define the name of the tokenizer name
+    * default: keep the default to :yaml:`{..model_name}` if you are not using different tokenizer else provide the string of the tokenizer_name from the huggingface.
+* :yaml:`label2id`: dict: this is the most important part of the tokenizer, as :yaml:`label2id` within :python:`model.config` form the transformer might assume different ground truth labels
+    * For example, MNLI dataset contains three classes: entailment, contradiction, and neutral. Hence, define this mapping.
+    * **Note:** Please refer to your selected dataset.
+    * Consider the below snippet for sample:
+
+.. code-block:: yaml
+
+    label2id:
+        contradiction: 0
+        neutral: 1
+        entailment: 2
+
+* :yaml:`args`: dict: define the unrestricted set of arguments for the tokenizer from huggingface.
+    * For example, it can contain :python:`max_len:512`, :python:`truncation:false` or any other custom arguments.
+
+The final :yaml:`tokenizer` level-2 config looks like:
+
+.. code-block:: yaml
+
+    tokenizer:
+        model_name: ${..model_name}
+        label2id:
+            contradiction: 0
+            neutral: 1
+            entailment: 2
+        args:
+            truncation: true
+
+
+**data_processing:**
+
+This is by far the most important and latest feature which should be carefully defined.
+Suppose your model is trained using prompt enginerring or instruction learning. And in these cases it is important to define the prompts/instructions.
+At the same time, some models do not require any of these like BERT/RoBERTa and in this case we can ignore these parameters except for the :yaml:`separator`.
+
+* :yaml:`header`: str: define the global instruction
+    * default: keep the default to :yaml:`{null}` if you are not using any instruction.
+* :yaml:`footer`: str: define the signal to signal the model to generate
+    * default: keep the default to :yaml:`{null}` if you are not using any instruction.
+* :yaml:`separator`: str: define the separator string depending on your model for mixing the different columns of the dataset such as premise and hypothesis
+    * For BERT/RoBERTa: :yaml:`separator=" [SEP] "`
+    * For generative model: :yaml:`separator="\n"`
+* :yaml:`columns`: dict: this requires the good level of understanding of the dataset being used
+    * default: keep the default to :yaml:`null` if your are not using prompting.
+    * Else define the prefix string for each column in the dataset.
+    * consider the following code snippet for the MT5 prompt enginerring based model:
+
+
+.. code-block:: yaml
+
+    data_processing:
+        header: null
+        footer: null
+        separator: " "
+        columns:
+            premise: "xnli: premise:"
+            hypothesis: "hypothesis:"

docs/source/index.rst

@@ -20,7 +20,7 @@ If you have already setup the tool, then feel free to refer following pages for
 
    :ref:`BYoM (Bring Your own Model) <byom>`
 
-If you are a developer and interested in contributing then we highly suggests to refer following pages:
+If you are a developer and interested in contributing then we highly suggest to refer following pages:
 
    Learn about the dependencies <tbd>
 

docs/source/installation.rst

@@ -23,9 +23,7 @@ Setup the anaconda environment:
 
 Install the requirements:
 -------------------------
-# install pytorch according to instructions
-
-# https://pytorch.org/get-started/
+install pytorch according to instructions: https://pytorch.org/get-started/
 
 .. code-block:: shell