Special Tokens (#2)

* add special tokens

* special tokens

* pylint

* flake8

Author: hepengfe

README.md

@@ -5,6 +5,7 @@ Some shared utilities across multiple ASYML projects.
 
 ## List of Modules
 * `asyml_utilities.hyperparams.HParams`
+* `asyml_utilities.special_tokens.SpecialTokens`
 
 ## License
 [Apache License 2.0](./LICENSE)

asyml_utilities/special_tokens.py

@@ -0,0 +1,14 @@
+__all__ = [
+    'SpecialTokens',
+]
+
+
+class SpecialTokens:
+    r"""Special tokens, including :attr:`PAD`, :attr:`BOS`, :attr:`EOS`,
+    :attr:`UNK`. These tokens will by default have token ids 0, 1, 2, 3,
+    respectively.
+    """
+    PAD = "<PAD>"
+    BOS = "<BOS>"
+    EOS = "<EOS>"
+    UNK = "<UNK>"