Retiarii Roadmap

[QuanluZhang]

Retiarii is a new NAS (exploratory-training) framework. We have the following goals when implementing it on NNI:

• Our basic design philosophy of model APIs is "writing DNN model as a space". That is why we have provided APIs like LayerChoice, InputChoice, ValueChoice, which can be directly embedded in a traditional PyTorch/TensorFlow model.
• High expressiveness. Any model space can be expressed with our provided APIs (e.g., mutators)
• High effectiveness and reusability of model search algorithms.
• Support more system-level optimizations to speedup the search, deduplicate computation as much as possible.

Guided by the abovementioned goals, we draft the following roadmap:

experimental in V2.0 -> alpha release in V2.1 -> beta release in V2.2 -> release in V2.3

V2.1

• Improved the robustness of graph generation and code generation for PyTorch models
• Supported the inline mutation API ValueChoice
• Improved the design and implementation of Model Evaluator
• Supported Random/Grid/Evolution exploration strategies (i.e., search algorithms)
• Support one system-level optimization: CSE optimization to deduplicate computation (deferred to V2.2)
• Migrate all the one-shot trainer from NAS v1 to Retiarii framework (i.e., NAS v2) (deferred to V2.2)

V2.2

• Support debug mode
• More high-level APIs for users to easily express model space
• Support a fancy NAS algorithm which could leverage architectural information
• Migrate more one-shot trainer from NAS v1 to Retiarii framework (i.e., NAS v2)
• Further improve the logic of graph generation and code generation
  • Support control flow in graph/code generation
  • Further integrated with TorchScript
• Support one system-level optimization: CSE optimization to deduplicate computation
• Initial support of TensorFlow

V2.3

• Deprecate NAS v1
• Improve system-level optimizations, especially on device placement and resource management
• Support fancy search algorithms which could jointly optimize architectures and hyper-parameters
• Improve system-level optimizations, distributed one-shot trainer with mixed parallelism