Integrate Automated QDQ placement tool - Part 2

[willg-nv]

What does this PR do?

Type of change: new feature

Overview: This PR integrate automated Q/DQ placement tool to ModelOpt. This PR is 2/4 parts of the cahnges.

Part 1: #701
Part 2: #702
Part 3: #703
Part 4: #704

This PR contains the following changes:

1. Implement RegionPattern to represent the topology structure of Regions. InsertionPoints are also defined on RegionPattern. Regions with same pattern are optimized at the same time
2. Implement RegionSearch class to divide ONNX graph into small regions
3. RegionSearch python file also provides an entry point to print out the region structures.
4. Unit tests for new classse.

Usage

python -m modelopt.onnx.quantization.autotune.region_search --model model.onnx --verbose

Example output:

    ├─ Region 212 (Level 0, Type: COMPOSITE)
    │  ├─ Direct nodes: 0
    │  ├─ Total nodes (recursive): 9
    │  ├─ Children: 1
    │  ├─ Inputs: 3 tensors
    │  │    - xxx
    │  │    - xxx
    │  │    - xxx
    │  └─ Outputs: 1 tensors
    │       - xxx
    │
    │  Child regions:
    │
      ├─ Region 209 (Level 2, Type: LEAF) 
      │  ├─ Direct nodes: 9
      │  ├─ Total nodes (recursive): 9
      │  ├─ Children: 0
      │  ├─ Inputs: 11 tensors
      │  │    - xxx

Testing

Implemented unit tests for new classes. All unit tests could get pass locally.

Before your PR is "Ready for review"

• Make sure you read and follow Contributor guidelines and your commits are signed.
• Is this change backward compatible?: Yes
• Did you write any new necessary tests?: Yes
• Did you add or update any necessary documentation?: No, document change will be in part 4.
• Did you update Changelog?: No. Change log will be included in part 4.

Additional Information

[copy-pr-bot]

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

[willg-nv]

Hi @ajrasane , could you help me review this PR, thanks!

[ajrasane]

Could you provide a small example of how this tree looks?

[willg-nv]

I have added tests for region search to print tree structure, below is an example:

tests/unit/onnx/quantization/autotune/test_region_search.py::TestPrintTree::test_print_tree_top_down_builder
============================================================
Region Tree Structure:
============================================================
├─ Region 0 (Level 0, Type: LEAF)
│  ├─ Direct nodes: 2
│  ├─ Total nodes (recursive): 2
│  ├─ Children: 0
│  ├─ Inputs: 1 tensors
│  │    - input
│  └─ Outputs: 1 tensors
│       - output
│
│  Nodes in this region:
│    - Node 0: Conv (name: conv)
│    - Node 1: Relu (name: relu)
|
  ├─ <child region if exists>
============================================================

Currently, the 2 stage region partitioner only creates regions with depth <= 2.

[ajrasane]

Can this be saved as LEAF(ops)

[willg-nv]

Why additional LEAF is needed? I think op1->op2->...opn is okay to represent op sequence. Adding LEAF would make complex region too long.

[ajrasane]

How will the signatures of custom ops look? Could you provide an example?

[willg-nv]

example res-block signature:

COMPOSITE(Conv[kernel_shape=3x3]->BatchNormalization->Relu->Conv[kernel_shape=3x3]->BatchNormalization+Conv[kernel_shape=1x1]->BatchNormalization+Add->Relu)

graph structure:

                         Input
                           │
            ┌──────────────┴──────────────┐
            │                             │
            ▼                             ▼
    ┌───────────────┐             ┌───────────────┐
    │ Conv (3x3)    │             │ Conv (1x1)    │ (projection)
    └───────────────┘             └───────────────┘
            │                             │
            ▼                             ▼
    ┌───────────────┐             ┌───────────────┐
    │ BatchNorm     │             │ BatchNorm     │
    └───────────────┘             └───────────────┘
            │                             │
            ▼                             │
    ┌───────────────┐                     │
    │ Relu          │                     │
    └───────────────┘                     │
            │                             │
            ▼                             │
    ┌───────────────┐                     │
    │ Conv (3x3)    │                     │
    └───────────────┘                     │
            │                             │
            ▼                             │
    ┌───────────────┐                     │
    │ BatchNorm     │                     │
    └───────────────┘                     │
            │                             │
            └──────────────┬──────────────┘
                           ▼
                      ┌─────────┐
                      │   Add   │
                      └─────────┘
                           │
                           ▼
                      ┌─────────┐
                      │  Relu   │
                      └─────────┘

For custom plugin node, thier name will also be added to the signature.

[gcunhase]

If --dq_only is enabled, there may only be the DQ node indicating that a tensor is being quantized. Please verify that those cases are supporting with this function.

See

Model-Optimizer/modelopt/onnx/quantization/__main__.py

Line 210 in 307fe71

"--dq_only",