Activation Hessian computation runtime optimization

[ofirgo]

Pull Request Description:

Improve Activation Hessian computation runtime for GPTQ and Mixed precision with the following optimizations:

• Enable batch computation.
• Enable computation on a set of nodes (instead of a single node only).
• Other minor loop and implementation modifications.

Major design remarks:

• TraceHessianRequest receives a list of target nodes (target_nodes) instead of a single BaseNode.
• HessianInfoService produces a batch of samples for each computation iteration. We also fixed a bug where the HessianInfoService representative generator would have been initiated for each iteration instead of running on the generator from end-to-end.
• • The output of the "fetch" call of HessianInfoService is of the structure: List (per target nodes) of List (per image) of Hessian approximations (tensor).
• • The service's cache mechanism still saves results per request for a single node. We split and construct requests per-node to save and retrieve results for a certain node.
• • Batch computation requires keeping track of remaining samples from a given representative dataset batch, in case the requested Hessian computation batch is smaller (we don't want to "throw" samples away).
• • In addition, we assume that the Hessians computation batch_size is <= to the representative dataset batch size.
• Weights Hessian computation is still limited to a single image and per-node computation.
• Hessian results tensor includes a batch dimension.
• Default values for the number of samples and number of iterations for Hessians computation for GPTQ and Mixed precision have been modified.

Checklist before requesting a review:

• I set the appropriate labels on the pull request.
• I have added/updated the release note draft (if necessary).
• I have updated the documentation to reflect my changes (if necessary).
• All function and files are well documented.
• All function and classes have type hints.
• There is a licenses in all file.
• The function and variable names are informative.
• I have checked for code duplications.
• I have added new unittest (if necessary).

[reuvenperetz]

Used, but not crucial.