Vectorized channel threshold search #971
Conversation
eladc-git
force-pushed
the
vectorized_channel_search
branch
from
584b936 to
59a4cc6
Compare
eladc-git
changed the title
| n_bins: int = 2048, | ||
| n_bits: int = 8) -> np.ndarray: | ||
| """ | ||
| Compute the error function between a tensor to its quantized version per channel. |
There was a problem hiding this comment.
Computes the error function between a tensor and its quantized version for each channel.
The error is based on the KL-divergence between the distributions.
The function uses a specified number of bins to compute the histogram of the float tensor.
It requires the threshold and number of bits used for quantization to determine the histogram's boundaries and the number of quantized bins.
There was a problem hiding this comment.
I didn't change the descriptions in this PR, but I can change them.
| n_bins: Number of bins for the float histogram. | ||
| n_bits: Number of bits the quantized tensor was quantized by. | ||
|
|
||
| Returns: |
There was a problem hiding this comment.
An array containing the KL-divergence between the float and quantized histograms of the tensor for each channel.
|
|
||
| Args: | ||
| x: Float tensor. | ||
| range_min: array of min bound on the quantization range. |
There was a problem hiding this comment.
range_min: Array specifying the minimum bound of the quantization range for each channel.
range_max: Array specifying the maximum bound of the quantization range for each channel.
n_bins: Number of bins for the float histogram.
n_bits: Number of bits used for quantization.
| """ | ||
|
|
||
| error_list = [] | ||
| for j in range(x.shape[0]): # iterate all channels of the tensor. |
There was a problem hiding this comment.
is this always the axis of the channels? for all frameworks?
There was a problem hiding this comment.
yes. We make it that way in the quantization search.
| @@ -349,6 +381,7 @@ def get_threshold_selection_tensor_error_function(quantization_method: Quantizat | |||
| quantization_method: Quantization method for threshold selection | |||
There was a problem hiding this comment.
quantization_method: Method used for selecting the quantization threshold.
quant_error_method: Type of error function requested.
p: P-norm to use for calculating the Lp-norm distance.
axis: Axis along which the operation has been performed.
norm: Indicates whether to normalize the result of the error function.
n_bits: Number of bits used to quantize the tensor.
signed: Indicates whether the input is signed.
| @@ -55,8 +55,9 @@ def power_of_two_selection_tensor(tensor_data: np.ndarray, | |||
| threshold = max_power_of_two(tensor_max, min_threshold) | |||
| else: | |||
| signed = True # weights are always signed | |||
| axis = -1 if per_channel else None | |||
There was a problem hiding this comment.
I don't think this is the case for Pytorch. I think you need to use the fw_info
There was a problem hiding this comment.
Numpy. This is in the common section.
| @@ -726,9 +726,7 @@ def _error_function_wrapper(error_function: Callable, | |||
| q_tensor: Numpy array with quantized tensor's content. | |||
| in_params: Quantization params the tensor is quantized by (used in specific error functions only). | |||
|
|
|||
| Returns: A list of error values per-channel for the quantized tensor, according to the error function. | |||
| Returns: A array of error values per-channel for the quantized tensor, according to the error function. | |||
There was a problem hiding this comment.
Returns: An array of error values for each channel of the quantized tensor, as determined by the specified error function.
| @@ -58,7 +58,8 @@ def symmetric_selection_tensor(tensor_data: np.ndarray, | |||
| threshold = get_init_threshold(min_threshold, tensor_max, per_channel) | |||
| else: | |||
| signed = True # weights are always signed | |||
| error_function = get_threshold_selection_tensor_error_function(QuantizationMethod.SYMMETRIC, quant_error_method, p, norm=False, n_bits=n_bits, signed=signed) | |||
| axis = -1 if per_channel else None | |||
There was a problem hiding this comment.
I don't think this is the case for Pytorch. I think you need to use the fw_info
| @@ -56,7 +56,8 @@ def uniform_selection_tensor(tensor_data: np.ndarray, | |||
| if quant_error_method == qc.QuantizationErrorMethod.NOCLIPPING: | |||
| mm = tensor_min, tensor_max | |||
| else: | |||
| error_function = get_threshold_selection_tensor_error_function(QuantizationMethod.UNIFORM, quant_error_method, p, norm=False) | |||
| axis = -1 if per_channel else None | |||
There was a problem hiding this comment.
I don't think this is the case for Pytorch. I think you need to use the fw_info
eladc-git
force-pushed
the
vectorized_channel_search
branch
from
59a4cc6 to
5e33f22
Compare
Pull Request Description:
This PR removes the for loop for each threshold search per channel and uses numpy vectorization capability.
For KL divergence error function, we keep the loop because this KL involves histogram functions which are not trivial to vectorize.
Checklist before requesting a review: