👋 Hi! Thank you for contributing to llm-compressor. Please add the ready label when the PR is ready for review.

Note: This is required to complete the testing suite, please only add the label once the PR is code complete and local testing has been performed.

@rahul-tuli That's a good enough reason to wait until some regression tests are finished. We should figure out why OOM occurs and potentially add that to the device map/ fix memory leaks

Yep, the title is a bit of a misnomer is more in reference to "fixing smoothquant" rather than smoothquant-specific implementation

Here's the results of memory profiling SmoothQuant under various implementations. Constants are the model used (llama3.2-1b-instruct) and the batch_size (16x2048).

Standard w/ calib_context: 10047127552
Smoothquant: 10584129536
SQ w/o empty_cache: 10584129536
SQ w/o empty_cache, w/ calib_context: 10047258624

# standard forward pass
with calibration_forward_context(model):
    for batch in tqdm.tqdm(dataloader):
        model(**batch)

Here's the results of memory profiling Quantization Modifier under various implementations. Constants are the model used (llama3.2-1b-instruct) and the batch_size (16x2048).

Standard w/ calib_context: 10047127552
Quantization Modifier: 11243553901
QM w/o empty_cache: 11243553901
QM w/o empty_cache, w/ calib_context: 10047128685

# standard forward pass
with calibration_forward_context(model):
    for batch in tqdm.tqdm(dataloader):
        model(**batch)

From this brief analysis, I believe it's safe to conclude that

1. Removing torch.cuda.empty_cache does not lead to increased peak memory usage
2. Adding calibration_forward_context does decrease peak memory usage (specifically due to having disabled the HF KV cache)