Confusion about the results

[wutong4012]

Hello, thank you for sharing with the open source community. @ranonrkm @jianc99
I ran the result of Token per second based on Baseline.
But the result of running according to this command ENABLE_INTRA_NODE_COMM=1 torchrun --standalone --nproc_per_node=8 tests/SnapKV/selfspec_benchmark.py --model checkpoints/meta-llama/Meta-Llama-3.1-8B/model.pth --model_name meta-llama/Meta-Llama-3.1-8B --rank_group 0 1 2 3 4 5 6 7 --gamma 3 --B 64 --prefix_len 16032 --max_len 16128 --draft_budget 257 --benchmark --compile is as shown below.

Could you tell me how to view this result? How to compare with Baseline?

[jianc99]

Thanks for your attention of our work.

We have updated the code to print the number of generate tokens per second (throughput) after each run to help directly compare the throughput between baseline and speculative decoding results.

[wutong4012]

MagicDec/tests/baseline_benchmark.py

Line 97 in 09cd671

print(f"Tokens per second :{BATCH_SIZE*(model_steps/total_time)}")

Thanks for your reply. I have a minor question.
In fact, the calculation result of your Baseline is latency, so the result of magicdec should also output latency.
longspec does output avg latency, but selfspec does not.

By the way, what is the main difference between Figure 5 and Table 1 in the paper?

[jianc99]

For the baseline benchmark, I think the BATCH_SIZE*(model_steps/total_time) is throughput, not latency. At the end of each speculative decoding script, the throughput is also printed:

MagicDec/tests/StreamingLLM/selfspec_benchmark.py

Line 262 in 09cd671

print(f"Final tokens per second :{num_gen_tokens/total_time}")

Figure 5 in the paper illustrates the optimal speedup for different settings, including draft budgets, prefill lengths, batch sizes and draft-target model pairs. The annotations in the figure indicate the optimal speculation length (gamma) for each setting. Table 1 shows the detailed experiment results for various batch sizes and prefill length, using the fixed gamma and draft budget.

[wutong4012]

Sorry, it's my mistake. It is indeed throughput.
But I think to calculate the speedup ratio, latency should be used, that is, total_time / model_steps * BATCH_SIZE?

Thank you for your detailed explanation of Figure 5 and Table 1.