[fix] Make add_special_tokens to false for completions API

[Pernekhan]

PR title

[fix] Change default for add_special_tokens to false, since the completions API shouldn't expect to modify the prompt.

Description

Change the default of add_special_tokens. It made some confusions, especially when running with speculative decoding which were trained with single bos tokens. The extra bos token messes up the rest of the output.

[jhaotingc]

/bot run --disable-fail-fast

[tensorrt-cicd]

PR_Github #6338 [ run ] triggered by Bot

[tensorrt-cicd]

PR_Github #6338 [ run ] completed with state SUCCESS
/LLM/main/L0_MergeRequest_PR pipeline #4628 completed with status: 'FAILURE'

[LinPoly]

It made some confusions, especially when running with speculative decoding which were trained with single bos tokens. The extra bos token messes up the rest of the output.

So you have prepended a BOS token to the prompt before sending to the server, correct?

[Pernekhan]

It made some confusions, especially when running with speculative decoding which were trained with single bos tokens. The extra bos token messes up the rest of the output.

So you have prepended a BOS token to the prompt before sending to the server, correct?

This is how it's usually done. For the completions API the prompt is already formatted, and the bos token is included most of the time. That's why I'm proposing the default to not prepend anything.

[LinPoly]

Thanks for explanation, I think it is reasonable to do, could you please also change this line to prompt_tokens = 5 to pass the CI?