Incorrect DeepSeek-R1 EOS token in LLMs section
Description
The "What are LLMs?" section currently lists the DeepSeek-R1 EOS token as:
<|end_of_sentence|>
However, the official DeepSeek-R1 tokenizer configuration defines the EOS token as:
<｜end▁of▁sentence｜>
The functionality is also currently described as "End of message text". Since this is the model's EOS token, it should be described as "End of sequence".
Expected correction
Change:
<|end_of_sentence|> → <｜end▁of▁sentence｜>
And change:
End of message text → End of sequence
Evidence
Official DeepSeek-R1 tokenizer configuration:
https://huggingface.co/deepseek-ai/DeepSeek-R1/blob/main/tokenizer_config.json
The eos_token field defines <｜end▁of▁sentence｜> as the EOS token.
Pull Request
A fix has already been prepared in PR #725:
#725
Incorrect DeepSeek-R1 EOS token in LLMs section
Description
The "What are LLMs?" section currently lists the DeepSeek-R1 EOS token as:
<|end_of_sentence|>However, the official DeepSeek-R1 tokenizer configuration defines the EOS token as:
<｜end▁of▁sentence｜>The functionality is also currently described as "End of message text". Since this is the model's EOS token, it should be described as "End of sequence".
Expected correction
Change:
<|end_of_sentence|>→<｜end▁of▁sentence｜>And change:
End of message text→End of sequenceEvidence
Official DeepSeek-R1 tokenizer configuration:
https://huggingface.co/deepseek-ai/DeepSeek-R1/blob/main/tokenizer_config.json
The
eos_tokenfield defines<｜end▁of▁sentence｜>as the EOS token.Pull Request
A fix has already been prepared in PR #725:
#725