-
Notifications
You must be signed in to change notification settings - Fork 1.7k
Open
Labels
documentationImprovements or additions to documentationImprovements or additions to documentation
Description
1. 遇到问题的章节 / Affected Chapter
Chapter5.3 ->SFTDataset->generate_loss_mask函数
2. 具体问题描述 / Problem Description
代码这样改是否更简洁?
def init() 添加:
self.bos_token_id = self.tokenizer('<|im_start|>assistant' , add_special_tokens = False)['input_ids']
self.eos_token_id = self.tokenizer('<|im_end|>' , add_special_tokens=False)['input_ids']

以及,是否能用KMP算法对时间进行优化?O(n^2) -> O(n * log n)
3. 问题重现材料 / Reproduction Materials
无
确认事项 / Verification
Metadata
Metadata
Assignees
Labels
documentationImprovements or additions to documentationImprovements or additions to documentation