大佬问一个大模型分词问题

[Polaris-zmy]

Reminder

• I have read the README and searched the existing issues.

System Info

[2024-08-09 16:57:37,757] [INFO] [real_accelerator.py:203:get_accelerator] Setting ds_accelerator to cuda (auto detect)
[WARNING] async_io requires the dev libaio .so object and headers but these were not found.
[WARNING] async_io: please install the libaio-dev package with apt
[WARNING] If libaio is already installed (perhaps from source), try setting the CFLAGS and LDFLAGS environment variables to where it can be found.
[WARNING] Please specify the CUTLASS repo directory as environment variable $CUTLASS_PATH
[WARNING] NVIDIA Inference is only supported on Ampere and newer architectures
[WARNING] sparse_attn requires a torch version >= 1.5 and < 2.0 but detected 2.2
[WARNING] using untested triton version (2.2.0), only 1.0.0 is known to be compatible

• llamafactory version: 0.8.3.dev0
• Platform: Linux-4.19.24-7.34.cbp.al7.x86_64-x86_64-with-glibc2.35
• Python version: 3.10.12
• PyTorch version: 2.2.0+cu118 (GPU)
• Transformers version: 4.42.3
• Datasets version: 2.18.0
• Accelerate version: 0.32.1
• PEFT version: 0.11.1
• TRL version: 0.9.4
• GPU type: Tesla V100-SXM2-16GB
• DeepSpeed version: 0.14.2

Reproduction

大佬问一个大模型分词问题：现在我不想使用某个大模型原有的分词器，我该修改哪些代码来禁用原始分词器并添加我自己的呢？求指教

Expected behavior

No response

Others

No response