Loading pretrained weights error: UnicodeDecodeError

[STEVENTAN100]

Hi, Sebastian

I've encounter a problem on loading pretrained weights from OpenAI:


UnicodeDecodeError: 'utf-8' codec can't decode byte 0xbf in position 26: invalid start byte

My running environment is Windows 11 23h2，conda install python==3.10.14

I think this issue is very likely related to the TensorFlow incompatibility! But I tried TensorFlow==2.16.2, it doesn't solved the problem.

Looking forward to your kind reply!

Related issues but no answer #362

[rasbt]

Arg, sorry about this hassle. Could you try tensorflow==2.15.0. That's a version that I know that worked on Windows for sure.

If this doesn't work, I think the best workaround for now, until we find a solution, is to consider alternative weight loading methods, e.g., one of the two I implemented here: https://github.com/rasbt/LLMs-from-scratch/tree/main/ch05/02_alternative_weight_loading

[d-kleine]

I am using tensorflow 2.18 and ran the code without any issues. I assume this is rather an encoding issue (chinese system) or a Windows path setting issue.

My idea would be to make sure that the downloaded files are uft-8 encoded, here:

LLMs-from-scratch/ch05/01_main-chapter-code/gpt_download.py

Line 67 in 5fc78ff

with open(destination, "wb") as file:

Please change this line in your code to

with open(destination, "wb", encoding="utf-8") as file:

and delete the downloaded files, restart the kernel of the notebook, running the code from the beginning again.

[rasbt]

Thanks, @d-kleine! I believe

with open(destination, "wb", encoding="utf-8") as file:

may not work because it is a binary file. However, I think you are on the right track, and it's this line where being explicit about the encoding might address the issue:

settings = json.load(open(os.path.join(model_dir, "hparams.json"), "r", encoding="utf-8"))

[d-kleine]

I think both solutions should work, but unfortunately I cannot verify this on my end. My approach would try to encode the files when downloading (the binary model file should be unaffected), yours would try to encode the hparams setting file when reading in.

Edit: Just tried, my solution does not work as indeed all files will be forced-encoded then, including the model file. So please delete the downloaded files and try @rasbt's solution. At least on my system, this worked perfectly.

[STEVENTAN100]

Thanks, @d-kleine! I believe

with open(destination, "wb", encoding="utf-8") as file:

may not work because it is a binary file. However, I think you are on the right track, and it's this line where being explicit about the encoding might address the issue:

settings = json.load(open(os.path.join(model_dir, "hparams.json"), "r", encoding="utf-8"))

Well, I've tried your solution and re-downloaded all the files, but it couldn't solve the problem though. I think the Chinese system is the underlying reason, however, the problem doesn't occur at the settings, but at the tf_ckpt_path params. I'm so sorry to present the full log late, which suddenly occur to me that I've been made a mistake to put my code under the folder with Chinese characters!

After I move my code to the pure English folder, I can run my code successfully. Above all, the issue is totally solved, no code should be changed & feel free to use the latest TensorFlow version, and again thanks for your hospitable reply @rasbt @d-kleine !

[d-kleine]

Yeah, in the full error log you can see that the issue is an encoding problem with the path, probably because of the folder that contains Chinese characters where the repo has been cloned into. Anyways, glad it got solved! 👍🏻

[rasbt]

I am glad this works now! Thanks for your help and insights here @d-kleine !