GLM-4 Series

[hhhhzl]

Add GLM-4

File Changed:

• add glm.py and glm_layer.py
• add GLM4 to AutoModel
• add GLM4 template to templates.py (double checked it's the correct template)

Test:

• python generate.py --model THUDM/glm-4-9b-chat --template glm4
• hf_result: python hf_generate.py --model THUDM/glm-4-9b-chat

Challenges/Questions:

• I got different results when I changed AutoModelForCausalLM to GlmForCausalLM. However, I can get a uniform unreadable result if I set top_k, top_p, and temperature = 0 with GlmForCausalLM. My intuition is that this is because GlmForCausalLM has slightly different settings for a GLM-like model.
• I also got different unreadable outputs from generate.py, which makes it difficult to compare them due to challenge 1.
• Do we need to pass trust-remote-code = true in for GLMs?