Broken openai api

[Alcanderian]

System Info

Docker image: tensorrt_llm-release:v0.20.0rc0
GPU: B200

Who can help?

No response

Information

• The official example scripts
• My own modified scripts

Tasks

• An officially supported task in the examples folder (such as GLUE/SQuAD, ...)
• My own task or dataset (give details below)

Reproduction

YML:

cat dsr1.yml 
pytorch_backend_config:
    enable_overlap_scheduler: true
    use_cuda_graph: true
    moe_backend: TRTLLM

Serve:

trtllm-serve serve     --tp_size 8 --ep_size 8     --backend pytorch     --extra_llm_api_options dsr1.yml /root/.cache/huggingf
ace/hub/models--nvidia--DeepSeek-R1-FP4/snapshots/574fdb8a5347fdbc06b2c18488699c0c17d71e05

Eval

lm_eval --model local-chat-completions --model_args model=/root/.cache/huggingface/hub/models--nvidia--DeepSeek-R1-FP4/snapshots/574fdb8a5347fdbc06b2c18488699c0c17d71e05/,base_url=http://127.0.0.1:8000/v1/chat/completions,num_concurrent=64,timeout=999999,max_gen_toks=8192 --tasks gsm8k --batch_size 64 --num_fewshot 8

Expected behavior

Correctly response the request

actual behavior

API request failed with error message: {"object":"error","message":"[{'type': 'literal_error', 'loc': ('body', 'messages', 0, 'typed-dict', 'role'), 'msg': "Input should be 'developer'", 'input': 'user', 'ctx': {'expected': "'developer'"}}, {'type': 'extra_forbidden', 'loc': ('body', 'messages', 0, 'typed-dict', 'type'), 'msg': 'Extra inputs are not permitted', 'input': 'text'}, {'type': 'literal_error', 'loc': ('body', 'messages', 0, 'typed-dict', 'role'), 'msg': "Input should be 'system'", 'input': 'user', 'ctx': {'expected': "'system'"}}, {'type': 'extra_forbidden', 'loc': ('body', 'messages', 0, 'typed-dict', 'type'), 'msg': 'Extra inputs are not permitted', 'input': 'text'}, {'type': 'extra_forbidden', 'loc': ('body', 'messages', 0, 'typed-dict', 'type'), 'msg': 'Extra inputs are not permitted', 'input': 'text'}, {'type': 'literal_error', 'loc': ('body', 'messages', 0, 'typed-dict', 'role'), 'msg': "Input should be 'assistant'", 'input': 'user', 'ctx': {'expected': "'assistant'"}}, {'type': 'extra_forbidden', 'loc': ('body', 'messages', 0, 'typed-dict', 'type'), 'msg': 'Extra inputs are not permitted', 'input': 'text'}, {'type': 'literal_error', 'loc': ('body', 'messages', 0, 'typed-dict', 'role'), 'msg': "Input should be 'tool'", 'input': 'user', 'ctx': {'expected': "'tool'"}}, {'type': 'missing', 'loc': ('body', 'messages', 0, 'typed-dict', 'tool_call_id'), 'msg': 'Field required', 'input': {'role': 'user', 'content': "Question: Farrah ordered 4 boxes from Amazon containing 20 matchboxes each. If each matchbox has 300 sticks, calculate the total number of match sticks that Farah ordered?\nAnswer: Farrah ordered 4 boxes from Amazon containing 20 matchboxes each, a total of 420=<<420=80>>80 matchboxes.\nIf each matchbox has 300 sticks, the total number of match sticks that Farah bought is 30080 = <<30080=24000>>24000\n#### 24000\n\nQuestion: Maya's organization hosted a weekly farmers' market to raise money for the church choir. They sold broccolis, carrots, spinach, and cauliflowers. After adding together all of their earnings, Maya found out that they had made $380. The organization made $57 from broccoli and the sales of the carrots are twice as much as the sales of broccoli. Then, their sales for the spinach is $16 more than half of the sales of carrots. How much did they make from cauliflower sales?\nAnswer: The sales of the carrot is 57 x 2 = $&lt;<572=114>>114\nHalf the sales of carrot is $114 / 2 = $<<114/2=57>>57.\nSo, the sales of the spinach is $57 + $16 = <<57+16=73>>73\nThe total sales of the broccolis, carrots, spinach is $57 + $114 + $73 =$<<57+114+73=244>>244.\nThey made $380 - $244 = $<<380-244=136>>136 from cauliflower sales.\n#### 136\n\nQuestion: Rica's group won in a dance competition. She got 3/8 of the prize money. From Rica's prize money, she spent 1/5 of it and is now left with $300. How much was the prize money that her group won?\nAnswer: Rica is left with 1 - 1/5 = 4/5 of her prize money which is equal to $300.\nSince 4/5 is worth $300, then 1/5 is worth $300/4 = $75.\nSo, Rica got $75 x 5 = $375 from their prize money which is 3/8 of the total prize.\nSince 3/8 is equal to $375, then 1/8 is worth $375/3 = $125.\nSo, the total prize money is $125 x 8 = $<<1258=1000>>1000.\n#### 1000\n\nQuestion: A wildlife team is monitoring the number of birds in a park. There are 3 blackbirds in each of the park’s 7 trees. There are also 13 magpies roaming around the park. How many birds are in the park in total?\nAnswer: In the trees, there are 7 trees * 3 blackbirds per tree = <<73=21>>21 blackbirds.\nIn addition to the magpies, there is a total of 21 blackbirds + 13 magpies = <<21+13=34>>34 birds in the park.\n#### 34\n\nQuestion: In a race, there are eight runners. The first five runners finish the race in 8 hours, while the rest of the runners finish the race 2 hours later. Calculate the total time the eight runners took to finish the race.\nAnswer: The first five runners took a combined total of 58 = <<58=40>>40 hours to finish the race.\nThe number of runners who finished the race 2 hours later after the first five is 8-5 = <<8-5=3>>3\nThe 3 runners who finished the race 2 hours later took a total of 8+2 =<<8+2=10>>10 hours to run the race.\nTogether, the three runners who finished the race in 10 hours took a total of 103 = <<103=30>>30 hours.\nAll eight runners took 30+40 = 70 hours altogether to finish the race.\n#### 70\n\nQuestion: John buys twice as many red ties as blue ties. The red ties cost 50% more than blue ties. He spent $200 on blue ties that cost $40 each. How much did he spend on ties?\nAnswer:", 'type': 'text'}}, {'type': 'extra_forbidden', 'loc': ('body', 'messages', 0, 'typed-dict', 'type'), 'msg': 'Extra inputs are not permitted', 'input': 'text'}, {'type': 'missing', 'loc': ('body', 'messages', 0, 'typed-dict', 'name'), 'msg': 'Field required', 'input': {'role': 'user', 'content': "Question: Farrah ordered 4 boxes from Amazon containing 20 matchboxes each. If each matchbox has 300 sticks, calculate the total number of match sticks that Farah ordered?\nAnswer: Farrah ordered 4 boxes from Amazon containing 20 matchboxes each, a total of 420=<<420=80>>80 matchboxes.\nIf each matchbox has 300 sticks, the total number of match sticks that Farah bought is 30080 = <<30080=24000>>24000\n#### 24000\n\nQuestion: Maya's organization hosted a weekly farmers' market to raise money for the church choir. They sold broccolis, carrots, spinach, and cauliflowers. After adding together all of their earnings, Maya found out that they had made $380. The organization made $57 from broccoli and the sales of the carrots are twice as much as the sales of broccoli. Then, their sales for the spinach is $16 more than half of the sales of carrots. How much did they make from cauliflower sales?\nAnswer: The sales of the carrot is 57 x 2 = $<<572=114>>114\nHalf the sales of carrot is $114 / 2 = $&lt;<114/2=57>>57.\nSo, the sales of the spinach is $57 + $16 = <<57+16=73>>73\nThe total sales of the broccolis, carrots, spinach is $57 + $114 + $73 =$&lt;<57+114+73=244>>244.\nThey made $380 - $244 = $&lt;<380-244=136>>136 from cauliflower sales.\n#### 136\n\nQuestion: Rica's group won in a dance competition. She got 3/8 of the prize money. From Rica's prize money, she spent 1/5 of it and is now left with $300. How much was the prize money that her group won?\nAnswer: Rica is left with 1 - 1/5 = 4/5 of her prize money which is equal to $300.\nSince 4/5 is worth $300, then 1/5 is worth $300/4 = $75.\nSo, Rica got $75 x 5 = $375 from their prize money which is 3/8 of the total prize.\nSince 3/8 is equal to $375, then 1/8 is worth $375/3 = $125.\nSo, the total prize money is $125 x 8 = $&lt;<1258=1000>>1000.\n#### 1000\n\nQuestion: A wildlife team is monitoring the number of birds in a park. There are 3 blackbirds in each of the park’s 7 trees. There are also 13 magpies roaming around the park. How many birds are in the park in total?\nAnswer: In the trees, there are 7 trees * 3 blackbirds per tree = <<73=21>>21 blackbirds.\nIn addition to the magpies, there is a total of 21 blackbirds + 13 magpies = <<21+13=34>>34 birds in the park.\n#### 34\n\nQuestion: In a race, there are eight runners. The first five runners finish the race in 8 hours, while the rest of the runners finish the race 2 hours later. Calculate the total time the eight runners took to finish the race.\nAnswer: The first five runners took a combined total of 58 = <<58=40>>40 hours to finish the race.\nThe number of runners who finished the race 2 hours later after the first five is 8-5 = <<8-5=3>>3\nThe 3 runners who finished the race 2 hours later took a total of 8+2 =<<8+2=10>>10 hours to run the race.\nTogether, the three runners who finished the race in 10 hours took a total of 103 = <<103=30>>30 hours.\nAll eight runners took 30+40 = 70 hours altogether to finish the race.\n#### 70\n\nQuestion: John buys twice as many red ties as blue ties. The red ties cost 50% more than blue ties. He spent $200 on blue ties that cost $40 each. How much did he spend on ties?\nAnswer:", 'type': 'text'}}, {'type': 'literal_error', 'loc': ('body', 'messages', 0, 'typed-dict', 'role'), 'msg': "Input should be 'function'", 'input': 'user', 'ctx': {'expected': "'function'"}}, {'type': 'extra_forbidden', 'loc': ('body', 'messages', 0, 'typed-dict', 'type'), 'msg': 'Extra inputs are not permitted', 'input': 'text'}, {'type': 'extra_forbidden', 'loc': ('body', 'messages', 0, 'typed-dict', 'type'), 'msg': 'Extra inputs are not permitted', 'input': 'text'}]","type":"BadRequestError","param":null,"code":400}

additional notes

Ref to the openai api

[VALLIS-NERIA]

@LinPoly for viz