[Bug] Continuous batching (OpenAI Server) with greedy search return different results

[thangld201]

Checklist

• 1. I have searched related issues but cannot get the expected help.
• 2. The bug has not been fixed in the latest version.
• 3. Please note that if the bug-related issue you submitted lacks corresponding environment info and a minimal reproducible demo, it will be challenging for us to reproduce and resolve the issue, reducing the likelihood of receiving feedback.
• 4. If the issue you raised is not a bug but a question, please raise a discussion at https://github.com/sgl-project/sglang/discussions/new/choose Otherwise, it will be closed.
• 5. Please use English, otherwise it will be closed.

Describe the bug

I am using greedy decoding (temperature==0.0) for the same gpu and every time we run inference on the same data, the results are a whole lot different

Results are similar if (i) I use for loop (no batching, one example at a time) OR (ii) use offline inference i.e. model.chat(...)
I believe there's a critical bug with continuous batching at the moment (since (ii) works).

Reproduction

To reproduce, first run the api server

python -m sglang.launch_server --model-path  meta-llama/Llama-3.1-8B-Instruct  --dtype bfloat16  --host 0.0.0.0 --port 8011

Then run (batching with multithread)

from openai import OpenAI
from tqdm.auto import tqdm
from concurrent.futures import ThreadPoolExecutor, as_completed

def multithread(func_to_call,list_data,max_workers=8):
    res = []
    with tqdm(total=len(list_data)) as pbar:
        with ThreadPoolExecutor(max_workers=min(len(list_data),max_workers)) as ex:
            futures = [ex.submit(func_to_call, k) for k in list_data]
            for future in as_completed(futures):
                result = future.result()
                res.append(result)
                pbar.update(1)
    return res

def multithread_wrapper(func_to_call,list_data,max_workers=8):
    from tqdm import trange
    mthread_inputs = []
    for rnd_idx in range(len(list_data)): mthread_inputs.append((rnd_idx,list_data[rnd_idx]))
    wrapper_func = lambda x:[x[0],func_to_call(x[1])]
    wrapped_outputs = multithread(wrapper_func,mthread_inputs,max_workers=max_workers)
    wrapped_outputs = sorted(wrapped_outputs,key=lambda x:x[0])
    wrapped_outputs = [x[1].choices[0].message.content.strip() for x in wrapped_outputs]
    return wrapped_outputs

from openai import OpenAI
openai_api_key ='WHATEVER' 
openai_api_base =  'http://localhost:8011/v1'
client = OpenAI(
    api_key=openai_api_key,
    base_url=openai_api_base,
)


models = client.models.list()
model = models.data[0].id

func_to_call = lambda x: client.chat.completions.create(
            messages=x,
            model=model,
            max_tokens=1024,
            temperature=0.0,
            seed=9,
        )

single_prompt = [
{
    "role":"system",
    "content":"You are a superhelpful assistant"
},
 {
    "role":"user",
    "content":"Hi how are you feeling ? Respond in a random non-English language"
 },
 {
     "role":"assistant",
     "content":"元気です！あなたはどうですか？"
 },
 {
    "role":"user",
    "content":"Great, now teach me which language you used, and its grammar and basic phrases"
 }
]

list_prompts = [single_prompt]*100

# batch infer - for loop
# outs_1 = [func_to_call(single_prompt).choices[0].message.content.strip() for single_prompt in list_prompts]
# outs_2 = [func_to_call(single_prompt).choices[0].message.content.strip() for single_prompt in list_prompts]


# batch infer - multithread
outs_1 = multithread_wrapper(func_to_call,list_prompts)
outs_2 = multithread_wrapper(func_to_call,list_prompts)

cc=0
for x,y in zip(outs_1,outs_2): 
    if not x==y: # inconsistent output
        cc+=1
        # print(x)
        # print('-'*5)
        # print(y)
        # print('='*10)
print("Num mismatch:",cc) 

#Output: Num mismatch: 35
#Tested on a A100 PCIE GPU

Environment

vllm: 0.6.4.post1
openai: 1.58.1
PyTorch: 2.5.1+cu124
sglang: 0.4.1.post3
flashinfer: 0.1.6+cu124torch2.4
triton: 3.1.0
transformers: 4.45.2

[zhaochenyang20]

@thangld201 Thanks for pointing out. You can check this:

https://sgl-project.github.io/references/faq.html

[thangld201]

@zhaochenyang20 At this point I believe there's a bug and not simply "slight numeric error" due to "dynamic batch size" (It even changes the top-1 token). I would also like to help (though I'm not sure where to start).

And the same issue persists with vLLM, but everything works fine with offline inference (which AFAIK similarly uses dynamic batch). Even if assuming that different batch sizes invoke different kernels, so long as the request orders and contents stay the same, we should expect exact same results for same hardware (obviously not the case here).

[zhaochenyang20]

@merrymercy I don't know what can we do beyond this https://sgl-project.github.io/references/faq.html now.