[Bug] HuggingFace and SGLang inference don't match

[pratcooper]

Checklist

• 1. I have searched related issues but cannot get the expected help.
• 2. The bug has not been fixed in the latest version.
• 3. Please note that if the bug-related issue you submitted lacks corresponding environment info and a minimal reproducible demo, it will be challenging for us to reproduce and resolve the issue, reducing the likelihood of receiving feedback.
• 4. If the issue you raised is not a bug but a question, please raise a discussion at https://github.com/sgl-project/sglang/discussions/new/choose Otherwise, it will be closed.
• 5. Please use English, otherwise it will be closed.

Describe the bug

The accuracy of the model is degraded due to inconsistent outputs from SGLang. While HF and vLLM produce consistent results such as "A" or "B," SGLang occasionally outputs responses like "I can't process that request." or "A." / "B." This inconsistency impacts overall accuracy.

Reproduction

What command or script did you run?
A script for generating outputs using a LLaMA 3.1 8B Istruct model with LoRA.

Which model are you using?
LLaMA 3.1 with LoRA applied.

Steps to reproduce:

1. Run the script with HF, vLLM, and SGLang configurations.
2. Compare the single-token outputs between the frameworks.
3. Observe the inconsistent behavior in SGLang.

---

Hugging Face Code Snippet :

tokenizer = AutoTokenizer.from_pretrained(model_name)
model = LlamaForCausalLM.from_pretrained(
model_name,
load_in_8bit=False,
torch_dtype=torch.float16,
device_map='auto',
)
adaptor_path = './model_spec/checkpoints/checkpoint-200-vllm'
model = PeftModel.from_pretrained(
model,
adaptor_path,
torch_dtype=torch.float16,
)

model.config.pad_token_id = tokenizer.pad_token_id = 0
model.config.bos_token_id = 1
model.config.eos_token_id = 2
model.generation_config.pad_token_id = tokenizer.pad_token_id
model.eval()

def evaluate(
instruction,
input=None,
temperature=0,
top_p=1,
top_k=-1,
num_beams=4,
max_new_tokens=128,
stream_output=False,
**kwargs,
):
prompt = f"<|begin_of_text|><|start_header_id|>system<|end_header_id|>\n\n{instruction}<|eot_id|><|start_header_id|>user<|end_header_id|>\n\n{input}<|eot_id|><|start_header_id|>assistant<|end_header_id|>\n\n"

inputs = tokenizer(prompt, return_tensors="pt")
input_ids = inputs["input_ids"].to('cuda')
generation_config = GenerationConfig(
    temperature=temperature,
    top_p=top_p,
    top_k=top_k,
    num_beams=num_beams,
    **kwargs,
)

with torch.no_grad():
    generation_output = model.generate(
        input_ids=input_ids,
        generation_config=generation_config,
        return_dict_in_generate=True,
        output_scores=True,
        max_new_tokens=max_new_tokens
    )
s = generation_output.sequences[0]
output = tokenizer.decode(s, skip_special_tokens=True)
result = output.split('assistant')[1].strip()
return result

---

SGLang Code Snippet :

import sglang as sgl
from sglang import *
import logging
import json
import torch

from typing import Union, List
from vllm import LLM, SamplingParams
from vllm.inputs import TokensPrompt
from vllm.lora.request import LoRARequest
from collections import defaultdict

from utils.usecase_prompts import UseCasePrompter
from utils.lora_adapters import UseCaseLoraAdapters
from utils.prompter import Prompter

logging.basicConfig(format='%(asctime)s %(message)s')
logger = logging.getLogger()
logger.setLevel(logging.INFO)

class SimpleSGLangLlama2:
def init(self, base_model_path, number_of_gpu=1, gpu_memory_utilization=0.4):
self.base_model_path = base_model_path
self._model = sgl.Engine(model_path=self.base_model_path)

def generate(
    self,
    prompt: Union[str, List[int]],
    temperature: float = 0.0,
    top_p: float = 1.0,
    top_k: int = -1,
    use_beam_search: bool = True,
    max_new_tokens: int = 128,
    best_of: int = 4
) -> List[str]:
    sampling_params = {"temperature": temperature, "top_p": top_p, "top_k":top_k, "max_new_tokens": max_new_tokens}
    # Build final_prompt_text, then:
    outputs = self._model.generate(
        [final_prompt_text],
        sampling_params,
        lora_path=adaptor_path
    )
    results = [output['text'] for output in outputs]
    return results

if name == "main":
model_path = "./models/meta-llama/Meta-Llama-3.1-8B-Instruct"
llm = SimpleSGLangLlama2(model_path)

---

Environment

Run the class in notebooks environment.

SGLang 0.4.0 (with flashinfer 0.1.6+cu121torch2.4)

[pratcooper]

@Ying1123