HiggsAudioM3 Starter Code

Audio-in, text-out predictions using the HiggsAudioM3 model via an OpenAI-compatible API.

Setup

pip install -r requirements.txt

Set your API key (if required):

export BOSONAI_API_KEY="your-api-key-here"

Quick Start

# Basic usage — send audio, get text response
python predict.py sample.wav --system-prompt "Chat naturally with the user."

# Stream the response token by token
python predict.py sample.wav --system-prompt "Chat naturally with the user." --stream

# Use a different endpoint and/or different model (e.g., local server)
python predict.py sample.wav --base-url https://hackathon.boson.ai/v1 --model higgs-audio-understanding-v3-Hackathon

Using as a Library

from audio_utils import chunk_audio_file
from predict import predict

# Option 1: Use the high-level predict function
response = predict("recording.wav", system_prompt="Summarize this audio.")

# Option 2: Build your own pipeline
chunks, meta = chunk_audio_file("recording.wav")
print(f"Audio: {meta['duration_s']}s → {meta['num_chunks']} chunks")

# Then use chunks with build_messages() and your own OpenAI client
from predict import build_messages
messages = build_messages(chunks, system_prompt="You are a helpful assistant.")

How Audio Chunking Works

Important: The VAD + 4-second chunking logic is essential for correct API behavior. We strongly recommend using the provided chunk_audio_file() helper as-is. Modifying the chunking pipeline (e.g., skipping VAD, changing the chunk size, or altering the gap-filling logic) may produce unexpected or degraded results. Only make changes if you fully understand the preprocessing requirements.

The API accepts audio in chunks of up to 4 seconds each. The chunking pipeline handles this automatically:

1. Load — Read the audio file (WAV, FLAC, OGG, etc.)
2. Resample — Convert to 16kHz (API requirement)
3. VAD — Silero Voice Activity Detection finds speech segments
4. Fill gaps — Expand segments to cover the full audio (no dropped frames)
5. Split — Break any segment > 4s into sub-chunks
6. Encode — Each chunk becomes a base64-encoded WAV string

Each chunk is sent as a separate audio_url content part in the API request. The server reassembles them in order using the chunk index in the MIME type (audio/wav_0, audio/wav_1, ...).

API Call Parameters

Important: The STOP_SEQUENCES and EXTRA_BODY parameters in predict.py are required for correct API behavior. Do not modify them — they are specific to the HiggsAudioM3 API and changing them will likely break the response format.

Parameter	Value	Can modify?
STOP_SEQUENCES	["<|eot_id|>", "<|endoftext|>", "<|audio_eos|>", "<|im_end|>"]	No — required by the API
EXTRA_BODY	{"skip_special_tokens": False}	No — required by the API
temperature	0.2	Yes — but 0.2 is the recommended default
top_p	0.9	Yes — but 0.9 is the recommended default
max_tokens	2048	Yes — adjust based on expected response length

The combination of temperature=0.2 and top_p=0.9 is the best practice for most use cases. You may adjust these for your specific needs, but the defaults are well-tested.

Prompt Examples

The following examples demonstrate different use cases. All prompts go in the system message unless noted otherwise. Audio chunks are appended as audio_url content parts AFTER any text in the user message. See predict.py for the full wire format.

1. ASR (Automatic Speech Recognition)

For ASR, you need both a system prompt and a user text prompt. The user text goes before the audio in the user message content.

python predict.py sample.wav \
    --system-prompt "You are an automatic speech recognition (ASR) system." \
    --user-text "Your task is to listen to audio input and output the exact spoken words as plain text in English."

Prompt templates:

SYSTEM_MESSAGE_ASR = "You are an automatic speech recognition (ASR) system."

# Language-specific (replace {source_language} with: English, Chinese, Spanish, Korean, Japanese, etc.)
USER_MESSAGE_ASR = "Your task is to listen to audio input and output the exact spoken words as plain text in {source_language}."

# Language-agnostic (auto-detect language)
USER_MESSAGE_ASR_AUTO = "Your task is to listen to audio input and output the exact spoken words as plain text."

2. General Chat / Audio Understanding

For general chat, specify the system prompt only. Leave --user-text empty — the audio itself is the user's input.

python predict.py sample.wav --system-prompt "You are a helpful assistant."

3. Tool Use / Function Calling [v3.5 only]

Embed tool definitions in the system prompt wrapped in <tools>...</tools>. The API response will contain tool calls wrapped in <tool_call>...</tool_call>. Pass tool results back as the next user message wrapped in <tool_response>...</tool_response>.

Flow:

1. Send system prompt with <tools> + user audio → API response with <tool_call>
2. Execute the function locally
3. Send tool result as next user message in <tool_response> tags
4. API responds with final natural language answer

python predict.py sample.wav \
    --system-prompt "You are a helpful function-calling AI assistant.
<tools>
{\"tools\": [{\"type\": \"function\", \"function\": {\"name\": \"get_horoscope\", ...}}]}
</tools>"

Full tool definition example:

import json

# Step 1: Define your tools
tools = [
    {
        "type": "function",
        "function": {
            "name": "get_horoscope",
            "description": "Get today's horoscope for an astrological sign.",
            "parameters": {
                "type": "object",
                "properties": {
                    "sign": {
                        "type": "string",
                        "description": "An astrological sign like Taurus or Aquarius",
                    },
                },
                "required": ["sign"],
            },
        }
    },
]

# Step 2: Build the system prompt with tools embedded
tools_wrapper = {"tools": tools}
tools_json = json.dumps(tools_wrapper)
SYSTEM_MSG_WITH_TOOLS = (
    "You are a helpful AI assistant with access to function calling tools.\n"
    f"<tools>\n{tools_json}\n</tools>"
)

# Step 3: After receiving a <tool_call> response, execute the function and
# send the result back as the next user message:
tool_response_msg = {
    "role": "user",
    "content": '<tool_response>\n{"name": "get_horoscope", "result": {"horoscope": "..."}}\n</tool_response>'
}
# Then call the API again with the updated message history to get the final answer.

4. Thinking Mode [v3.5 only]

Add "Use Thinking." (note: both words must be capitalized) at the end of the system prompt. The API will include reasoning in <think>...</think> tags before the actual response.

python predict.py sample.wav \
    --system-prompt "You are a helpful voice assistant. Use Thinking."

5. Multi-Turn Chat

For multi-turn conversations, keep appending to the existing ChatML message list. Each new user turn with audio is prepared the same way as a single-turn request — chunk the audio using VAD, encode as base64, and add as audio_url content parts in the user message.

The message history follows standard OpenAI chat format:

messages = [
    {"role": "system", "content": "You are a helpful assistant."},

    # Turn 1: user audio
    {"role": "user", "content": [
        {"type": "audio_url", "audio_url": {"url": "data:audio/wav_0;base64,..."}},
        {"type": "audio_url", "audio_url": {"url": "data:audio/wav_1;base64,..."}},
    ]},
    {"role": "assistant", "content": "I heard you say ..."},

    # Turn 2: user audio (chunked the same way)
    {"role": "user", "content": [
        {"type": "audio_url", "audio_url": {"url": "data:audio/wav_0;base64,..."}},
    ]},
    {"role": "assistant", "content": "Sure, here's ..."},

    # Turn 3: continue appending ...
]

Each turn's audio is independently chunked via the same VAD pipeline (chunk_audio_file). The chunk indices (wav_0, wav_1, ...) reset per user message — they only need to be unique within a single message.

Best Practices

Prompt Tips

The quality of your results depends heavily on how you write your system prompt (user prompt is usually not needed except for ASR):

• Be specific about the model's role. Instead of "You are a helpful assistant", describe what kind of assistant: "You are a medical transcription specialist who converts doctor-patient audio recordings into structured clinical notes."
• Define expected behavior explicitly. State what the model should and shouldn't do: output format, tone, language, level of detail, how to handle ambiguity or unclear audio.
• Provide domain context. If the task involves specialized knowledge (e.g., legal, medical, financial), include relevant terminology, constraints, or rules the model should follow.
• Specify output format. If you need JSON, bullet points, a specific schema, or a particular structure — say so in the system prompt. The model follows formatting instructions well when they are clear.
• Keep instructions unambiguous. Vague prompts produce vague results. If two interpretations are possible, the model will guess. Remove ambiguity by being explicit.
• Iterate and refine. Start with a simple prompt, review the output, then add constraints or clarifications where the model's behavior doesn't match your expectations.

Key takeaway: Clarity is the single most important factor. A detailed, well-structured system prompt consistently outperforms a short, generic one.

Audio & API Usage

The model is designed for ASR and audio-based conversation, not long-form audio understanding. Keep these guidelines in mind:

• Keep each audio file under 30 seconds. Split longer audio into sentence-level segments and call the API once per segment. This produces better results and also reduces per-request latency.
• Be mindful of API rate limits. The hackathon API has request rate limitations. Avoid sending many requests in rapid succession — add appropriate spacing between calls, and prefer sequential sentence-wise requests over bulk parallel requests.

File Overview

File	Purpose
predict.py	Main script — CLI and predict() function
audio_utils.py	Audio loading, VAD chunking, encoding utilities
requirements.txt	Python dependencies

Available Models

Two model versions are provided:

Model	Model ID	Notes
v3.5 (recommended)	higgs-audio-understanding-v3.5-Hackathon	Stronger overall — better instruction following, tool use support, and better adherence to long system prompts. Use this as your default.
v3 (fallback)	higgs-audio-understanding-v3-Hackathon	Solid baseline. Try this if you encounter any issues with v3.5 (e.g., unexpected outputs or edge-case regressions).

Switch models via the --model flag:

# Default (v3.5)
python predict.py sample.wav --model higgs-audio-understanding-v3.5-Hackathon

# Fallback (v3)
python predict.py sample.wav --model higgs-audio-understanding-v3-Hackathon

Recommendation: Start with v3.5. If you run into problems, switch to v3 to see if the issue is model-specific.

API Details

• Endpoint: https://hackathon.boson.ai/v1
• Models: See Available Models above
• Format: OpenAI-compatible chat completions
• Auth: API key via BOSONAI_API_KEY env var

Note for hackathon participants: The endpoint, model name, and API key may be customized for your event. Check with your hackathon organizers for the correct values and update predict.py or pass them via CLI flags (--base-url, --model) accordingly.