[ Discussion ] Realtime voice agents

[asiff00]

I was planning to use csm-1b as the final layer of a real-time voice agent (VAD -> ASR -> LLM -> TTS) that I was working on. However, due to its super slow inference time, it seems like an unrealistic goal at this point.

This is the repo I'm working on: https://github.com/asiff00/On-Device-Speech-to-Speech-Conversational-AI

I currently use kokoro as the TTS engine, which what makes it possible to run in real-time.

[francescogruner]

Hi @asiff00 , so CSM1b is too slow? Do you think it can improve with systems like livekit and fastRTC? I saw your project, I'll try it :) does it work with coqui tts too?

[asiff00]

Hi @asiff00 , so CSM1b is too slow? Do you think it can improve with systems like livekit and fastRTC? I saw your project, I'll try it :) does it work with coqui tts too?

@francescogruner, CSM takes a couple of seconds to generate 1 second of audio even with a GPU (RTX 3060). It has to be faster than that to be effective as real-time local TTS.

The proof of concept project was based on Kokoro because I wanted it to run on an average CPU. Therefore, other TTS won't directly work out of the box, but the code can be adjusted to work with any TTS. As a matter of fact, I'm refactoring the code as we speak so it is more flexible and actually useful.

[francescogruner]

@asiff00 Great! I bookmarked your project, I'd love to experiment with this one! Thanks for the feedback, if you want let's talk on LinkedIn, good work 💪🔝

[harisrab]

Why don't you guys use Pipecat (https://docs.pipecat.ai/getting-started/overview)? Surprising to me no one talked about it. I'm building a solution for it.

[asiff00]

Why don't you guys use Pipecat (https://docs.pipecat.ai/getting-started/overview)? Surprising to me no one talked about it. I'm building a solution for it.

@harisrab Interesting! Will check it out!

[etohimself]

I got stuck at same issue. Used webrtcvad > whisper > ollama > csm1b. first three runs almost instantly but csm1b takes 5-10 seconds even at fp16. we need to find/make a fp4 and try

[asiff00]

I got stuck at same issue. Used webrtcvad > whisper > ollama > csm1b. first three runs almost instantly but csm1b takes 5-10 seconds even at fp16. we need to find/make a fp4 and try

@etohimself Definitely share if you manage to reduce generation latency of this model. It has so much potential but is limited by it's speed.

[shekharmeena2896]

@asiff00 Try to load the model in float 16, and keep the model loaded in the gpu, do not load the model again and again for each request, just keep it loaded in gpu, hit it with websocket and see if it works.

[asiff00]

@asiff00 Try to load the model in float 16, and keep the model loaded in the gpu, do not load the model again and again for each request, just keep it loaded in gpu, hit it with websocket and see if it works.

@shekharmeena2896 did you try it?

[herraristotle]

Hello @asiff00 tried looking at dia yet?

[TheAjaykrishnanR]

@asiff00 I see that you have included orpheus in your project, i have made a realtime voice to voice setup using it here, it does stutter in 3060 but on a 3090 its realtime, i even have a colab but on normal T4 its the same as a 3060

[asiff00]

@TheAjaykrishnanR That's so cool! I would check it out and let you know!

[asiff00]

@herraristotle I did, and it is fantastic!

[Horschig]

With all these new fancy TTS (Orpheus, CSM, Dia) the biggest issue is indeed that you need a somewhat beefy GPU. CSM requires something like 100 tokens per second (and it is based in Llama 1B, so you can do your research what it means). Orpheus needs about 84 tokens per second (while being based in Llama 3B). I do not know about Dia, because people already say that it is slow as f...

For CSM there were some attempts to make it faster. One is using quantized versions, e.g. https://huggingface.co/johnbenac/sesame-csm-1b-GGUF-encoder/blob/main/examples/main/README.md
you can also try torch.compile(), which might speed things up considerably (but it's a bit of a hell to get it to work properly often).
This guy said that they improved CSM streaming (not sure how or what, maybe code tweaks): https://huggingface.co/johnbenac/sesame-csm-1b-GGUF-encoder/blob/main/examples/main/README.md
This guy also said it is "near realtime" with an RTF of 0.6 n a Ti4070 Super:
https://github.com/ReisCook/VoiceAssistant

But with limited GPU power, Kokoro is still the way to go.