Added support for Groq

[haailabs]

Groq is delivering the world's fastest LLM inference with very generous free API credits. The default usage is with llama3-70b as it is currently Groq's most advanced free model, though paying users have preview access to llama3.1-405B. To use llama3.1-405B, replace all instances of llama3-70b-8192 by llama-3.1-405b-reasoning

[alberto98fx]

You can actually use the current OpenAI implementation :)
What you need to do is just allow the user to pass the base_url like this:

import openai

openai.api_key = '...'
openai.base_url = "https://api.groq.com/openai/v1/chat/completions"

Taken from here: https://github.com/openai/openai-python?tab=readme-ov-file#module-level-client

[haailabs]

You can actually use the current OpenAI implementation :) What you need to do is just allow the user to pass the base_url like this:

import openai

openai.api_key = '...'
openai.base_url = "https://api.groq.com/openai/v1/chat/completions"

Taken from here: https://github.com/openai/openai-python?tab=readme-ov-file#module-level-client

There isn't a single instance of the term "groq" in the link you attached:

And Google doesn't return any decent results for "https://api.groq.com/openai/v1/chat/completions"

So, no you cannot use the current OpenAI implementation. Not to mention that OpenAI itself recommends not using the module level client you link in application code

Thanks ;)

[alberto98fx]

Oh, looks like I needed to update myself a little bit on the lib and such, here is the thing:

Here (https://console.groq.com/docs/quickstart) you can find the endpoint I mentioned:

And here is how to use it:

Note: Groq is OpenAI compatible. Means that you can use any implementation of that API.

from openai import OpenAI

client = OpenAI(
    api_key='xxxxx',
    base_url="https://api.groq.com/openai/v1"
)


def chat_with_groq(prompt):
    try:
        response = client.chat.completions.create(
            model="mixtral-8x7b-32768",  # Or another model supported by Groq
            messages=[
                {"role": "system", "content": "You are a helpful assistant."},
                {"role": "user", "content": prompt}
            ]
        )
        return response.choices[0].message.content
    except Exception as e:
        return f"An error occurred: {str(e)}"

def main():
    print("Welcome to the Groq Chat! (Type 'quit' to exit)")
    while True:
        user_input = input("You: ")
        if user_input.lower() == 'quit':
            print("Goodbye!")
            break
        response = chat_with_groq(user_input)
        print("Assistant:", response)

if __name__ == "__main__":
    main()

Here's the output:

╰─➤  python3 test.py
Welcome to the Groq Chat! (Type 'quit' to exit)
You: Which model are you
Assistant: I am a large language model trained by Mistral AI. I was designed to be able to assist with a wide range of tasks, including providing information, answering questions, and helping with various tasks.
You:

Please let me know if it works for you!
You just need the API key and it should work straight out of the box :)

[xprabhudayal]

there has been a rate limit on groq api, does anyone know how to bypass it & use it in free tier?