Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Allow the use of multiple providers #64

Open
flakey5 opened this issue May 16, 2024 · 0 comments
Open

Allow the use of multiple providers #64

flakey5 opened this issue May 16, 2024 · 0 comments

Comments

@flakey5
Copy link
Collaborator

flakey5 commented May 16, 2024

As titled, but my thoughts on how it could work:

Config schema changes

We just allow multiple AI providers in the config,

{
  aiProviders: {
    openai: { /* ... */ },
    mistral: { /* ... */ },
    // ...
  }
}

Request Body Changes

Add an optional provider property, so the body could look something like

{
  prompt: 'asd123',
  provider: 'openai'
}

Api changes

fastify.ai.warp and fastify.ai.warpStream could then take in the provider property from the request's body. The call to build() in warp.ts could return a lookup map (provider name:implementation). We then check that lookup map to see if the provider is enabled. If it is, we call it. Otherwise, throw a 400.

Concerns

Since the provider property in the body is optional, we need to figure out how to decide which provider to default to if we're not given one in the body.

I think we leave this up to the user, adding a defaultProvider option in the config for hardcoding it, e.g.

{
  defaultProvider: 'openai',
  aiProviders: {
    openai: { /* ... */ },
    mistral: { /* ... */ },
    // ...
  }
}

then in the api we have fastify.ai.getDefaultProvider() to dynamically decide what the default provider is. Could be useful for managing rate limits with the provider, ab testing, etc. By default, the function will just return what's set in the config.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant