Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

feat: Allow remote backend configuration #75

Open
askulkarni2 opened this issue Feb 7, 2025 · 1 comment
Open

feat: Allow remote backend configuration #75

askulkarni2 opened this issue Feb 7, 2025 · 1 comment
Assignees
@ApostaC
Labels
feature request New feature or request

Comments

@askulkarni2
Copy link

For production deployments on smaller gpus, it would be great to allow configuration of remote backend options as shown here. This will allow sharing of the KV Cache across multiple vllm instances resulting in better performance.
@ApostaC ApostaC added the feature request New feature or request label Feb 7, 2025
@ApostaC
Copy link
Collaborator

ApostaC commented Feb 7, 2025

Thanks for submitting the issue! That's on our todo list and it should be ready in the next week.

@ApostaC ApostaC self-assigned this Feb 7, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@ApostaC ApostaC

Labels
feature request New feature or request
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@askulkarni2 @ApostaC