web-llm with Deno?

[lonnietc]

Hello,

I am experimenting with Web-Based LLM's to take advantage of the WebGPU type acceleration and have been looking into Transformers.js and Web-llm. In particular, I am interested in seeing if it is possible to run the Node,js Web-LLM inside a Deno v8 runtime engine (https://deno.com/) by putting an API or Web Server in the front so that the user just needs to connect to the Deno engine or call the API.

Still working out some flow ideas here, but the idea is to basically start up Deno with the Web-LLM and then users can connect to it via the web server pages or possible as a headless OpenAI compatible API from other systems so it can use the local resources.

Any thoughts would be greatly appreciated.