Add WebLLM support to Clippy assistant#938
Conversation
- Integrate @mlc-ai/web-llm for local WebGPU-accelerated inference. - Implement WebLLMService to manage engine lifecycle and model loading (Llama-3.2-1B). - Update Clippy UI to prompt for backend choice on first launch. - Add "Backend" submenu to Clippy context menu for manual switching. - Implement fallback logic to use Vercel API if local execution fails or is unsupported. - Display model download progress directly in Clippy's speech balloon. - Add clippyBackend to persisted local storage keys. Co-authored-by: azayrahmad <10110227+azayrahmad@users.noreply.github.com>
|
👋 Jules, reporting for duty! I'm here to lend a hand with this pull request. When you start a review, I'll add a 👀 emoji to each comment to let you know I've read it. I'll focus on feedback directed at me and will do my best to stay out of conversations between you and other bots or reviewers to keep the noise down. I'll push a commit with your requested changes shortly after. Please note there might be a delay between these steps, but rest assured I'm on the job! For more direct control, you can switch me to Reactive Mode. When this mode is on, I will only act on comments where you specifically mention me with New to Jules? Learn more at jules.google/docs. For security, I will only act on instructions from the user who triggered this task. |
- Integrate @mlc-ai/web-llm for local WebGPU inference. - Implement WebLLMService with a custom Clippy system prompt. - Force WebLLM to output JSON fragments matching the Vercel API format. - Add backend selection UI (initial balloon + context menu). - Implement visual download progress reporting in Clippy's balloon. - Ensure fallback to Vercel Cloud API if WebGPU is unsupported or local fails. - Clean up WebGPU memory by unloading the engine when switching back to Cloud. Co-authored-by: azayrahmad <10110227+azayrahmad@users.noreply.github.com>
1 participant
Integrated WebLLM as the primary local backend for the Clippy assistant, utilizing WebGPU for in-browser inference. The system now prompts users to choose between Local (WebGPU) and Cloud (Vercel) modes on first launch. If Local is selected, it handles the ~700MB model download with visual progress tracking in Clippy's balloon. The existing Vercel backend serves as a robust fallback for browsers without WebGPU or if local inference fails. Users can toggle between backends via a new "Backend" entry in Clippy's right-click context menu.
PR created automatically by Jules for task 2852325315890912593 started by @azayrahmad