Skip to content

Commit ec80147

Browse files
committed
feat: add provider plugin architecture
1 parent b1f5395 commit ec80147

23 files changed

Lines changed: 3361 additions & 132 deletions

.gitignore

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -24,3 +24,4 @@ Thumbs.db
2424
# Config (may contain local paths)
2525
config.yaml
2626
config.json
27+
.env

CHANGELOG.md

Lines changed: 25 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,30 @@
11
# Changelog
22

3+
## Unreleased
4+
5+
## v1.4.0 - 2026-06-14
6+
7+
Feature release for provider/plugin architecture and MCP-backed search/read providers.
8+
9+
### Added
10+
11+
- Provider/plugin configuration with built-in `searxng`, `duckduckgo`, and `builtin-reader` providers.
12+
- `web-search --provider` with compatibility for existing `--engine` workflows.
13+
- `web-reader --provider` with `builtin-reader` default behavior preserved.
14+
- MCP provider adapter for Streamable HTTP responses, including SSE event parsing and double-encoded `content[0].text` JSON payloads.
15+
- Optional BigModel/Zhipu MCP provider support through `ZHIPU_APIKEY`.
16+
- `doctor --json` provider summaries with auth configured status and no secret value leakage.
17+
- Provider architecture docs, Mermaid flow diagrams, and a provider plugin development guide.
18+
19+
### Verified
20+
21+
- `go test ./...`
22+
- `go vet ./...`
23+
- `./scripts/smoke.sh`
24+
- `git diff --check`
25+
- Live smoke with `ZHIPU_APIKEY`: BigModel Search MCP via `--provider bigmodel`.
26+
- Live smoke with `ZHIPU_APIKEY`: BigModel Reader MCP via `--provider bigmodel`.
27+
328
## v1.3.2 - 2026-06-14
429

530
Patch release from real-world Agent workflow testing.

README.md

Lines changed: 86 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -2,14 +2,14 @@
22

33
Local-first web search and reading CLI for AI agents.
44

5-
Zero cost. No API keys. No third-party dependencies.
5+
Zero cost by default. No API keys required for the local-first path.
66

77
[Releases](https://github.com/koda-claw/web-tools/releases)
88

99
## What it does
1010

11-
- **web-search**: Search the web via DuckDuckGo Lite by default, with optional local SearXNG
12-
- **web-reader**: Extract readable content from URLs or convert local files (PDF, DOCX, PPTX, XLSX) to Markdown
11+
- **web-search**: Search the web via DuckDuckGo Lite by default, optional local SearXNG, or configured MCP providers
12+
- **web-reader**: Extract readable content from URLs or convert local files (PDF, DOCX, PPTX, XLSX) to Markdown, with optional MCP providers
1313

1414
## Agent Quick Start
1515

@@ -102,9 +102,11 @@ npm i -g agent-browser
102102
web-tools web-search "latest AI news"
103103
web-tools web-search "AI latest developments" --locale en-US --limit 3
104104
web-tools web-search "golang readability" --include-domain github.com --exclude-domain reddit.com
105+
web-tools web-search "golang readability" --provider duckduckgo --json
105106

106107
# Read a URL
107108
web-tools web-reader https://example.com/article
109+
web-tools web-reader https://example.com/article --provider builtin-reader
108110

109111
# Convert a file
110112
web-tools web-reader ./report.pdf
@@ -147,13 +149,17 @@ Config file (optional): `~/.config/web-tools/config.json` or `./web-tools.json`
147149
"default_timeout": 15,
148150
"browser_fallback": true,
149151
"markitdown_path": "markitdown",
150-
"agent_browser_path": "agent-browser"
152+
"agent_browser_path": "agent-browser",
153+
"default_provider": "auto",
154+
"default_provider_chain": ["builtin-reader"]
151155
},
152156
"search": {
153157
"searxng_url": "http://localhost:8888",
154158
"default_limit": 5,
155159
"default_locale": "auto",
156-
"default_engine": "auto"
160+
"default_engine": "auto",
161+
"default_provider": "auto",
162+
"default_provider_chain": ["searxng", "duckduckgo"]
157163
}
158164
}
159165
```
@@ -162,6 +168,55 @@ CLI flags override config defaults when provided. `--format=html` is only availa
162168

163169
`web-reader --json` includes a `quality` object with extraction score, word count, minimum word threshold, fallback recommendation, and reasons. Sparse extraction warnings are written to stderr so stdout remains machine-consumable.
164170

171+
### Provider configuration
172+
173+
`--provider` is the preferred selector for new integrations. `--engine` remains supported for compatibility with `auto`, `duckduckgo`, and `searxng`.
174+
175+
The default no-key path stays local-first:
176+
177+
```text
178+
search auto: searxng -> duckduckgo
179+
reader auto: builtin-reader
180+
```
181+
182+
Optional MCP providers can be enabled through config. For BigModel/Zhipu:
183+
184+
```json
185+
{
186+
"providers": {
187+
"bigmodel": {
188+
"type": "mcp",
189+
"auth_env": "ZHIPU_APIKEY",
190+
"enabled_if_env": "ZHIPU_APIKEY",
191+
"timeout": 30,
192+
"capabilities": ["search", "reader"],
193+
"search": {
194+
"url": "https://open.bigmodel.cn/api/mcp/web_search_prime/mcp",
195+
"tool": "web_search_prime"
196+
},
197+
"reader": {
198+
"url": "https://open.bigmodel.cn/api/mcp/web_reader/mcp",
199+
"tool": "webReader"
200+
}
201+
}
202+
},
203+
"search": {
204+
"default_provider_chain": ["searxng", "bigmodel", "duckduckgo"]
205+
}
206+
}
207+
```
208+
209+
Then run:
210+
211+
```bash
212+
export ZHIPU_APIKEY=...
213+
web-tools doctor --json
214+
web-tools web-search "Go readability library" --provider bigmodel --json
215+
web-tools web-reader "https://github.com/go-shiori/go-readability" --provider bigmodel --json
216+
```
217+
218+
Secrets are read only from environment variables. `doctor --json` reports whether auth is configured, but never prints the token value.
219+
165220
## Install as Agent Skill
166221

167222
Compatible with [vercel-labs/skills](https://github.com/vercel-labs/skills) CLI:
@@ -188,15 +243,41 @@ After installing the skill, ask the agent to use `web-tools` for web search,
188243
webpage reading, article extraction, or file-to-Markdown conversion. The skill
189244
contains the Agent research workflow.
190245

246+
## Provider Development
247+
248+
To add a new search or reader backend, start with
249+
[`docs/provider-plugin-development-guide.md`](docs/provider-plugin-development-guide.md).
250+
The provider model is configuration-first: use `providers.<id>` with an existing
251+
adapter when possible, and add adapter code only when the protocol or response
252+
mapping cannot be reused.
253+
191254
## Architecture
192255

256+
```mermaid
257+
flowchart TB
258+
Agent["Agent / Skill"] --> CLI["web-tools CLI"]
259+
CLI --> Search["web-search"]
260+
CLI --> Reader["web-reader"]
261+
CLI --> Doctor["doctor"]
262+
Config["Config\nproviders + defaults"] --> Registry["Provider Registry"]
263+
Search --> Registry
264+
Reader --> Registry
265+
Doctor --> Registry
266+
Registry --> Builtins["builtin providers\nsearxng / duckduckgo / builtin-reader"]
267+
Registry --> MCP["MCP adapter\nStreamable HTTP + SSE + JSON-RPC"]
268+
MCP --> BigModel["BigModel/Zhipu MCP"]
269+
Builtins --> Output["stable JSON output"]
270+
MCP --> Output
271+
```
272+
193273
```
194274
web-tools
195275
├── cmd/web-reader/ # web-reader CLI entry point
196276
├── cmd/web-search/ # web-search CLI entry point
197277
├── internal/
198278
│ ├── config/ # Configuration loading (file + env + defaults)
199279
│ ├── errors/ # Structured error handling for agent consumption
280+
│ ├── provider/ # Provider registry and MCP adapters
200281
│ ├── reader/ # HTTP fetch, readability extraction, cache, converter, browser fallback
201282
│ └── search/ # SearXNG client, result parsing, output formatting
202283
├── docker/ # SearXNG docker-compose.yml + settings

cmd/doctor/main.go

Lines changed: 111 additions & 24 deletions
Original file line numberDiff line numberDiff line change
@@ -27,25 +27,39 @@ type Check struct {
2727
}
2828

2929
type ConfigSummary struct {
30-
Reader ReaderSummary `json:"reader"`
31-
Search SearchSummary `json:"search"`
30+
Providers map[string]ProviderSummary `json:"providers,omitempty"`
31+
Reader ReaderSummary `json:"reader"`
32+
Search SearchSummary `json:"search"`
33+
}
34+
35+
type ProviderSummary struct {
36+
Type string `json:"type"`
37+
Capabilities []string `json:"capabilities,omitempty"`
38+
AuthEnv string `json:"auth_env,omitempty"`
39+
AuthConfigured bool `json:"auth_configured,omitempty"`
40+
EnabledIfEnv string `json:"enabled_if_env,omitempty"`
41+
Enabled bool `json:"enabled"`
3242
}
3343

3444
type ReaderSummary struct {
35-
CacheDir string `json:"cache_dir"`
36-
CacheTTL int `json:"cache_ttl"`
37-
DefaultTimeout int `json:"default_timeout"`
38-
BrowserFallback bool `json:"browser_fallback"`
39-
MarkitdownPath string `json:"markitdown_path"`
40-
AgentBrowserPath string `json:"agent_browser_path"`
41-
MinContentLength int `json:"min_content_length"`
45+
CacheDir string `json:"cache_dir"`
46+
CacheTTL int `json:"cache_ttl"`
47+
DefaultTimeout int `json:"default_timeout"`
48+
BrowserFallback bool `json:"browser_fallback"`
49+
MarkitdownPath string `json:"markitdown_path"`
50+
AgentBrowserPath string `json:"agent_browser_path"`
51+
MinContentLength int `json:"min_content_length"`
52+
DefaultProvider string `json:"default_provider"`
53+
DefaultProviderChain []string `json:"default_provider_chain"`
4254
}
4355

4456
type SearchSummary struct {
45-
SearXNGURL string `json:"searxng_url"`
46-
DefaultLimit int `json:"default_limit"`
47-
DefaultLocale string `json:"default_locale"`
48-
DefaultEngine string `json:"default_engine"`
57+
SearXNGURL string `json:"searxng_url"`
58+
DefaultLimit int `json:"default_limit"`
59+
DefaultLocale string `json:"default_locale"`
60+
DefaultEngine string `json:"default_engine"`
61+
DefaultProvider string `json:"default_provider"`
62+
DefaultProviderChain []string `json:"default_provider_chain"`
4963
}
5064

5165
type Report struct {
@@ -141,6 +155,7 @@ func (c checker) Run() Report {
141155
c.executableCheck("agent-browser", cfg.Reader.AgentBrowserPath, "optional browser fallback dependency"),
142156
c.searxngCheck(cfg.Search.SearXNGURL),
143157
}
158+
checks = append(checks, providerChecks(*cfg)...)
144159

145160
return Report{
146161
OK: allRequiredChecksOK(checks),
@@ -216,24 +231,88 @@ func errorCheck(name string, message string, err error, details map[string]strin
216231

217232
func summarizeConfig(cfg config.Config) ConfigSummary {
218233
return ConfigSummary{
234+
Providers: summarizeProviders(cfg.Providers),
219235
Reader: ReaderSummary{
220-
CacheDir: cfg.Reader.CacheDir,
221-
CacheTTL: cfg.Reader.CacheTTL,
222-
DefaultTimeout: cfg.Reader.DefaultTimeout,
223-
BrowserFallback: cfg.Reader.BrowserFallback,
224-
MarkitdownPath: cfg.Reader.MarkitdownPath,
225-
AgentBrowserPath: cfg.Reader.AgentBrowserPath,
226-
MinContentLength: cfg.Reader.MinContentLength,
236+
CacheDir: cfg.Reader.CacheDir,
237+
CacheTTL: cfg.Reader.CacheTTL,
238+
DefaultTimeout: cfg.Reader.DefaultTimeout,
239+
BrowserFallback: cfg.Reader.BrowserFallback,
240+
MarkitdownPath: cfg.Reader.MarkitdownPath,
241+
AgentBrowserPath: cfg.Reader.AgentBrowserPath,
242+
MinContentLength: cfg.Reader.MinContentLength,
243+
DefaultProvider: cfg.Reader.DefaultProvider,
244+
DefaultProviderChain: append([]string(nil), cfg.Reader.DefaultProviderChain...),
227245
},
228246
Search: SearchSummary{
229-
SearXNGURL: cfg.Search.SearXNGURL,
230-
DefaultLimit: cfg.Search.DefaultLimit,
231-
DefaultLocale: cfg.Search.DefaultLocale,
232-
DefaultEngine: cfg.Search.DefaultEngine,
247+
SearXNGURL: cfg.Search.SearXNGURL,
248+
DefaultLimit: cfg.Search.DefaultLimit,
249+
DefaultLocale: cfg.Search.DefaultLocale,
250+
DefaultEngine: cfg.Search.DefaultEngine,
251+
DefaultProvider: cfg.Search.DefaultProvider,
252+
DefaultProviderChain: append([]string(nil), cfg.Search.DefaultProviderChain...),
233253
},
234254
}
235255
}
236256

257+
func providerChecks(cfg config.Config) []Check {
258+
checks := make([]Check, 0, len(cfg.Providers))
259+
for id, provider := range cfg.Providers {
260+
details := map[string]string{
261+
"type": provider.Type,
262+
"capabilities": strings.Join(provider.Capabilities, ","),
263+
}
264+
if provider.AuthEnv != "" {
265+
details["auth_env"] = provider.AuthEnv
266+
details["auth_configured"] = fmt.Sprintf("%t", os.Getenv(provider.AuthEnv) != "")
267+
}
268+
if provider.EnabledIfEnv != "" {
269+
details["enabled_if_env"] = provider.EnabledIfEnv
270+
details["enabled"] = fmt.Sprintf("%t", os.Getenv(provider.EnabledIfEnv) != "")
271+
} else {
272+
details["enabled"] = "true"
273+
}
274+
status := StatusOK
275+
message := "provider configured"
276+
if provider.EnabledIfEnv != "" && os.Getenv(provider.EnabledIfEnv) == "" {
277+
status = StatusWarn
278+
message = "provider configured but activation env is missing"
279+
}
280+
checks = append(checks, Check{
281+
Name: "provider." + id,
282+
Status: status,
283+
Message: message,
284+
Details: details,
285+
})
286+
}
287+
return checks
288+
}
289+
290+
func summarizeProviders(providers map[string]config.ProviderConfig) map[string]ProviderSummary {
291+
if len(providers) == 0 {
292+
return nil
293+
}
294+
out := make(map[string]ProviderSummary, len(providers))
295+
for id, provider := range providers {
296+
enabled := true
297+
if provider.EnabledIfEnv != "" {
298+
enabled = os.Getenv(provider.EnabledIfEnv) != ""
299+
}
300+
authConfigured := false
301+
if provider.AuthEnv != "" {
302+
authConfigured = os.Getenv(provider.AuthEnv) != ""
303+
}
304+
out[id] = ProviderSummary{
305+
Type: provider.Type,
306+
Capabilities: append([]string(nil), provider.Capabilities...),
307+
AuthEnv: provider.AuthEnv,
308+
AuthConfigured: authConfigured,
309+
EnabledIfEnv: provider.EnabledIfEnv,
310+
Enabled: enabled,
311+
}
312+
}
313+
return out
314+
}
315+
237316
func (r Report) RenderJSON() string {
238317
data, err := json.MarshalIndent(r, "", " ")
239318
if err != nil {
@@ -257,6 +336,14 @@ func (r Report) RenderText() string {
257336
sb.WriteString(fmt.Sprintf(" reader.cache_dir: %s\n", r.Config.Reader.CacheDir))
258337
sb.WriteString(fmt.Sprintf(" reader.browser_fallback: %t\n", r.Config.Reader.BrowserFallback))
259338
sb.WriteString(fmt.Sprintf(" search.default_engine: %s\n", r.Config.Search.DefaultEngine))
339+
sb.WriteString(fmt.Sprintf(" search.default_provider: %s\n", r.Config.Search.DefaultProvider))
260340
sb.WriteString(fmt.Sprintf(" search.searxng_url: %s\n", r.Config.Search.SearXNGURL))
341+
if len(r.Config.Providers) > 0 {
342+
sb.WriteString("Providers:\n")
343+
for id, provider := range r.Config.Providers {
344+
sb.WriteString(fmt.Sprintf(" %s: type=%s enabled=%t auth_env=%s auth_configured=%t\n",
345+
id, provider.Type, provider.Enabled, provider.AuthEnv, provider.AuthConfigured))
346+
}
347+
}
261348
return sb.String()
262349
}

0 commit comments

Comments
 (0)