Skip to content

README.en.md

Latest commit

 

History

History
130 lines (95 loc) · 4.87 KB

README.en.md

File metadata and controls

130 lines (95 loc) · 4.87 KB
Raw

🇮🇹 Italiano | 🇬🇧 English

agentic-web-clipper

Agentic web clipper: turn any URL into clean, Obsidian-ready Markdown by combining a deterministic extractor (headless Chrome + Defuddle) with an LLM agent that recovers the content the extractor would wrongly discard.

An agnostic (no per-site selectors) and portable (drop it into any project/vault, Windows·macOS·Linux, no code changes) Node.js toolkit. It reproduces — and improves on — the official Obsidian Web Clipper, with a two-tier hybrid pipeline.

What it is

  • Tier 1 — deterministic (clipper/clip.mjs): renders the page with Chrome, dismisses cookie banners, expands accordions/tabs, extracts the main content, adds the commerce layer from JSON-LD, and writes a .md with frontmatter — plus a sidecar of the blocks it discarded.
  • Tier 2 — hybrid (/clip skill): a judge agent decides which discarded blocks are actually on-topic and stitches them back into the .md; a quality check (lint + inspector agent) verifies completeness. Judgments are cached, so re-clips cost almost nothing.

Repository layout

clipper/            The tool. Copy THIS folder into your project to use it.
  clip.mjs            deterministic extractor (CLI)
  merge-candidates.mjs  merges the judge's recoveries (CLI)
  lint-clip.mjs / score-clip.mjs  QA and scoring (CLI)
  ocr.mjs             OCR scaffold (not wired up yet)
  README.md           ← FULL GUIDE (installation, usage, configuration, portability)
  ROADMAP.md          living changelog
.claude/            Claude Code integration (hybrid tier)
  skills/clip/SKILL.md      /clip pipeline orchestration
  agents/clip-judge.md      judge agent (keep/discard)
  agents/clip-inspector.md  completeness inspector agent

Quick start

cd clipper
npm install
node clip.mjs "https://example.com/some-page"
# → writes ../raw/<Title>.md

The full documentation (installation, usage, configuration, security, portability, troubleshooting) lives in clipper/README.en.md.

Installing with Claude Code

The hybrid tier (judge + QA) runs inside Claude Code: the /clip skill orchestrates the pipeline, and the agents handle judging and inspection.

Prerequisites: Node.js ≥ 22.12, Google Chrome installed, Claude Code.

  1. Clone the repo and install the dependencies:

    git clone https://github.com/3stelth/agentic-web-clipper.git
cd agentic-web-clipper/clipper
npm install

  2. Open the repo folder with Claude Code: the /clip skill and the agents (clip-judge, clip-inspector) in .claude/ are picked up automatically — no manual registration.

  3. Alternatively, to use it inside an existing project: copy the clipper/ folder and the contents of .claude/ (skills/clip/ + agents/) into your project, then cd clipper && npm install.

Usage. In Claude Code, type:

/clip <url>            # full pipeline: extraction + judge + merge + QA

Or, deterministic tier only from the terminal (zero LLM, zero cost):

node clipper/clip.mjs <url>

Obsidian integration

The clipper can replace the official Obsidian Web Clipper. An Obsidian vault is just a regular folder: point the clipper at the vault and the note shows up on its own — Obsidian automatically indexes externally created files, no "import" step needed.

Configuration. Create clipper/clipper.config.json (by copying clipper.config.example.json) with outDir pointing to a folder inside the vault:

{
  "outDir": "C:/Users/you/Documents/MyVault/Clippings"
}

Alternatively, use the --out <dir> flag or the CLIP_OUT_DIR environment variable (precedence: --out > CLIP_OUT_DIR > config > default ../raw).

Flow. /clip <url> in Claude Code → the .md with frontmatter (title, source, author, published, created, tags) appears in the vault, exactly like with the official Web Clipper — but with the hybrid pipeline: content recovered by the judge, automatic completeness QA.

Differences from the official Web Clipper: runs headless without an open browser, expands collapsed accordions/tabs, captures commerce data from JSON-LD (## Dati commerciali), and recovers, via agent, the blocks the extractor wrongly discarded.

Security

The tool renders untrusted pages and ships with defenses on by default: Chrome sandbox enabled, http/https URLs only, robust frontmatter escaping, defensive caps against DoS. Details in the Security section of clipper/README.en.md.

To report a vulnerability, see SECURITY.md.

License

MIT.

Obsidian, Defuddle, Puppeteer and the products mentioned belong to their respective owners. Independent project, not affiliated.