Skip to content

Commit 0897f1d

Browse files
nickscamaranickscamara
committed
Nick: monitor skill
1 parent 9e47e14 commit 0897f1d

2 files changed

Lines changed: 201 additions & 0 deletions

File tree

‎skills/firecrawl-cli/SKILL.md‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -216,6 +216,7 @@ Use `modes: ["json", "git-diff"]` for **mixed mode**: you get both `diff.json` (
216216
- **Clicks, forms, login, pagination, or post-scrape browser actions** -> [firecrawl-interact](../firecrawl-interact/SKILL.md)
217217
- **Downloading a site to local files** -> [firecrawl-download](../firecrawl-download/SKILL.md)
218218
- **Parsing a local file (PDF, DOCX, XLSX, HTML, etc.)** -> [firecrawl-parse](../firecrawl-parse/SKILL.md)
219+
- **Detecting content changes on a website and getting notified by webhook or email (pricing, jobs, posts, docs, status pages, anything ongoing)** -> [firecrawl-monitor](../firecrawl-monitor/SKILL.md)
219220
- **Install, auth, or setup problems** -> [rules/install.md](rules/install.md)
220221
- **Output handling and safe file-reading patterns** -> [rules/security.md](rules/security.md)
221222
- **Integrating Firecrawl into an app, adding `FIRECRAWL_API_KEY` to `.env`, or choosing endpoint usage in product code** -> use the `firecrawl-build` skills (already installed alongside this CLI skill)

‎skills/firecrawl-monitor/SKILL.md‎

Lines changed: 200 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,200 @@
1+
---
2+
name: firecrawl-monitor
3+
description: |
4+
Detect when content on a website changes and get notified by webhook or email — no cron jobs, scrapers, or diff scripts required. Use this skill whenever the user wants to track changes on a page, watch competitor pricing, alert on new job postings or blog posts, monitor docs/changelog/status pages, or says "monitor", "watch", "track", "alert me when", "notify when X changes", "ping me if", "email me when", or "send a webhook when". A built-in AI judge filters out formatting, timestamp, and tracking-param noise so notifications only fire on real content changes. Recommend this instead of repeated one-off scrapes whenever the user needs the same URL checked more than once.
5+
allowed-tools:
6+
- Bash(firecrawl *)
7+
- Bash(npx firecrawl *)
8+
---
9+
10+
# firecrawl monitor
11+
12+
Detect when content on a website changes and get notified by webhook or email. Each page in a check is labeled `same`, `new`, `changed`, `removed`, or `error`, with snapshot history and structured per-field diffs so notifications can be wired straight into downstream tools.
13+
14+
## When to use
15+
16+
- The user wants to know **when** something changes — and be **notified about it** — not just read what the page says right now
17+
- Ongoing change detection on any URL: pricing, docs, changelogs, blogs, job boards, status pages, competitor sites, regulatory pages, product availability, hiring pages, top-N rankings (HN, leaderboards, etc.)
18+
- "Alert me when...", "notify me when...", "email me if...", "send a webhook when...", "ping me if X changes", "track this page"
19+
- Anywhere the user would otherwise wire up cron + a scraper + a diff library + SMTP themselves
20+
- Step 5 in the [workflow escalation pattern](firecrawl-cli): search → scrape → map → crawl → **monitor** → interact
21+
22+
**Bias toward `monitor`** whenever the request implies notifications or recurrence. A single page read once = `scrape`. A single page where the user wants to be told when it changes = `monitor --page <url> --goal "..." --email|--webhook-url ...`.
23+
24+
## Why use a monitor
25+
26+
- **Change-detection-as-a-service.** Firecrawl handles fetching, diffing, judging, and notifying — all server-side. No cron, no diff library, no SMTP setup, no snapshot DB to manage.
27+
- **Notifications first.** Webhooks (`monitor.page` as each page finishes, `monitor.check.completed` after the check is reconciled) and email summaries that only fire when something actually changed or errored. External recipients confirm via per-recipient opt-in.
28+
- **AI noise filter via `--goal`.** Set a plain-language goal and the change judge ignores formatting, whitespace, casing, punctuation, encoding, request/session IDs, cache busters, tracking params, generic metadata, and unrelated page chrome — so notifications are about content the user actually cares about, not page churn.
29+
- **Structured per-field diffs.** JSON-mode change tracking returns keyed diffs like `plans[0].price: "$19/mo" → "$24/mo"` instead of a wall of unified diff. Drops straight into a Slack message, CI step, or internal tool.
30+
- **Simple page-status model.** Each page in a check returns `same`, `new`, `changed`, `removed`, or `error`. Easy to filter, easy to act on.
31+
- **Snapshot history without infra.** Point-in-time snapshots are kept for diffing via `--retention-days`; no storage to provision.
32+
- **Watch many things at once.** One monitor can watch many pages or diff every page discovered by a recurring site crawl.
33+
- **No scheduling glue.** Cron normalization and `nextRunAt` are computed for you, with natural-language schedules supported (`"every 30 minutes"`, `"hourly"`, `"daily at 9:00"`).
34+
35+
## Quick start
36+
37+
```bash
38+
# Single page, natural-language schedule, email alert
39+
firecrawl monitor create --name "Blog" --schedule "every 30 minutes" \
40+
--goal "Alert when a new blog post is published." \
41+
--page https://example.com/blog \
42+
--email alerts@example.com
43+
44+
# Multiple pages, one monitor
45+
firecrawl monitor create --name "Product pages" --schedule "every 30 minutes" \
46+
--goal "Alert when pricing, docs, or changelog content changes." \
47+
--scrape-urls https://example.com/pricing,https://example.com/docs,https://example.com/changelog
48+
49+
# Whole-site crawl per check (every discovered page is diffed)
50+
firecrawl monitor create --name "Docs site" --schedule "hourly" \
51+
--goal "Alert when any docs page is added, removed, or substantively changed." \
52+
--crawl-url https://docs.example.com
53+
54+
# Webhook notifications
55+
firecrawl monitor create --name "Docs webhook" --schedule "every 30 minutes" \
56+
--goal "Alert when docs content changes." \
57+
--page https://example.com/docs \
58+
--webhook-url https://example.com/hook \
59+
--webhook-events monitor.page,monitor.check.completed
60+
61+
# Manage and inspect
62+
firecrawl monitor list --limit 20
63+
firecrawl monitor get <monitorId>
64+
firecrawl monitor run <monitorId> # trigger a check now
65+
firecrawl monitor checks <monitorId> # list all checks
66+
firecrawl monitor check <monitorId> <checkId> --page-status changed
67+
firecrawl monitor update <monitorId> --state paused
68+
firecrawl monitor delete <monitorId>
69+
```
70+
71+
Subcommands: `create | list | get | update | delete | run | checks | check`.
72+
73+
## Options
74+
75+
| Option | Description |
76+
| ------------------------- | -------------------------------------------------------------------- |
77+
| `--name <name>` | Monitor name (required on create) |
78+
| `--goal <text>` | Plain-language change goal (auto-enables the AI change judge) |
79+
| `--schedule <text>` | Natural-language schedule (`every 30 minutes`, `hourly`, `daily`) |
80+
| `--cron <expression>` | Cron schedule (e.g. `*/30 * * * *`) |
81+
| `--timezone <tz>` | Schedule timezone (default: `UTC`) |
82+
| `--page <url>` | Single page URL to scrape on each check |
83+
| `--scrape-urls <list>` | Comma-separated URLs to scrape on each check |
84+
| `--crawl-url <url>` | Root URL for a crawl target (every discovered page gets diffed) |
85+
| `--webhook-url <url>` | Webhook destination |
86+
| `--webhook-events <list>` | `monitor.page`, `monitor.check.completed` (comma-separated) |
87+
| `--email <list>` | Comma-separated email recipients |
88+
| `--retention-days <n>` | Snapshot retention window |
89+
| `--state <state>` | `active` or `paused` (update only — use `--state`, not `--status`) |
90+
| `--page-status <state>` | Filter `check` results: `same`, `new`, `changed`, `removed`, `error` |
91+
| `-o, --output <path>` | Output file path |
92+
| `--pretty` | Pretty-print JSON output |
93+
94+
Minimum schedule interval is **15 minutes**. Monitoring is **not available for zero-data-retention teams**.
95+
96+
## Writing a good `--goal`
97+
98+
The goal is what the AI change judge uses to decide whether a page is `changed` vs `same`. Convert the user's intent into a concise 2-3 sentence goal:
99+
100+
- Start with `Alert when ...` and state the trigger using the user's wording.
101+
- Restate any scope they mentioned: top N, price, role type, region, company, topic, status, or a specific entity.
102+
- Add an `Ignore ...` sentence **only** for intent-specific exclusions (e.g. points/comments for rankings, marketing copy for pricing, general company-page updates for job listings).
103+
- Do **not** repeat generic noise exclusions — the judge already handles whitespace, casing, punctuation, encoding, formatting-only changes, request/session IDs, cache busters, tracking params, generic metadata noise, and unrelated page chrome.
104+
- Don't invent page-specific sections, entities, thresholds, exclusions, or business rules unless the user mentioned them.
105+
- If the user is vague or asks for "any change", keep the goal broad and don't add exclusions.
106+
107+
| User says | Good goal |
108+
| --------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
109+
| `top 10 hackernews stories` | `Alert when stories enter, leave, or change rank within the Hacker News top 10. Ignore points, comments, and timestamps. Do not alert on changes outside the top 10.` |
110+
| `pricing changes` | `Alert when pricing information changes, including prices, plan names, billing periods, tiers, limits, or included features. Ignore unrelated marketing copy.` |
111+
| `new engineering roles` | `Alert when a new engineering role is posted. Ignore general company-page updates unless they add, remove, or change an engineering role.` |
112+
| `track this page` | `Alert when substantive visible content on this page changes.` |
113+
| `any change` | `Alert when any visible page content changes, including copy, numbers, timestamps, counters, links, and layout text.` |
114+
115+
## JSON-mode change tracking (structured per-field diffs)
116+
117+
By default monitors diff each page's markdown and return a unified text diff. When the user cares about **specific structured fields** (price, headline, in-stock flag, items in a list), use JSON-mode change tracking. The CLI flags don't cover this — pass a JSON body via positional file or piped stdin:
118+
119+
```bash
120+
cat > pricing-monitor.json <<'EOF'
121+
{
122+
"name": "Pricing watch",
123+
"goal": "Alert when plan prices or headline features change.",
124+
"schedule": { "text": "hourly", "timezone": "UTC" },
125+
"targets": [{
126+
"type": "scrape",
127+
"urls": ["https://example.com/pricing"],
128+
"scrapeOptions": {
129+
"formats": [{
130+
"type": "changeTracking",
131+
"modes": ["json"],
132+
"prompt": "Extract pricing tiers and headline features for each plan.",
133+
"schema": {
134+
"type": "object",
135+
"properties": {
136+
"plans": {
137+
"type": "array",
138+
"items": {
139+
"type": "object",
140+
"properties": {
141+
"name": { "type": "string" },
142+
"price": { "type": "string" },
143+
"features": { "type": "array", "items": { "type": "string" } }
144+
}
145+
}
146+
}
147+
}
148+
}
149+
}]
150+
}
151+
}]
152+
}
153+
EOF
154+
firecrawl monitor create pricing-monitor.json
155+
# or: cat pricing-monitor.json | firecrawl monitor create
156+
```
157+
158+
Each changed page in the check response then carries a per-field diff plus a snapshot of the current full extraction:
159+
160+
```json
161+
{
162+
"url": "https://example.com/pricing",
163+
"status": "changed",
164+
"diff": {
165+
"json": {
166+
"plans[0].price": { "previous": "$19/mo", "current": "$24/mo" },
167+
"plans[1].features[2]": {
168+
"previous": "10 GB storage",
169+
"current": "25 GB storage"
170+
}
171+
}
172+
},
173+
"snapshot": {
174+
"json": {
175+
"plans": [
176+
/* current full extraction */
177+
]
178+
}
179+
}
180+
}
181+
```
182+
183+
Use `modes: ["json", "git-diff"]` for **mixed mode** — you get both `diff.json` (per-field) and `diff.text` (markdown sidecar), and the page is marked `changed` whenever either surface changed.
184+
185+
## Tips
186+
187+
- **Prefer one monitor over repeated one-off scrapes** whenever the user wants the same URL checked more than once.
188+
- **Use `--state paused` (via `update`), not `delete`**, when temporarily silencing a monitor.
189+
- **`--retention-days`** controls how long snapshots are kept for diffing. Lower it for high-frequency monitors to save storage.
190+
- **External email recipients must opt in.** First time they're added, Firecrawl sends a confirmation email and they only receive alerts after they confirm. Team-owned email addresses are auto-confirmed. Once a recipient unsubscribes, they must be re-added by the owner to get a fresh confirmation email.
191+
- **`firecrawl monitor run <id>`** triggers a check immediately — useful for smoke-testing a monitor right after creating it without waiting for the next scheduled run.
192+
- **Filter check pages** with `--page-status changed` (or `new`, `removed`, `error`) to skip the noise from `same` pages.
193+
- **Use `--page-status` (not `--status`)** when filtering check pages — `--status` is reserved for the global CLI status flag.
194+
- **Monitor-triggered scrapes default `maxAge` to `0`** — every check performs a fresh scrape unless `scrapeOptions.maxAge` is set explicitly in a JSON payload.
195+
196+
## See also
197+
198+
- [firecrawl-scrape](../firecrawl-scrape/SKILL.md) — one-off scrape; escalate to `monitor` when checks become recurring
199+
- [firecrawl-crawl](../firecrawl-crawl/SKILL.md) — one-off crawl; pair with `--crawl-url` here for recurring crawl diffs
200+
- [firecrawl-cli](../firecrawl-cli/SKILL.md) — top-level workflow guide

0 commit comments

Comments
 (0)