Migration lego-hunter to Tinyfish SDK

lego-hunter/.env.example

@@ -1,7 +1,3 @@
-# TinyFish API key for browser automation agents
-# Get yours at https://tinyfish.ai
+# TinyFish Web Agent API key (server-side only)
+# Get yours at: https://agent.tinyfish.ai/api-keys
 TINYFISH_API_KEY=
-
-# OpenAI API key for URL generation and deal analysis (uses gpt-4o-mini)
-# Get yours at https://platform.openai.com/api-keys
-OPENAI_API_KEY=

lego-hunter/.gitignore

@@ -1,42 +1,22 @@
-# See https://help.github.com/articles/ignoring-files/ for more about ignoring files.
+# Dependencies
+node_modules/
 
-# dependencies
-/node_modules
-/.pnp
-.pnp.*
-.yarn/*
-!.yarn/patches
-!.yarn/plugins
-!.yarn/releases
-!.yarn/versions
-
-# testing
-/coverage
-
-# next.js
-/.next/
-/out/
-
-# production
-/build
-
-# misc
-.DS_Store
-*.pem
+# Next.js
+.next/
+out/
+next-env.d.ts
 
-# debug
-npm-debug.log*
-yarn-debug.log*
-yarn-error.log*
-.pnpm-debug.log*
+# Environment files
+.env
+.env.local
+.env.*.local
 
-# env files
-.env*
-!.env.example
+# Build outputs
+*.tsbuildinfo
 
-# vercel
+# Vercel
 .vercel
 
-# typescript
-*.tsbuildinfo
-next-env.d.ts
+# OS
+.DS_Store
+Thumbs.db

lego-hunter/README.md

@@ -1,108 +1,152 @@
 # Lego Restock Hunter
+**Live Demo:** _add URL after deploy_
 
-Search 15+ retailers simultaneously to find sold-out Lego sets back in stock. Uses TinyFish browser agents for parallel scraping and OpenAI for deal analysis.
+**Find sold-out LEGO sets across multiple retailers simultaneously — TinyFish Search discovers real product pages, then parallel browser agents check stock and price in real time.**
 
-**Live Demo:** https://cookbook-lego-hunter.vercel.app/
+Enter a LEGO set name or number and the app runs two parallel TinyFish Search queries to find real retailer product pages for that specific set. One browser agent fires per retailer simultaneously — each checking stock availability, extracting price and shipping, and streaming results back as it finishes.
 
-![Lego Hunter Demo](./demo-screenshot.jpg)
+## Architecture
 
-## What This Project Is
+```
+┌─────────────────────────────────────────────────────────────┐
+│                      Browser (Client)                       │
+│                                                             │
+│  Search input + budget → RetailerStatusCard grid            │
+│  Best deal banner → Results table                           │
+│  (results + iframes stream in as agents finish)             │
+└──────────────────────────┬──────────────────────────────────┘
+                           │
+                  ┌────────┴────────┐
+                  ▼                 ▼
+   POST /api/discover-retailers   POST /api/search-lego
+                  │                 │
+                  ▼                 ▼
+┌─────────────────────┐  ┌──────────────────────────────────┐
+│ TinyFish Search API │  │        TinyFish SDK              │
+│                     │  │                                  │
+│ Two parallel queries│  │ client.agent.stream({ url, goal })│
+│                     │  │                                  │
+│ 1. Targeted at known│  │ EventType.STREAMING_URL          │
+│    LEGO retailers   │  │   → live iframe per agent        │
+│    (lego.com,       │  │ EventType.PROGRESS               │
+│    amazon, target,  │  │   → step updates                 │
+│    bricklink, etc.) │  │ EventType.COMPLETE               │
+│                     │  │   + RunStatus.COMPLETED          │
+│ 2. Broader search   │  │   → stock/price JSON → SSE       │
+│    for any retailer │  │                                  │
+│    carrying the set │  │ Promise.allSettled (all parallel) │
+│                     │  │                                  │
+│ Aggregators filtered│  │ Built-in deal analysis           │
+│ Deduped by domain   │  │ (no LLM needed)                  │
+└─────────────────────┘  └──────────────────────────────────┘
+
+No database. No cache. No OpenAI. Pure in-memory — results fetched live.
+```
 
-A cookbook example showing how to use the [TinyFish API](https://tinyfish.ai) to deploy parallel browser automation agents. Given a Lego set name, the app scrapes 15 retailer websites at the same time, extracts stock and pricing data, and recommends the best deal.
+### TinyFish SDK event flow
 
-## How It Works
+```
+client.agent.stream({ url, goal })
+  │
+  ├── EventType.STREAMING_URL → live iframe URL forwarded to client
+  ├── EventType.PROGRESS      → step description forwarded to client
+  └── EventType.COMPLETE
+        └── RunStatus.COMPLETED
+              // COMPLETED only means the browser ran without crashing
+              // — always validate result content, not just the status
+              → parse event.result → { inStock, price, shipping, productUrl }
+              → retailer_stock_found (if inStock) + retailer_complete → SSE
+```
 
-1. **User enters a Lego set name and budget** -- e.g. "75192 Millennium Falcon", $900
-2. **OpenAI generates retailer search URLs** -- AI creates direct search URLs for 15 retailers (Amazon, Walmart, LEGO.com, Target, BrickLink, etc.)
-3. **TinyFish agents scrape each retailer in parallel** -- 15 browser agents launch simultaneously, each navigating a retailer site and extracting stock status, price, currency, and shipping info as structured JSON
-4. **OpenAI analyzes the best deal** -- all results are passed to OpenAI, which recommends the best retailer based on price, shipping, and reliability
+## What Each Agent Extracts
 
-## TinyFish API Usage
+Each agent navigates the retailer's product or search page and returns:
 
-The core integration lives in `app/api/search-lego/route.ts`. Each retailer gets its own TinyFish agent via the SSE endpoint:
+- **In stock** — true/false
+- **Price** — numeric, in local currency
+- **Shipping** — free / cost / check website
+- **Product URL** — direct link to the listing
 
-```typescript
-const response = await fetch('https://agent.tinyfish.ai/v1/automation/run-sse', {
-  method: 'POST',
-  headers: {
-    'X-API-Key': TINYFISH_API_KEY,
-    'Content-Type': 'application/json'
-  },
-  body: JSON.stringify({
-    url: retailer.url,
-    goal: 'Search for "75192 Millennium Falcon" Lego set. Extract inStock, price, currency, shipping, and productUrl as JSON.',
-    browser_profile: 'lite'
-  })
-})
+## Search Flow
 
-// Stream SSE events for progress and results
-const reader = response.body.getReader()
-// ... parse SSE events with streaming_url, purpose, status, result_json
-```
+1. User enters LEGO set name/number and optional max budget
+2. `/api/discover-retailers` runs two parallel TinyFish Search queries to find real product pages
+3. All discovered retailer URLs fire browser agents simultaneously via `Promise.allSettled`
+4. Each agent navigates the page, extracts stock and price data
+5. `EventType.STREAMING_URL` → live browser iframe shown in the retailer card
+6. `EventType.COMPLETE` + `RunStatus.COMPLETED` → parse result → stream to client
+7. Results appear as each retailer finishes — no waiting for the slowest one
+8. Best deal calculated in-memory (lowest price among in-stock results)
 
-The SSE stream provides:
-- `streaming_url` -- live browser preview URL
-- `purpose` -- progress updates describing what the agent is doing
-- `status: "COMPLETED"` + `result_json` -- final extracted data
-- `status: "FAILED"` + `error` -- error information
+## Setup
 
-## Tech Stack
+### Prerequisites
 
-- **Next.js 16** (App Router) -- full-stack framework
-- **TinyFish API** -- parallel browser automation agents
-- **OpenAI GPT-4o Mini** (via Vercel AI SDK) -- URL generation and deal analysis
-- **Tailwind CSS 4** -- styling with custom Lego brick theme
-- **canvas-confetti** -- celebration effects when stock is found
-- **TypeScript** -- end-to-end type safety
+- Node.js 22.x
+- TinyFish API key
 
-## Setup
+### Environment Variables
 
-1. Install dependencies:
-```bash
-npm install
-```
-
-2. Copy `.env.example` to `.env.local` and fill in your keys:
 ```bash
 cp .env.example .env.local
 ```
 
-```
-TINYFISH_API_KEY=your-tinyfish-api-key
-OPENAI_API_KEY=your-openai-api-key
+Then fill in:
+
+```env
+# TinyFish Web Agent API key (server-side only)
+# Get yours at: https://agent.tinyfish.ai/api-keys
+TINYFISH_API_KEY=
 ```
 
-3. Start the dev server:
+### Install & Run
+
 ```bash
+npm install
 npm run dev
 ```
 
-4. Open [http://localhost:3000](http://localhost:3000)
+Open http://localhost:3000
 
-## Architecture
+## Project Structure
 
 ```
-User
-  |
-  v
-[Next.js Frontend] -- page.tsx (SSE consumer, Lego brick UI)
-  |
-  |-- POST /api/generate-urls
-  |     |
-  |     v
-  |   [GPT-4o Mini] -- generates 15 retailer search URLs
-  |
-  |-- POST /api/search-lego (SSE stream)
-        |
-        |-- [TinyFish Agent 1]  --> Amazon
-        |-- [TinyFish Agent 2]  --> Walmart
-        |-- [TinyFish Agent 3]  --> LEGO.com
-        |-- [TinyFish Agent 4]  --> Target
-        |-- ...                 --> (11 more retailers)
-        |
-        v
-      [GPT-4o Mini] -- analyzes all results, picks best deal
-        |
-        v
-      SSE: analysis_complete (best retailer + reasoning)
+lego-hunter/
+├── app/
+│   ├── layout.tsx
+│   ├── page.tsx                          # Main UI — search, agent grid, results
+│   ├── globals.css                       # LEGO brick design system
+│   └── api/
+│       ├── discover-retailers/route.ts   # POST — TinyFish Search → retailer URLs
+│       └── search-lego/route.ts          # POST — TinyFish Agent stream → SSE
+├── components/
+│   └── lego-confetti.tsx                 # Confetti on stock found
+├── lib/
+│   └── retailers.ts                      # Default retailer logos
+├── types/
+│   └── index.ts                          # TypeScript definitions
+├── .env.example
+├── .gitignore
+└── package.json
 ```
+
+## Constraint Checklist
+
+| Constraint | Status |
+|---|---|
+| External database used? | NO (pure in-memory) |
+| OpenAI / LLM for URL generation? | NO (TinyFish Search finds real pages) |
+| LLM for deal analysis? | NO (built-in price sort logic) |
+| Scraping parallel? | YES (`Promise.allSettled` across all retailers) |
+| Live browser preview? | YES (`EventType.STREAMING_URL` → iframe per agent) |
+| Result validation? | YES (COMPLETED status ≠ goal achieved — content validated) |
+| Aggregator sites filtered? | YES (reddit, quora, youtube, etc. excluded) |
+
+## Tech Stack
+
+- **Framework:** Next.js 16 (App Router), TypeScript, Tailwind CSS 4
+- **Browser Agents:** TinyFish SDK (`client.agent.stream`)
+- **URL Discovery:** TinyFish Search API (`client.search.query`)
+- **Confetti:** canvas-confetti
+- **Icons:** Lucide React
+- **Deployment:** Vercel

lego-hunter/app/api/discover-retailers/route.ts

@@ -0,0 +1,67 @@
+import { NextRequest, NextResponse } from "next/server";
+import { TinyFish } from "@tiny-fish/sdk";
+
+export const runtime = "nodejs";
+
+const AGGREGATORS = [
+  "reddit", "quora", "youtube", "twitter", "facebook",
+  "instagram", "pinterest", "trustpilot", "yelp",
+];
+
+export async function POST(req: NextRequest) {
+  const apiKey = process.env.TINYFISH_API_KEY;
+  if (!apiKey) {
+    return NextResponse.json({ error: "Missing TINYFISH_API_KEY" }, { status: 500 });
+  }
+
+  const { legoSetName } = await req.json();
+  if (!legoSetName) {
+    return NextResponse.json({ error: "legoSetName is required" }, { status: 400 });
+  }
+
+  try {
+    const client = new TinyFish({ apiKey });
+
+    // Two parallel searches — one targeting known retailers, one broader
+    const [res1, res2] = await Promise.all([
+      client.search.query({
+        query: `LEGO "${legoSetName}" buy in stock lego.com OR amazon.com OR target.com OR walmart.com OR bricklink.com OR smythstoys.com OR argos.co.uk OR johnlewis.com OR zavvi.com OR entertainmentearth.com`,
+      }),
+      client.search.query({
+        query: `"${legoSetName}" LEGO set price stock retailer 2025`,
+      }),
+    ]);
+
+    const allResults = [
+      ...(res1.results || []),
+      ...(res2.results || []),
+    ];
+
+    const seenDomains = new Set<string>();
+    const retailers: { name: string; url: string }[] = [];
+
+    for (const r of allResults) {
+      try {
+        const urlObj = new URL(r.url);
+        const domain = urlObj.hostname.replace("www.", "");
+        const isAggregator = AGGREGATORS.some((agg) => domain.includes(agg));
+        if (isAggregator || seenDomains.has(domain)) continue;
+        seenDomains.add(domain);
+
+        // Derive clean retailer name from title
+        const name = r.title?.split(/[-|–]/)[0].trim() || domain.split(".")[0];
+        retailers.push({ name, url: r.url });
+        if (retailers.length >= 12) break;
+      } catch {
+        // skip malformed URLs
+      }
+    }
+
+    return NextResponse.json({ retailers });
+  } catch (error) {
+    return NextResponse.json(
+      { error: error instanceof Error ? error.message : "Discovery failed" },
+      { status: 500 }
+    );
+  }
+}