Skip to content

Latest commit

 

History

History
484 lines (354 loc) · 15.7 KB

File metadata and controls

484 lines (354 loc) · 15.7 KB

HeadlessX Logo

The World's Most Advanced Anti-Detection Scraping Full Stack Web App


Version Release Detection Status


Node.js Next.js Camoufox TypeScript Playwright


License PRs Welcome Maintenance Made with Love


🎯 0% Detection Rate on all major anti-bot systems

Powered by Camoufox — Firefox with C++ level fingerprint spoofing

📖 Documentation🚀 Quick Start🐛 Report Bug✨ Request Feature


🆕 What's New in V2.0

🦊 Camoufox Engine 🛡️ 0% Detection 3x Faster
Firefox with binary-level stealth patches Passes CreepJS, Sannysoft, & all major tests Optimized context pooling

V2.0 Changelog

+ 🦊 NEW: Camoufox browser engine (replaces Chromium + stealth plugins)
+ 🛡️ NEW: 0% headless detection rate (was 67%+)
+ 🔒 NEW: Built-in WebRTC leak protection
+ 🎭 NEW: C++ level Canvas/WebGL/AudioContext fingerprint spoofing
+ ⚡ IMPROVED: Faster browser launches with persistent contexts
- 🧹 REMOVED: playwright-extra (no longer needed)
- 🧹 REMOVED: puppeteer-extra-plugin-stealth (no longer needed)

📊 Detection Benchmark Comparison

Tool Headless Detection Stealth Score Engine
🦊 HeadlessX V2.0 (Camoufox) 0% 0% Firefox (C++ patched)
HeadlessX V1 (Playwright Extra) ❌ 67% ⚠️ 50% Chromium (JS patches)
Puppeteer Stealth ❌ 33% ❌ 80% Chromium (JS patches)
Vanilla Playwright ❌ 100% ❌ 100% Chromium
Vanilla Selenium ❌ 100% ❌ 100% Chrome

Lower scores = better (0% = undetectable)


🤖 Scrapers

Google SERP 🌐 Website Scraping
Extract search results with zero detection Scrape any website with full JS rendering
Live Live

Coming Soon

Amazon LinkedIn Twitter Instagram
Product & Reviews Job Listings & Profiles Tweets & Trends Posts & Stories

🖥️ UI Screenshots

Google SERP Scraper

Google SERP UI

Website Scraper

Website Scraper UI


📸 Proof of Undetectability

BrowserScan (100% Passed)

BrowserScan

Pixelscan (Human Detection) Profile Tested with Proxy
Pixelscan Proxy Test

📋 Table of Contents


⚡ Quick Start

Prerequisites:

  • Node.js 22+
  • pnpm 9+ (npm install -g pnpm)
  • Mise (Task Runner)
  • Docker (Optional, for containerized deployment)
  • PostgreSQL Database — either:
    • Supabase (recommended, free tier available)
    • Self-hosted PostgreSQL 14+

1️⃣ Clone & Configure Environment

git clone https://github.com/your-repo/HeadlessX.git
cd HeadlessX

# Copy environment template
cp .env.example .env

Edit .env with your database connection:

# REQUIRED: Your PostgreSQL connection string
DATABASE_URL="postgresql://user:password@host:5432/database"

# Optional: Server configuration
PORT=8000
NODE_ENV=development

2️⃣ Install Dependencies & Setup

Using Mise (Recommended):

# Install dependencies, setup database, and download Camoufox
mise run setup

Or manually:

pnpm install
pnpm --filter api db:push
pnpm --filter api exec camoufox-js fetch

3️⃣ Start Development Server

Using Mise (Recommended):

# Start both frontend and backend concurrently
mise run dev

Or manually:

pnpm dev

🐳 Docker Deployment

HeadlessX can be easily deployed using Docker Compose. See the Docker Setup Guide for detailed instructions.

# Start the application in detached mode
docker compose -f infra/docker/docker-compose.yml up -d

6️⃣ Access the Application

Service URL Notes
🖥️ Dashboard http://localhost:3000 Frontend UI
🔗 Backend API http://localhost:8000 REST API

Custom Ports

You can customize ports via environment variables:

# Backend (from root)
PORT=8000 pnpm --filter api dev

# Frontend
PORT=3000 pnpm --filter web dev

🔥 Key Features

🦊 Camoufox Stealth Engine (V2.0)

Feature Description
🔒 Binary-Level Patches Firefox modified at C++ level
🎭 Fingerprint Spoofing Canvas, WebGL, AudioContext
🌐 WebRTC Protection No IP leaks in headless mode
🖱️ Human Behavior Natural mouse movements & scrolling
📱 Device Emulation Realistic viewport & user agents
🚫 Zero Detection Passes all anti-bot tests

🖥️ Modern Dashboard

Feature Description
⚛️ Next.js 16 React 19 with Turbopack
🎨 HeroUI Beautiful dark/light mode UI
⚙️ Live Config Change settings without restart
📊 Request Logs Full history with screenshots
🧪 Playground Test scraping in real-time
🔑 API Keys Secure key management

🌐 API Endpoints

Website Scraping APIs

Endpoint Method Description
/api/website/html POST Get raw HTML (fast)
/api/website/html-js POST Get HTML with JS rendering
/api/website/content POST Get Markdown content
/api/website/screenshot POST Capture full-page Screenshot
/api/website/stream POST Real-time SSE Stream

Google SERP APIs

Endpoint Method Description
/api/google-serp/search POST Extract Google search results
/api/google-serp/stream GET Real-time SSE Stream

Example Request

curl -X POST http://localhost:8000/api/website/html \
  -H "Content-Type: application/json" \
  -H "X-API-Key: your-api-key" \
  -d '{"url": "https://example.com", "stealth": true}'

Example Response

{
  "success": true,
  "data": {
    "url": "https://example.com",
    "title": "Example Domain",
    "html": "<!DOCTYPE html>...",
    "statusCode": 200
  }
}

Configuration

Environment Variables

⚠️ Note: Most configuration (Browsers, Stealth, Proxy, Timeouts) is managed directly via the Dashboard Settings.

Only the following core variables are required in .env:

Variable Default Description
PORT 8000 Backend API port
DATABASE_URL - PostgreSQL connection (Supabase or self-hosted)
NEXT_PUBLIC_API_URL http://localhost:8000 Frontend API URL

Dashboard Settings

Configure these live at /settings:

General Configuration

  • Headless Mode: Run browser without UI (faster execution)
  • Browser Timeout: Max execution time per job (default: 60000ms)
  • Max Concurrent Jobs: Concurrent scraping jobs (recommended: 3-8)

Browser Engine (Camoufox)

  • Block WebRTC: Prevent IP leaks via WebRTC
  • Camoufox GeoIP: Spoof location based on IP
  • Enable Cache: Cache resources for speed

Proxies

  • Manage proxy configurations and rotation settings

🛠️ Manual Setup

Backend

cd apps/api

# 1. Configure environment
cp ../.env.example .env
# Edit .env with your DATABASE_URL

# 2. Install dependencies
pnpm install

# 3. Download Camoufox browser
npx camoufox-js fetch

# 4. Setup database
npx prisma db push

# 5. Start dev server
pnpm dev

Frontend

cd apps/web
pnpm install
pnpm dev

🤝 Contributing

Contributions are welcome! Please read our Contributing Guide for details.

  1. Fork the repository
  2. Create your feature branch (git checkout -b feature/amazing-feature)
  3. Commit your changes (git commit -m 'Add amazing feature')
  4. Push to the branch (git push origin feature/amazing-feature)
  5. Open a Pull Request

📝 License

This project is licensed under the MIT License - see the LICENSE file for details.


🌟 Star History

If you find HeadlessX useful, please consider giving it a star! ⭐


Built with ❤️ using cutting-edge technologies


Node.js Next.js Firefox Playwright TypeScript React TailwindCSS Prisma


🦊 HeadlessX V2.0 — Undetectable by Design

© 2026 HeadlessX. All rights reserved.