Playwright Feature Validator

It logs into your app, clicks through every feature, and tells you what broke before your customer does.

Features • Quick Start • Configuration • Documentation

---

Why this exists

A customer demo fails for the dumbest reasons: an API key expired overnight, a feature flag flipped, the seed data got wiped. Nobody notices until you're sharing your screen.

So I built a robot that does the boring check for you. It logs in (OAuth, MFA, the works), walks through every feature on a schedule, screenshots the evidence, scores each screen, and pings Slack the moment something stops working. By the time you open the demo, you already know it's green.

This is the framework behind a validation engine I run across 29 features of a real product.

Features

• 🔐 Auto Login - Handles OAuth, MFA/TOTP, session management
• 🧭 Smart Navigation - Declarative routing with fallbacks
• 📸 Evidence Capture - Screenshots with annotations
• ✅ Validation Engine - Check elements, data, API responses
• 🛡️ Content Scanner - Bilingual profanity/gaffe/PII shield for demo safety
• 🔔 Alerting - Slack, email, webhooks on failure
• 📊 Reporting - Excel/JSON reports with history
• ⏰ Scheduling - Run daily, hourly, or on-demand

Quick Start

Prerequisites

• Python 3.9+
• Chrome browser
• Playwright

Installation

# Clone the repository
git clone https://github.com/thiagoger/playwright-feature-validator.git
cd playwright-feature-validator

# Create virtual environment
python -m venv venv
source venv/bin/activate  # On Windows: venv\Scripts\activate

# Install dependencies
pip install -r requirements.txt

# Install Playwright browsers
playwright install chromium

First Run

# Copy environment template
cp .env.example .env

# Edit .env with your credentials
# Then run:
feature-validator run --product example --project demo

Architecture

flowchart LR
    Config[Config<br/>JSON] --> Auth[Auth<br/>OAuth + MFA]
    Auth --> Nav[Navigate<br/>routing]
    Nav --> Check[Validate<br/>checks]
    Check --> Rep[Reporter<br/>screenshots + Excel]
    Check --> Scan[Content Scanner<br/>profanity / PII / gaffes]
    Rep --> Alert[Alerter<br/>Slack / email]
    Scan --> Alert

Loading

Configuration

Product Configuration

Define your product in config/products/:

{
  "name": "ExampleApp",
  "base_url": "https://app.example.com",
  "auth": {
    "type": "oauth_totp",
    "login_url": "/sign-in"
  },
  "projects": {
    "demo": {
      "companies": ["Acme", "Globex", "Initech"]
    }
  }
}

Health Checks

Define checks in config/checks/:

{
  "product": "ExampleApp",
  "checks": [
    {
      "id": "login-works",
      "name": "Login Works",
      "type": "auth",
      "priority": "critical",
      "expect": {
        "url_contains": "/homepage"
      }
    },
    {
      "id": "customers-exist",
      "name": "Customers Have Data",
      "type": "navigation",
      "priority": "high",
      "route": "/app/customers",
      "expect": {
        "selector": "[data-testid='customer-row']",
        "min_count": 1
      }
    }
  ]
}

CLI Reference

# Run all checks for a product
feature-validator run --product example

# Run specific project
feature-validator run --product example --project demo

# Run single check
feature-validator run --product example --check login-works

# Dry run (no screenshots, no alerts)
feature-validator run --product example --dry-run

# Generate report only
feature-validator report --product example --format excel

# List available checks
feature-validator list --product example

Check Types

Type	Description	Example
auth	Validate login works	Login → expect homepage
navigation	Navigate and validate	Go to page → check element exists
content_scan	Scan page for profanity/PII/gaffes	Navigate → scan text → flag violations
api	Call API endpoint	GET /health → expect 200
data	Validate data exists	Query → expect count >= N
ui	Check UI element	Find element → validate text

Alerting

Configure alerts in .env:

# Slack
SLACK_WEBHOOK_URL=https://hooks.slack.com/services/xxx
SLACK_CHANNEL=#demo-health

# Email
SMTP_HOST=smtp.gmail.com
SMTP_PORT=587
ALERT_EMAIL=team@company.com

Documentation

Document	Description
Getting Started	Installation and first run
Adding Checks	How to write a new check

Project Structure

playwright-feature-validator/
├── src/feature_checker/
│   ├── cli.py              # Command-line interface
│   ├── core/
│   │   ├── checker.py      # Main check orchestrator
│   │   ├── browser.py      # Browser management
│   │   ├── content_scanner.py  # Profanity/PII/gaffe detection
│   │   └── reporter.py     # Report generation
│   ├── auth/
│   │   ├── login.py        # Login handlers
│   │   └── totp.py         # TOTP generation
│   ├── navigation/
│   │   └── navigator.py    # Page navigation
│   └── utils/
│       ├── screenshot.py   # Screenshot capture
│       └── config.py       # Configuration loader
├── config/
│   ├── checks/             # Check definitions
│   └── products/           # Product configurations
└── docs/                   # Documentation

Tests

pip install -e . pytest
pytest -q

Covers the pure-logic core: TOTP generation (checked against the RFC 6238 reference vector) and the content scanner's profanity, placeholder, and PII detection across sensitivity levels. CI runs it on Python 3.10 to 3.12 on every push.

Contributing

1. Fork the repository
2. Create a feature branch (git checkout -b feature/amazing)
3. Commit changes (git commit -m 'Add amazing feature')
4. Push to branch (git push origin feature/amazing)
5. Open a Pull Request

License

MIT License - see LICENSE for details.

---

Built and maintained by Thiago Rodrigues