Security Misconfiguration Scanner

Python/FastAPI security scanner for headers, CORS, exposed files, SSL issues, PDF reports, scraping, Docker, PostgreSQL, and CI.

This is a learning and portfolio project, not a replacement for a professional penetration test. Only scan systems you own or have explicit permission to test.

Table of Contents

• Why I Built This
• Repository Metadata
• Features
• Tech Stack
• Architecture
• Requirements
• Quick Start
• Docker Compose
• CLI Usage
• API Endpoints
• API Examples
• Example Output
• Optional Portfolio Assets
• Local Python Setup
• Tests And Quality
• Project Structure
• Known Limitations
• Roadmap
• Contributing
• Security Notes
• License

Why I Built This

I built this project to go deeper than typical tutorial-level CRUD apps and practice building something closer to a real security tool: a FastAPI backend with JWT auth, background job processing, PostgreSQL persistence, PDF reporting, and a full CI/CD pipeline. It's meant to demonstrate:

• Designing and documenting a REST API from scratch.
• Working with async background jobs and persistent state.
• Writing checks that reason about real HTTP security issues (headers, CORS, SSL, exposed files).
• Containerizing a multi-service app with Docker Compose, including an intentionally vulnerable target for safe local testing.
• Setting up automated testing, linting, type checking, and CI.

Repository Metadata

Description:

Python/FastAPI security scanner for headers, CORS, exposed files, SSL issues, PDF reports, scraping, Docker, PostgreSQL, and CI.

GitHub topics:

python
fastapi
postgresql
sqlalchemy
alembic
security-scanner
security-automation
web-security
docker
pytest
github-actions
web-scraping
playwright
jwt-authentication
pdf-report

Features

• 10+ security checks with severity scoring and a total risk score.
• CLI scanner with table and JSON output.
• JWT-authenticated REST API under /api/v1.
• Background scan jobs with persistent scan history.
• PostgreSQL storage for users, scans, findings, and scraped jobs.
• PDF report generation for completed scans.
• Web scraping module with static HTML parsing and optional Playwright rendering.
• Saved scraped-job results with per-user isolation and CSV export.
• Dockerized app, database, and intentionally misconfigured target site.
• CI/CD with GitHub Actions, pytest, Ruff, mypy, and coverage tooling.

Tech Stack

Area	Tools
Language	Python 3.11+
API	FastAPI, Starlette, Uvicorn
Database	PostgreSQL, SQLAlchemy, Alembic
Auth	JWT, python-jose, passlib, bcrypt
Scanner	HTTPX, Cryptography, custom security checks
Scraping	Static HTML extraction, Playwright support
Reports	ReportLab PDF generation, CSV export
Testing	pytest, pytest-asyncio, pytest-cov
Quality	Ruff, mypy
Runtime	Docker Compose
CI	GitHub Actions

Architecture

                         +-----------------------------+
                         | Security Scanner CLI        |
                         | python -m security_scanner  |
                         +--------------+--------------+
                                        |
                                        v
+----------------+       +-------------+--------------+       +----------------+
| Target Website | <---- | FastAPI app /api/v1        | ----> | PostgreSQL     |
| or Target Site |       | auth, scans, scrape, PDF   |       | users/scans    |
+----------------+       +-------------+--------------+       | findings/jobs  |
                                        |                      +----------------+
                                        v
                         +--------------+--------------+
                         | PDF Report Generator        |
                         | completed scan -> PDF       |
                         +-----------------------------+

The Docker integration stack adds a local target-site container that intentionally exposes weak headers, permissive CORS, and a test .env file so the scanner has known misconfigurations to detect.

Requirements

For the Docker quick start, you only need:

• Docker with the Compose plugin.
• A free host port for the API, default 8000.
• A free host port for the target site, default 8099.
• A free host port for PostgreSQL, default 5432.

For local Python development without Docker, use Python 3.11+ and install the project dependencies from requirements.txt and requirements-dev.txt.

Quick Start

Clone the repository and enter the project directory:

git clone https://github.com/attarehman962/security-misconfiguration-scanner.git
cd security-misconfiguration-scanner

Start the Docker stack:

POSTGRES_PORT=5432 docker compose -f docker/docker-compose.yml -f docker/docker-compose.ci.yml up -d --build app db target-site

The app runs at:

API:        http://localhost:8000
API docs:   http://localhost:8000/docs
Target:     http://localhost:8099
Postgres:   localhost:5432

Check container status:

POSTGRES_PORT=5432 docker compose -f docker/docker-compose.yml -f docker/docker-compose.ci.yml ps

Shut down the stack and remove the test database volume:

docker compose -f docker/docker-compose.yml -f docker/docker-compose.ci.yml down -v

You do not need to copy .env.example for the Docker quick start. Local development defaults to PostgreSQL with DATABASE_URL pointing at the Compose database on localhost when running Python directly on your machine. Docker Compose maps its own DOCKER_DATABASE_URL to the app container so the service uses the internal db hostname.

Docker Compose

The Docker Compose files live in the docker/ directory:

docker/docker-compose.yml      base app, database, and optional Playwright service
docker/docker-compose.ci.yml   integration target site and CI/test overrides

The CI compose override runs database migrations before starting Uvicorn:

alembic upgrade head && uvicorn security_scanner.main:app --host 0.0.0.0 --port 8000

View logs:

POSTGRES_PORT=5432 docker compose -f docker/docker-compose.yml -f docker/docker-compose.ci.yml logs -f

Run the full Docker-backed integration flow:

scripts/run_integration.sh

CLI Usage

After local installation, show CLI help:

security-scanner --help

Run a scan from inside the app container against the Docker target site:

POSTGRES_PORT=5432 docker compose -f docker/docker-compose.yml -f docker/docker-compose.ci.yml exec app \
  python -m security_scanner --url http://target-site:80

After local installation, run a scan with table output:

security-scanner --url https://example.com --format table

Print JSON:

security-scanner --url https://example.com --format json

Save a JSON report:

security-scanner --url https://example.com --output artifacts/example-scan.json

The CLI currently writes table or JSON output. PDF reports are generated through the API for completed background scans.

API Endpoints

Method	Endpoint	Auth	Purpose
GET	/api/v1/health	No	Health check
POST	/api/v1/auth/register	No	Register a user
POST	/api/v1/auth/login	No	Login and receive a JWT
GET	/api/v1/auth/me	Yes	Return the current user
POST	/api/v1/scans	Yes	Start a background scan
GET	/api/v1/scans	Yes	List the current user's scans
GET	/api/v1/scans/{scan_id}	Yes	Get scan status and findings
GET	/api/v1/scans/{scan_id}/report	Yes	Download a completed scan as PDF
POST	/api/v1/scrape/	No	Scrape a URL immediately
POST	/api/v1/scrape/results	Yes	Save scraped job rows
GET	/api/v1/scrape/results	Yes	List saved scraped jobs
GET	/api/v1/scrape/results/export	Yes	Export saved scraped jobs as CSV

Interactive API documentation is available after startup:

http://localhost:8000/docs

API Examples

Register a user:

curl -X POST http://127.0.0.1:8000/api/v1/auth/register \
  -H "Content-Type: application/json" \
  -d '{"email": "learner@example.com", "password": "StrongPassword123!", "full_name": "Learner"}'

Login and capture the token:

TOKEN=$(curl -s -X POST http://127.0.0.1:8000/api/v1/auth/login \
  -H "Content-Type: application/json" \
  -d '{"email": "learner@example.com", "password": "StrongPassword123!"}' \
  | python -c "import json,sys; print(json.load(sys.stdin)['access_token'])")

Start a scan against the Docker target site:

SCAN_ID=$(curl -s -X POST http://127.0.0.1:8000/api/v1/scans \
  -H "Authorization: Bearer $TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"target_url": "http://target-site:80"}' \
  | python -c "import json,sys; print(json.load(sys.stdin)['scan_id'])")

Check scan status:

curl -H "Authorization: Bearer $TOKEN" \
  "http://127.0.0.1:8000/api/v1/scans/$SCAN_ID"

Download the PDF report after the scan completes:

mkdir -p artifacts

curl -L -H "Authorization: Bearer $TOKEN" \
  -o artifacts/target-site-scan.pdf \
  "http://127.0.0.1:8000/api/v1/scans/$SCAN_ID/report"

Run a live scrape without storing the result:

curl -X POST http://127.0.0.1:8000/api/v1/scrape/ \
  -H "Content-Type: application/json" \
  -d '{"url": "https://example.com/jobs", "css_selector": "a", "use_javascript": false}'

Save scraped job records:

curl -X POST http://127.0.0.1:8000/api/v1/scrape/results \
  -H "Authorization: Bearer $TOKEN" \
  -H "Content-Type: application/json" \
  -d '[
    {
      "source_url": "https://jobs.example.com/1",
      "title": "Security Engineer",
      "company": "Example Labs",
      "location": "Remote"
    }
  ]'

Export saved scraped jobs as CSV:

curl -H "Authorization: Bearer $TOKEN" \
  http://127.0.0.1:8000/api/v1/scrape/results/export

Example Output

Terminal output for the intentionally misconfigured Docker target looks like:

Scan result for: http://target-site
Total score: 54

Strict-Transport-Security  Fail  High    Missing Strict-Transport-Security header.
Content-Security-Policy    Fail  High    Missing Content-Security-Policy header.
Weak CORS policy           Fail  High    Access-Control-Allow-Origin is set to '*'.
Exposed .env file          Fail  High    http://target-site:80/.env returned HTTP 200.
ssl                        Fail  High    The target is not using HTTPS.

The sample PDF generated during local testing is written to:

artifacts/target-site-scan.pdf

Optional Portfolio Assets

Screenshots and demo video coming soon.

Screenshots and demo videos are not required to run or test the project. If you want them for a portfolio page, useful captures are:

View	Suggested file	Notes
Terminal scan output	docs/screenshots/terminal-scan.png	Capture the CLI table output for http://target-site:80.
API docs page	docs/screenshots/api-docs.png	Capture http://localhost:8000/docs.
Sample PDF report	docs/screenshots/pdf-report.png	Capture a page from artifacts/target-site-scan.pdf.

A short demo can follow this flow:

1. Start Docker Compose.
2. Open http://localhost:8000/docs.
3. Register and login.
4. Start a scan against http://target-site:80.
5. Show findings and download the PDF report.
6. Run the CLI scan from the app container.

Local Python Setup

For local development without Docker, create a virtual environment and install dependencies:

python3 -m venv .venv
source .venv/bin/activate
python -m pip install -r requirements.txt
python -m pip install -r requirements-dev.txt
python -m pip install -e .

Copy the environment template for local development:

cp .env.example .env

Apply database migrations before using database-backed API features:

alembic upgrade head

Start the development server:

uvicorn security_scanner.main:app --reload

Install Playwright browsers if you use JavaScript rendering for scraping:

python -m playwright install chromium

Tests And Quality

Run API and unit tests:

.venv/bin/python -m pytest -q tests/api tests/unit

With the Docker stack running, run the Docker-backed integration test:

BASE_URL=http://127.0.0.1:8000 \
TARGET_SCAN_URL=http://target-site \
.venv/bin/python -m pytest -q tests/integration

Run the full integration script, including Docker startup, readiness checks, pytest, logs, and cleanup:

scripts/run_integration.sh

Run coverage:

.venv/bin/python -m pytest --cov=security_scanner

Run linting and type checks:

ruff check .
mypy security_scanner tests

Project Structure

security_scanner/
  api/v1/          FastAPI routes and dependencies
  core/            Settings, logging, security helpers, exception handling
  crud/            Database-focused insert/query helpers
  db/              SQLAlchemy session setup and Alembic migrations
  models/          Domain models and SQLAlchemy records
  repositories/    User repository helpers
  reporting/       PDF, JSON, table, and CSV helpers
  scanner/         Scanner runner, HTTP client, and checks
  schemas/         Pydantic request and response schemas
  scraper/         Playwright scraper support and scrape data classes
  services/        Application service layer
  tasks/           Task entry points
  utils/           URL, TLS, validation, and fetch utilities

tests/
  api/
  integration/
  unit/

docker/
  Dockerfile
  Dockerfile.playwright
  docker-compose.yml
  docker-compose.ci.yml

docs/
  api.md
  architecture.md
  setup.md

Known Limitations

• The security checks cover common misconfigurations (headers, CORS, exposed files, SSL) and are not a substitute for a full penetration test or a commercial DAST/SAST tool.
• Background scan jobs are processed in-process; there is no dedicated task queue (e.g. Celery, RQ) yet, so scan throughput is limited under load.
• Playwright-based scraping increases resource usage and is optional by design — static scraping is used by default.
• Authentication is JWT-based with no refresh-token rotation or rate limiting on auth endpoints yet.
• Test coverage focuses on API and unit-level checks; broader end-to-end coverage across all checks is still growing.

Roadmap

• Add refresh-token support and rate limiting on auth endpoints.
• Move background scan jobs to a proper task queue.
• Expand the security check library (e.g. cookie flags, subresource integrity, open redirect detection).
• Add HTML report output alongside PDF and JSON.
• Publish demo screenshots and a walkthrough video.

Contributing

This started as a solo learning/portfolio project, but suggestions and pull requests are welcome.

1. Fork the repository and create a feature branch.
2. Follow the existing code style — run ruff check . and mypy security_scanner tests before opening a PR.
3. Add or update tests under tests/ for any behavior change.
4. Open a pull request describing the change and why it's useful.

If you find a bug or have an idea, opening an issue first is a good way to start a discussion before writing code.

Security Notes

• Only scan systems you own or have authorization to test.
• The included target-site is intentionally misconfigured for safe local testing.
• Do not use the CI/demo secrets in production.
• Replace JWT_SECRET_KEY with a strong random value before deploying.

License

This project is licensed under the MIT License.