repo-auditor

repo-auditor is a LangGraph-based agent workflow for GitHub repository audits. It reviews repositories for CI/CD drift, engineering standards gaps, security hygiene issues, cost opportunities, and performance improvements.

The project shows how an AI agent can move beyond a chat prompt and into a structured engineering workflow: load scoped configuration, gather repository evidence through tools, reason over the current state, compare against previous findings, and produce a report a team can act on.

As AI becomes part of everyday software delivery, workflows like this point toward systems that do more than summarize code. They can monitor engineering standards, keep operational knowledge current, identify drift early, and prepare the next unit of work for a human to approve.

It is designed to be easy to run and easy to adapt: point it at one or more repositories, define the standards you care about in Markdown, run an audit, and get a readable report with findings, evidence, and recommendations.

What It Does

The workflow uses a bounded agent to help answer questions like:

• Are GitHub Actions workflows missing permissions, timeouts, or concurrency?
• Do repositories follow the engineering standards documented for this team?
• Are there obvious build, dependency, Docker, or runtime hygiene gaps?
• Did findings from a previous audit get fixed, stay open, or become unclear?

The audit is read-only. It does not create issues, branches, commits, pull requests, or workflow changes.

Audit Workflow

1. You configure the repository scope in config/github_repositories.yaml.
2. You define audit standards in config/standards.md.
3. The FastAPI backend loads and validates that configuration.
4. If an organization wildcard is configured, the app resolves it into concrete repositories through the GitHub MCP server.
5. The auditor gathers repository context, key files, and GitHub Actions workflows using read-only GitHub MCP tools.
6. A bounded agent reasons over the evidence, reviews previous findings, and checks the repository against your standards and selected focus areas.
7. The result is saved in SQLite and shown in the Streamlit UI as a progress timeline, Markdown report, JSON report, and finding summary.

flowchart TD
    repoScope["Configure repository scope<br/>config/github_repositories.yaml"]
    standards["Define audit standards<br/>config/standards.md"]
    backend["FastAPI loads and validates configuration"]
    wildcard{"Organization wildcard configured?"}
    resolve["Resolve wildcard to concrete repositories<br/>GitHub MCP server"]
    gather["Gather repository context, key files,<br/>and GitHub Actions workflows"]
    prior["Load matching prior completed audit<br/>when available"]
    agent["Bounded agent reviews evidence,<br/>prior findings, standards, and focus areas"]
    store["Save audit result in SQLite"]
    ui["Show progress timeline, Markdown report,<br/>JSON report, and finding summary"]

    repoScope --> backend
    standards --> backend
    backend --> wildcard
    wildcard -- Yes --> resolve --> gather
    wildcard -- No --> gather
    gather --> agent
    prior -. optional context .-> agent
    agent --> store --> ui

Loading

If a matching completed audit already exists, the next run includes that prior report as context so previous findings can be reviewed first.

Technology And Design

The runtime is intentionally small and local-first:

• Streamlit provides the operator UI.
• FastAPI owns validation, audit lifecycle endpoints, progress streaming, and integration endpoints.
• The optional Slack integration exposes signed slash commands for audit status, job lists, and compact report reviews without requiring a Slack bot token.
• LangGraph coordinates the bounded audit workflow.
• LangChain model adapters support OpenAI and Anthropic chat models.
• GitHub's MCP server provides read-only repository, Actions, pull request, Dependabot, and code security context where the token has access.
• SQLite stores audit records, events, reports, and model usage locally.

At a high level, the flow is:

Streamlit UI --------\
                     -> FastAPI API -> LangGraph audit graph -> GitHub MCP tools
Slack slash commands-/
                                           |
                                           v
                                     SQLite audit store

Reasoning Strategy

The current reasoning strategy is bounded and tool-driven. The app does not try to place an entire organization or repository history into one prompt. Instead, it loads the configured scope, gathers key repository evidence, summarizes the available files and standards, and lets the agent use read-only MCP tools when it needs more context. The result is a focused audit loop with a finite tool-call budget and evidence-backed findings.

For larger organizations, this could be extended with retrieval-augmented generation. A future version could index repository metadata, workflow files, dependency manifests, infrastructure files, standards, and previous findings, then retrieve the most relevant evidence before each audit. The agent could use that retrieved context to spot organization-wide patterns, while still verifying important findings against the current repository state before reporting them.

Prerequisites

• Python 3.11 or newer. On macOS, verify the exact interpreter because the system python3 may be older than 3.11.
• A GitHub personal access token with read access to the repositories you want to audit.
• An OpenAI or Anthropic API key

The default setup uses GitHub's hosted MCP server at https://api.githubcopilot.com/mcp/. Docker is only needed if you choose to run the GitHub MCP server locally.

For a fine-grained GitHub personal access token, start with the smallest scope that can read the selected repositories:

• Required for normal repository audits: repository Metadata: read and Contents: read.
• Needed when the configured GITHUB_TOOLSETS use those areas: Actions: read, Pull requests: read, Dependabot alert read access, and code scanning or code security alert read access.

If your token or repository plan does not support one of those areas, remove the matching value from GITHUB_TOOLSETS in .env. The default value keeps the tool surface small for lower MCP traffic and prompt overhead:

GITHUB_TOOLSETS=repos,actions

Keep GITHUB_READ_ONLY=1. The backend rejects audits when read-only mode is disabled and also blocks mutating GitHub MCP tools before calls execute.

Public Access And Data Handling

This project is suitable to publish as source code as long as local secrets and generated data stay out of the repository. The included .gitignore excludes .env, virtual environments, caches, and local SQLite data.

Do not expose the FastAPI or Streamlit processes directly to the public internet without adding authentication, TLS, and an appropriate deployment boundary. The local development commands are intended for trusted local use.

During an audit, repository metadata and selected file contents can be sent to the configured LLM provider and to the configured GitHub MCP endpoint. Audit reports are stored in SQLite at REPO_AUDITOR_DB_PATH and may contain repository names, file paths, findings, evidence snippets, and recommendations. Protect or delete that database before sharing a demo environment.

Quick Start

Clone the repository and create a virtual environment with Python 3.11 or newer:

git clone <repo-url>
cd repo-auditor
python3.11 --version
python3.11 -m venv .venv
source .venv/bin/activate
python --version
python -m pip install --upgrade pip
python -m pip install -e ".[dev]"

If your machine uses a different executable, such as python3.12 or python3.13, use that executable instead. After activation, python --version should report Python 3.11 or newer. Python dependencies are declared in pyproject.toml; use that file as the source of truth for dependency changes.

Create your environment file:

cp .env.example .env

Edit .env and set at least:

AUDITOR_LLM_PROVIDER=openai
AUDITOR_LLM_MODEL=<model-available-in-your-account>
OPENAI_API_KEY=<your-openai-api-key>
GITHUB_PERSONAL_ACCESS_TOKEN=<your-read-only-github-token>

For Anthropic, set AUDITOR_LLM_PROVIDER=anthropic, set ANTHROPIC_API_KEY, and choose an Anthropic model for AUDITOR_LLM_MODEL.

Configure the repository scope before starting your first audit:

$EDITOR config/github_repositories.yaml

Replace the sample repository target with a repository your GitHub token can read. For example:

audit_targets:
  provider: github
  default_ref: main
  targets:
    - type: repository
      owner: your-org
      name: your-repo
      ref: main

Validate the local install:

python -m pip check
python -m pytest

Start the API:

python -m uvicorn backend.api.main:app --reload

Optional API health check:

curl http://127.0.0.1:8000/health

In another terminal, start the frontend:

source .venv/bin/activate
python -m streamlit run frontend/app.py

Open the Streamlit URL shown in the terminal, usually http://localhost:8501.

Docker

Create a local environment file and set the required provider and GitHub values:

cp .env.example .env
$EDITOR .env

Build and run the API and Streamlit frontend:

docker compose up --build

The API is available at http://localhost:8000, and the frontend is available at http://localhost:8501. Compose stores the SQLite database in the repo-auditor-data Docker volume and mounts the configured host config directory read-only as /app/config so local configuration edits are picked up without rebuilding the image.

By default, Compose mounts ./config. To use ignored local configuration, set REPO_AUDITOR_HOST_CONFIG_DIR in .env:

REPO_AUDITOR_HOST_CONFIG_DIR=./local-config

That directory must contain the same files the app expects, such as github_repositories.yaml and standards.md. Inside Docker, REPO_AUDITOR_CONFIG_DIR should stay set to /app/config.

For development with API reload and Streamlit reruns, layer in the dev compose file:

docker compose -f docker-compose.yml -f docker-compose.dev.yml up --build

To run the test suite in the development container:

docker compose -f docker-compose.yml -f docker-compose.dev.yml run --rm api python -m pytest

If local ports 8000 or 8501 are already in use, override them when starting Compose:

REPO_AUDITOR_API_PORT=18000 REPO_AUDITOR_FRONTEND_PORT=18501 docker compose up --build

Configure Repositories

Repository targets live in config/github_repositories.yaml.

The committed example may not be readable with your GitHub token. Replace it with your own repository or organization scope before running an audit.

Example single-repository target:

audit_targets:
  provider: github
  default_ref: main
  targets:
    - type: repository
      owner: example-org
      name: service-api
      ref: main

Example organization target:

audit_targets:
  provider: github
  default_ref: main
  targets:
    - type: organization
      owner: example-org
      repositories: "*"
      include_archived: false
      include_forks: false
      visibility: all
      exclude:
        - experimental-*
        - archived-service

Keep repository selection in this file. The API intentionally starts audits from validated configuration instead of accepting arbitrary repository names at runtime.

Configure Standards

Audit standards live in config/standards.md.

The format is intentionally readable:

# Audit Standards

## GitHub Actions

- Standard: GitHub Actions jobs should set timeout-minutes
  - Severity: medium
  - Enabled: true

- Standard: GitHub Actions workflows should define explicit permissions
  - Severity: high
  - Enabled: true

Use standards that match the engineering expectations you want the audit to check. The text is treated as review guidance, so it can describe practical team conventions rather than rigid rule IDs.

Environment Variables

Variable	Required	Purpose
REPO_AUDITOR_CONFIG_DIR	No	Directory containing github_repositories.yaml and standards.md. Defaults to config.
REPO_AUDITOR_DB_PATH	No	SQLite path for audit records, events, and reports. Defaults to data/repo_auditor.sqlite3.
REPO_AUDITOR_API_URL	No	FastAPI base URL used by Streamlit. Defaults to http://127.0.0.1:8000.
AUDITOR_LLM_PROVIDER	Yes	openai or anthropic.
AUDITOR_LLM_MODEL	Yes	Model name used by the selected provider.
AUDITOR_MAX_TOOL_CALLS	No	Agent tool-call budget per audit. Defaults to 8; valid range is 1 to 100.
OPENAI_API_KEY	For OpenAI	Required when AUDITOR_LLM_PROVIDER=openai.
ANTHROPIC_API_KEY	For Anthropic	Required when AUDITOR_LLM_PROVIDER=anthropic.
GITHUB_PERSONAL_ACCESS_TOKEN	Yes	GitHub token used by the GitHub MCP server. Prefer a fine-grained read-only token.
GITHUB_TOOLSETS	No	GitHub MCP toolsets. Defaults in .env.example cover repository contents and Actions reads; add pull request, Dependabot, or code security toolsets only when needed.
GITHUB_READ_ONLY	No	Set to 1 to request read-only GitHub MCP tools. The app rejects audits when this is disabled.
GITHUB_MCP_URL	No	Streamable HTTP GitHub MCP endpoint. Defaults to https://api.githubcopilot.com/mcp/.
GITHUB_MCP_COMMAND	No	Optional local MCP command override. Set this only when running a local GitHub MCP server.
GITHUB_MCP_ARGS	No	Optional shell-style args for a local GitHub MCP server.
GITHUB_HOST	No	Optional GitHub Enterprise Server or Enterprise Cloud host URL for local MCP mode. Omit for github.com.
SLACK_SIGNING_SECRET	For Slack	Signing secret from the Slack app. The slash-command endpoint rejects requests without a valid Slack signature.
SLACK_ALLOWED_COMMANDS	No	Comma-separated slash command names accepted by the endpoint. Defaults to /repo-auditor.

The app loads .env automatically. Exported shell variables take precedence. Do not commit .env.

GitHub MCP Notes

GitHub access is handled through the official GitHub MCP server. By default, the app connects to GitHub's hosted Streamable HTTP endpoint:

GITHUB_MCP_URL=https://api.githubcopilot.com/mcp/

The backend sends the configured GITHUB_TOOLSETS through the X-MCP-Toolsets header and enables read-only mode through X-MCP-Readonly.

To run a local GitHub MCP server instead, set GITHUB_MCP_COMMAND and GITHUB_MCP_ARGS. The .env.example file includes a commented Docker command for that path.

The app also applies a read-only tool interceptor before MCP calls execute. For normal usage, keep GITHUB_READ_ONLY=1 and avoid exposing broad write-capable toolsets.

GitHub MCP server reference: github/github-mcp-server

Development

Run dependency validation:

python -m pip check

Run tests:

python -m pytest

Run the API and frontend:

python -m uvicorn backend.api.main:app --reload
python -m streamlit run frontend/app.py

Useful API endpoints:

• GET /health checks configuration and storage readiness.
• GET /config returns the validated repository targets and standards.
• POST /audits starts an audit from the configured repository scope.
• GET /audits lists persisted audit runs.
• GET /audits/{audit_id} returns status and summary details.
• GET /audits/{audit_id}/events streams run events as server-sent events.
• GET /audits/{audit_id}/events.json returns run events as JSON.
• GET /audits/{audit_id}/report returns the final report.
• GET /audits/{audit_id}/status-summary returns a short status sentence.
• GET /audits/{audit_id}/usage returns model and token usage for one run.
• GET /usage returns recent model and token usage.
• POST /integrations/slack/commands handles Slack slash command requests.

Possible Next Steps

The current workflow is intentionally read-only: it identifies findings, captures evidence, and produces a report. Natural extensions would turn those findings into follow-up work:

• Create GitHub issues from selected findings, including severity, evidence, affected files, and recommended fixes.
• Open Jira tickets or Notion tasks for teams that track engineering work outside GitHub.
• Group related findings into initiatives, such as CI/CD hardening, Docker hygiene, dependency maintenance, or cost cleanup.
• Create fix branches for selected findings and attach generated remediation notes or starter patches.
• Open pull requests once a branch has a proposed fix, keeping the finding, evidence, and audit run linked from the PR body.
• Add approval gates so write actions only happen after a user selects specific findings and confirms the destination.

Slack Integration (Optional)

The API can receive Slack slash command requests at POST /integrations/slack/commands. Slack sends a signed form request to that endpoint, and repo-auditor returns an ephemeral Slack response with status, job lists, or a compact report review. No Slack bot token is required for the current slash-command flow.

Requirements

• A running FastAPI backend at an HTTPS URL Slack can reach.
• A Slack app installed in the target workspace with a slash command such as /repo-auditor.
• SLACK_SIGNING_SECRET set to the Slack app signing secret.
• SLACK_ALLOWED_COMMANDS set if you use command names other than /repo-auditor.
• The normal repo-auditor configuration and secrets required to run audits, including the GitHub token and LLM provider settings.

For local development, expose the backend with a tunnel and point Slack at the tunnel URL. The request URL should include the API route, for example:

https://some-tunnel.app/integrations/slack/commands

Slack App Setup

1. Create or open a Slack app for the workspace.
2. Enable Slash Commands and create /repo-auditor.
3. Set the command request URL to https://<public-api-host>/integrations/slack/commands.
4. Copy the app signing secret into .env as SLACK_SIGNING_SECRET.
5. If the command name is not /repo-auditor, set SLACK_ALLOWED_COMMANDS to the exact command name. Multiple commands can be comma-separated.
6. Restart the FastAPI backend and run /repo-auditor help in Slack.

Example .env values:

SLACK_SIGNING_SECRET=your-slack-signing-secret
SLACK_ALLOWED_COMMANDS=/repo-auditor

Command Reference

Slack text	Behavior
help	Returns usage help.
start	Queues an audit with the default config profile.
start <config_profile>	Queues an audit using a named config profile.
status <audit_id>	Returns the status summary for one audit.
status latest	Returns the newest persisted audit status.
<audit_id>	Shorthand for status <audit_id>.
report <audit_id>	Returns a readable Slack review with summary, scope, and key finding details.
report latest	Returns the newest completed report review.
list, jobs, or audits	Lists persisted audit jobs.
list all	Lists persisted audit jobs; currently equivalent to list.

latest is not a standalone command. Use status latest for the newest audit status or report latest for the newest completed report review.

Response Format

Slack responses are ephemeral by default, so command output is visible to the person who ran the command. Reports are intentionally summarized for Slack readability instead of dumping the full Markdown file into the channel. Command examples, audit IDs, profiles, and focus values are rendered as plain text so they can be copied back into Slack commands without carrying inline-code formatting.

The Slack report review includes:

• Finding counts by severity.
• Repository scope.
• Previous finding review count, when present.
• Up to the first 10 findings with severity, repository, category, confidence, location, first evidence item, and recommendation.
• A pointer to GET /audits/{audit_id}/report for the full Markdown and JSON report.

Long Slack messages are split into Slack-sized block payloads. If a slash command response exceeds Slack's block limit, the API returns the first payload immediately and posts the remaining payloads to Slack's response_url in the background.

Security And Operations

The endpoint verifies Slack's X-Slack-Request-Timestamp and X-Slack-Signature headers before handling commands. Requests outside the five-minute timestamp tolerance or with an invalid signature are rejected.

Set SLACK_ALLOWED_COMMANDS when multiple Slack apps or command aliases could reach the same API host. Requests from command names outside that allowlist are acknowledged with an ephemeral rejection message.

Troubleshooting

• 503 or SLACK_SIGNING_SECRET is required for Slack commands: set SLACK_SIGNING_SECRET and restart the backend.
• 401 Slack request verification failed: check that Slack is using the same signing secret and that the backend clock is accurate.
• This slash command is not configured for repo-auditor: add the slash command name to SLACK_ALLOWED_COMMANDS.
• Slack shows a timeout: confirm the request URL reaches the FastAPI backend over HTTPS and that /health is healthy.

About InfiniumTek

InfiniumTek builds AI workflow automation for businesses and software engineering teams. Its work focuses on practical agentic systems, internal tools, and integrations that connect models with the operational systems teams already use.

Learn more at infiniumtek.com.