Skip to content

Latest commit

Β 

History

16 Commits

Folders and files

NameName
Last commit message
Last commit date
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 

Repository files navigation

NeuroOps

NeuroOps is an autonomous AI SRE engine that detects, diagnoses, and remediates Kubernetes incidents end-to-end β€” with zero human intervention for high-confidence scenarios.


πŸ“Š Results

Metric Result
🎯 Chaos Incidents Resolved 15 / 15 β€” 100%
⚑ Average MTTR < 4 minutes
πŸ† DORA Tier Elite Performer
πŸŽͺ False Positive Rate 0%
πŸ’° Cost vs Manual On-Call > 1,600Γ— cheaper
πŸ€– Autonomous Resolution Rate 50–100% (confidence-gated)

🧠 How It Works

flowchart TD
    A([πŸ”₯ Fault Injected]) --> B

    subgraph DETECT["πŸ” DETECT"]
        B[Prometheus Scrapes\nGolden Signals] --> C[Isolation Forest\nAnomaly Scoring]
        C --> D{Anomaly?}
        D -- No --> B
        D -- Yes --> E[Alert Dedup\n+ Severity Triage]
    end

    E --> F

    subgraph DIAGNOSE["🧬 DIAGNOSE β€” LangGraph Fan-Out"]
        F[Supervisor Init\nIncident ID + OTel Trace] --> G1 & G2 & G3 & G4
        G1[πŸ”Ž Detective\nMetric Correlation]
        G2[πŸ—ΊοΈ Topologist\nTrace Analysis]
        G3[πŸ“œ Historian\nDeploy Timeline]
        G4[πŸ“‹ Log Analyser\nPod Logs]
        G1 & G2 & G3 & G4 --> H[🧠 Supervisor\nRoot Cause + Confidence]
    end

    H -- "< 0.55" --> I[πŸ‘€ Escalate\nto Human]
    H -- ">= 0.65" --> J

    subgraph REMEDIATE["πŸ”§ REMEDIATE"]
        J{Action?} -- "restart / scale" --> K[⚑ Auto-Execute]
        J -- "rollback / PR" --> L[πŸ”” Slack Approval\nThen Execute]
        K & L --> M[βœ… Verify Resolution]
    end

    M --> N[πŸ“„ Post-Mortem\nStored]

    style DETECT fill:#1a1a2e,stroke:#00D9FF,color:#fff
    style DIAGNOSE fill:#1a1a2e,stroke:#FF6B35,color:#fff
    style REMEDIATE fill:#1a1a2e,stroke:#00C853,color:#fff
Loading

πŸ—οΈ Architecture

graph TB
    subgraph UI["🌐 Web UI :3000"]
        WEB[Orbital Command Hub\nReal-time Dashboard]
    end

    subgraph CORE["βš™οΈ Core Services"]
        DET["πŸ” Detector :8001"]
        AGT["🧠 Agent :8002"]
        REM["πŸ”§ Remediator :8003"]
    end

    subgraph OBS["πŸ“‘ Observability"]
        PROM[Prometheus]
        JAEGER[Jaeger]
        GRAFANA[Grafana]
        OTEL[OTel Collector]
    end

    subgraph K8S["☸️ Kubernetes"]
        WORKLOADS[Target Workloads]
        CHAOS[LitmusChaos]
    end

    WEB --> AGT & DET
    DET --> PROM & OTEL
    AGT --> PROM & JAEGER & OTEL
    REM --> WORKLOADS & OTEL
    CHAOS --> WORKLOADS
    WORKLOADS --> PROM
    OTEL --> JAEGER
    PROM & JAEGER --> GRAFANA

    style UI fill:#0d1117,stroke:#00D9FF,color:#00D9FF
    style CORE fill:#0d1117,stroke:#FF6B35,color:#FF6B35
    style OBS fill:#0d1117,stroke:#7C3AED,color:#7C3AED
    style K8S fill:#0d1117,stroke:#326CE5,color:#326CE5
Loading

πŸš€ Quickstart

Prerequisites: Python 3.11+, Docker, Kubernetes cluster (Minikube / kind / EKS)

git clone https://github.com/Tayab-Ahamed/neuroops.git
cd neuroops
cp .env.example .env      # fill in your API keys

Docker Compose β€” full stack

docker compose up --build

Services: :8001 Detector Β· :8002 Agent Β· :8003 Remediator
Open web-ui/index.html for the live dashboard.

Run services individually

# Detector
cd detector && pip install -r requirements.txt && uvicorn server:app --port 8001

# Agent
cd agent && pip install -r requirements.txt && uvicorn main:app --port 8002

# Remediator
cd remediator && pip install -r requirements.txt && uvicorn server:app --port 8003

πŸ”‘ Environment Variables

# LLM
ANTHROPIC_API_KEY=sk-ant-...          # Required β€” RCA agents
OPENAI_API_KEY=sk-...                 # Optional fallback

# Observability
PROMETHEUS_URL=http://localhost:9090
JAEGER_QUERY_URL=http://localhost:16686
OTEL_COLLECTOR_ENDPOINT=http://localhost:4317

# Kubernetes
KUBECONFIG=~/.kube/config
TARGET_NAMESPACE=neuroops-demo

# GitHub
GITHUB_TOKEN=ghp_...                  # Historian agent + PR actions
GITHUB_REPO=your-username/repo

# Tuning
CONFIDENCE_THRESHOLD=0.65             # Below β†’ human escalation
AUTONOMOUS_CONFIDENCE_THRESHOLD=0.65  # Actions above this run autonomously
ANOMALY_CONTAMINATION=0.05

# ChatOps
SLACK_WEBHOOK_URL=https://hooks.slack.com/...   # Optional

πŸ“‘ API Reference

Detector β€” :8001

Method Endpoint Description
GET /health Health + anomaly model status
GET /alerts Active alerts
GET /metrics Prometheus scrape endpoint
POST /baseline/train Trigger baseline training

Agent β€” :8002

Method Endpoint Description
POST /investigate Trigger RCA for an alert
GET /incidents All persisted incidents
GET /incidents/{id} Single incident + full RCA trace
GET /incidents/{id}/similar Top-K similar incidents (RAG)
GET /analytics/mttr p50/p95/p99 MTTR per service
GET /analytics/sla SLA breach + autonomous resolution rate
GET /analytics/cost LLM token + USD cost tracking

Remediator β€” :8003

Method Endpoint Description
POST /remediate Execute remediation action
GET /health Health + action count
GET /metrics Prometheus scrape endpoint

πŸ“ Project Structure

neuroops/
β”œβ”€β”€ detector/              # Anomaly detection β€” Isolation Forest + Ridge Regression
β”œβ”€β”€ agent/                 # LangGraph RCA β€” Detective, Topologist, Historian, Supervisor
β”œβ”€β”€ remediator/            # Actions β€” restart, rollback, scale, patch, PR + approval gate
β”œβ”€β”€ observability/         # OTel collector config, Grafana dashboards, CLI replay tool
β”œβ”€β”€ benchmarks/            # Chaos benchmark runner + report generator
β”œβ”€β”€ web-ui/                # Orbital Command Hub (real-time HTML dashboard)
β”œβ”€β”€ cluster/               # Kubernetes manifests + LitmusChaos experiments
β”œβ”€β”€ docker-compose.yml
β”œβ”€β”€ Makefile
└── pyproject.toml

πŸ“œ License

MIT β€” see LICENSE


Built by Tayab Ahamed

About

Autonomous AI SRE system for Kubernetes. Detects, diagnoses, and remediates cluster incidents using LangGraph multi-agent RCA, OpenTelemetry self-observability, and chaos engineering benchmarks.

Topics

Resources

Stars

0 stars

Watchers

0 watching

Forks

Releases

Packages

Contributors

Languages