Skip to content

Commit 7c3f3d6

Browse files
Copilotzloeber
Copilot
and
zloeber
committed
feat: Add AI-powered secret discovery with LangGraph/LangChain integration
Co-authored-by: zloeber <4702624+zloeber@users.noreply.github.com>
1 parent e199698 commit 7c3f3d6

8 files changed

Lines changed: 2552 additions & 0 deletions

File tree

‎docs/user-guide/discovery.md‎

Lines changed: 176 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,176 @@
1+
# AI-Powered Secret Discovery
2+
3+
The `secretzero discover` command uses artificial intelligence to automatically
4+
scan your project and generate a starter `Secretfile.detect.yml` — the fastest
5+
way to bootstrap your secrets configuration.
6+
7+
## Quick Start
8+
9+
```bash
10+
# Scan current directory with default settings (Ollama/local)
11+
secretzero discover
12+
13+
# Privacy-first: only use local LLM models
14+
secretzero discover --local-only
15+
16+
# Dry-run to preview without writing any files
17+
secretzero discover --dry-run --no-llm
18+
19+
# Use OpenAI for deeper semantic analysis
20+
secretzero discover --provider openai
21+
```
22+
23+
## How It Works
24+
25+
Discovery runs in two complementary stages:
26+
27+
1. **Pattern-based detection** (always available)
28+
Regular-expression heuristics identify common secret patterns such as API
29+
keys, database passwords, JWT secrets, cloud credentials, and OAuth tokens.
30+
Each match receives a *confidence score* based on how specific and
31+
recognisable the pattern is.
32+
33+
2. **LLM-enhanced analysis** (optional, requires `secretzero[ai]`)
34+
When an LLM backend is configured and reachable, file snippets are sent to
35+
the model for deeper semantic analysis. LLM candidates are merged with
36+
pattern results and weighted by combined confidence. The analysis runs
37+
locally when Ollama is selected, so sensitive data never leaves your machine.
38+
39+
## Output
40+
41+
Running `secretzero discover` produces `Secretfile.detect.yml` — a valid
42+
SecretZero configuration that you can review, edit, and merge into your
43+
`Secretfile.yml`:
44+
45+
```yaml
46+
version: "1.0"
47+
metadata:
48+
description: Auto-generated by secretzero discover
49+
generated_by: secretzero-discovery-agent
50+
providers:
51+
local:
52+
kind: local
53+
config: {}
54+
secrets:
55+
- name: database_password
56+
description: Detected database password in .env
57+
kind: random_password
58+
config: {}
59+
targets:
60+
- provider: local
61+
kind: file
62+
config:
63+
path: .env
64+
format: dotenv
65+
merge: true
66+
```
67+
68+
## Configuration
69+
70+
Create a `secretzero.yml` file to control discovery behaviour:
71+
72+
```yaml
73+
# secretzero.yml
74+
version: "1.0"
75+
76+
llm:
77+
default_provider: ollama # ollama | openai | anthropic | azure_openai
78+
providers:
79+
ollama:
80+
base_url: "${OLLAMA_HOST:-http://localhost:11434}"
81+
model: "${OLLAMA_MODEL:-llama3.2:3b}"
82+
timeout: 120
83+
temperature: 0.7
84+
openai:
85+
api_key: "${OPENAI_API_KEY}"
86+
model: "gpt-4o-mini"
87+
anthropic:
88+
api_key: "${ANTHROPIC_API_KEY}"
89+
model: "claude-3-5-sonnet-20241022"
90+
91+
discovery:
92+
confidence_threshold: 0.6 # Minimum score to include (0.0–1.0)
93+
max_files: 1000 # Cap on files scanned
94+
include_patterns:
95+
- "*.env*"
96+
- "*.yml"
97+
- "*.yaml"
98+
- "*.json"
99+
- "*.toml"
100+
- "*.tf"
101+
- "*.tfvars"
102+
exclude_patterns:
103+
- "**/node_modules/**"
104+
- "**/venv/**"
105+
- "**/.venv/**"
106+
- "**/dist/**"
107+
- "**/build/**"
108+
- "**/.git/**"
109+
110+
output:
111+
format: text # text | json | yaml
112+
verbosity: 1 # 0–3
113+
color: true
114+
```
115+
116+
### Configuration Loading Priority
117+
118+
The CLI loads `secretzero.yml` from the first location found:
119+
120+
1. `SECRETZERO_CONFIG` environment variable (absolute path)
121+
2. `./secretzero.yml` in the current working directory
122+
3. `~/.config/secretzero/secretzero.yml` in your home directory
123+
124+
If no file is found, built-in defaults are used.
125+
126+
## CLI Options
127+
128+
| Option | Default | Description |
129+
|--------|---------|-------------|
130+
| `--path` / `-p` | `.` | Project root directory to scan |
131+
| `--output` / `-o` | `<path>/Secretfile.detect.yml` | Output file path |
132+
| `--dry-run` | `false` | Analyse without writing files |
133+
| `--no-llm` | `false` | Pattern matching only; skip LLM |
134+
| `--provider` | From config | LLM provider to use |
135+
| `--model` | From config | LLM model override |
136+
| `--local-only` | `false` | Restrict to local LLM providers |
137+
| `--config` / `-c` | Auto-discovered | Path to `secretzero.yml` |
138+
| `--format` / `-f` | `text` | Output format: text, json, yaml |
139+
| `--threshold` | From config | Confidence threshold override |
140+
141+
## Installing AI Extras
142+
143+
```bash
144+
# All AI providers (recommended)
145+
pip install "secretzero[ai]"
146+
147+
# Individual providers
148+
pip install langchain-ollama # Ollama (local, free)
149+
pip install langchain-openai # OpenAI
150+
pip install langchain-anthropic # Anthropic Claude
151+
```
152+
153+
## Privacy & Security
154+
155+
- **Local-only mode** (`--local-only` or `--no-llm`) never sends any data
156+
to external APIs.
157+
- Raw secret *values* are never stored, logged, or included in the output.
158+
Only the variable name, description, confidence score, and recommended
159+
generator type are written to the output file.
160+
- Obvious placeholder values (e.g. `your_api_key_here`, `${VAR_NAME}`,
161+
`<your-secret>`) are automatically skipped.
162+
163+
## Next Steps
164+
165+
After running discovery:
166+
167+
1. Review `Secretfile.detect.yml` and remove false positives.
168+
2. Rename or merge entries into your `Secretfile.yml`.
169+
3. Run `secretzero validate` to validate the merged configuration.
170+
4. Run `secretzero sync --dry-run` to preview secret generation.
171+
172+
## Related
173+
174+
- [CLI Reference: discover](cli/discover.md)
175+
- [Configuration Reference](../reference/configuration.md)
176+
- [Getting Started](../getting-started/index.md)

‎examples/secretzero.yml‎

Lines changed: 132 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,132 @@
1+
# secretzero.yml
2+
# SecretZero CLI configuration for AI-powered features.
3+
#
4+
# Copy this file to ./secretzero.yml (or ~/.config/secretzero/secretzero.yml)
5+
# and customise for your environment.
6+
#
7+
# Configuration loading priority:
8+
# 1. SECRETZERO_CONFIG env var (absolute path)
9+
# 2. ./secretzero.yml (current working directory)
10+
# 3. ~/.config/secretzero/secretzero.yml
11+
12+
version: "1.0"
13+
14+
# ---------------------------------------------------------------------------
15+
# LLM provider configuration for AI-powered features (e.g. secretzero discover)
16+
# ---------------------------------------------------------------------------
17+
llm:
18+
# Which provider to use by default:
19+
# ollama – local, free, no data leaves your machine
20+
# openai – OpenAI cloud API (requires OPENAI_API_KEY)
21+
# anthropic – Anthropic Claude (requires ANTHROPIC_API_KEY)
22+
# azure_openai – Azure-hosted OpenAI (requires AZURE_OPENAI_* vars)
23+
default_provider: ollama
24+
25+
providers:
26+
# -----------------------------------------------------------------------
27+
# Ollama – local LLM server (recommended for privacy-first workflows)
28+
# Install: https://ollama.com | Model: ollama pull llama3.2:3b
29+
# -----------------------------------------------------------------------
30+
ollama:
31+
base_url: "${OLLAMA_HOST:-http://localhost:11434}"
32+
model: "${OLLAMA_MODEL:-llama3.2:3b}"
33+
# Heavier model for reasoning-intensive analysis (optional)
34+
reasoning_model: "${OLLAMA_REASONING_MODEL:-}"
35+
timeout: 120
36+
temperature: 0.7
37+
max_tokens: 4096
38+
39+
# -----------------------------------------------------------------------
40+
# OpenAI – cloud API
41+
# -----------------------------------------------------------------------
42+
openai:
43+
api_key: "${OPENAI_API_KEY}"
44+
model: "${OPENAI_MODEL:-gpt-4o-mini}"
45+
organization: "${OPENAI_ORG_ID}"
46+
timeout: 120
47+
temperature: 0.7
48+
max_tokens: 4096
49+
50+
# -----------------------------------------------------------------------
51+
# Anthropic – Claude models
52+
# -----------------------------------------------------------------------
53+
anthropic:
54+
api_key: "${ANTHROPIC_API_KEY}"
55+
model: "${ANTHROPIC_MODEL:-claude-3-5-sonnet-20241022}"
56+
timeout: 120
57+
temperature: 0.7
58+
max_tokens: 4096
59+
60+
# -----------------------------------------------------------------------
61+
# Azure OpenAI – Azure-hosted deployment
62+
# -----------------------------------------------------------------------
63+
azure_openai:
64+
api_key: "${AZURE_OPENAI_API_KEY}"
65+
endpoint: "${AZURE_OPENAI_ENDPOINT}"
66+
deployment: "${AZURE_OPENAI_DEPLOYMENT}"
67+
api_version: "2024-02-15-preview"
68+
timeout: 120
69+
temperature: 0.7
70+
max_tokens: 4096
71+
72+
# ---------------------------------------------------------------------------
73+
# Discovery settings (used by: secretzero discover)
74+
# ---------------------------------------------------------------------------
75+
discovery:
76+
# Disable to prevent external script execution during discovery
77+
allow_script_execution: false
78+
79+
# Minimum confidence score (0.0–1.0) for a secret to be included in output.
80+
# Higher values = fewer false positives but may miss real secrets.
81+
confidence_threshold: 0.6
82+
83+
# Maximum number of files to scan (performance guard)
84+
max_files: 1000
85+
86+
# Glob patterns for files to scan (relative to project root)
87+
include_patterns:
88+
- "*.env*"
89+
- ".env"
90+
- ".env.*"
91+
- "*.yml"
92+
- "*.yaml"
93+
- "*.json"
94+
- "*.toml"
95+
- "*.tf"
96+
- "*.tfvars"
97+
- "**/.github/workflows/*.yml"
98+
- "**/k8s/**/*.yaml"
99+
- "**/kubernetes/**/*.yaml"
100+
101+
# Glob patterns for paths to exclude (relative to project root)
102+
exclude_patterns:
103+
- "**/node_modules/**"
104+
- "**/venv/**"
105+
- "**/.venv/**"
106+
- "**/dist/**"
107+
- "**/build/**"
108+
- "**/.git/**"
109+
- "**/vendor/**"
110+
- "**/__pycache__/**"
111+
- "**/*.pyc"
112+
- "**/target/**" # Java/Rust build outputs
113+
114+
# ---------------------------------------------------------------------------
115+
# Output preferences
116+
# ---------------------------------------------------------------------------
117+
output:
118+
# Default summary format when not overridden by --format:
119+
# text – human-readable Rich console output
120+
# json – machine-readable JSON (useful for CI pipelines)
121+
# yaml – YAML summary
122+
format: text
123+
124+
# Verbosity level:
125+
# 0 = quiet (errors only)
126+
# 1 = normal (default)
127+
# 2 = verbose
128+
# 3 = debug
129+
verbosity: 1
130+
131+
# Enable colour output (auto-disabled in non-TTY environments)
132+
color: true

‎pyproject.toml‎

Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -89,6 +89,13 @@ docs = [
8989
"mkdocs-click>=0.8.0",
9090
"mkdocs-swagger-ui-tag>=0.6.0",
9191
]
92+
ai = [
93+
"langchain-core>=1.0.0",
94+
"langgraph>=1.0.0",
95+
"langchain-ollama>=1.0.0",
96+
"langchain-openai>=1.0.0",
97+
"langchain-anthropic>=1.0.0",
98+
]
9299
security = [
93100
"bandit",
94101
"pip-audit",

0 commit comments

Comments
 (0)