Skip to content

Commit 84ff010

Browse files
authored
Merge pull request #14 from willwebster5/feature/v2-authoring-layer
feat(v2): authoring layer + State v4 + provider refactor + migrate (Sections 1–4)
2 parents 0a07600 + dc86dd6 commit 84ff010

68 files changed

Lines changed: 4278 additions & 449 deletions

File tree

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

.gitignore

Lines changed: 8 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -46,3 +46,11 @@ src/talonctl/_version.py
4646

4747
# Git worktrees (isolated workspaces, not checked in)
4848
.worktrees/
49+
50+
# direnv per-developer environment (not committed)
51+
.envrc
52+
53+
# Coverage artifacts
54+
.coverage
55+
.coverage.*
56+
htmlcov/

CHANGELOG.md

Lines changed: 57 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -2,6 +2,63 @@
22

33
## [Unreleased]
44

5+
## v0.5.0 — v2 authoring layer, State v4, provider refactor, and `talonctl migrate`
6+
7+
The foundation for talonctl v2: a unified Kubernetes-style resource envelope,
8+
`resource_id`-keyed state, and an idempotent migration command. **Backward
9+
compatible** — existing v1 templates and v3 state continue to load unchanged; v2
10+
adoption is opt-in via `talonctl migrate`.
11+
12+
### Added
13+
14+
- **Canonical v2 envelope** (`apiVersion: talon/v2` with `kind`/`metadata`/`spec`/`status`)
15+
as the in-memory model every resource normalizes to. A dual-read loader
16+
transparently reads both legacy v1 flat templates and v2 envelopes, so no
17+
template changes are required to upgrade.
18+
- **Multi-resource files** — a single YAML file may declare one or many
19+
independent resources (multi-doc `---` or a top-level list), each keyed by its
20+
own `resource_id`. (There is no `kind: Module`; a file is 1..N independent
21+
resources.)
22+
- **In-package JSON Schema + schema-driven validator** for the v2 envelope;
23+
`talonctl validate` understands both v1 and v2 files.
24+
- **State v4** — state is keyed by stable `resource_id`; v3 (name-keyed) state
25+
auto-migrates in memory on load, non-destructively. A read-only `status`
26+
projection (`server_id`, `rule_id`, `deployed_at`, `content_hash`) is derived
27+
from existing state fields; authors cannot write `status`.
28+
- **`talonctl migrate`** — idempotent, dry-run-by-default command that rewrites
29+
v1 templates to v2 in place and reconciles v3 state to v4:
30+
- `--write` applies changes (dry-run writes nothing — neither templates nor
31+
state); `--templates-only` / `--state-only` scope each half;
32+
`--format json -o FILE` emits a machine-readable report.
33+
- Reports orphans (state with no template), unmanaged templates (template with
34+
no state), and conflicts — **report-only; never deletes or creates** resources.
35+
- Content hashes are preserved byte-for-byte through a rewrap, so already-deployed
36+
resources do not show as changed on the next `plan`.
37+
38+
### Changed
39+
40+
- All seven providers now consume the canonical `Envelope`; `plan`, `apply`,
41+
`validate`, and `drift` operate on v1 and v2 templates end-to-end through a
42+
single parse path.
43+
- API-consumed fields (e.g. `_search_domain`) are classified into `spec`; the v1
44+
top-level `metadata:` block routes to envelope `metadata`.
45+
46+
### Internal
47+
48+
- New modules: `core/envelope.py`, `core/envelope_loader.py`, `core/v1_compat.py`,
49+
`core/envelope_validation.py`, `core/envelope_serializer.py`,
50+
`core/status_projection.py`, `core/migrate.py`; `schemas/envelope.schema.json`.
51+
`TemplateDiscovery` delegates to the shared envelope loader (one parse path).
52+
- Merge-blocking content-hash stability tests guarantee the v1→v2 transform and
53+
the serializer never alter a deployed resource's content hash.
54+
55+
### Migration
56+
57+
Run `talonctl migrate` (dry-run) to preview, then `talonctl migrate --write` to
58+
convert templates to v2 and re-key state to v4 in place. Git is the rollback. v1
59+
templates and v3 state continue to work unmigrated; a future release will
60+
announce the v1-reader removal deadline.
61+
562
## v0.4.0 — `find` command + fleet-wide CQL validation
663

764
### Added

CLAUDE.md

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -59,6 +59,11 @@ talonctl sync # Reconcile state with live tenant
5959
talonctl drift # Detect manual console changes
6060
talonctl show # Show current state
6161
talonctl destroy # Destroy managed resources
62+
talonctl migrate # Dry-run: preview v1->v2 template + v3->v4 state migration
63+
talonctl migrate --write # Apply migration in place (git is the rollback)
64+
talonctl migrate --templates-only # Rewrap templates only
65+
talonctl migrate --state-only # Reconcile state only
66+
talonctl migrate --format json -o m.json # Machine-readable report (orphans/unmanaged/conflicts)
6267

6368
# Credential management
6469
talonctl auth setup # Interactive credential setup wizard

pyproject.toml

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -25,6 +25,7 @@ dependencies = [
2525
"rich>=13.0.0",
2626
"requests>=2.28.0",
2727
"click>=8.0.0",
28+
"jsonschema>=4.0",
2829
]
2930

3031
[project.optional-dependencies]
@@ -35,6 +36,7 @@ talonctl = "talonctl.cli:cli"
3536

3637
[tool.hatch.build.targets.wheel]
3738
packages = ["src/talonctl"]
39+
force-include = { "src/talonctl/schemas/envelope.schema.json" = "talonctl/schemas/envelope.schema.json" }
3840

3941
[tool.hatch.version]
4042
source = "vcs"

src/talonctl/cli.py

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -83,6 +83,7 @@ def cli(ctx, verbose):
8383
from talonctl.commands.health import health # noqa: E402
8484
from talonctl.commands.metrics import metrics # noqa: E402
8585
from talonctl.commands.find import find # noqa: E402
86+
from talonctl.commands.migrate import migrate # noqa: E402
8687

8788
cli.add_command(validate)
8889
cli.add_command(plan)
@@ -101,3 +102,4 @@ def cli(ctx, verbose):
101102
cli.add_command(health)
102103
cli.add_command(metrics)
103104
cli.add_command(find)
105+
cli.add_command(migrate)

src/talonctl/commands/init.py

Lines changed: 5 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -7,6 +7,7 @@
77
import click
88

99
from talonctl.commands._common import console
10+
from talonctl.core.state_manager import StateManager
1011

1112
RESOURCE_DIRS = [
1213
"detections",
@@ -47,9 +48,11 @@ def init(ctx, path):
4748
templates_dir = Path(__file__).parent.parent / "templates" / "init"
4849
_copy_templates(templates_dir, project_dir)
4950

50-
# Create state file
51+
# Create state file. Use the same `version` key/value StateManager reads/writes
52+
# (sourced from STATE_VERSION) so a freshly-scaffolded project is never stamped
53+
# with a stale or mismatched format version.
5154
(project_dir / ".crowdstrike").mkdir(exist_ok=True)
52-
state = {"format_version": "3.0", "resources": {}}
55+
state = {"version": StateManager.STATE_VERSION, "resources": {}}
5356
(project_dir / ".crowdstrike" / "deployed_state.json").write_text(json.dumps(state, indent=2) + "\n")
5457

5558
# Create .gitignore

src/talonctl/commands/migrate.py

Lines changed: 117 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,117 @@
1+
"""talonctl migrate — rewrap v1 templates to v2 and reconcile state to v4.
2+
3+
Dry-run by default. `--write` is the only flag that mutates disk. Idempotent.
4+
Orphans/unmanaged/conflicts are reported, never deleted or created.
5+
"""
6+
7+
from __future__ import annotations
8+
9+
import json
10+
from pathlib import Path
11+
12+
import click
13+
from rich.table import Table
14+
15+
from talonctl.commands._common import console, get_state_file_path, state_options
16+
from talonctl.core.migrate import (
17+
MigrationReport,
18+
build_template_index,
19+
reconcile_state,
20+
scan_templates,
21+
)
22+
from talonctl.core.state_manager import StateManager
23+
from talonctl.core.template_discovery import TemplateDiscovery
24+
from talonctl.project import find_project_root
25+
26+
27+
@click.command()
28+
@state_options
29+
@click.option("--write", is_flag=True, help="Apply changes (default: dry-run, writes nothing).")
30+
@click.option("--templates-only", is_flag=True, help="Only rewrap templates; skip state reconciliation.")
31+
@click.option("--state-only", is_flag=True, help="Only reconcile state; skip template rewrap.")
32+
@click.option("--format", "fmt", type=click.Choice(["text", "json"]), default="text")
33+
@click.option("--output", "-o", type=click.Path(), help="Write JSON report to file.")
34+
def migrate(state_file, write, templates_only, state_only, fmt, output):
35+
"""Migrate v1 templates -> v2 and v3 state -> v4 (dry-run by default)."""
36+
if templates_only and state_only:
37+
raise click.UsageError("--templates-only and --state-only are mutually exclusive.")
38+
39+
do_templates = not state_only
40+
do_state = not templates_only
41+
42+
report = MigrationReport(dry_run=not write)
43+
44+
if do_templates:
45+
resources_dir = find_project_root() / TemplateDiscovery.DEFAULT_RESOURCES_DIR
46+
report.rewraps = scan_templates(resources_dir)
47+
if write:
48+
for fr in report.rewraps:
49+
if fr.status == "rewrap" and fr.new_text is not None:
50+
fr.path.write_text(fr.new_text)
51+
52+
if do_state:
53+
state_path = get_state_file_path(state_file)
54+
sm = StateManager(state_file_path=state_path) if state_path.exists() else None
55+
resources = sm.export_to_dict().get("resources", {}) if sm else {}
56+
index = build_template_index(TemplateDiscovery().discover_all())
57+
report.state = reconcile_state(resources, index)
58+
if write and sm is not None and report.state.rekeyed:
59+
for rtype, old, new in report.state.rekeyed:
60+
rs = sm.get_resource(rtype, old)
61+
if rs is not None:
62+
sm.set_resource(rtype, new, rs)
63+
sm.delete_resource(rtype, old)
64+
sm.save()
65+
66+
if fmt == "json" or output:
67+
data = json.dumps(report.to_dict(), indent=2)
68+
if output:
69+
Path(output).write_text(data)
70+
console.print(f"[green]Report written to {output}[/green]")
71+
else:
72+
console.print_json(data)
73+
else:
74+
_render_text(report)
75+
76+
77+
def _render_text(report: MigrationReport) -> None:
78+
mode = "DRY-RUN (no changes written — pass --write to apply)" if report.dry_run else "WRITE"
79+
console.print(f"[bold blue]talonctl migrate[/bold blue] [dim]{mode}[/dim]\n")
80+
81+
if report.rewraps:
82+
t = Table(title="Templates")
83+
t.add_column("status")
84+
t.add_column("file")
85+
t.add_column("details")
86+
for fr in report.rewraps:
87+
detail = ""
88+
if fr.status == "rewrap":
89+
detail = f"{', '.join(fr.kinds)} · {fr.comments_dropped} comment(s) dropped"
90+
elif fr.status == "error":
91+
detail = "; ".join(fr.errors)
92+
t.add_row(fr.status, str(fr.path), detail)
93+
console.print(t)
94+
if any(fr.status == "rewrap" and fr.comments_dropped for fr in report.rewraps):
95+
console.print("[yellow]Comments dropped — originals preserved in git history.[/yellow]")
96+
97+
s = report.state
98+
if s.rekeyed or s.orphans or s.unmanaged or s.conflicts:
99+
t = Table(title="State")
100+
t.add_column("category")
101+
t.add_column("detail")
102+
for rtype, old, new in s.rekeyed:
103+
t.add_row("rekey", f"{rtype}: {old} -> {new}")
104+
for rtype, key in s.orphans:
105+
t.add_row("orphan", f"{rtype}.{key} (state has no template)")
106+
for rtype, rid in s.unmanaged:
107+
t.add_row("unmanaged", f"{rtype}.{rid} (template has no state)")
108+
for rtype, key, target, reason in s.conflicts:
109+
t.add_row("conflict", f"{rtype}.{key} -> {target}: {reason}")
110+
console.print(t)
111+
if s.orphans or s.unmanaged or s.conflicts:
112+
console.print("[dim]Orphans/unmanaged/conflicts are reported only — never deleted or created.[/dim]")
113+
114+
actionable_templates = any(fr.status in ("rewrap", "error") for fr in report.rewraps)
115+
actionable_state = bool(s.rekeyed or s.orphans or s.unmanaged or s.conflicts)
116+
if not actionable_templates and not actionable_state:
117+
console.print("[green]Nothing to migrate — templates are v2 and state is reconciled.[/green]")

src/talonctl/commands/validate.py

Lines changed: 56 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -12,6 +12,61 @@
1212
from talonctl.utils.auth import load_credentials
1313

1414

15+
def _validate_v2_files(console) -> bool:
16+
"""Validate any v2 envelope files under resources/. Returns True if any errors.
17+
18+
Additive: only files whose first document declares apiVersion talon/v2 are
19+
handled here; v1 files are left to the existing validation path.
20+
"""
21+
import yaml
22+
from talonctl.project import find_project_root
23+
from talonctl.core.envelope import API_VERSION
24+
from talonctl.core.envelope_loader import load_envelopes
25+
from talonctl.core.envelope_validation import validate_authored_envelope, check_depends_on_cycles
26+
27+
try:
28+
resources_dir = find_project_root() / "resources"
29+
except Exception:
30+
return False
31+
if not resources_dir.exists():
32+
return False
33+
34+
all_envs = []
35+
had_errors = False
36+
for yaml_file in sorted(resources_dir.rglob("*.yaml")):
37+
try:
38+
first = next(iter(yaml.safe_load_all(yaml_file.read_text())), None)
39+
except yaml.YAMLError as e:
40+
console.print(f"[red]✗ {yaml_file}: YAML parse error: {e}[/red]")
41+
had_errors = True
42+
continue
43+
except Exception:
44+
# Non-YAML error (e.g. unreadable file) — skip defensively.
45+
continue
46+
if not (isinstance(first, dict) and first.get("apiVersion") == API_VERSION):
47+
continue # leave v1 files to the existing path
48+
try:
49+
envs = load_envelopes(yaml_file)
50+
except ValueError as e:
51+
console.print(f"[red]✗ {yaml_file}: {e}[/red]")
52+
had_errors = True
53+
continue
54+
for env in envs:
55+
errs = validate_authored_envelope(env)
56+
if errs:
57+
had_errors = True
58+
for msg in errs:
59+
console.print(
60+
f"[red]✗ {yaml_file} [{env.kind} {env.metadata.get('resource_id', '?')}]: {msg}[/red]"
61+
)
62+
all_envs.extend(envs)
63+
64+
for msg in check_depends_on_cycles(all_envs):
65+
console.print(f"[red]✗ {msg}[/red]")
66+
had_errors = True
67+
return had_errors
68+
69+
1570
@click.command()
1671
@filter_options
1772
@state_options
@@ -49,6 +104,7 @@ def validate(ctx, resources, tags, names, state_file, parse_queries):
49104
formatter.format_validation_results(results)
50105

51106
has_errors = any(errors for errors in results.values() if errors)
107+
has_errors = _validate_v2_files(console) or has_errors # additive: v2 envelope validation
52108
if has_errors:
53109
raise SystemExit(1)
54110

src/talonctl/commands/validate_query.py

Lines changed: 42 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -36,20 +36,57 @@ def validate_query(ctx, query, query_file, template):
3636
raise SystemExit(1)
3737
return
3838
try:
39+
from talonctl.core.envelope_loader import load_envelopes
40+
from talonctl.core.template_discovery import TemplateDiscovery
41+
42+
# Peek the raw YAML to derive the default resource type needed for v1
43+
# documents (v2 docs derive their type from `kind`). The raw `type`
44+
# field is only a resource category for some v1 docs; for detections
45+
# it carries the rule SUBTYPE (e.g. "behavioral"/"correlation"),
46+
# which is NOT a valid resource type. Feeding a subtype to
47+
# load_envelopes raises KeyError, so map non-category `type` values
48+
# to a real resource type before handing it off. The exact type
49+
# barely matters here — query extraction reads search/queryString
50+
# regardless — the goal is just to pass a VALID type.
3951
with open(template_path) as f:
40-
template_data = yaml.safe_load(f)
41-
search = template_data.get("search", {})
42-
resolved_query = search.get("filter") or search.get("query")
43-
if not resolved_query:
44-
resolved_query = template_data.get("queryString")
52+
raw = yaml.safe_load(f)
53+
raw = raw or {}
54+
raw_type = raw.get("type")
55+
valid_types = set(TemplateDiscovery.VALID_RESOURCE_TYPES)
56+
if raw_type in valid_types:
57+
default_resource_type = raw_type
58+
elif raw_type in {"behavioral", "correlation"} or "search" in raw:
59+
default_resource_type = "detection"
60+
elif "queryString" in raw or "query_string" in raw:
61+
default_resource_type = "saved_search"
62+
else:
63+
default_resource_type = "saved_search"
64+
65+
envelopes = load_envelopes(template_path, default_resource_type=default_resource_type)
66+
for env in envelopes:
67+
working = env.to_working_dict()
68+
search = working.get("search", {}) or {}
69+
resolved_query = search.get("filter") or search.get("query")
70+
if not resolved_query:
71+
resolved_query = working.get("queryString")
72+
if resolved_query:
73+
break
4574
if not resolved_query:
4675
console.print("INVALID: No search.filter, search.query, or queryString found in template")
4776
raise SystemExit(1)
4877
return
78+
except SystemExit:
79+
raise
4980
except yaml.YAMLError as e:
5081
console.print(f"INVALID: YAML parse error: {e}")
5182
raise SystemExit(1)
5283
return
84+
except (ValueError, KeyError) as e:
85+
# KeyError surfaces if a v1 `type`/`kind` maps to no known resource
86+
# type/kind; map it to the INVALID contract instead of a traceback.
87+
console.print(f"INVALID: {e}")
88+
raise SystemExit(1)
89+
return
5390
else:
5491
console.print("INVALID: Must specify --query, --file, or --template")
5592
raise SystemExit(1)

0 commit comments

Comments
 (0)