Add scylla-monitor-ctl: Go CLI for monitoring stack management

[dkropachev]

Summary

Single-binary Go CLI (scylla-monitor-ctl) that replaces the shell-script-based monitoring stack management. Handles dashboard generation, container orchestration, configuration, migration, and health checking.

Commands

Command	What it does
deploy	Start the full monitoring stack (Prometheus, Grafana, AlertManager, Loki)
destroy	Stop and remove all stack containers
status	Show container-level status of all components
check	API-level health checks (Grafana, Prometheus, AlertManager, datasources, alerts)
configure	Set up datasources and dashboards on an existing Grafana
upgrade	Push new dashboards to a running stack without restart
dashboards generate	Generate Grafana JSON from templates (replaces make_dashboards.py)
dashboards upload/download/list	Manage dashboards via Grafana API
migrate export	Export a running stack (configs + dashboards + optional TSDB data)
migrate import PATH	Import from an export archive or unpacked directory
migrate clone	Clone a running stack to new ports in a single operation
migrate copy	Live-copy dashboards/datasources between two Grafana instances
prometheus config	Generate prometheus.yml from template
prometheus reload	Hot-reload Prometheus config
tune	Adjust scrape intervals, drop metrics, toggle native histograms on the fly
targets generate	Generate Prometheus target YAML from a node list

Examples

1. Deploy a monitoring stack for ScyllaDB 6.2

scylla-monitor-ctl deploy \
  --scylla-version 6.2 \
  --targets-file prometheus/scylla_servers.yml \
  --data-dir /var/lib/prometheus

2. Clone a running stack

scylla-monitor-ctl migrate clone \
  --grafana-url http://localhost:3000 \
  --prometheus-url http://localhost:9090 \
  --prometheus-port 9091 \
  --grafana-port 3001 \
  --alertmanager-port 9095

Auto-discovers targets from source Prometheus, deploys a new stack, rewrites datasource URLs to container-internal addresses, uploads dashboards. One command.

3. Check stack health

$ scylla-monitor-ctl check
COMPONENT      CHECK                STATUS DETAIL
--------------------------------------------------------------------------------
Grafana        API Health           [OK]   http://localhost:3000
Grafana        Dashboards           [OK]   10 dashboards loaded
Prometheus     API Health           [OK]   http://localhost:9090
Prometheus     Scrape targets       [OK]   3 up, 0 down
AlertManager   API Health           [OK]   http://localhost:9093
Datasource     prometheus           [OK]   prometheus -> http://localhost:9090
Prometheus     Alerts               [OK]   no alerts firing

4. Export and import

# Export (metric data included when --prometheus-url is provided)
scylla-monitor-ctl migrate export \
  --grafana-url http://localhost:3000 \
  --prometheus-url http://localhost:9090

# Import from archive or unpacked directory
scylla-monitor-ctl migrate import ./stack-export.tar.gz \
  --grafana-url http://localhost:3001 \
  --prometheus-url http://aprom-9091:9090

5. Generate dashboards for a specific version

scylla-monitor-ctl dashboards generate --scylla-version 6.2

6. Live-copy dashboards between two Grafana instances

scylla-monitor-ctl migrate copy \
  --source-grafana-url http://prod-grafana:3000 \
  --target-grafana-url http://staging-grafana:3000

7. Drop expensive metrics and reload

scylla-monitor-ctl tune \
  --drop-metrics cas,cdc \
  --scrape-interval 30s \
  --reload

Key implementation details

• Datasource upsert: migrate import and migrate clone use upsert (create-or-update) for datasources, so URL rewrites always take effect even when the datasource already exists
• Network-aware clone: migrate clone auto-discovers targets from the Prometheus API, mounts them at the correct container-internal paths, and rewrites datasource URLs to container names
• Archive or directory: migrate import accepts both .tar.gz archives and unpacked directories
• No backup command: removed — it was a duplicate of migrate export/import with slightly different defaults

Test plan

• go build ./... passes
• go vet ./... passes
• go test ./... — all unit tests pass
• check command verified against live stack (all OK)
• migrate clone end-to-end: clone → check → all services healthy, scraping targets
• migrate import with archive and with unpacked directory
• Datasource upsert verified: URL rewrite takes effect on existing datasources
• CI workflow (build-and-test + integration-test) passes