Skip to content

troubleshooting.md

Latest commit

 

History

History
220 lines (157 loc) · 5.52 KB

troubleshooting.md

File metadata and controls

220 lines (157 loc) · 5.52 KB
Raw

Aiden v4.5 daemon — troubleshooting

Common failure modes for the v4.5 daemon, with diagnostic commands and fixes.

"Daemon already running"

You ran aiden daemon start (or another aiden invocation tried to acquire the runtime lock) and saw:

[daemon] daemon already running: instanceId=... pid=12345

Confirm whether the listed PID is actually alive:

# Linux / macOS
ps -p 12345

# Windows
tasklist /FI "PID eq 12345"

If the PID is dead, the runtime lock is stale. Remove it:

rm ~/.aiden/daemon/runtime.lock

Then try again. (The lock is auto-released on graceful shutdown, but kill -9 and certain crash paths leave it behind.)

If the PID is alive, the daemon is already up — you don't need a second one. Use aiden daemon status to inspect, aiden daemon stop to terminate.

Webhook returns 401 despite valid HMAC

A signed POST returns 401 even though openssl dgst confirms the signature is correct. Three causes:

  1. Wrong HMAC format. GitHub uses sha256=<hex>; generic uses bare <hex>. Check the route's hmacFormat via aiden trigger show <id>.
  2. Missing header. GitHub → X-Hub-Signature-256; GitLab → X-Gitlab-Token; generic → X-Webhook-Signature.
  3. Body mutated upstream. Some proxies re-serialize JSON, which changes the body bytes the HMAC was computed over. Send the exact raw bytes through (Cloudflare, nginx, ngrok all preserve by default).

Inspect recent attempts:

sqlite3 ~/.aiden/daemon/daemon.db \
  "SELECT received_at, status_code, signature_verified FROM webhook_deliveries
   WHERE route_id = '<id>' ORDER BY received_at DESC LIMIT 10;"

IMAP authentication fails

[email] failed to start <name>: IMAP authentication failed

Most common: Gmail / Outlook now require app passwords rather than your account password. Generate one:

Use that 16-character string as --password. OAuth2 is deferred to v4.6+ — app passwords are the supported path in v4.5.

Test connectivity manually:

openssl s_client -crlf -connect imap.example.com:993
# At the prompt:
. login me@example.com myapppassword
. logout

"no daemon running" on aiden daemon stop

The runtime lock at ~/.aiden/daemon/runtime.lock is absent or points to a stale PID. If you suspect an orphaned daemon process:

# Linux / macOS
lsof -i :9301
ps aux | grep aiden

# Windows
netstat -ano | findstr :9301

Kill the orphan, then start fresh:

kill <pid>
rm ~/.aiden/daemon/runtime.lock
aiden daemon start

Drain timeout exceeded on shutdown

[daemon] drain timeout 30000ms exceeded — forcing exit

A run hung past the drain window. The dispatcher will mark it interrupted with resume_pending=1 so the next boot's crash-recovery pass picks it up. To allow longer drains:

export AIDEN_DAEMON_DRAIN_TIMEOUT_MS=120000   # 2 minutes

Set this in the systemd unit's Environment= or launchd's EnvironmentVariables for the change to persist across aiden daemon restart.

Cron jobs not firing in daemon mode

  1. Confirm the migration ran:

    sqlite3 ~/.aiden/daemon/daemon.db \
  "SELECT COUNT(*) FROM scheduled_workflows;"

    If 0 and you have rows in ~/.aiden/cron_jobs.json, the migration was skipped. Check the daemon log for [cron-migration] lines.

  2. Confirm the emitter is installed:

    aiden daemon status | grep -i emitter

  3. Misfire policy may be skipping stale fires after a long suspend:

    aiden cron show <id>      # check misfire_policy in sqlView

    Adjust if skip_stale is dropping fires you want to keep:

    aiden cron remove <id>
aiden cron add --label X --schedule ... --command ... \
  --misfire-policy run_once_if_late

File watcher misses changes

  1. NFS / Docker bind mount: native inotify/FSEvents don't work. Use --polling when adding:

    aiden trigger add file --label X --path /mnt/nfs --polling

  2. Debounce eats fast bursts: lower --debounce-ms (default 300) if you're saving rapidly:

    aiden trigger add file --label X --path . --debounce-ms 50

  3. Glob doesn't match: bare *.md is auto-prefixed to **/*.md, but src/*.md is left as-is (anchored). Verify with aiden trigger show <id> and adjust globs as needed.

High RSS / slow drain

Capture a metric snapshot:

curl -s http://127.0.0.1:9301/metrics | grep -E "rss|claim|resource"

If RSS climbs steadily over hours, run the 72-hour soak harness in tests/v4/daemon/soak/README.md and file the results with the recap shape it documents.

Port conflicts

Default AIDEN_DAEMON_PORT=9301. To use a different port:

export AIDEN_DAEMON_PORT=9401
aiden daemon restart

If you bind to a non-loopback interface, AIDEN_API_KEY is required (the bind-safety check refuses to start otherwise):

export AIDEN_DAEMON_BIND=0.0.0.0
export AIDEN_API_KEY=<32-byte-base64url>

Still stuck?

The daemon database is small enough to inspect directly:

sqlite3 ~/.aiden/daemon/daemon.db
.tables
.schema trigger_events
SELECT * FROM trigger_events WHERE status = 'dead_letter' LIMIT 10;
SELECT * FROM crash_reports ORDER BY detected_at DESC LIMIT 5;

File an issue with the output of aiden daemon status and the last 50 lines of journalctl --user -u aiden.service (or ~/Library/Logs/aiden-daemon.log on macOS).