What it means: Watchflow has multiple AI agents (like workers), but they work alone. They don't coordinate.
Real-world example: Imagine you have 3 security guards, but they never talk. Guard 1 sees something suspicious, but Guard 2 doesn't know about it. They can't work together to solve complex problems.
Why it matters: For complex rules that need multiple checks, the agents can't combine their knowledge. They each do their own thing independently.
What needs to happen: Make agents work together. When one agent finds something, others should know. They should be able to discuss and make better decisions together.
What it means: If someone breaks a rule, Watchflow might tell you about it 5 times instead of once.
Real-world example: Like getting 5 emails about the same meeting reminder. Annoying, right?
Why it matters: Developers get spammed with the same violation messages. It's noise, not useful information.
What needs to happen: Track what violations have already been reported. If we've seen this exact violation before, don't report it again (or at least mark it as "already reported").
What it means: Watchflow makes the same wrong decisions over and over. It doesn't learn.
Real-world example: Like a teacher who keeps giving the same wrong answer to students, never learning from feedback.
Why it matters: If Watchflow incorrectly blocks a PR (false positive), it will keep doing it. If it misses a real violation (false negative), it keeps missing it. No improvement over time.
What needs to happen: When developers say "this was wrong" or "this was right", Watchflow should remember and adjust. Over time, it gets smarter.
What it means: When something goes wrong, the system sometimes says "everything is fine" instead of "something broke."
Real-world example: Your car's check engine light is broken, so it never lights up even when there's a problem. You think everything is fine, but it's not.
Why it matters: If a validator (rule checker) crashes, Watchflow might say "no violations found" when really it just couldn't check. This is dangerous - it looks like everything passed, but actually we don't know.
What needs to happen: Clearly distinguish between:
- ✅ "Rule passed - everything is good"
- ❌ "Rule failed - violation found"
⚠️ "Error - couldn't check, need to investigate"
What it means: In programming, there are "abstract" classes - templates that other classes must fill in. Currently, if someone forgets to fill in a required part, the code just says "pass" (do nothing) instead of raising an error.
Real-world example: Like a job application form where you can skip required fields and it still accepts it, instead of saying "you must fill this out."
Why it matters: If a developer forgets to implement something, the code will silently fail later, making it hard to debug.
What needs to happen: Change pass to raise NotImplementedError so if someone forgets to implement something, they get an immediate, clear error message.
What it means: Many parts of the code don't have automated tests to verify they work correctly.
Real-world example: Like a car manufacturer that only tests the engine, but never tests the brakes, steering, or lights.
Why it matters: When you change code, you don't know if you broke something. Tests catch bugs before they reach production.
What needs to happen: Write tests for:
- Acknowledgment agent (handles when developers say "I know about this violation")
- Repository analysis agent (analyzes repos to suggest rules)
- Deployment processors (handles deployment events)
- End-to-end workflows (test the whole process from PR to decision)
What it means: You can't create complex rules like "Block if (author is X AND file is /auth) OR (author is Y AND it's weekend)"
Real-world example: Like a security system that can check "is door locked?" OR "is window closed?" but can't check "is door locked AND window closed at the same time?"
Why it matters: Real-world policies are complex. You might want: "Prevent John from modifying the authentication code, unless it's an emergency and he has approval." That needs multiple conditions combined.
What needs to happen: Add support for combining validators with AND, OR, and NOT operators. Allow nested conditions.
What it means: The system uses exactly 5 workers (background processes) to handle tasks. This number is written in code, not configurable.
Real-world example: Like a restaurant that always has exactly 5 waiters, even if it's super busy (needs 10) or empty (needs 1).
Why it matters: Can't scale up when busy, wastes resources when idle.
What needs to happen: Make worker count configurable via environment variable. Allow auto-scaling based on load.
What it means: The system caches (stores) some data to avoid re-fetching it, but we don't know:
- How long data is cached
- When cache is cleared
- How much memory is used
Real-world example: Like a library that caches books, but you don't know how long books stay in cache, when they're removed, or if the cache is full.
Why it matters: Without understanding caching, you can't optimize performance or debug issues.
What needs to happen: Document the caching strategy. Make cache settings (TTL, size limits) configurable.
What it means: Every time Watchflow uses AI (LLM), it costs money. There's no clear strategy to reduce these costs.
Real-world example: Like making expensive phone calls every time you need information, instead of writing it down and reusing it.
Why it matters: AI calls are expensive. If you're checking 100 PRs per day, costs add up quickly.
What needs to happen:
- Track how much each AI call costs
- Cache similar rule evaluations (if we checked this before, reuse the result)
- Batch multiple rules together when possible
What it means: Documentation says "Prometheus and Grafana" but they're not actually implemented.
Real-world example: Like a car with no dashboard - you can't see speed, fuel level, or if the engine is overheating.
Why it matters: In production, you need to know:
- Is the system healthy?
- How fast are responses?
- How many errors are happening?
- How much is this costing?
What needs to happen:
- Add Prometheus metrics endpoint (exposes metrics)
- Create Grafana dashboards (visualize metrics)
- Track: response times, error rates, AI costs, cache performance
What it means: Lots of debug logs everywhere, but no clear structure. Hard to find what you need.
Real-world example: Like a diary with no dates, no organization, just random thoughts scattered everywhere.
Why it matters: When something breaks in production, you need to find the relevant logs quickly. Too much noise makes it hard.
What needs to happen:
- Standardize log levels (INFO for normal operations, DEBUG for development)
- Use structured logging (JSON format, easier to search)
- Add correlation IDs (track one request across multiple log entries)
What it means: Documentation says "complete audit trail" but it's unclear where logs are stored, how long they're kept, or how to search them.
Real-world example: Like a security camera system that records everything, but you don't know where the recordings are stored, how long they're kept, or how to find a specific event.
Why it matters: For compliance (SOC2, GDPR, etc.), you need to prove what decisions were made and why. You need to be able to search and retrieve audit logs.
What needs to happen:
- Implement audit log storage (database or file-based)
- Define retention policy (how long to keep logs)
- Add search/query API for audit logs
What it means: GitHub App private keys are stored as base64-encoded environment variables.
Real-world example: Like writing your password on a sticky note and putting it on your desk. It works, but not secure.
Why it matters: If environment variables are logged, exposed in error messages, or accessed by unauthorized people, secrets are compromised.
What needs to happen:
- Use a secret management service (AWS Secrets Manager, HashiCorp Vault)
- Support secret rotation (change keys periodically)
- Never log secrets, even in debug mode
What it means: Documentation describes a "Decision Orchestrator" that combines rule-based and AI-based decisions, but it doesn't actually exist in code.
Real-world example: Like a recipe that says "combine ingredients in the mixer" but you don't have a mixer - you're just mixing by hand inconsistently.
Why it matters: Without a central orchestrator, decisions are made inconsistently. Sometimes rules win, sometimes AI wins, but there's no smart way to combine them.
What needs to happen: Build the Decision Orchestrator that:
- Takes input from both rule engine and AI agents
- Intelligently combines them (maybe rules for simple cases, AI for complex)
- Handles conflicts (what if rule says "pass" but AI says "fail"?)
What it means: Watchflow only works with GitHub. Documentation mentions GitLab and Azure DevOps as future features, but they're not implemented.
Real-world example: Like a phone that only works with one carrier, when you could support multiple carriers and reach more customers.
Why it matters: Limits market reach. Many companies use GitLab or Azure DevOps.
What needs to happen:
- Abstract the provider interface (make it easy to add new platforms)
- Implement GitLab support
- Implement Azure DevOps support
What it means: All agents are general-purpose. There are no specialized agents for security, compliance, or performance.
Real-world example: Like having general doctors but no specialists. A general doctor can help, but a cardiologist is better for heart problems.
Why it matters: Specialized agents would be better at their specific domains. A security agent would understand security patterns better than a general agent.
What needs to happen:
- Create security-focused agent (specializes in security rules)
- Create compliance-focused agent (specializes in compliance rules)
- Create performance-focused agent (specializes in performance rules)
What it means: FastAPI auto-generates API docs, but they're missing examples, error codes, and rate limiting info.
Real-world example: Like a product manual that lists features but doesn't show how to use them or what to do when something goes wrong.
Why it matters: Developers using the API need clear examples and error handling guidance.
What needs to happen: Enhance API documentation with:
- Example requests and responses
- All possible error codes and what they mean
- Rate limiting information (how many requests per minute)
What it means: Configuration options are spread across multiple files. Hard to know all available options.
Real-world example: Like settings for your phone scattered across 10 different menus instead of one settings page.
Why it matters: Hard to configure the system. You might miss important settings.
What needs to happen:
- Create comprehensive configuration guide
- Add configuration validation (warn if settings are wrong)
- Provide examples for common scenarios
What it means: No tests to see how the system performs under heavy load (many PRs at once).
Real-world example: Like opening a restaurant without testing if the kitchen can handle a full house.
Why it matters: In production, you might get 100 PRs at once. Will the system handle it? Will it crash? Slow down? We don't know.
What needs to happen:
- Add load testing with Locust (mentioned in docs but not implemented)
- Define performance SLAs (e.g., "must respond in < 2 seconds")
- Add performance regression tests (make sure new code doesn't slow things down)
What it means: All tests use mocks (fake GitHub API). Never tested against real GitHub.
Real-world example: Like practicing driving in a parking lot but never on real roads. It's good practice, but real conditions are different.
Why it matters: Real GitHub API might behave differently than mocks. API might change. We need to know it actually works.
What needs to happen:
- Add optional integration tests with real GitHub (behind a flag, so they don't run in CI by default)
- Use a test GitHub App for CI/CD
- Test against GitHub API changes
What it means: Users can't create their own custom agents. They're stuck with what Watchflow provides.
Real-world example: Like a LEGO set with fixed pieces - you can only build what the instructions say, not your own creations.
Why it matters: Different companies have different needs. They should be able to create custom agents for their specific use cases.
What needs to happen:
- Create agent plugin system (allow users to add custom agents)
- Provide agent development SDK (tools to build agents)
- Add examples of custom agents
What it means: Documentation mentions analytics, but there's no dashboard to see:
- Which rules are violated most often?
- How many false positives?
- How effective are rules?
Real-world example: Like a business with no sales reports. You don't know what's working and what's not.
Why it matters: Can't measure effectiveness. Can't improve. Can't show value to management.
What needs to happen:
- Build analytics dashboard
- Track: violation rates, acknowledgment patterns, false positive rates
- Show trends over time
What it means: When you change a rule, there's no history. Can't see what changed, when, or rollback if something breaks.
Real-world example: Like editing a document without "track changes" - you can't see what you changed or go back.
Why it matters: If a rule change breaks things, you need to rollback quickly. You also need to see rule history for compliance.
What needs to happen:
- Add rule versioning (track all changes)
- Add rollback capability (revert to previous version)
- Track who changed what and when
What it means: If a validator crashes, the system says "no violation found" instead of "error occurred."
Real-world example: Like a smoke detector that breaks and just stays silent. You think everything is fine, but it's actually broken.
Why it matters: Dangerous - looks like rules passed, but actually we don't know.
What needs to happen: Return error state instead of treating as "passed." Maybe block PR to be safe, or retry.
What it means: When AI returns a response, sometimes it's malformed (truncated JSON). The fallback logic is complex and might miss violations.
Real-world example: Like a translator that sometimes gets cut off mid-sentence, and you have to guess what they meant.
Why it matters: Might miss real violations if parsing fails.
What needs to happen: Improve error handling and retry logic for malformed responses.
What it means: Code starts the deployment scheduler twice (line 44 and line 68). It's safe (has a check), but redundant and confusing.
Real-world example: Like pressing the "start" button twice on your car - it's already running, so nothing happens, but why press it twice?
Why it matters: Confusing code. Future developers might think it's intentional and add more redundant code.
What needs to happen: Remove one of the calls. Keep the one with the safety check.
- Agent Coordination - Make agents work together
- Regression Prevention - Stop duplicate violation reports
- Error Handling - Don't hide errors as "passed"
- Test Coverage - Test all the things
- Learning Agent - Learn from feedback
- Decision Orchestrator - Smart decision combining
- Monitoring - Know what's happening
- Validator Combinations - Support complex rules
- Enterprise Policies - More rule types
- Cross-Platform - Support GitLab/Azure DevOps
- Custom Agents - Let users build their own
- Analytics - Measure effectiveness
- Agent Specialization - Specialized agents
- Rule Versioning - Track rule changes
- Performance - Optimize costs and speed