-
Notifications
You must be signed in to change notification settings - Fork 551
fix(runtime): ensure stop flag is set for policy violations in parallel rails #1467
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: develop
Are you sure you want to change the base?
Conversation
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Greptile Overview
Greptile Summary
This PR fixes a critical bug in the parallel rails execution path where policy violations (stopped rails) were not properly recorded in the processing log. The fix ensures that when a rail detects a policy violation and stops, its processing log is now included in the aggregated result and the OutputRailFinished
/InputRailFinished
post-event is correctly omitted. This allows compute_generation_log()
to create the required ActivatedRail
entry with stop=True
, enabling client code to identify which specific guardrail triggered and why.
How it works:
- In
runtime.py
, when a parallel rail stops (detected via a "stop"BotIntent
event), the code now captures that rail's processing log before canceling the task - The stopped rail's processing log is filtered (removing
Listen
andstart_flow
events) and merged into the main processing log - Post-events are now conditionally appended only when no stop event is present, preventing invalid event sequences
- Five new test cases verify that stopped rails appear in
activated_rails
withstop=True
, exactly one rail is marked as stopped per violation, and stopped/non-stopped rails are mutually exclusive
Potential Issues:
- Lines 446-453: The filtering logic that removes
Listen
andstart_flow
events from stopped task logs may be too aggressive—if these events contain critical context about why the rail stopped, discarding them could make debugging harder - Lines 315-324: The stop detection relies on string matching
event["intent"] == "stop"
without validating the event structure; if the event dictionary is malformed (missing "intent" key), this will raise aKeyError
- Test coverage gap: The new tests only validate the Colang 1.0 runtime path but the fix is isolated to
v1_0/runtime/runtime.py
; if parallel rails exist in v2_0, they may still have the bug - Line 389: The
stopped_task_processing_log
is captured after the task is canceled—there's a potential race condition where the processing log could be incomplete if the cancellation happens before all events are recorded
Confidence: 3/5 — The fix correctly addresses the stated problem and test coverage is good, but the event filtering logic and lack of error handling for malformed events introduce risk. Recommend adding validation for the event structure and clarifying whether Listen
/start_flow
events should really be discarded from stopped rails.
2 files reviewed, 1 comment
Codecov Report❌ Patch coverage is
📢 Thoughts on this report? Let us know! |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Pull Request Overview
This PR fixes a bug where policy violations in parallel rails were not properly tracked. Specifically, when a rail stopped due to a policy violation, its processing log was excluded from aggregation, preventing the creation of an ActivatedRail
entry for the stopped rail. Additionally, the OutputRailFinished
/InputRailFinished
events were incorrectly added even when a rail stopped.
Key Changes:
- Modified the
task_call_helper
to check for stop events and conditionally add post_events - Added tracking of stopped rail processing logs to include them in the final aggregation
- Added comprehensive test coverage for parallel rail stop behavior
Reviewed Changes
Copilot reviewed 2 out of 2 changed files in this pull request and generated 3 comments.
File | Description |
---|---|
tests/test_parallel_rails.py |
Adds four new test cases to verify stop flag behavior for input rails, output rails, client code patterns, and multiple activated rails |
nemoguardrails/colang/v1_0/runtime/runtime.py |
Implements stop event detection, conditional post_event addition, and stopped rail processing log aggregation |
Tip: Customize your code reviews with copilot-instructions.md. Create the file or learn how to get started.
19a574f
to
c3594ae
Compare
Co-authored-by: Copilot <[email protected]> Signed-off-by: Pouyan <[email protected]>
When rails ran in parallel, the stopped rail's processing log was excluded from aggregation, causing
compute_generation_log()
to never create anActivatedRail
entry for the stopped rail. Additionally,the
post_event
(OutputRailFinished
/InputRailFinished
) was incorrectly added even when a rail stopped.