Skip to content

Commit faabbe5

Browse files
nccatoninccatoni
authored
Update manifest/test activation doc and AI rules (#6211)
1 parent 4d7274e commit faabbe5

File tree

11 files changed

+648
-86
lines changed

11 files changed

+648
-86
lines changed

‎.cursor/rules/doc.mdc‎

Lines changed: 28 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,28 @@
1+
---
2+
description: Rules to edit the documentation
3+
globs: docs/*
4+
alwaysApply: true
5+
---
6+
# Editing the documenation
7+
8+
## Key Documentation
9+
10+
- @docs/glossary.md: Technical definitions to use
11+
12+
## Rules
13+
14+
### Links
15+
16+
Repetitions should be avoided. Prefer using links to other pages of the documentation
17+
instead of repeating information.
18+
19+
### Outdated information
20+
21+
When you find a contradiction in the documentation or when the documentation contradicts
22+
know behavior of the system signal it to the user and propose a fix.
23+
24+
### Glossary
25+
26+
When appropriate use the words from the glossary. You should avoid using similar
27+
words to avoid ambiguities. When you use many words from the glossary you can
28+
add a link to the glossary to help the reader.

‎.cursor/rules/test-activation.mdc‎

Lines changed: 212 additions & 38 deletions
Original file line numberDiff line numberDiff line change
@@ -1,54 +1,228 @@
11
---
2-
description:
2+
description: Rules for activating and deactivating tests using manifests or decorators
33
globs:
44
alwaysApply: true
55
---
6-
# Activate or deactivate tests
6+
# Test Activation and Deactivation
77

8-
For enabling/disabling tests, refer to these key documentation files:
8+
**Always prefer manifests over decorators.**
99

10-
- @docs/edit/enable-test.md: How to enable tests and test against unmerged changes
11-
- @docs/edit/manifest.md: Using manifest files for test activation/deactivation
12-
- @docs/edit/skip-tests.md: Using decorators and marking tests as skipped
13-
- @docs/edit/versions.md: Version specification guidelines for different languages
10+
## Key Documentation
1411

15-
Key points to remember to activate or deactivate tests:
12+
- @docs/edit/manifest.md: Manifest file format, entry formats, and syntax rules
13+
- @docs/edit/skip-tests.md: Decorator usage (when manifests cannot be used)
14+
- @docs/edit/enable-test.md: How to enable tests
15+
- @docs/edit/versions.md: Version specification guidelines
1616

17-
1. For test classes/files: Use manifest files in `manifests/` directory
18-
2. For individual test methods: Use decorators in test files
19-
3. Always include JIRA references for bugs using the `reason` parameter.
20-
4. Never assume that not adding a test to a language's manifest will disable it - tests run by default unless explicitly disabled
21-
5. Entries in the manifest file MUST be sorted in alphabetical order. After edit a manifest file ALWAYS run the scenario "TEST_THE_TEST" and format.sh before committing changes to ensure code follows the manifest rules.
22-
6. When using @missing_feature/@irrelevant decorators, the condition specifies when to SKIP the test.
23-
7. For update a manifest yml file, never add fields that don't exist in any other manifests files. Use the documentation [manifests files specification](mdc:docs/edit/manifest.md) to be accurate in your modifications.
17+
---
18+
19+
## Decision: Manifest vs Decorator
20+
21+
```
22+
Does the condition depend ONLY on:
23+
- Library name (java, python, nodejs, etc.)
24+
- Library version
25+
- Weblog variant
26+
- Agent version (alone)
27+
?
28+
│
29+
├─► YES → Use MANIFEST (preferred)
30+
│ - Library conditions → manifests/{library}.yml
31+
│ - Agent version conditions → manifests/agent.yml
32+
│
33+
└─► NO (uses context.scenario, context.vm_name,
34+
or other attributes, or complex combinations)
35+
→ Use DECORATOR (only option)
36+
```
37+
38+
---
39+
40+
## Manifest Rules
41+
42+
### Do NOT Create New Manifest Files
43+
44+
The existing manifests cover all supported libraries. If creating a new manifest seems needed:
45+
1. STOP and ask the user first
46+
2. Suggest asking in **#apm-shared-testing** Slack channel
47+
3. Only proceed if explicitly confirmed
48+
4. If confirmed, also update `utils/manifest/_internal/parser.py`
49+
50+
### Using YAML References (Anchors)
51+
52+
is a repeating pattern in your modifications.
53+
54+
References are defined in the `refs:` section at the top of manifests:
55+
56+
```yaml
57+
refs:
58+
- &ref_5_0_0 v5.0.0
59+
- &ref_5_6_0 '>=5.6.0'
60+
- &django_weblogs "django-poc, django-py3.13, python3.12"
61+
```
62+
63+
**Rules:**
64+
1. Check for existing references before adding a version
65+
2. Verify what a reference contains before using it
66+
3. Don't mix reference types (version refs vs declaration refs)
67+
4. You should not change existing references without explicit approval from the user.
68+
5. You can create new references only in manifests that already have references and
69+
with approval from the user. You should propose creating new references when there
70+
71+
**Mistake to avoid:**
72+
```yaml
73+
# WRONG - using a version ref where a declaration is expected
74+
tests/path/test.py::TestClass:
75+
- declaration: *ref_5_0_0 # ref_5_0_0 is "v5.0.0", not a valid declaration!
76+
component_version: '<5.0.0'
77+
78+
# CORRECT
79+
tests/path/test.py::TestClass:
80+
- declaration: missing_feature (Description)
81+
component_version: '<5.0.0'
82+
```
83+
84+
### After Modifying Manifests
85+
86+
**ALWAYS run `./format.sh` after any manifest modification.**
87+
88+
---
89+
90+
## Decorator Rules
91+
92+
### Version Format
93+
94+
**CRITICAL**: Always use `library@version` format:
95+
96+
```python
97+
# CORRECT
98+
@missing_feature(context.library < "python@2.5.0", reason="...")
99+
100+
# WRONG - missing library specifier
101+
@missing_feature(context.library < "2.5.0", reason="...")
102+
```
103+
104+
### Always Include Reason
105+
106+
** Rules **
107+
1. When you don't know the reason you should ask the user
108+
2. Bugs must reference a valid JIRA ticket, you must ask the user for one
109+
110+
```python
111+
# CORRECT
112+
@missing_feature(condition, reason="Feature not implemented yet")
113+
@bug(condition, reason="JIRA-1234")
114+
115+
# WRONG - missing reason
116+
@missing_feature(condition)
117+
```
118+
119+
---
120+
121+
## Quick Reference: User Request → Action
122+
123+
| User Request | Use | Format |
124+
|--------------|-----|--------|
125+
| "Bug for {library}" | Manifest | `bug (JIRA-XXX)` in `{library}.yml` |
126+
| "Bug for {library} >= X.Y.Z" | Manifest | Explicit with `component_version` |
127+
| "Feature available from X.Y.Z" | Manifest | `vX.Y.Z` |
128+
| "Irrelevant for {library}" | Manifest | `irrelevant (reason)` |
129+
| "Flaky for {library}" | Manifest | `flaky (JIRA-XXX)` |
130+
| "Bug on {weblog} variant" | Manifest | `weblog_declaration` format |
131+
| "Bug for agent version X" | Manifest | Entry in `agent.yml` |
132+
| "Bug for scenario X" | **Decorator** | `@bug(context.scenario == ...)` |
133+
| "Bug on specific VM" | **Decorator** | `@bug(context.vm_name == ...)` |
134+
| "Bug for agent + library combo" | **Decorator** | Complex condition |
135+
| "Skip test completely" | **Decorator** | With `force_skip=True` |
136+
137+
---
138+
139+
## Common Mistakes to Avoid
140+
141+
1. ❌ Using decorators when manifests can express the condition
142+
2. ❌ Forgetting the `reason` parameter in decorators
143+
3. ❌ Using `library < "2.5.0"` instead of `library < "python@2.5.0"`
144+
4. ❌ Not sorting manifest entries alphabetically
145+
5. ❌ Using `false` or invalid markers in manifests (use only: `irrelevant`, `bug`, `flaky`, `missing_feature`, `incomplete_test_app`)
146+
6. ❌ Forgetting to quote values with special YAML characters (`>`, `<`, `:`, `#`)
147+
7. ❌ Creating new manifest files without asking first
148+
8. ❌ Using `X.Y.Z` instead of `vX.Y.Z` for versions (see warning below)
149+
150+
### Version Format Warning
151+
152+
**If you see a version like `1.2.0` (without `v` prefix) in a manifest, ASK THE USER if this is intentional:**
153+
154+
- `vX.Y.Z` = `>=X.Y.Z` (enabled for this version AND all future versions) ✓ Common case
155+
- `X.Y.Z` = exactly that version only ⚠️ Rarely intended
156+
157+
```yaml
158+
# COMMON - enables test for v1.2.0 and all later versions
159+
tests/path/test.py::TestClass: v1.2.0
160+
161+
# RARE - enables test ONLY for exactly version 1.2.0 (not 1.2.1, not 1.3.0)
162+
tests/path/test.py::TestClass: 1.2.0 # ⚠️ Ask user: "Did you mean v1.2.0?"
163+
```
164+
165+
---
166+
167+
## Examples
168+
169+
### Example 1: Deactivate for library (all versions)
170+
171+
**Request:** "Test X is irrelevant for java"
172+
173+
**Action:** Add to `manifests/java.yml`:
174+
```yaml
175+
tests/path/test.py::TestClass::test_method: irrelevant (Not applicable to Java)
176+
```
177+
178+
### Example 2: Bug from specific version
179+
180+
**Request:** "Bug in test X for nodejs >= 5.0.0, ticket APPSEC-123"
181+
182+
**Action:** Add to `manifests/nodejs.yml`:
183+
```yaml
184+
tests/path/test.py::TestClass::test_method:
185+
- declaration: bug (APPSEC-123)
186+
component_version: '>=5.0.0'
187+
```
188+
189+
### Example 3: Feature available from version
190+
191+
**Request:** "Test X requires python 2.5.0 or higher"
192+
193+
**Action:** Add to `manifests/python.yml`:
194+
```yaml
195+
tests/path/test.py::TestClass::test_method: v2.5.0
196+
```
197+
198+
### Example 4: Weblog-specific issue
24199

25-
## CRITICAL: Decorator Usage Rules
200+
**Request:** "Test X fails on fastify for nodejs"
26201

27-
8. **Version Format**: ALWAYS use `library@version` format in version comparisons (e.g., `context.library < "python@2.5.0"`, NOT `context.library < "2.5.0"`). The language specifier is MANDATORY.
202+
**Action:** Add to `manifests/nodejs.yml`:
203+
```yaml
204+
tests/path/test.py::TestClass:
205+
- weblog_declaration:
206+
"*": v5.0.0
207+
fastify: bug (JIRA-123)
208+
```
28209

29-
9. **Reason Parameter**: ALWAYS include the `reason` parameter for decorators:
30-
- `@missing_feature(condition, reason="Feature not implemented yet")`
31-
- `@bug(condition, reason="JIRA-1234")`
32-
- `@irrelevant(condition, reason="Not applicable to this language")`
33-
- `@incomplete_test_app(condition, reason="Weblog endpoint missing")`
210+
### Example 5: Scenario-specific (requires decorator)
34211

35-
10. **Decorator Behavior Differences**:
36-
- `@missing_feature` and `@bug`: Test STILL RUNS, results in XPASS (if passes) or XFAIL (if fails)
37-
- `@irrelevant`: Test is SKIPPED completely, never runs
38-
- `@flaky`: Test is NOT executed by default
39-
- Use `force_skip=True` parameter to prevent execution of `@missing_feature` or `@bug` decorated tests
212+
**Request:** "Test X doesn't apply to profiling scenario"
40213

41-
11. **Manifest Disable Markers**: To disable tests in manifest files, use ONLY these valid markers:
42-
- `irrelevant` (with optional comment/link)
43-
- `bug` (with JIRA reference in comment)
44-
- `flaky`
45-
- `missing_feature`
46-
- NEVER use `false` or other arbitrary values
214+
**Action:** Add decorator in test file:
215+
```python
216+
@irrelevant(context.scenario == scenarios.profiling, reason="Not applicable to profiling")
217+
def test_method(self):
218+
pass
219+
```
47220

48-
12. **Language-Specific Conditions**: When using decorators with version checks, ensure the condition properly targets the intended language. Example: `@missing_feature(context.library < "python@2.5.0")` will NOT skip for other languages like Node.js or Java.
221+
### Example 6: Multiple libraries
49222

50-
For common test activation patterns and examples of enabling/disabling tests, see:
223+
**Request:** "Test X is irrelevant for both python and nodejs"
51224

52-
1. @docs/edit/manifest.md#example
53-
2. @docs/edit/versions.md#use-cases
54-
3. @docs/edit/skip-tests.md#decorators
225+
**Action:** Add to BOTH `manifests/python.yml` AND `manifests/nodejs.yml`:
226+
```yaml
227+
tests/path/test.py::TestClass: irrelevant (Reason)
228+
```

‎.promptfoo/tests_activate_tests.yaml‎

Lines changed: 37 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -21,4 +21,40 @@
2121
- "golang.yml"
2222
- "dotnet.yml"
2323
- "cpp_httpd.yml"
24-
- "cpp_nginx.yml"
24+
- "cpp_nginx.yml"
25+
- description: "[test activation] Scenario-specific condition requires decorator"
26+
vars:
27+
user_prompt: "I want to mark the test class Test_RaspSqli as a bug only when running on the APPSEC_RASP scenario. The JIRA ticket is APPSEC-789."
28+
assert:
29+
- type: icontains-all
30+
value:
31+
- "decorator"
32+
- "context.scenario"
33+
- "@bug"
34+
- "APPSEC-789"
35+
- type: not-icontains
36+
value: "manifests/java.yml"
37+
- description: "[test activation] Weblog-specific bug in manifest"
38+
vars:
39+
user_prompt: "Test Test_Endpoint_Discovery works on express4 from version 5.0.0 but fails on fastify for nodejs. The JIRA ticket for fastify is APPSEC-456."
40+
assert:
41+
- type: icontains-all
42+
value:
43+
- "manifests/nodejs.yml"
44+
- "weblog_declaration"
45+
- "fastify"
46+
- "bug"
47+
- "APPSEC-456"
48+
- "v5.0.0"
49+
- description: "[test activation] Version format with v prefix in manifest"
50+
vars:
51+
user_prompt: "I need to activate the test Test_NewFeature for the python tracer starting from version 2.15.0 and all future versions. What should I put in the manifest?"
52+
assert:
53+
- type: icontains-all
54+
value:
55+
- "manifests/python.yml"
56+
- "v2.15.0"
57+
- type: icontains-any
58+
value:
59+
- "format.sh"
60+
- "./format.sh"

‎AGENTS.md‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -21,6 +21,7 @@ Read and follow these rules in EVERY interaction:
2121
7. **`.cursor/rules/end-to-end-testing.mdc`** - End-to-end testing guidelines
2222
8. **`.cursor/rules/k8s-ssi.mdc`** - Kubernetes library injection testing
2323
9. **`.cursor/rules/test-activation.mdc`** - Test activation/deactivation rules
24+
10. **`.cursor/rules/doc.mdc`** - Rules for editing the documentation
2425

2526
## Manual Rules (Apply Only When Explicitly Requested)
2627

0 commit comments

Comments
 (0)