[FR] Reset deprecated lock to the latest state during lock

[Mikaayenson]

Pull Request

Issue link(s): #5714

Summary - What I changed

After each branch except the last, reset deprecated_rules.json to the branch's committed state. This prevents branch-specific deprecation state from leaking across checkouts. The last branch (e.g. 9.3) is closest to main and carries the correct deprecation state, including any legitimate new deprecations, back to main for the PR.

• Removing D4C from _deprecated/ on 8.19 release branch breaks existing deprecation steel threads for 8.x users and orphans version lock previous["8.19"] entries which will require more changes to address
• Adding D4C entries to deprecated_rules.json on release branches is problematic because the file is shared across release branches.

How To Test

Here's a test script to simulate the release (executing just the shell commands).

Simulation Script

# From main, verify the reset logic works
git stash  # save any other changes
BRANCHES=(8.19 9.1 9.2 9.3)
LAST_INDEX=$((${#BRANCHES[@]} - 1))

for i in "${!BRANCHES[@]}"; do
    BRANCH=${BRANCHES[$i]}
    echo "=== $BRANCH ==="
    git checkout $BRANCH
    git pull

    # Simulate what manage_versions does: add a marker to deprecated_rules.json
    python3 -c "
import json
with open('detection_rules/etc/deprecated_rules.json') as f:
    d = json.load(f)
d['TEST_MARKER_$BRANCH'] = {'rule_name': 'test', 'stack_version': '$BRANCH', 'deprecation_date': '2025/01/01'}
with open('detection_rules/etc/deprecated_rules.json', 'w') as f:
    json.dump(d, f, indent=2, sort_keys=True)
"
    echo "After build: $(python3 -c "import json; d=json.load(open('detection_rules/etc/deprecated_rules.json')); print([k for k in d if k.startswith('TEST_MARKER')])")"

    if [ $i -lt $LAST_INDEX ]; then
        git checkout -- detection_rules/etc/deprecated_rules.json
    fi

    echo "After reset: $(python3 -c "import json; d=json.load(open('detection_rules/etc/deprecated_rules.json')); print([k for k in d if k.startswith('TEST_MARKER')])")"
done

git checkout main
echo "=== Back on main ==="
echo "Final state: $(python3 -c "import json; d=json.load(open('detection_rules/etc/deprecated_rules.json')); print([k for k in d if k.startswith('TEST_MARKER')])")"
git checkout -- detection_rules/etc/deprecated_rules.json  # cleanup
git stash pop 2>/dev/null

Checklist

• Added a label for the type of pr: bug, enhancement, schema, maintenance, Rule: New, Rule: Deprecation, Rule: Tuning, Hunt: New, or Hunt: Tuning so guidelines can be generated
• Added the meta:rapid-merge label if planning to merge within 24 hours
• Secret and sensitive material has been managed correctly
• Automated testing was updated or added to match the most common scenarios
• Documentation and comments were added for features that require explanation

[github-actions]

Enhancement - Guidelines

These guidelines serve as a reminder set of considerations when addressing adding a feature to the code.

Documentation and Context

• Describe the feature enhancement in detail (alternative solutions, description of the solution, etc.) if not already documented in an issue.
• Include additional context or screenshots.
• Ensure the enhancement includes necessary updates to the documentation and versioning.

Code Standards and Practices

• Code follows established design patterns within the repo and avoids duplication.
• Ensure that the code is modular and reusable where applicable.

Testing

• New unit tests have been added to cover the enhancement.
• Existing unit tests have been updated to reflect the changes.
• Provide evidence of testing and validating the enhancement (e.g., test logs, screenshots).
• Validate that any rules affected by the enhancement are correctly updated.
• Ensure that performance is not negatively impacted by the changes.
• Verify that any release artifacts are properly generated and tested.
• Conducted system testing, including fleet, import, and create APIs (e.g., run make test-cli, make test-remote-cli, make test-hunting-cli)

Additional Checks

• Verify that the enhancement works across all relevant environments (e.g., different OS versions).
• Confirm that the proper version label is applied to the PR patch, minor, major.

[eric-forte-elastic]

Simulation Testing 🟢

❯ bash simulation.bash 
Saved working directory and index state WIP on 5714-fr-d4c-rules-are-erroneously-added-to-deprecated_rulesjson: ef02eed9c bump version
=== 8.19 ===
Branch '8.19' set up to track remote branch '8.19' from 'origin'.
Switched to a new branch '8.19'
Already up to date.
After build: ['TEST_MARKER_8.19']
After reset: []
=== 9.1 ===
Branch '9.1' set up to track remote branch '9.1' from 'origin'.
Switched to a new branch '9.1'
Already up to date.
After build: ['TEST_MARKER_9.1']
After reset: []
=== 9.2 ===
Branch '9.2' set up to track remote branch '9.2' from 'origin'.
Switched to a new branch '9.2'
Already up to date.
After build: ['TEST_MARKER_9.2']
After reset: []
=== 9.3 ===
Branch '9.3' set up to track remote branch '9.3' from 'origin'.
Switched to a new branch '9.3'
Already up to date.
After build: ['TEST_MARKER_9.3']
After reset: ['TEST_MARKER_9.3']
M       detection_rules/etc/deprecated_rules.json
Switched to branch 'main'
Your branch is up to date with 'origin/main'.
=== Back on main ===
Final state: ['TEST_MARKER_9.3']
Auto-merging detection_rules/etc/lock-multiple.sh
CONFLICT (content): Merge conflict in detection_rules/etc/lock-multiple.sh
On branch main
Your branch is up to date with 'origin/main'.

[eric-forte-elastic]

I think we should add a safeguard to the logic similar to this. However, the approach functions as desired and does not introduce more risk than that which currently exists in the release process.