Agent terminal command safety

[tobezdev]

Add Command Safety System for AI Agent Terminal Operations

Pre-summary disclaimer: yes I vibe coded this. Yes it is all in one commit. Yes, I am sorry for the huge diff. No, I am not the most experienced person in the world with Rust so there is probably some stupid logic error or something somewhere (although all checks passed, so I am kinda hopeful).

Summary

Implements a comprehensive command safety system that prevents AI agents from executing dangerous terminal commands without user approval. The system includes built-in dangerous command detection, user-configurable whitelists/blacklists, and cross-platform support for Windows, macOS, and Linux.

Motivation

As mentioned in #37343:

Claude sonnet 4 used rm - rf $HOME/ after I asked it to make a commit to my git repo.

Changes

• New: crates/assistant_tools/src/command_safety.rs - Core safety logic with dangerous command patterns
• Modified: crates/assistant_tools/src/terminal_tool.rs - Integration with existing terminal tool
• Modified: crates/settings/src/settings_content/agent.rs - New settings schema
• Modified: crates/agent_settings/src/agent_settings.rs - Settings integration

Key Features

1. Built-in Dangerous Command Detection

Automatically detects and categorizes dangerous commands:

• Destructive: rm -rf, del /s /q, format, dd, shred
• System Modification: fdisk, chmod 777, reg add, systemctl disable
• Execution Risks: curl | sh, sudo rm, download-and-execute patterns
• Sensitive Access: Reading passwd/shadow files, SSH keys, certificates
• Network Risks: nc -l, netsh modifications

2. User-Configurable Options

{
  "agent": {
    "command_safety": {
      "whitelist": ["git *", "npm install", "cargo build"],
      "blacklist": ["rm -rf *", "curl * | sh"],
      "use_builtin_blacklist": true
    }
  }
}

3. Enhanced UX

• Safe commands execute immediately
• Dangerous commands show clear warnings: rm -rf /

⚠️ Destructive (Recursive file deletion can destroy important data)

• Whitelisted commands bypass all safety checks (including those seen as "dangerous" if the user has whitelisted it)

Backward Compatibility

• Fully backward compatible with existing always_allow_tool_actions setting
• No breaking changes to existing workflows
• Fail-safe design: when in doubt, require confirmation

Testing

Includes comprehensive tests for:

• Cross-platform dangerous command detection
• Whitelist/blacklist pattern matching
• Wildcard support (git *)
• Integration with existing confirmation system

Additionally, all changes made by this PR have passed checks via cargo check -p assistant_tools and cargo check -p settings.

DISCLAIMER - AI assistance was used (in small amounts) during the creation of this PR. In specific, Copilot was used to generate a structure of the Zed repository and identify potential places where changes were needed (even though it wasn't even right some of the time lol), however, Copilot (nor any other AI model) was used to generate code aside from the odd tab-to-complete here and there. For any further questions regarding this, please let me know.

[tobezdev]

cricket noises