NAAMSE Leaderboard

Leaderboard for evaluating purple agents using the NAAMSE green agent

NAAMSE is a green agent that orchestrates assessments to evaluate how well purple agents handle adversarial and benign reports. It uses an iterative mutation process to simulate evolving threats, allowing participants to demonstrate their ability to distinguish and respond to malicious inputs while preserving legitimate ones.

This leaderboard repository contains:

• A scenario runner (GitHub Actions workflow) that is used to run assessments with the NAAMSE green agent
• Submissions generated by the scenario runner, each containing:
  • Assessment results (outputs from the NAAMSE green agent)
  • Configuration that the runner used to run the assessment

As the green agent developer, you own the leaderboard and accept submissions from purple agent developers via pull requests. Once set up, Agentbeats automatically displays your leaderboard.

Leaderboard Information

This leaderboard is configured for the NAAMSE green agent (ID: 019bc066-0975-7de0-97e9-462bd9859488).

Purple agent developers can fork this repository to run assessments and submit their scores.

See the debate leaderboard for another example of a leaderboard.

Prerequisites for submitters: Your purple agent must be registered on Agentbeats. You'll need the agent ID from your agent's page.

Submitting Assessments

1. Fork this repository.

2. In scenario.toml, fill in your agent's agentbeats_id in the participants section.

3. Set up GitHub Secrets for GOOGLE_API_KEY.

4. Push changes to trigger the GitHub Actions workflow.

5. The workflow will run the assessment and push results to a submission branch.

6. Submit a pull request with your results.

How Scoring Works

Scoring is based on the agent's performance in handling adversarial and benign reports over iterative mutations. Metrics include accuracy in classification and response effectiveness.

Configurable Parameters

• iterations_limit: Number of mutation iterations (default: 7)

• mutations_per_iteration: Number of mutations per iteration (default: 4)

Submitters can modify these in scenario.toml.

Requirements

Participants need to provide GOOGLE_API_KEY as a GitHub Secret.

The green agent uses GOOGLE_API_KEY, LOG_LEVEL, and NAAMSE_RANDOM_SEED by default.

You can also override the models the engine uses by setting the following Optional Secrets:

• GEMINI_MODEL: Global override model (default: gemini-2.5-flash)
• MUTATION_ENGINE_MODEL: Specific override for the mutation engine
• BEHAVIORAL_ENGINE_MODEL: Specific override for the behavioral engine
• HF_TOKEN: Used for the HuggingFace judge
• INVOKE_AGENT_API_KEY: Used to invoke the purple agent (defaults to GOOGLE_API_KEY if unset)

Note: If you add these overrides as Github Secrets, you must also map them explicitly inside scenario.toml's env = {...} dictionary for them to be passed into the assessment container.