OKD-370: Add promptfoo evals for agentic-docs plugin

.claude-plugin/marketplace.json

@@ -246,6 +246,12 @@
       "source": "./plugins/marketplace-ops",
       "description": "Maintenance commands for Claude Code plugin marketplaces",
       "version": "0.1.2"
+    },
+    {
+      "name": "agentic-docs",
+      "source": "./plugins/agentic-docs",
+      "description": "Create and maintain AI-optimized documentation for OpenShift",
+      "version": "1.0.0"
     }
   ]
 }

docs/data.json

@@ -1822,6 +1822,26 @@
       "name": "marketplace-ops",
       "skills": [],
       "version": "0.1.2"
+    },
+    {
+      "commands": [],
+      "description": "Create and maintain AI-optimized documentation for OpenShift",
+      "has_readme": true,
+      "hooks": [],
+      "name": "agentic-docs",
+      "skills": [
+        {
+          "description": "Create lean component documentation for OpenShift repositories",
+          "id": "component",
+          "name": "component-docs"
+        },
+        {
+          "description": "Update existing platform documentation with automatic gap detection in openshift/enhancements",
+          "id": "update-platform-docs",
+          "name": "update-platform-docs"
+        }
+      ],
+      "version": "1.0.0"
     }
   ]
 }

plugins/agentic-docs/.claude-plugin/plugin.json

@@ -0,0 +1,8 @@
+{
+  "name": "agentic-docs",
+  "description": "Create and maintain AI-optimized documentation for OpenShift",
+  "version": "1.1.0",
+  "author": {
+    "name": "github.com/openshift-eng"
+  }
+}

plugins/agentic-docs/commands/evaluate.md

@@ -0,0 +1,55 @@
+---
+description: Evaluate agentic documentation quality using promptfoo-based behavioral validation
+argument-hint: "[repository-path]"
+---
+
+## Name
+agentic-docs:evaluate
+
+## Synopsis
+```
+/agentic-docs:evaluate [repository-path]
+```
+
+## Description
+The `agentic-docs:evaluate` command evaluates documentation quality by testing whether AI agents naturally discover and correctly apply repository conventions without being explicitly told to read documentation.
+
+This command validates **documentation-first natural discovery behavior** using the OpenShift Enhancements Agentic Docs Evaluation framework. It measures:
+- **Natural discovery**: Does the agent find documentation without instruction?
+- **Correct navigation**: Does the agent follow documentation structure?
+- **Pattern application**: Does the agent apply repository conventions correctly?
+- **Anti-pattern rejection**: Does the agent reject incorrect patterns?
+
+The evaluation uses promptfoo to run assertions from `promptfooconfig.yaml` and generates detailed HTML reports with pass/fail grades.
+
+## Implementation
+When this command is invoked, Claude will execute the `agentic-docs:evaluate` skill, which:
+1. Loads evaluation configuration from `promptfooconfig.yaml`
+2. Runs coding sub-agents with task descriptions (no explicit file instructions)
+3. Evaluates whether agents naturally discovered and applied documentation
+4. Generates graded results with pass/fail assertions
+5. Creates HTML reports for review
+
+The skill maintains strict separation between coding agents (who must discover docs naturally) and evaluation agents (who grade the results).
+
+## Return Value
+- Evaluation results with pass/fail grades for each test case
+- HTML report showing which documentation was discovered and applied
+- Metrics on natural discovery patterns
+
+## Examples
+
+1. **Evaluate current directory**:
+   ```
+   /agentic-docs:evaluate
+   ```
+   Evaluates documentation in the current working directory.
+
+2. **Evaluate specific repository**:
+   ```
+   /agentic-docs:evaluate /path/to/openshift/repo
+   ```
+   Evaluates documentation in the specified repository.
+
+## Arguments
+- `repository-path` (optional): Path to the repository to evaluate. Defaults to current directory if not specified.

plugins/agentic-docs/commands/generate-evals.md

@@ -0,0 +1,54 @@
+---
+description: Generate repository-specific promptfoo evaluation suites for OpenShift documentation
+argument-hint: "[repository-path]"
+---
+
+## Name
+agentic-docs:generate-evals
+
+## Synopsis
+```
+/agentic-docs:generate-evals [repository-path]
+```
+
+## Description
+The `agentic-docs:generate-evals` command generates a tailored `promptfooconfig.yaml` evaluation suite for a specific OpenShift repository. Instead of using a generic evaluation configuration, it analyzes the repository's documentation structure, code patterns, and conventions to create repository-specific test cases.
+
+The generated evaluation suite tests whether AI agents can:
+- Naturally discover repository documentation
+- Apply repository-specific patterns correctly
+- Follow established conventions without explicit instruction
+- Reject anti-patterns specific to the repository
+
+This follows the OpenShift Enhancements Agentic Docs Evaluation framework, which emphasizes documentation-first natural discovery.
+
+## Implementation
+When this command is invoked, Claude will execute the `agentic-docs:generate-evals` skill, which:
+1. Analyzes repository documentation structure (CLAUDE.md, ai-docs/, ARCHITECTURE.md)
+2. Identifies code patterns (API versions, operator patterns, controller structure)
+3. Extracts repository-specific conventions
+4. Generates test cases that validate natural documentation discovery
+5. Creates `promptfooconfig.yaml` with assertions tailored to the repository
+6. Saves configuration to the repository root
+
+## Return Value
+- Generated `promptfooconfig.yaml` file in the repository root
+- Test cases specific to the repository's patterns and conventions
+- Assertions configured for natural discovery validation
+
+## Examples
+
+1. **Generate evals for current directory**:
+   ```
+   /agentic-docs:generate-evals
+   ```
+   Analyzes the current repository and generates `promptfooconfig.yaml`.
+
+2. **Generate evals for specific repository**:
+   ```
+   /agentic-docs:generate-evals /path/to/openshift/repo
+   ```
+   Analyzes the specified repository and generates tailored evaluation configuration.
+
+## Arguments
+- `repository-path` (optional): Path to the target repository for analysis. Defaults to current directory if not specified.