Add ai_confidence metadata field to rules to indicate AI fix likelihood

[jwmatthews]

Summary

We’d like to introduce a new metadata field, ai_confidence, within rule definitions to communicate how likely an AI assistant is to automatically resolve the issue identified by that rule. This value will help users prioritize which findings are low-risk for automated fixes versus those that require more human review.

The field could use a three-level scale: High, Medium, or Low, representing increasing difficulty or risk of unsuccessful AI remediation. Initially, the confidence level can be manually assigned by rule authors based on their understanding of the rule’s complexity. Over time, the Solution Server may provide feedback data to refine or validate these levels based on real-world success metrics.

Proposed Implementation

Add ai_confidence as an optional field in the rule definition schema.

Accept discrete values: High, Medium, Low.

Display this value in the IDE or report views where rule metadata is surfaced to end users.

Seed the existing https://github.com/konveyor/rulesets Rules with a suitable value for ai_confidence, explore if we can leverage an approach like in https://github.com/tsanders-rh/konveyor-iq to generates a test snippet per rule and score a few LLMs on how well they handle fixing the issue given the info in the rule.

Future enhancement: integrate telemetry from the Solution Server to auto-adjust confidence levels based on observed fix success rates.

Prior Work

This work was originally planned during the initial Solution Server implementation as #761. We discussed updating Rules metadata to track this confidence level. As work progressed we implemented the ability to capture the accept and reject interactions from IDE and store in Solution Server tied to a given Rule and to surface this to end user in IDE (implemented in konveyor/editor-extensions#568). We missed the work to think through and plan how to update the Rules metadata to track this.