Mlflow eval

[EoghanOConnor]

Includes PR #14 (to allow for testing while its in WIP)
MLFlow Eval loads annotator.json files from jumpbox
Then logs this info onto MLFlow
These can be viewed in evaluation runs under the experiment name that was set.

Needs to include the Human expectations.

[coderabbitai]

Important

Review skipped

Ignore keyword(s) in the title.

⛔ Ignored keywords (1)

• WIP

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro Plus

Run ID: bae0eac8-7df9-4a1c-83e2-2ad12b4ead9d

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

• 🔍 Trigger review

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

[jerry-ng2]

Are there any examples on the jumpbox?

[jerry-ng2]

should be annotation.json

[jerry-ng2]

There should be a condition that checks if an eval already exists for a certain job_id. wdyt? Maybe look into mlflow.search_runs