[Online Scoring] Automation Rules endpoints #945

ldaugusto · 2024-12-20T13:57:38Z

Details

Data model, DAO, service layer and endpoints for Automation Rule Evaluators.

We're using a polymorphic type for the evaluator code, which at least for now for LLM-as-Judge, we will use a JsonNode type

Issues

OPIK-590
OPIK-591

Testing

Documentation

Data model, DAO, service layer and endpoints for Automation Rule Evaluators. I've started with a more complicated approach, but in the end decided for a much simpler one which hopefully is easy enough to extend later into non-evaluator automation rules. Also as we've decided for a non-validation approach in the code, we're just going with a text field. ## Issues OPIK-590 OPIK-591

andrescrz

As discussed in the design, the rule, action and evaluator can be up to 3 different entities. Fine to go with an approach where they're all collapsed into 1 in order to deliver faster. However, this can make extendability for future requirements harder.

For this PR, focus on making the evaluator payload a polymorphic value, so we can define any type of evaluator (python, llm as a judge). The current code field of type string is a concerning implementation.

Then review a bit the paths of the new resource and adjust them.

Everything else is minor polishings. I didn't go in the tests deeply, as still commented out and because this is in-progress.

Thank you very much for putting this together and congrats for your first PR in Opik!

apps/opik-backend/src/main/java/com/comet/opik/api/AutomationRuleEvaluator.java

andrescrz · 2024-12-20T15:50:50Z

apps/opik-backend/src/main/java/com/comet/opik/api/AutomationRuleEvaluator.java

+            return new AutomationRuleEvaluator.AutomationRuleEvaluatorPage(page, 0, 0, List.of());
+        }
+    }
+}


Minor: run spotless on your PRs, so code is automatically formatted.

This is a previous miss from our side, we need to automate this.

New line in the end would be nice.

...end/src/main/java/com/comet/opik/api/resources/v1/priv/AutomationRuleEvaluatorsResource.java

apps/opik-backend/src/main/java/com/comet/opik/domain/AutomationRuleDAO.java

apps/opik-backend/src/main/java/com/comet/opik/domain/AutomationRuleService.java

...d/src/main/resources/liquibase/db-app-state/migrations/000008_add_automation_rule_tables.sql

…o be based in a sample

…rogramming in case the string changes.

andrescrz

This is in better state, but let's keep polishing.

apps/opik-backend/src/main/java/com/comet/opik/api/AutomationRule.java

apps/opik-backend/src/main/java/com/comet/opik/api/AutomationRuleEvaluatorUpdate.java

...end/src/main/java/com/comet/opik/api/resources/v1/priv/AutomationRuleEvaluatorsResource.java

...d/src/main/resources/liquibase/db-app-state/migrations/000009_add_automation_rule_tables.sql

apps/opik-backend/src/main/java/com/comet/opik/api/AutomationRuleEvaluator.java

BorisTkachenko

Overall looks good, left some comments

apps/opik-backend/src/main/java/com/comet/opik/api/AutomationRule.java

apps/opik-backend/src/main/java/com/comet/opik/api/AutomationRuleEvaluator.java

apps/opik-backend/src/main/java/com/comet/opik/api/AutomationRule.java

apps/opik-backend/src/main/java/com/comet/opik/api/AutomationRuleEvaluatorUpdate.java

...end/src/main/java/com/comet/opik/api/resources/v1/priv/AutomationRuleEvaluatorsResource.java

apps/opik-backend/src/main/java/com/comet/opik/domain/AutomationRuleEvaluatorService.java

BorisTkachenko · 2025-01-02T17:16:42Z

apps/opik-backend/src/main/java/com/comet/opik/domain/AutomationRuleEvaluatorService.java

+
+
+    @Override
+    public void update(@NonNull UUID id, @NonNull UUID projectId, @NonNull String workspaceId,


For consistency it would be better to add logs to all methods. Or remove from all, if we could use logs from Resource class.

save and delete methods do not have logging

apps/opik-backend/src/main/java/com/comet/opik/domain/AutomationRuleEvaluatorService.java

…uleEvaluator.java Co-authored-by: BorisTkachenko <[email protected]>

apps/opik-backend/src/main/java/com/comet/opik/api/AutomationRule.java

apps/opik-backend/src/main/java/com/comet/opik/api/AutomationRuleEvaluator.java

BorisTkachenko · 2025-01-03T08:48:08Z

apps/opik-backend/src/main/java/com/comet/opik/api/AutomationRuleEvaluator.java

+            return new AutomationRuleEvaluator.AutomationRuleEvaluatorPage(page, 0, 0, List.of());
+        }
+    }
+}


New line in the end would be nice.

apps/opik-backend/src/main/java/com/comet/opik/api/AutomationRuleEvaluatorType.java

BorisTkachenko · 2025-01-03T08:51:13Z

apps/opik-backend/src/main/java/com/comet/opik/domain/AutomationRuleEvaluatorService.java

+
+
+    @Override
+    public void update(@NonNull UUID id, @NonNull UUID projectId, @NonNull String workspaceId,


save and delete methods do not have logging

...src/test/java/com/comet/opik/api/resources/v1/priv/AutomationRuleEvaluatorsResourceTest.java

BorisTkachenko · 2025-01-03T08:54:36Z

...src/test/java/com/comet/opik/api/resources/v1/priv/AutomationRuleEvaluatorsResourceTest.java

+    private UUID create(AutomationRuleEvaluator<?> evaluator, String apiKey, String workspaceName) {
+        try (var actualResponse = client.target(URL_TEMPLATE.formatted(baseURI, evaluator.getProjectId()))
+                .request()
+                .accept(MediaType.APPLICATION_JSON_TYPE)
+                .header(HttpHeaders.AUTHORIZATION, apiKey)
+                .header(WORKSPACE_HEADER, workspaceName)
+                .post(Entity.json(evaluator))) {
+
+            assertThat(actualResponse.getStatusInfo().getStatusCode()).isEqualTo(201);
+
+            return TestUtils.getIdFromLocation(actualResponse.getLocation());
+        }


Please move it to test client (ex. TraceResourceClient), in case we need it in other tests. Now we have a lot of code duplication in tests, due to making such same calls in each test class.

BorisTkachenko

Looks good

ldaugusto added the work in progress label Dec 20, 2024

ldaugusto requested a review from andrescrz December 20, 2024 13:57

ldaugusto requested a review from a team as a code owner December 20, 2024 13:57

cleaning up

59cb8c2

andrescrz reviewed Dec 20, 2024

View reviewed changes

ldaugusto added 8 commits December 20, 2024 23:53

Splitting Rule and Evaluator types

bdf11fe

cleaning up

6d731f6

moving 'sampling rate' to the base rule; other rule actions would als…

eccaa4b

…o be based in a sample

removing RequestContext references from Service

a79b400

Read-only rule evaluator ids

a20a452

cleaning up

a544940

moving project to endpoint root and cleaning up descriptions

84f5197

Removing hard coded 'evaluator' from sqls to Action bind. Defensive p…

f4310cb

…rogramming in case the string changes.

andrescrz reviewed Dec 23, 2024

View reviewed changes

Using polymorphic type for the evaluator code

574acbf

ldaugusto requested a review from BorisTkachenko December 31, 2024 13:56

ldaugusto and others added 2 commits December 31, 2024 17:31

Adding search by name support in find endpoint

71427d7

Merge branch 'main' into daniel/automation-rule-evaluator-endpoints

1ff1b14

ldaugusto removed the work in progress label Jan 2, 2025

Merge branch 'main' into daniel/automation-rule-evaluator-endpoints

10af9ca

BorisTkachenko reviewed Jan 2, 2025

View reviewed changes

ldaugusto and others added 3 commits January 2, 2025 18:36

Update apps/opik-backend/src/main/java/com/comet/opik/api/AutomationR…

9b0d13b

…uleEvaluator.java Co-authored-by: BorisTkachenko <[email protected]>

PR suggestions/requests fixed

4a475fe

missing float -> Float

c734e5e

BorisTkachenko reviewed Jan 3, 2025

View reviewed changes

ldaugusto added 2 commits January 3, 2025 10:35

missing small cleaning from last commit

7aacec9

Requiring non null list of ids to delete

3065ace

BorisTkachenko approved these changes Jan 3, 2025

View reviewed changes

ldaugusto merged commit 7f5175d into main Jan 3, 2025
7 checks passed

ldaugusto deleted the daniel/automation-rule-evaluator-endpoints branch January 3, 2025 11:01

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Online Scoring] Automation Rules endpoints #945

[Online Scoring] Automation Rules endpoints #945

ldaugusto commented Dec 20, 2024 •

edited

Loading

andrescrz left a comment •

edited

Loading

andrescrz Dec 20, 2024

BorisTkachenko Jan 3, 2025

andrescrz left a comment

BorisTkachenko left a comment

BorisTkachenko Jan 2, 2025

ldaugusto Jan 2, 2025

BorisTkachenko Jan 3, 2025

BorisTkachenko Jan 3, 2025

BorisTkachenko Jan 3, 2025

BorisTkachenko Jan 3, 2025

BorisTkachenko left a comment



		@Override
		public void update(@NonNull UUID id, @NonNull UUID projectId, @NonNull String workspaceId,

[Online Scoring] Automation Rules endpoints #945

[Online Scoring] Automation Rules endpoints #945

Conversation

ldaugusto commented Dec 20, 2024 • edited Loading

Details

Issues

Testing

Documentation

andrescrz left a comment • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

andrescrz left a comment

Choose a reason for hiding this comment

BorisTkachenko left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

BorisTkachenko left a comment

Choose a reason for hiding this comment

ldaugusto commented Dec 20, 2024 •

edited

Loading

andrescrz left a comment •

edited

Loading