|
1 | | -# Braintrust Java Tracing & Eval SDK |
2 | | - |
3 | | -[](https://javadoc.io/doc/dev.braintrust/braintrust-sdk-java) |
4 | | -[](https://github.com/braintrustdata/braintrust-sdk-java/actions/workflows/ci.yml) |
5 | | - |
6 | | -## Overview |
7 | | - |
8 | | -This library provides tools for **evaluating** and **tracing** AI applications in [Braintrust](https://www.braintrust.dev). Use it to: |
9 | | - |
10 | | -- **Evaluate** your AI models with custom test cases and scoring functions |
11 | | -- **Trace** LLM calls and monitor AI application performance with OpenTelemetry |
12 | | -- **Integrate** seamlessly with OpenAI, Anthropic, and other LLM providers |
13 | | - |
14 | | -This SDK is currently in BETA status and APIs may change. |
15 | | - |
16 | | -## Quickstart |
17 | | - |
18 | | -The fastest way to report data to Braintrust is to add the [braintrust java agent](https://central.sonatype.com/artifact/dev.braintrust/braintrust-java-agent) to your jvm startup args. |
19 | | - |
20 | | -springboot+gradle example: |
21 | | - |
22 | | -```build.gradle |
23 | | -configurations { |
24 | | - btAgent |
25 | | -} |
26 | | -dependencies { |
27 | | - implementation 'org.springframework.boot:spring-boot-starter-web' |
28 | | - btAgent "dev.braintrust:braintrust-java-agent:<version-goes-here>" |
29 | | -} |
30 | | -bootRun { |
31 | | - jvmArgs = [ |
32 | | - // NOTE: if you're running with other java agents, add the braintrust agent last |
33 | | - "-javaagent:${configurations.btAgent.singleFile.absolutePath}", |
34 | | - ] |
35 | | -} |
36 | | -``` |
37 | | - |
38 | | -This will automatically instrument major AI clients and frameworks. No code changes required. A list of supported instrumentation can be found [here](./braintrust-sdk/instrumentation) |
39 | | - |
40 | | -NOTE: Additional steps may be required for users running with other `-javaagent` flags. Consult [braintrust docs](https://www.braintrust.dev/docs/instrument/trace-llm-calls#java) for details. |
41 | | - |
42 | | -## Eval Quickstart |
43 | | - |
44 | | -Add the [Braintrust SDK](https://central.sonatype.com/artifact/dev.braintrust/braintrust-sdk-java) to your package manager. |
45 | | - |
46 | | -gradle example: |
47 | | -```gradle |
48 | | -dependencies { |
49 | | - implementation 'dev.braintrust:braintrust-sdk-java:<version-goes-here>' |
50 | | -} |
51 | | -``` |
52 | | - |
53 | | -Use the SDK to create and send your eval: |
54 | | -```java |
55 | | -var braintrust = Braintrust.get(); |
56 | | -braintrust.openTelemetryCreate(); |
57 | | -var openAIClient = OpenAIOkHttpClient.fromEnv(); |
58 | | - |
59 | | -Function<String, String> getFoodType = |
60 | | - (String food) -> { |
61 | | - var request = |
62 | | - ChatCompletionCreateParams.builder() |
63 | | - .model(ChatModel.GPT_4O_MINI) |
64 | | - .addSystemMessage("Return a one word answer") |
65 | | - .addUserMessage("What kind of food is " + food + "?") |
66 | | - .build(); |
67 | | - var response = openAIClient.chat().completions().create(request); |
68 | | - return response.choices().get(0).message().content().orElse("").toLowerCase(); |
69 | | - }; |
70 | | - |
71 | | -var eval = braintrust.<String, String>evalBuilder() |
72 | | - .name("java-eval-x-" + System.currentTimeMillis()) |
73 | | - .cases( |
74 | | - DatasetCase.of("asparagus", "vegetable"), |
75 | | - DatasetCase.of("banana", "fruit")) |
76 | | - .taskFunction(getFoodType) |
77 | | - .scorers( |
78 | | - Scorer.of( |
79 | | - "exact_match", |
80 | | - (expected, result) -> expected.equals(result) ? 1.0 : 0.0)) |
81 | | - .build(); |
82 | | -var result = eval.run(); |
83 | | -System.out.println("\n\n" + result.createReportString()); |
84 | | -``` |
85 | | - |
86 | | -### Manual Instrumentation |
87 | | - |
88 | | -Alternatively, sdk users can manually apply instrumentation instead of using the java agent. |
89 | | - |
90 | | -```java |
91 | | -var braintrust = Braintrust.get(); |
92 | | -var openTelemetry = braintrust.openTelemetryCreate(); |
93 | | -OpenAIClient openAIClient = BraintrustOpenAI.wrapOpenAI(openTelemetry, OpenAIOkHttpClient.fromEnv()); |
94 | | - |
95 | | -var request = |
96 | | - ChatCompletionCreateParams.builder() |
97 | | - .model(ChatModel.GPT_4O_MINI) |
98 | | - .addUserMessage("What is the capital of France?") |
99 | | - .build(); |
100 | | -// openai calls will be traced and reported to braintrust |
101 | | -var response = openAIClient.chat().completions().create(request); |
102 | | -``` |
103 | | - |
104 | | -A list of supported instrumentation can be found [here](./braintrust-sdk/instrumentation) |
105 | | - |
106 | | -## Running Examples |
107 | | - |
108 | | -Example source code can be found [here](./examples/src/main/java/dev/braintrust/examples) |
109 | | - |
110 | | -```bash |
111 | | -export BRAINTRUST_API_KEY="your-braintrust-api-key" |
112 | | -export OPENAI_API_KEY="your-oai-api-key" # to run oai examples |
113 | | -export ANTHROPIC_API_KEY="your-anthropic-api-key" # to run anthropic examples |
114 | | -# install java 17 or later |
115 | | -brew install openjdk@17 # macOS |
116 | | -sudo apt install openjdk-17-jdk # ubuntu |
117 | | -# to run a specific example |
118 | | -./gradlew :examples:runSimpleOpenTelemetry |
119 | | -# to see all examples |
120 | | -./gradlew :examples:tasks --group="Braintrust SDK Examples" |
121 | | -``` |
122 | | - |
123 | | -## Logging |
124 | | - |
125 | | -The SDK uses a standard slf4j logger and will use the default log level (or not log at all if slf4j is not installed). |
126 | | - |
127 | | -All Braintrust loggers will log into the `dev.braintrust` namespace. To adjust the log level, consult your logger documentation. |
128 | | - |
129 | | -For example, to enable debug logging for slf4j-simple you would set the system property `org.slf4j.simpleLogger.log.dev.braintrust=DEBUG` |
| 1 | +I'm just testing CI. Don't actually merge this |
0 commit comments