Skip to content

Commit 0e4a144

Browse files
authored
add anthropic caching to system prompt in agent (#1655)
# Add Anthropic Prompt Caching for Agent System Prompts ## Summary Added Anthropic's ephemeral cache control to system prompts in `v3AgentHandler.ts`. Refactored into a single `prependSystemMessage()` function used by both `execute()` and `stream()`. ## Why Reduces token costs and latency for Claude models by caching the system prompt server-side. The `providerOptions` are ignored by non-Anthropic models, so this is safe for all providers. ## Limitations Currently limited to the system prompt only. Extending to conversation messages requires restructuring message processing to insert cache breakpoints at stable boundaries. ## Test Plan - [x] Verify agent works with Claude and non Anthropic models - [x] Confirm `cached_input_tokens` appears in usage metrics with Claude <!-- This is an auto-generated description by cubic. --> --- ## Summary by cubic Added Anthropic ephemeral caching to the agent’s system prompt to lower Claude token usage and reduce latency. Safe for non-Anthropic models; the options are ignored. - **New Features** - System messages now include Anthropic providerOptions with cacheControl: ephemeral. - **Refactors** - Introduced prependSystemMessage() and used it in both text generation and streaming to centralize system prompt handling. <sup>Written for commit 6e2eef6. Summary will update on new commits. <a href="https://cubic.dev/pr/browserbase/stagehand/pull/1655">Review in cubic</a></sup> <!-- End of auto-generated description by cubic. -->
1 parent b27c04d commit 0e4a144

File tree

1 file changed

+24
-4
lines changed

1 file changed

+24
-4
lines changed

packages/core/lib/v3/handlers/v3AgentHandler.ts

Lines changed: 24 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -42,6 +42,28 @@ function getErrorMessage(error: unknown): string {
4242
return error instanceof Error ? error.message : String(error);
4343
}
4444

45+
/**
46+
* Prepends a system message with cache control to the messages array.
47+
* The cache control providerOptions are used by Anthropic and ignored by other providers.
48+
*/
49+
function prependSystemMessage(
50+
systemPrompt: string,
51+
messages: ModelMessage[],
52+
): ModelMessage[] {
53+
return [
54+
{
55+
role: "system",
56+
content: systemPrompt,
57+
providerOptions: {
58+
anthropic: {
59+
cacheControl: { type: "ephemeral" },
60+
},
61+
},
62+
},
63+
...messages,
64+
];
65+
}
66+
4567
export class V3AgentHandler {
4668
private v3: V3;
4769
private logger: (message: LogLine) => void;
@@ -289,8 +311,7 @@ export class V3AgentHandler {
289311

290312
const result = await this.llmClient.generateText({
291313
model: wrappedModel,
292-
system: systemPrompt,
293-
messages,
314+
messages: prependSystemMessage(systemPrompt, messages),
294315
tools: allTools,
295316
stopWhen: (result) => this.handleStop(result, maxSteps),
296317
temperature: 1,
@@ -414,8 +435,7 @@ export class V3AgentHandler {
414435

415436
const streamResult = this.llmClient.streamText({
416437
model: wrappedModel,
417-
system: systemPrompt,
418-
messages,
438+
messages: prependSystemMessage(systemPrompt, messages),
419439
tools: allTools,
420440
stopWhen: (result) => this.handleStop(result, maxSteps),
421441
temperature: 1,

0 commit comments

Comments
 (0)