Skip to content

Commit 6d82c61

Browse files
jmharoldjmharold
authored
Docs/langfuse
1 parent 1328db1 commit 6d82c61

File tree

2 files changed

+218
-0
lines changed

2 files changed

+218
-0
lines changed

‎lib/docs/.vitepress/config.mts‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -57,6 +57,7 @@ const navLinks = [
5757
{ text: 'Model Management UI', link: '/config/model-management-ui' },
5858
{ text: 'Usage & Features', link: '/config/usage' },
5959
{ text: 'RAG Vector Stores', link: '/config/vector-stores' },
60+
{ text: 'Langfuse Tracing', link: '/config/langfuse-tracing'},
6061
{
6162
text: 'Configuration Schema',
6263
link: '/config/configuration',

‎lib/docs/config/langfuse-tracing.md‎

Lines changed: 217 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,217 @@
1+
# LISA Langfuse Integration Guide
2+
3+
Langfuse is an open source tool that supports advanced traces, evals, and metrics. This guide provides step-by-step instructions for integrating Langfuse with LISA to enable tracing and monitoring of LLM interactions.
4+
5+
## Prerequisites
6+
7+
First ensure that Langfuse is properly deployed by either:
8+
9+
- Creating a managed Langfuse account at [Langfuse Cloud](https://langfuse.com)
10+
- Self-hosting Langfuse using the [official self-hosting documentation](https://langfuse.com/self-hosting)
11+
12+
### Initial Langfuse Setup
13+
14+
After deploying Langfuse, complete the following setup steps:
15+
16+
1. Navigate to the Langfuse web interface
17+
2. Create a user account
18+
3. Create an organization
19+
4. Create a project within the organization
20+
5. Generate API credentials (Public Key and Secret Key)
21+
22+
Retain the generated API credentials as they will be required for the LISA configuration in Step 1.
23+
24+
## Configuration Steps
25+
26+
### Step 1: Update LiteLLM Configuration
27+
28+
Configure the LiteLLM integration by updating the `litellmConfig` section in the `config-base.yaml` file:
29+
30+
```yaml
31+
litellmConfig:
32+
callbacks: ["langfuse"]
33+
LANGFUSE_HOST: {YOUR_LANGFUSE_ENDPOINT}
34+
LANGFUSE_PUBLIC_KEY: pk-{YOUR_PUBLIC_KEY}
35+
LANGFUSE_SECRET_KEY: sk-{YOUR_SECRET_KEY}
36+
```
37+
38+
Replace the placeholder values with the appropriate Langfuse endpoint and API credentials obtained during the initial setup.
39+
40+
### Step 2: Update Python Dependencies
41+
42+
Add the Langfuse Python package to the LISA Serve REST API dependencies by including the following line in the [`requirements.txt`](https://github.com/awslabs/LISA/blob/main/lib/serve/rest-api/src/requirements.txt) file located at `lib/serve/rest-api/src/`:
43+
44+
```
45+
langfuse>=3.0.0
46+
```
47+
48+
### Step 3: Configure Environment Variables
49+
50+
#### Update Configuration Schema
51+
52+
Modify the `LiteLLMConfig` schema in the [`configSchema.ts`](https://github.com/awslabs/LISA/blob/main/lib/schema/configSchema.ts#L758) file located at `lib/schema/configSchema.ts` to include the Langfuse environment variables:
53+
54+
```typescript
55+
const LiteLLMConfig = z.object({
56+
db_key: z.string().refine(
57+
...
58+
),
59+
general_settings: z.any().optional(),
60+
litellm_settings: z.any().optional(),
61+
router_settings: z.any().optional(),
62+
environment_variables: z.any().optional(),
63+
callbacks: z.array(z.string()).optional().describe('LiteLLM callbacks to enable (e.g., ["langfuse"])'),
64+
LANGFUSE_HOST: z.string().optional().describe('Langfuse host URL (e.g., https://us.cloud.langfuse.com)'),
65+
LANGFUSE_PUBLIC_KEY: z.string().optional().describe('Langfuse public key for authentication'),
66+
LANGFUSE_SECRET_KEY: z.string().optional().describe('Langfuse secret key for authentication'),
67+
})
68+
```
69+
70+
#### Update FastAPI Container Environment
71+
72+
Modify the [`fastApiContainer.ts`](https://github.com/awslabs/LISA/blob/95a38b055044b0930df7b66ca4fa25dc58fddcd8/lib/api-base/fastApiContainer.ts#L64) file located at `lib/api-base/fastApiContainer.ts` to include the Langfuse environment variables in the `baseEnvironment`:
73+
74+
```typescript
75+
if (config.litellmConfig.LANGFUSE_HOST) {
76+
baseEnvironment.LANGFUSE_HOST = config.litellmConfig.LANGFUSE_HOST;
77+
}
78+
if (config.litellmConfig.LANGFUSE_PUBLIC_KEY) {
79+
baseEnvironment.LANGFUSE_PUBLIC_KEY = config.litellmConfig.LANGFUSE_PUBLIC_KEY;
80+
}
81+
if (config.litellmConfig.LANGFUSE_SECRET_KEY) {
82+
baseEnvironment.LANGFUSE_SECRET_KEY = config.litellmConfig.LANGFUSE_SECRET_KEY;
83+
}
84+
```
85+
86+
### Step 4: Implement Langfuse
87+
88+
#### Decorate the LiteLLM Passthrough Function
89+
90+
> [!WARNING] The implementation in this section is designed for non-streamed responses. Streaming responses require additional implementation considerations for properly handling `StreamingResponse` objects.
91+
92+
Add the Langfuse observe decorator to the [`litellm_passthrough`](https://github.com/awslabs/LISA/blob/main/lib/serve/rest-api/src/api/endpoints/v2/litellm_passthrough.py#L94) function located at `lib/serve/rest-api/src/api/endpoints/v2/litellm_passthrough.py`:
93+
94+
```python
95+
from langfuse import observe
96+
97+
@router.api_route("/{api_path:path}", methods=["GET", "POST", "OPTIONS", "PUT", "PATCH", "DELETE", "HEAD"])
98+
@observe()
99+
async def litellm_passthrough(request: Request, api_path: str) -> Response:
100+
...
101+
```
102+
103+
> [!NOTE] The decorator order is significant. The `@observe()` decorator must be positioned directly above the function definition.
104+
105+
## Verification and Monitoring
106+
107+
### Deployment and Testing
108+
109+
After completing all configuration changes, redeploy LISA. Once the deployment is successful, interactions with models via LISA will automatically send telemetry data to Langfuse.
110+
111+
Access the Langfuse tracing interface to view collected traces.
112+
113+
### Trace Structure
114+
115+
#### Non-Streamed Response Traces
116+
117+
Non-streamed responses generate traces with the following structure:
118+
119+
**Input:**
120+
```json
121+
{
122+
"args": [],
123+
"kwargs": {
124+
"api_path": "chat/completions",
125+
"request": {}
126+
}
127+
}
128+
```
129+
130+
**Output:**
131+
```json
132+
{
133+
"status_code": 200,
134+
"background": null,
135+
"body": {
136+
"id": "chatcmpl-4c0f3c88-12d9-4e4d-a38e-9c83fabfa9df",
137+
"created": 1758912436,
138+
"model": "us.anthropic.claude-3-7-sonnet-20250219-v1:0",
139+
"object": "chat.completion",
140+
"choices": [
141+
{
142+
"finish_reason": "stop",
143+
"index": 0,
144+
"message": {
145+
"content": "Hi there! I notice we're just exchanging greetings. Is there something I can help you with today? I'm happy to assist with questions, provide information on a topic you're interested in, or help with a specific task. Just let me know what you need!",
146+
"role": "assistant"
147+
}
148+
}
149+
],
150+
"usage": {
151+
"completion_tokens": 60,
152+
"prompt_tokens": 1324,
153+
"total_tokens": 1384,
154+
"prompt_tokens_details": {
155+
"cached_tokens": 0
156+
},
157+
"cache_creation_input_tokens": 0,
158+
"cache_read_input_tokens": 0
159+
}
160+
},
161+
"raw_headers": [
162+
[
163+
"content-length",
164+
"674"
165+
],
166+
[
167+
"content-type",
168+
"application/json"
169+
]
170+
]
171+
}
172+
```
173+
174+
#### Streamed Response Traces
175+
176+
Streamed responses maintain identical input structure to non-streamed responses but the output differs.
177+
178+
The default trace output for streamed responses are:
179+
```xml
180+
<starlette.responses.StreamingResponse object at 0x7fd112d31b50>
181+
```
182+
183+
#### Advanced Streaming Implementation
184+
185+
For enhanced observability into streaming responses, implement a custom transformation function for the [`generate_response`](https://github.com/awslabs/LISA/blob/main/lib/serve/rest-api/src/api/endpoints/v2/litellm_passthrough.py#L84) function located at `lib/serve/rest-api/src/api/endpoints/v2/litellm_passthrough.py`:
186+
187+
```python
188+
def custom_transformer(line):
189+
return f"{line}\n\n"
190+
191+
@observe(transform_to_string=custom_transformer)
192+
def generate_response(iterator: Iterator[Union[str, bytes]]) -> Iterator[str]:
193+
...
194+
```
195+
196+
The [`transform_to_string`](https://python.reference.langfuse.com/langfuse#observe) parameter enables custom handling of generator chunks, allowing for proper string concatenation and formatting.
197+
198+
## Additional Resources
199+
200+
### Langfuse Docs MCP
201+
202+
For enhanced troubleshooting and integration support, Langfuse provides:
203+
204+
- [Langfuse Docs MCP Server](https://langfuse.com/docs/docs-mcp)
205+
- [Custom Integration Prompt](https://langfuse.com/docs/observability/get-started)
206+
207+
### Reference Documentation
208+
209+
**LiteLLM Integration:**
210+
- [Langfuse Logging with LiteLLM](https://docs.litellm.ai/docs/proxy/logging#langfuse)
211+
- [OpenTelemetry Integration with LiteLLM Proxy](https://litellm.vercel.app/docs/observability/langfuse_otel_integration#with-litellm-proxy)
212+
213+
**Langfuse Documentation:**
214+
- [LiteLLM Proxy with @observe Decorator](https://langfuse.com/guides/cookbook/integration_litellm_proxy)
215+
- [LiteLLM SDK Integration Guide](https://langfuse.com/integrations/frameworks/litellm-sdk)
216+
- [Python SDK Documentation](https://python.reference.langfuse.com/langfuse)
217+
- [Langfuse Self-Hosting Guide](https://langfuse.com/self-hosting)

0 commit comments

Comments
 (0)