Skip to content

Commit 2c6c5ef

Browse files
cmilesbcmilesb
authored
Merge pull request #1798 from redis/DOC-5172-move-docs
LangCache: Move certain docs to develop/ai
2 parents 2776f2f + 1c2dbef commit 2c6c5ef

File tree

7 files changed

+280
-257
lines changed

7 files changed

+280
-257
lines changed

‎content/develop/ai/langcache.md

Lines changed: 0 additions & 96 deletions
This file was deleted.

‎content/develop/ai/langcache/_index.md

Lines changed: 111 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,111 @@
1+
---
2+
Title: Redis LangCache
3+
alwaysopen: false
4+
categories:
5+
- docs
6+
- develop
7+
- ai
8+
description: Store LLM responses for AI apps in a semantic cache.
9+
linkTitle: LangCache
10+
hideListLinks: true
11+
weight: 30
12+
---
13+
14+
Redis LangCache is a fully-managed semantic caching service that reduces large language model (LLM) costs and improves response times for AI applications.
15+
16+
## LangCache overview
17+
18+
LangCache uses semantic caching to store and reuse previous LLM responses for repeated queries. Instead of calling the LLM for every request, LangCache checks if a similar response has already been generated and is stored in the cache. If a match is found, LangCache returns the cached response instantly, saving time and resources.
19+
20+
Imagine you’re using an LLM to build an agent to answer questions about your company's products. Your users may ask questions like the following:
21+
22+
- "What are the features of Product A?"
23+
- "Can you list the main features of Product A?"
24+
- "Tell me about Product A’s features."
25+
26+
These prompts may have slight variations, but they essentially ask the same question. LangCache can help you avoid calling the LLM for each of these prompts by caching the response to the first prompt and returning it for any similar prompts.
27+
28+
Using LangCache as a semantic caching service has the following benefits:
29+
30+
- **Lower LLM costs**: Reduce costly LLM calls by easily storing the most frequently-requested responses.
31+
- **Faster AI app responses**: Get faster AI responses by retrieving previously-stored requests from memory.
32+
- **Simpler Deployments**: Access our managed service using a REST API with automated embedding generation, configurable controls, and no database management required.
33+
- **Advanced cache management**: Manage data access and privacy, eviction protocols, and monitor usage and cache hit rates.
34+
35+
LangCache works well for the following use cases:
36+
37+
- **AI assistants and chatbots**: Optimize conversational AI applications by caching common responses and reducing latency for frequently asked questions.
38+
- **RAG applications**: Enhance retrieval-augmented generation performance by caching responses to similar queries, reducing both cost and response time.
39+
- **AI agents**: Improve multi-step reasoning chains and agent workflows by caching intermediate results and common reasoning patterns.
40+
- **AI gateways**: Integrate LangCache into centralized AI gateway services to manage and control LLM costs across multiple applications..
41+
42+
### LLM cost reduction with LangCache
43+
44+
{{< embed-md "langcache-cost-reduction.md" >}}
45+
46+
## LangCache architecture
47+
48+
The following diagram displays how you can integrate LangCache into your GenAI app:
49+
50+
{{< image filename="images/rc/langcache-process.png" alt="The LangCache process diagram." >}}
51+
52+
1. A user sends a prompt to your AI app.
53+
1. Your app sends the prompt to LangCache through the `POST /v1/caches/{cacheId}/search` endpoint.
54+
1. LangCache calls an embedding model service to generate an embedding for the prompt.
55+
1. LangCache searches the cache to see if a similar response already exists by matching the embeddings of the new query with the stored embeddings.
56+
1. If a semantically similar entry is found (also known as a cache hit), LangCache gets the cached response and returns it to your app. Your app can then send the cached response back to the user.
57+
1. If no match is found (also known as a cache miss), your app receives an empty response from LangCache. Your app then queries your chosen LLM to generate a new response.
58+
1. Your app sends the prompt and the new response to LangCache through the `POST /v1/caches/{cacheId}/entries` endpoint.
59+
1. LangCache stores the embedding with the new response in the cache for future use.
60+
61+
See the [LangCache API reference]({{< relref "/develop/ai/langcache/api-reference" >}}) for more information on how to use the LangCache API.
62+
63+
## Get started
64+
65+
LangCache is currently in preview:
66+
67+
- Public preview on [Redis Cloud]({{< relref "/operate/rc/langcache" >}})
68+
- Fully-managed [private preview](https://redis.io/langcache/)
69+
70+
{{< multitabs id="langcache-get-started"
71+
tab1="Redis Cloud"
72+
tab2="Private preview" >}}
73+
74+
{{< embed-md "rc-langcache-get-started.md" >}}
75+
76+
-tab-sep-
77+
78+
### Prerequisites
79+
80+
To use LangCache in private preview, you need:
81+
82+
- An AI application that makes LLM API calls
83+
- A use case involving repetitive or similar queries
84+
- Willingness to provide feedback during the preview phase
85+
86+
### Access
87+
88+
LangCache is offered as a fully-managed service. During the private preview:
89+
90+
- Participation is free
91+
- Usage limits may apply
92+
- Dedicated support is provided
93+
- Regular feedback sessions are conducted
94+
95+
### Data security and privacy
96+
97+
LangCache stores your data on your Redis servers. Redis does not access your data or use it to train AI models. The service maintains enterprise-grade security and privacy standards.
98+
99+
### Support
100+
101+
Private preview participants receive:
102+
103+
- Dedicated onboarding resources
104+
- Documentation and tutorials
105+
- Email and chat support
106+
- Regular check-ins with the product team
107+
- Exclusive roadmap updates
108+
109+
For more information about joining the private preview, visit the [Redis LangCache website](https://redis.io/langcache/).
110+
111+
{{< /multitabs >}}

‎content/develop/ai/langcache/api-reference.md

Lines changed: 129 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,129 @@
1+
---
2+
alwaysopen: false
3+
categories:
4+
- docs
5+
- develop
6+
- ai
7+
description: Learn to use the Redis LangCache API for semantic caching.
8+
hideListLinks: true
9+
linktitle: API and SDK reference
10+
title: LangCache API and SDK reference
11+
weight: 10
12+
---
13+
14+
You can use the LangCache API from your client app to store and retrieve LLM, RAG, or agent responses.
15+
16+
To access the LangCache API, you need:
17+
18+
- LangCache API base URL
19+
- LangCache service API key
20+
- Cache ID
21+
22+
When you call the API, you need to pass the LangCache API key in the `Authorization` header as a Bearer token and the Cache ID as the `cacheId` path parameter.
23+
24+
For example, to check the health of the cache using `cURL`:
25+
26+
```bash
27+
curl -s -X GET "https://$HOST/v1/caches/$CACHE_ID/health" \
28+
-H "accept: application/json" \
29+
-H "Authorization: Bearer $API_KEY"
30+
```
31+
32+
- The example expects several variables to be set in the shell:
33+
34+
- **$HOST** - the LangCache API base URL
35+
- **$CACHE_ID** - the Cache ID of your cache
36+
- **$API_KEY** - The LangCache API token
37+
38+
{{% info %}}
39+
This example uses `cURL` and Linux shell scripts to demonstrate the API; you can use any standard REST client or library.
40+
{{% /info %}}
41+
42+
You can also use the [LangCache SDKs](#langcache-sdk) for Javascript and Python to access the API.
43+
44+
## API examples
45+
46+
### Check cache health
47+
48+
Use `GET /v1/caches/{cacheId}/health` to check the health of the cache.
49+
50+
```sh
51+
GET https://[host]/v1/caches/{cacheId}/health
52+
```
53+
54+
### Search LangCache for similar responses
55+
56+
Use `POST /v1/caches/{cacheId}/search` to search the cache for matching responses to a user prompt.
57+
58+
```sh
59+
POST https://[host]/v1/caches/{cacheId}/search
60+
{
61+
"prompt": "User prompt text"
62+
}
63+
```
64+
65+
Place this call in your client app right before you call your LLM's REST API. If LangCache returns a response, you can send that response back to the user instead of calling the LLM.
66+
67+
If LangCache does not return a response, you should call your LLM's REST API to generate a new response. After you get a response from the LLM, you can [store it in LangCache](#store-a-new-response-in-langcache) for future use.
68+
69+
You can also scope the responses returned from LangCache by adding an `attributes` object to the request. LangCache will only return responses that match the attributes you specify.
70+
71+
```sh
72+
POST https://[host]/v1/caches/{cacheId}/search
73+
{
74+
"prompt": "User prompt text",
75+
"attributes": {
76+
"customAttributeName": "customAttributeValue"
77+
}
78+
}
79+
```
80+
81+
### Store a new response in LangCache
82+
83+
Use `POST /v1/caches/{cacheId}/entries` to store a new response in the cache.
84+
85+
```sh
86+
POST https://[host]/v1/caches/{cacheId}/entries
87+
{
88+
"prompt": "User prompt text",
89+
"response": "LLM response text"
90+
}
91+
```
92+
93+
Place this call in your client app after you get a response from the LLM. This will store the response in the cache for future use.
94+
95+
You can also store the responses with custom attributes by adding an `attributes` object to the request.
96+
97+
```sh
98+
POST https://[host]/v1/caches/{cacheId}/entries
99+
{
100+
"prompt": "User prompt text",
101+
"response": "LLM response text",
102+
"attributes": {
103+
"customAttributeName": "customAttributeValue"
104+
}
105+
}
106+
```
107+
108+
### Delete cached responses
109+
110+
Use `DELETE /v1/caches/{cacheId}/entries/{entryId}` to delete a cached response from the cache.
111+
112+
You can also use `DELETE /v1/caches/{cacheId}/entries` to delete multiple cached responses at once. If you provide an `attributes` object, LangCache will delete all responses that match the attributes you specify.
113+
114+
```sh
115+
DELETE https://[host]/v1/caches/{cacheId}/entries
116+
{
117+
"attributes": {
118+
"customAttributeName": "customAttributeValue"
119+
}
120+
}
121+
```
122+
## LangCache SDK
123+
124+
If your app is written in Javascript or Python, you can also use the LangCache Software Development Kits (SDKs) to access the API.
125+
126+
To learn how to use the LangCache SDKs:
127+
128+
- [LangCache SDK for Javascript](https://www.npmjs.com/package/@redis-ai/langcache)
129+
- [LangCache SDK for Python](https://pypi.org/project/langcache/)

‎content/embeds/langcache-cost-reduction.md

Lines changed: 21 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,21 @@
1+
LangCache reduces your LLM costs by caching responses and avoiding repeated API calls. When a response is served from cache, you don’t pay for output tokens. Input token costs are typically offset by embedding and storage costs.
2+
3+
For every cached response, you'll save the output token cost. To calculate your monthly savings with LangCache, you can use the following formula:
4+
5+
```bash
6+
Est. monthly savings with LangCache =
7+
(Monthly output token costs) × (Cache hit rate)
8+
```
9+
10+
The more requests you serve from LangCache, the more you save, because you’re not paying to regenerate the output.
11+
12+
Here’s an example:
13+
- Monthly LLM spend: $200
14+
- Percentage of output tokens in your spend: 60%
15+
- Cost of output tokens: $200 × 60% = $120
16+
- Cache hit rate: 50%
17+
- Estimated savings: $120 × 50% = $60/month
18+
19+
{{<note>}}
20+
The formula and numbers above provide a rough estimate of your monthly savings. Actual savings will vary depending on your usage.
21+
{{</note>}}

‎content/embeds/rc-langcache-get-started.md

Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,7 @@
1+
To set up LangCache on Redis Cloud:
2+
3+
1. [Create a database]({{< relref "/operate/rc/databases/create-database" >}}) on Redis Cloud.
4+
2. [Create a LangCache service]({{< relref "/operate/rc/langcache/create-service" >}}) for your database on Redis Cloud.
5+
3. [Use the LangCache API]({{< relref "/operate/rc/langcache/use-langcache" >}}) from your client app.
6+
7+
After you set up LangCache, you can [view and edit the cache]({{< relref "/operate/rc/langcache/view-edit-cache" >}}) and [monitor the cache's performance]({{< relref "/operate/rc/langcache/monitor-cache" >}}).

0 commit comments

Comments
 (0)