Your Smart Caching Layer for LLM Applications
Stop wasting money on redundant API calls. cachelm intelligently caches LLM responses by understanding the meaning of queries, not just the exact words. Slash costs, accelerate response times, and build smarter, faster AI products.
LLMs are powerful but costly. Users often ask the same question in slightly different ways, leading to identical, expensive API calls that traditional key-value caches can't detect.
"Explain quantum computing" vs. "Break down quantum computing basics"
A traditional cache sees two different requests. cachelm sees one.
By understanding semantic intent, cachelm serves a cached response for the second query, saving you money and delivering the answer in milliseconds.
| Feature | Benefit |
|---|---|
| Semantic Caching | Intelligently handles paraphrased queries to maximize cache hits. |
| Cost & Latency Reduction | Cut LLM API costs by 20-40% and slash response times from seconds to milliseconds. |
| Seamless Integration | Drop it into your existing openai client code with just a few lines. No major refactoring needed. |
| Pluggable Architecture | Modular design lets you easily swap vector databases (Chroma, Redis, etc.) and models. |
| Streaming Support | Full, out-of-the-box compatibility with streaming chat completions. |
| Production-Ready | Battle-tested and built for scale with enterprise-grade integrations. |
Perfect For:
- High-traffic LLM applications where API costs are a concern.
- Real-time chatbots and virtual assistants that require instant responses.
- Cost-sensitive production deployments and internal tools.
cachelm intercepts your LLM API calls and adds a smart caching layer.
- Intercept: A user sends a prompt through the
cachelm-enhanced client. - Vectorize: The prompt is converted into a numerical representation (an embedding) that captures its semantic meaning.
- Search:
cachelmsearches your vector database for a similar, previously cached prompt. - Decision:
- Cache Hit: If a semantically similar prompt is found within a configurable threshold, the cached response is returned instantly. ⚡
- Cache Miss: If no match is found, the request is sent to the LLM provider (e.g., OpenAI). The new response is then vectorized and stored in the cache for future use.
Install cachelm with the default dependencies (ChromaDB & FastEmbed):
pip install "cachelm[chroma,fastembed]"Enhance your OpenAI client with caching in just a few lines.
from openai import OpenAI
from cachelm import OpenAIAdaptor, ChromaDatabase, FastEmbedVectorizer
# 1. Initialize the caching components
# By default, ChromaDatabase runs in-memory.
database = ChromaDatabase(
vectorizer=FastEmbedVectorizer(),
distance_threshold=0.1 # Lower = stricter matching, Higher = looser matching
)
# 2. Create the adaptor and get your enhanced client
# Replace with your actual OpenAI API key
client = OpenAI(api_key="sk-...")
adaptor = OpenAIAdaptor(module=client, database=database)
smart_client = adaptor.get_adapted()
# 3. Use the client as you normally would!
print("--- First call (will be slow and hit the API) ---")
response = smart_client.chat.completions.create(
model="gpt-4o-mini",
messages=[{"role": "user", "content": "Explain the basics of quantum computing in simple terms."}],
)
print(response.choices[0].message.content)
print("\n--- Second call (will be fast and served from cache) ---")
# This query is phrased differently but has the same meaning
cached_response = smart_client.chat.completions.create(
model="gpt-4o-mini",
messages=[{"role": "user", "content": "Could you break down how quantum computers work?"}],
)
print(cached_response.choices[0].message.content)
# The 'x-cachelm-status' header confirms if the response was a cache HIT or MISS
print(f"\nCache status: {cached_response.headers.get('x-cachelm-status')}")To persist your cache across application restarts, configure your database.
import chromadb
# Configure ChromaDB to save data to disk
persistent_settings = chromadb.config.Settings(
is_persistent=True,
persist_directory="./my_llm_cache" # Directory to store the database
)
database = ChromaDatabase(
vectorizer=FastEmbedVectorizer(),
chromaSettings=persistent_settings,
distance_threshold=0.1
)
# The rest of your setup remains the same!
# adaptor = OpenAIAdaptor(...)The middleware system lets you hook into the caching process to modify or filter data. This is perfect for handling variable data (like names or IDs) and protecting sensitive information.
Key hooks:
pre_cache_save: Runs before a new response is saved to the cache.post_cache_retrieval: Runs after a response is retrieved from the cache.
Imagine your prompts contain usernames that change but the core question is the same. The Replacer middleware substitutes placeholders to ensure these queries result in a cache hit.
from cachelm.middlewares.replacer import Replacer, Replacement
# Define replacements: "Anmol" will be treated as {{name}} for caching
replacements = [
Replacement(key="{{name}}", value="Anmol"),
Replacement(key="{{user_id}}", value="user_12345"),
]
adaptor = OpenAIAdaptor(
...,
middlewares=[Replacer(replacements)]
)
# Now these two queries will map to the same cache entry:
# 1. "My name is Anmol. What's my order status for ID user_12345?"
# 2. "My name is Bob. What's my order status for ID user_67890?" (if Bob and user_67890 are also in replacements)Before caching, "Anmol" becomes {{name}}. After retrieval, {{name}} is changed back to "Anmol". This dramatically improves cache hits for template-like queries.
cachelm is designed to be modular. Install only what you need.
| Category | Technology | pip install "cachelm[...]" |
|---|---|---|
| Databases | ChromaDB | [chroma] |
| Redis | [redis] |
|
| ClickHouse | [clickhouse] |
|
| Qdrant | [qdrant] |
|
| Vectorizers | FastEmbed | [fastembed] |
| RedisVL | [redis] |
|
| Text2Vec-Chroma | [chroma] |
|
| LLMs | OpenAI | (Included by default) |
More integrations for providers like Anthropic and Cohere are coming soon!
cachelm is ready for demanding production environments.
from cachelm.databases.redis import RedisDatabase
from cachelm.vectorizers.redisvl import RedisVLVectorizer
# Assumes you have a Redis instance with the RediSearch module
database = RedisDatabase(
vectorizer=RedisVLVectorizer(model="sentence-transformers/all-MiniLM-L6-v2"),
redis_url="redis://localhost:6379",
index_name="llm_cache_prod"
)from cachelm.databases.clickhouse import ClickHouse
from cachelm.vectorizers.fastembed import FastEmbedVectorizer
# Connect to a self-hosted or ClickHouse Cloud instance
database = ClickHouse(
vectorizer=FastEmbedVectorizer(),
host="your.clickhouse.cloud.host",
port=8443,
username="default",
password="your-password"
)We welcome contributions! The modular design makes it easy to add new components.
Implement the Vectorizer interface to support a new embedding model.
from cachelm.vectorizers.vectorizer import Vectorizer
class MyVectorizer(Vectorizer):
def embed(self, text: str) -> list[float]:
return my_embedding_model.encode(text).tolist()
def embed_many(self, texts: list[str]) -> list[list[float]]:
return my_embedding_model.encode(texts).tolist()Implement the Database interface to connect to a different vector store.
from cachelm.databases.database import Database
from cachelm.types.chat_history import Message
class MyDatabase(Database):
def find(self, history: list[Message], distance_threshold=0.1) -> Message | None:
# Your logic to search for a similar history vector
pass
def write(self, history: list[Message], response: Message):
# Your logic to store the history vector and response
pass
# ... implement connect() and disconnect()See our Contribution Guide to get started. We're excited to see what you build!
cachelm is licensed under the MIT License. It is free for both personal and commercial use.
Ready to Accelerate Your LLM Workloads? Report an Issue | View the Source