Implementing redis cache in LLM

Table of contents

Intro

Caching not only improves application performance but for LLMs, it reduces costs significantly and also benefits the environment. As we all know, running an LLM in a data center consumes a lot of resources; eventually, it will be sustainable, but we need to wait and see.

So, as architects/devs, we have the responsibility to create energy-efficient applications. We will see how to implement caching in LLMs to make more faster responses and reduce costs.

Types of Cache available

Traditionally, we find the cached data by key; the application requests the key that holds the cached data. But in LLM, we have the input that differs every time. Say the user might ask Weather today in Chennai; also, next time he might ask Today's weather in Chennai. Even though both have the same meaning, the exact string wouldn’t match.

We will be using Redis cache for our use case. Redis offers both exact string match and semantic match to handle questions that hold the same semantic value. It’s up to us, based on the use case, to decide.

• Exact match
• Semantic match

Step 1 : Packages required

We have used both OpenAI and Gemini for the example, and you would see those packages in the command below:

uv add "google-genai>=2.22.0" "langchain>=1.4.0" "langchain-core>=0.3.0" "langchain-openai>=1.6.0" "langchain-redis>=0.2.5" "python-dotenv>=1.2.3" "redis>=5.0.0"

Step 2: Exact match cache

We are initializing Redis and creating a method to measure the performance of the same query asked by the user twice.

#redis url
REDIS_URL = os.getenv("REDIS_URL", "redis://localhost:6379")

# Initialize RedisCache
redis_cache = RedisCache(redis_url=REDIS_URL)

# Set the cache for LangChain to use
set_llm_cache(redis_cache)

#model

llm = init_chat_model(
 model="gemini-2.5-flash",
 model_provider="google_genai")

# Function to measure execution time
def timed_completion(prompt):
 start_time = time.time()
 result = llm.invoke(prompt)
 end_time = time.time()
 return result, end_time - start_time

# First call (not cached)
prompt = "Explain the concept of caching in three sentences."
result1, time1 = timed_completion(prompt)
print(f"First call (not cached):\nResult: {result1}\nTime: {time1:.2f} seconds\n")

# Second call (should be cached)
result2, time2 = timed_completion(prompt)
print(f"Second call (cached):\nResult: {result2}\nTime: {time2:.2f} seconds\n")

print(f"Speed improvement: {time1 / time2:.2f}x faster")

Below is the performance:

Step 3 : Semantic cache

The exact match will work fine for the the same repeated query; for different queries with the same semantic meaning, there will be a cache miss. Below is the code where we implemented the semantic search.

REDIS_URL = os.getenv("REDIS_URL", "redis://localhost:6379")

#embeddings required
embeddings = OpenAIEmbeddings()

#Initialize semantic_cache from redis
semantic_cache = RedisSemanticCache(
 redis_url=REDIS_URL, embeddings=embeddings, distance_threshold=0.2
)

#model
llm = init_chat_model(
 model="gpt-5.4-mini",
 model_provider="openai")

# Function to test semantic cache
def test_semantic_cache(prompt):
 start_time = time.time()
 result = llm.invoke(prompt)
 end_time = time.time()
 return result, end_time - start_time

# Original query
original_prompt = "Capital of United States?"
result1, time1 = test_semantic_cache(original_prompt)
print(
 f"Original query:\nPrompt: {original_prompt}\nResult: {result1}\nTime: {time1:.2f} seconds\n"
)

# Semantically similar query
similar_prompt = "Can you tell me the capital of United States?"
result2, time2 = test_semantic_cache(similar_prompt)
print(
 f"Similar query:\nPrompt: {similar_prompt}\nResult: {result2}\nTime: {time2:.2f} seconds\n"
)

Below are the performance results:

Conclusion

We saw the performance improvement in the subsequent calls, and it’s up to the business use cases to set up the TTL configuration for the cache. There are several other cache providers available. If you want to see the source code of the above example, refer here.