Redis

Redis vector database introduction and langchain integration guide.

What is Redis?

Most developers from a web services background are familiar with Redis. At its core, Redis is an open-source key-value store that is used as a cache, message broker, and database. Developers choose Redis because it is fast, has a large ecosystem of client libraries, and has been deployed by major enterprises for years.

On top of these traditional use cases, Redis provides additional capabilities like the Search and Query capability that allows users to create secondary index structures within Redis. This allows Redis to be a Vector Database, at the speed of a cache.

Redis as a Vector Database

Redis uses compressed, inverted indexes for fast indexing with a low memory footprint. It also supports a number of advanced features such as:

• Indexing of multiple fields in Redis hashes and JSON
• Vector similarity search (with HNSW (ANN) or FLAT (KNN))
• Vector Range Search (e.g. find all vectors within a radius of a query vector)
• Incremental indexing without performance loss
• Document ranking (using tf-idf, with optional user-provided weights)
• Field weighting
• Complex boolean queries with AND, OR, and NOT operators
• Prefix matching, fuzzy matching, and exact-phrase queries
• Support for double-metaphone phonetic matching
• Auto-complete suggestions (with fuzzy prefix suggestions)
• Stemming-based query expansion in many languages (using Snowball)
• Support for Chinese-language tokenization and querying (using Friso)
• Numeric filters and ranges
• Geospatial searches using Redis geospatial indexing
• A powerful aggregations engine
• Supports for all utf-8 encoded text
• Retrieve full documents, selected fields, or only the document IDs
• Sorting results (for example, by creation date)

RedisVectorStore

This notebook demonstrates the usage of RedisVectorStore from the langchain-redis partner package. RedisVectorStore leverages Redis as a vector database, enabling efficient storage, retrieval, and similarity search of vector embeddings.

Installation

First, we need to install the necessary packages. Run the following command to install langchain-redis, sentence-transformers, and scikit-learn:

%pip install -U langchain-redis langchain-huggingface sentence-transformers scikit-learn

Importing Required Libraries

We'll import the necessary libraries for our tasks:

import os

from langchain.docstore.document import Document
from langchain_redis import RedisVectorStore
from sklearn.datasets import fetch_20newsgroups

API Reference:Document

Setting up Redis Connection

To use RedisVectorStore, you need a running Redis instance. For this example, we assume a local Redis instance running on the default port. Modify the URL if your setup differs:

import os

# Use the environment variable if set, otherwise default to localhost
REDIS_URL = os.getenv("REDIS_URL", "redis://localhost:6379")
print(f"Connecting to Redis at: {REDIS_URL}")

Connecting to Redis at: redis://redis:6379

Let's check that Redis is up an running by pinging it:

import redis

redis_client = redis.from_url(REDIS_URL)
redis_client.ping()

True

Preparing Sample Data

We'll use a subset of the 20 Newsgroups dataset for this demonstration. This dataset contains newsgroup posts on various topics. We'll focus on two categories: 'alt.atheism' and 'sci.space':

categories = ["alt.atheism", "sci.space"]
newsgroups = fetch_20newsgroups(
    subset="train", categories=categories, shuffle=True, random_state=42
)

# Use only the first 250 documents
texts = newsgroups.data[:250]
metadata = [
    {"category": newsgroups.target_names[target]} for target in newsgroups.target[:250]
]

documents = [
    Document(page_content=text, metadata=meta) for text, meta in zip(texts, metadata)
]
len(documents)

250

Let's inspect the first document:

documents[0]

Document(metadata={'category': 'alt.atheism'}, page_content='From: bil@okcforum.osrhe.edu (Bill Conner)\nSubject: Re: Not the Omni!\nNntp-Posting-Host: okcforum.osrhe.edu\nOrganization: Okcforum Unix Users Group\nX-Newsreader: TIN [version 1.1 PL6]\nLines: 18\n\nCharley Wingate (mangoe@cs.umd.edu) wrote:\n: \n: >> Please enlighten me.  How is omnipotence contradictory?\n: \n: >By definition, all that can occur in the universe is governed by the rules\n: >of nature. Thus god cannot break them. Anything that god does must be allowed\n: >in the rules somewhere. Therefore, omnipotence CANNOT exist! It contradicts\n: >the rules of nature.\n: \n: Obviously, an omnipotent god can change the rules.\n\nWhen you say, "By definition", what exactly is being defined;\ncertainly not omnipotence. You seem to be saying that the "rules of\nnature" are pre-existant somehow, that they not only define nature but\nactually cause it. If that\'s what you mean I\'d like to hear your\nfurther thoughts on the question.\n\nBill\n')

Creating Embeddings

We'll use the SentenceTransformer model to create embeddings. This model runs locally and doesn't require an API key:

from langchain_huggingface import HuggingFaceEmbeddings

embeddings = HuggingFaceEmbeddings(model_name="msmarco-distilbert-base-v4")

API Reference:HuggingFaceEmbeddings

Basic Usage with LangChain's RedisVectorStore

Now we'll demonstrate basic usage of RedisVectorStore, including creating an instance, inserting data, and performing a simple similarity search.

Creating a RedisVectorStore instance and inserting data

We'll create a RedisVectorStore instance and populate it with our sample data:

vector_store = RedisVectorStore.from_documents(
    documents,
    embeddings,
    redis_url=REDIS_URL,
    index_name="newsgroups",
    metadata_schema=[
        {"name": "category", "type": "tag"},
    ],
)

Performing a simple similarity search

Let's perform a basic similarity search using a query about space exploration:

query = "Tell me about space exploration"
results = vector_store.similarity_search(query, k=2)

print("Simple Similarity Search Results:")
for doc in results:
    print(f"Content: {doc.page_content[:100]}...")
    print(f"Metadata: {doc.metadata}")
    print()

Simple Similarity Search Results:
Content: From: aa429@freenet.carleton.ca (Terry Ford)
Subject: A flawed propulsion system: Space Shuttle
X-Ad...
Metadata: {'category': 'sci.space'}

Content: From: nsmca@aurora.alaska.edu
Subject: Space Design Movies?
Article-I.D.: aurora.1993Apr23.124722.1
...
Metadata: {'category': 'sci.space'}

Advanced Queries with RedisVectorStore

RedisVectorStore supports more advanced query types. We'll demonstrate similarity search with metadata filtering, maximum marginal relevance search, and similarity search with score.

Similarity search with metadata filtering

We can filter our search results based on metadata:

from redisvl.query.filter import Tag

query = "Tell me about space exploration"

# Create a filter expression
filter_condition = Tag("category") == "sci.space"

filtered_results = vector_store.similarity_search(query, k=2, filter=filter_condition)

print("Filtered Similarity Search Results:")
for doc in filtered_results:
    print(f"Content: {doc.page_content[:100]}...")
    print(f"Metadata: {doc.metadata}")
    print()

Filtered Similarity Search Results:
Content: From: aa429@freenet.carleton.ca (Terry Ford)
Subject: A flawed propulsion system: Space Shuttle
X-Ad...
Metadata: {'category': 'sci.space'}

Content: From: nsmca@aurora.alaska.edu
Subject: Space Design Movies?
Article-I.D.: aurora.1993Apr23.124722.1
...
Metadata: {'category': 'sci.space'}

Maximum marginal relevance search

Maximum marginal relevance search helps in getting diverse results:

# Maximum marginal relevance search with filter
mmr_results = vector_store.max_marginal_relevance_search(
    query, k=2, fetch_k=10, filter=filter_condition
)

print("Maximum Marginal Relevance Search Results:")
for doc in mmr_results:
    print(f"Content: {doc.page_content[:100]}...")
    print(f"Metadata: {doc.metadata}")
    print()

Maximum Marginal Relevance Search Results:
Content: From: aa429@freenet.carleton.ca (Terry Ford)
Subject: A flawed propulsion system: Space Shuttle
X-Ad...
Metadata: {'category': 'sci.space'}

Content: From: moroney@world.std.com (Michael Moroney)
Subject: Re: Vulcan? (No, not the guy with the ears!)
...
Metadata: {'category': 'sci.space'}

Similarity search with score

We can also get similarity scores along with our search results:

# Similarity search with score and filter
scored_results = vector_store.similarity_search_with_score(
    query, k=2, filter=filter_condition
)

print("Similarity Search with Score Results:")
for doc, score in scored_results:
    print(f"Content: {doc.page_content[:100]}...")
    print(f"Metadata: {doc.metadata}")
    print(f"Score: {score}")
    print()

Similarity Search with Score Results:
Content: From: aa429@freenet.carleton.ca (Terry Ford)
Subject: A flawed propulsion system: Space Shuttle
X-Ad...
Metadata: {'category': 'sci.space'}
Score: 0.569670796394

Content: From: nsmca@aurora.alaska.edu
Subject: Space Design Movies?
Article-I.D.: aurora.1993Apr23.124722.1
...
Metadata: {'category': 'sci.space'}
Score: 0.590400338173

Cleanup

After we're done, it's important to clean up our Redis indices:

# Delete the underlying index and it's data
vector_store.index.delete(drop=True)

LangChain Redis Kitchen Sink Example

This portion of the notebook demonstrates a comprehensive example that combines RedisVectorStore, RedisCache, and RedisChatMessageHistory to create a powerful, efficient, and context-aware chatbot system.

Setup and Imports

%pip install -U langchain-openai wikipedia

Ensure you have a Redis server running. You can start one using Docker with:

docker run -d -p 6379:6379 redis:latest

Or install and run Redis locally according to your operating system's instructions.

import os

# Use the environment variable if set, otherwise default to localhost
REDIS_URL = os.getenv("REDIS_URL", "redis://localhost:6379")
print(f"Connecting to Redis at: {REDIS_URL}")

Connecting to Redis at: redis://redis:6379

Importing Required Libraries

import wikipedia
from langchain.globals import set_llm_cache
from langchain.schema import Document
from langchain.text_splitter import CharacterTextSplitter
from langchain_core.output_parsers import StrOutputParser
from langchain_core.prompts import PromptTemplate
from langchain_core.runnables import RunnableLambda, RunnablePassthrough
from langchain_openai import OpenAI, OpenAIEmbeddings
from langchain_redis import RedisCache, RedisChatMessageHistory, RedisVectorStore

API Reference:set_llm_cache | Document | CharacterTextSplitter | StrOutputParser | PromptTemplate | RunnableLambda | RunnablePassthrough | OpenAI | OpenAIEmbeddings

Set OpenAI API key

from getpass import getpass

# Check if OPENAI_API_KEY is already set in the environment
openai_api_key = os.getenv("OPENAI_API_KEY")

if not openai_api_key:
    print("OpenAI API key not found in environment variables.")
    openai_api_key = getpass("Please enter your OpenAI API key: ")

    # Set the API key for the current session
    os.environ["OPENAI_API_KEY"] = openai_api_key
    print("OpenAI API key has been set for this session.")
else:
    print("OpenAI API key found in environment variables.")

OpenAI API key not found in environment variables.
``````output
Please enter your OpenAI API key:  ········
``````output
OpenAI API key has been set for this session.

Create an Index with RedisVL

In this section, we'll set up our vector store using RedisVL, which provides a powerful interface for creating and managing vector indexes in Redis. We'll define a schema for our Wikipedia data, create an index using RedisVL

from langchain_redis import RedisConfig, RedisVectorStore
from redis import Redis
from redisvl.index import SearchIndex
from redisvl.schema import IndexSchema

RedisVL Index Schema

We start by defining a schema for our index. This schema includes:

• A text field for the document content
• A text field for metadata
• A vector field for the document embeddings

The vector field is configured with 1536 dimensions (suitable for OpenAI embeddings), using cosine distance and a FLAT index algorithm.

schema = IndexSchema.from_dict(
    {
        "index": {
            "name": "kitchensink_docs",
            "storage_type": "hash",
            "prefix": "wiki:",
        },
        "fields": [
            {"name": "text", "type": "text"},
            {"name": "url", "type": "tag"},
            {"name": "title", "type": "text"},
            {
                "name": "embedding",
                "type": "vector",
                "attrs": {
                    "dims": 1536,
                    "distance_metric": "cosine",
                    "algorithm": "FLAT",
                },
            },
        ],
    }
)

Creating the RedisVL Index

Using the defined schema, we create a SearchIndex object and use it to create the actual index in Redis. This step sets up the structure that our vector store will use.

# Create the index using RedisVL
redisvl_index = SearchIndex(schema, redis_client)
redisvl_index.create(overwrite=True)

Initializing RedisVectorStore

With the RedisVL index in place, we can now initialize our RedisVectorStore. We use a RedisConfig object to specify the index name and Redis URL, ensuring that our vector store connects to the correct index.

# Initialize RedisVectorStore using the pre-constructed index
config = RedisConfig(
    index_name="kitchensink_docs", redis_url=REDIS_URL, from_existing=True
)
vector_store = RedisVectorStore(OpenAIEmbeddings(), config=config)

23:12:08 redisvl.index.index INFO   Index already exists, not overwriting.

Other Components

We also initialize other components like RedisCache for LLM caching, ChatOpenAI for our language model, and RedisChatMessageHistory for maintaining conversation history.

# Initialize RedisCache
redis_cache = RedisCache(redis_url=REDIS_URL)
set_llm_cache(redis_cache)

# Initialize ChatOpenAI with caching
llm = OpenAI(temperature=0)

# Initialize RedisChatMessageHistory
message_history = RedisChatMessageHistory("kitchensink_chat", redis_url=REDIS_URL)

Populate Vector Store with Wikipedia Data

## Populate Vector Store with Wikipedia Data

from langchain_core.documents import Document
from langchain_text_splitters import RecursiveCharacterTextSplitter


def fetch_wikipedia_content(titles):
    documents = []
    for title in titles:
        try:
            page = wikipedia.page(title)
            doc = Document(
                page_content=page.content, metadata={"title": title, "url": page.url}
            )
            documents.append(doc)
        except wikipedia.exceptions.DisambiguationError as e:
            # Choose the first option from the disambiguation list
            page = wikipedia.page(e.options[0])
            doc = Document(
                page_content=page.content,
                metadata={"title": e.options[0], "url": page.url},
            )
            documents.append(doc)
        except wikipedia.exceptions.PageError:
            print(f"Page not found for {title}")
    return documents


# Fetch some Wikipedia articles
titles = [
    "Artificial Intelligence",
    "Deep Learning",
    "Natural Language Processing",
    "Large Language Models",
    "Robotics",
]
documents = fetch_wikipedia_content(titles)

# Split documents into chunks
text_splitter = RecursiveCharacterTextSplitter(chunk_size=1000, chunk_overlap=200)
splits = text_splitter.split_documents(documents)

# Add to vector store
vector_store.add_documents(splits)

print(f"Added {len(splits)} document chunks to the vector store.")

API Reference:Document | RecursiveCharacterTextSplitter

Added 439 document chunks to the vector store.

Create the retriever

# Create the retriever
retriever = vector_store.as_retriever()

from langchain_core.runnables import RunnableLambda, RunnablePassthrough


def format_docs(docs):
    return "\n\n".join(doc.page_content for doc in docs)


def combine_chat_history_and_question(inputs):
    return f"Chat History: {inputs['chat_history']}\nHuman: {inputs['question']}"


# Update the prompt template to include chat history
prompt_template = """
    You are an AI assistant answering questions based on the provided context and chat history. Be concise and accurate.

    Context: {context}
    {question}
    AI Assistant:
"""
prompt = PromptTemplate(
    template=prompt_template, input_variables=["context", "question"]
)

# Create the RAG chain
rag_chain = (
    {
        "context": lambda x: format_docs(retriever.invoke(x["question"])),
        "question": combine_chat_history_and_question,
        "chat_history": lambda x: x["chat_history"],
    }
    | prompt
    | llm
    | StrOutputParser()
)

API Reference:RunnableLambda | RunnablePassthrough

Interactive Chat Loop

from langchain_core.messages import AIMessage, HumanMessage


def get_chat_history(history):
    return "\n".join(
        [f"{msg.type.capitalize()}: {msg.content}" for msg in history.messages[-5:]]
    )


print("Welcome to the AI Assistant! Type 'exit' to end the conversation.")

chat_history = []
while True:
    user_input = input("Human: ")
    if user_input.lower() == "exit":
        break

    # Add user message to history
    message_history.add_user_message(user_input)

    # Get response from RAG chain
    result = rag_chain.invoke(
        {"question": user_input, "chat_history": get_chat_history(message_history)}
    )

    # Add AI message to history
    message_history.add_ai_message(result)

    print(f"AI: {result}")

print("Thank you for using the AI Assistant!")

API Reference:AIMessage | HumanMessage

Welcome to the AI Assistant! Type 'exit' to end the conversation.
``````output
Human:  What are the core tenets of AI?
``````output
AI: 
The core tenets of AI include reasoning, knowledge representation, planning, learning, natural language processing, perception, and support for robotics. These are the traditional goals of AI research and are essential for creating intelligent machines that can perform tasks on par with humans. Additionally, AI also draws upon various fields such as psychology, linguistics, philosophy, neuroscience, and others to further advance its capabilities.
``````output
Human:  How does AI influence robotics, and viceversa?
``````output
AI: 
AI and robotics have a symbiotic relationship, as advancements in one field often lead to advancements in the other. AI influences robotics by providing the software and algorithms that enable robots to make decisions and learn from their environment. On the other hand, robotics influences AI by providing real-world data and challenges for AI systems to learn from and improve upon. Together, they are driving the development of advanced technologies that have the potential to greatly impact various industries and aspects of our daily lives.
``````output
Human:  exit
``````output
Thank you for using the AI Assistant!

Analysis of the Kitchen Sink Example

This example demonstrates the integration of multiple Redis-based components in LangChain:

1. RedisVectorStore: Used to store and retrieve document chunks from Wikipedia articles. It enables efficient similarity search for relevant context.

2. RedisCache: Implemented to cache LLM responses, potentially speeding up repeated or similar queries.

3. RedisChatMessageHistory: Stores the conversation history, allowing the AI to maintain context across multiple interactions.

The combination of these components creates a powerful, context-aware chatbot system with the following features:

• Efficient Information Retrieval: The vector store allows quick access to relevant information from a large dataset.
• Improved Response Time: Caching helps in reducing API calls for similar or repeated queries.
• Contextual Understanding: The chat history enables the AI to reference previous parts of the conversation.
• Scalability: Redis as a backend allows this system to handle large amounts of data and high traffic efficiently.

This kitchen sink example showcases how these Redis-based components can work together seamlessly in a real-world application, demonstrating the power and flexibility of the langchain-redis package.

Cleanup

# Clear vector store
vector_store.index.delete(drop=True)

# Clear cache
redis_cache.clear()

# Clear chat history
message_history.clear()

print("Cleanup completed.")

Cleanup completed.

Related

• Vector store conceptual guide
• Vector store how-to guides