I was debugging a client’s internal knowledge-base chatbot when the security team stopped me dead in my tracks. They refused to let any user query leave the corporate network, yet the LLM we were using lived in the cloud. The only way out was to bring the retrieval side on-premise and keep the vector store private. What I ended up with is a fully self-contained Retrieval-Augmented Generation (RAG) stack that lives on a single workstation and never sends data to a third-party service.

Why a private vector database matters

When you embed documents with an LLM, the resulting vectors are essentially fingerprints of the text. If those fingerprints are stored in a public service, anyone with API access can reconstruct similarity queries and potentially infer sensitive content. A private vector database eliminates that risk by keeping the index inside your firewall. It also removes latency spikes caused by round-trips to a remote service, a noticeable win when you’re serving interactive chat.

Choosing the right storage engine

I tried three options before settling on one that balanced speed, simplicity, and resource usage:

  1. SQLite with the vector extension, easy to ship, works on any OS, but struggles with more than a few hundred thousand vectors.
  2. FAISS (Facebook AI Similarity Search), blazing fast, great for large corpora, but requires compiled binaries and a separate process.
  3. Qdrant, an open-source vector DB with a REST API, built-in filtering, and persistent storage.

For my use case, a medium-size product catalog of ~30k items, Qdrant gave the best trade-off. It runs as a Docker container, persists data to disk, and offers a clean HTTP interface that any language can talk to.

Setting up Qdrant locally

First, pull the official image and start a container that maps the data folder to the host:

docker run -d \
  -p 6333:6333 \
  -v $(pwd)/qdrant_storage:/qdrant/storage \
  qdrant/qdrant
  • -p 6333:6333 exposes the REST port.
  • -v … makes the vectors survive container restarts.
  • The container runs in the background (-d).

Once the container is up, you can verify it with a simple curl:

curl http://localhost:6333/collections

You should see an empty JSON object, meaning no collections exist yet.

Indexing your documents

The workflow I follow is:

  1. Chunk the source material, split PDFs, HTML, or markdown into 200-300 word chunks.
  2. Generate embeddings, I use the sentence-transformers/all-MiniLM-L6-v2 model locally via the transformers library.
  3. Push vectors to Qdrant, each chunk becomes a point with a payload that stores the original text and metadata.

Here’s a minimal Python script that does all three steps:

# language: python
import os, json, glob
from transformers import AutoTokenizer, AutoModel
import torch
import requests

# 1. Load model
model_name = "sentence-transformers/all-MiniLM-L6-v2"
tokenizer = AutoTokenizer.from_pretrained(model_name)
model = AutoModel.from_pretrained(model_name)

def embed(text):
    inputs = tokenizer(text, return_tensors='pt', truncation=True, max_length=256)
    with torch.no_grad():
        embeddings = model(**inputs).last_hidden_state[:,0,:]
    return embeddings.squeeze().cpu().numpy().tolist()

# 2. Prepare chunks (very naive split)
chunks = []
for file_path in glob.glob('docs/*.txt'):
    with open(file_path, 'r', encoding='utf-8') as f:
        text = f.read()
    for i in range(0, len(text), 500):
        chunk = text[i:i+500]
        chunks.append({"id": f"{os.path.basename(file_path)}_{i}", "text": chunk})

# 3. Send to Qdrant
url = "http://localhost:6333/collections/my_docs/points"
points = []
for c in chunks:
    vec = embed(c['text'])
    points.append({
        "id": c['id'],
        "vector": vec,
        "payload": {"text": c['text']}
    })

payload = {"points": points}
requests.put(url, json=payload)
print('Indexed', len(points), 'chunks')
  • embed converts a string into a 384-dimensional vector.
  • The loop creates overlapping chunks of 500 characters, adjust to your needs.
  • The final PUT call creates the collection my_docs if it does not exist and uploads all points.

Querying the store for relevant context

When a user asks a question, you embed the query, ask Qdrant for the nearest neighbours, and feed the retrieved texts into the LLM as a prompt. The request looks like this:

curl -X POST http://localhost:6333/collections/my_docs/points/search \
  -H 'Content-Type: application/json' \
  -d '{
    "vector": [0.12, 0.34, …],
    "top": 5,
    "with_payload": true
  }'
  • "vector" is the query embedding.
  • "top": 5 returns the five most similar chunks.
  • "with_payload": true includes the original text in the response.

In Python you can wrap this in a helper:

# language: python
def retrieve(query, top_k=5):
    q_vec = embed(query)
    resp = requests.post(
        "http://localhost:6333/collections/my_docs/points/search",
        json={"vector": q_vec, "top": top_k, "with_payload": True}
    )
    results = resp.json()['result']
    return [r['payload']['text'] for r in results]

Now retrieve("How do I reset my password?") gives you the most relevant passages from your docs.

Wiring retrieval into a local LLM

I prefer running a quantised Llama-2 7B model with llama.cpp because it runs comfortably on a 16 GB laptop GPU. The prompt template I use is:

You are a helpful assistant. Use only the provided context to answer the question.

Context:
{retrieved_chunks}

Question: {user_query}
Answer:

The code that glues everything together is straightforward:

# language: python
from subprocess import Popen, PIPE

def ask_llm(query):
    context = "\n---\n".join(retrieve(query))
    prompt = f"You are a helpful assistant. Use only the provided context to answer the question.\n\nContext:\n{context}\n\nQuestion: {query}\nAnswer:"\n    proc = Popen(["./llama_cpp", "-m", "llama-2-7b.Q4_K_M.gguf", "-p", prompt], stdout=PIPE, stderr=PIPE)
    out, _ = proc.communicate()
    return out.decode('utf-8')
  • The function builds the prompt with the retrieved chunks.
  • llama_cpp reads the prompt from -p and streams the answer.
  • No network call ever leaves the machine.

Practical warnings you’ll hit first

  1. Embedding dimension mismatch, Qdrant expects a fixed vector size. If you switch models, you must delete the old collection or re-create it with the new dimension.
  2. Disk space, each vector takes roughly dim * 4 bytes. A million 384-dim vectors need ~1.5 GB. Keep an eye on the storage mount.
  3. Cold start latency, loading the LLM for the first time can take seconds. Warm it up in a background job if you need sub-second responses.

Scaling beyond a single machine

If your corpus grows to tens of millions of documents, consider sharding the index across multiple Qdrant nodes. The REST API supports a replication_factor parameter when you create a collection. You’ll also need a vector-aware load balancer; the open-source nginx-plus module can route based on query size, but for most teams a managed service like Weaviate Cloud is the easier path.

Security checklist for a private RAG stack

  • Run the vector DB behind a firewall and expose only the internal port.
  • Use TLS for the REST endpoint, Qdrant supports it via a reverse proxy.
  • Restrict container capabilities, drop all Linux capabilities you don’t need (--cap-drop ALL).
  • Audit payloads, never store raw user-generated text in the payload; keep only sanitized excerpts.
  • Rotate embeddings, when you upgrade the embedding model, re-index to avoid stale vectors that could leak older semantics.

FAQ

How do I choose an embedding model for a private setup?

Pick a model that can run on CPU or a modest GPU and fits your latency budget. all-MiniLM-L6-v2 is a good default; it balances quality and size. If you need multilingual support, sentence-transformers/paraphrase-multilingual-MPNET-base-v2 works similarly.

Can I use Qdrant without Docker?

Yes. Qdrant provides pre-compiled binaries for Linux, macOS, and Windows. Download the archive, run ./qdrant serve, and point the --service-url flag to your desired storage path.

What if my LLM needs more context than the top-5 chunks provide?

Increase top_k or experiment with a hybrid approach: first retrieve a broader set, then run a secondary similarity filter on the raw text to narrow it down. Remember that longer prompts increase inference time, so find the sweet spot for your hardware.

Wrapping up

If you ever need to keep user queries and proprietary knowledge inside your own perimeter, a private vector database plus a locally-run LLM is the cleanest solution. The stack I described, Qdrant, a tiny embedding model, and a quantised Llama, fits on a laptop, respects privacy, and avoids the latency of cloud calls. Start small, index a few hundred documents, and let the architecture grow with you.