RAG chatbot with semantic caching

Problem

The client needed a chatbot that answers from their own content. Calling a model for every question would repeat the same cost for questions already answered, and a model could invent answers when the content did not cover a question.

Approach

Ingestion. The client’s Word documents (.docx) are converted to HTML and split at their heading tags, so each chunk covers one topic, with overlap between neighbouring chunks. The chunks are embedded and saved to a FAISS index on disk.

Answering a question:

1. Exact cache: a question seen before is answered from the cache before anything is embedded.
2. Semantic cache: otherwise the question is embedded and compared with cached questions; a match above the similarity threshold returns the cached answer.
3. Retrieval with a relevance gate: the index returns the top matches. If even the best one is below the relevance threshold, the bot replies with a prewritten fallback message, chosen at random, and the model is never called.
4. Rerank: the candidates are reranked and the top 3 go to the model as context.
5. Answer with a second fallback: the model answers only from that context and falls back when it is not enough to answer.
6. Cache: the question, its embedding and the answer are stored, so the next identical or similar question is served from the cache.

Key decisions

1. Two cache layers: exact text first, then semantic

   Why

   The exact check skips even the embedding call; the semantic check catches rephrased questions.

   Trade-off

   The similarity threshold needs tuning: too low and a different question gets a wrong cached answer, too high and paraphrases miss. Cached answers can go stale when the content changes.

2. Chunks split at the documents' headings, with overlap

   Why

   Each chunk follows the document structure and covers one topic; the overlap keeps context across boundaries.

   Trade-off

   Chunk sizes vary, and it depends on the documents using real heading styles.

3. A relevance gate before the model

   Why

   Questions the content does not cover cost nothing and cannot produce an invented answer.

   Trade-off

   A genuine question phrased unusually can get the fallback reply.

4. Retrieve wide, rerank down to the top 3

   Why

   Wide retrieval protects recall; reranking puts only the best context in a short prompt.

   Trade-off

   An extra reranking step on every uncached question.

5. A second fallback inside the model step

   Why

   Even when retrieval passes the gate, the model declines rather than guesses if the context falls short.

   Trade-off

   Some answerable questions are declined when the best context is incomplete.

6. FAISS index saved on disk

   Why

   Fast similarity search without running a separate database.

   Trade-off

   The index has to be rebuilt when the source content changes.

Result

Repeated, rephrased and off-topic questions are answered without a model call, and questions the content cannot answer get a safe reply instead of an invented one.