Retrieval-Augmented Generation (RAG) is a technique for grounding a large language model (LLM)'s answers in a specific document collection at query time, without baking that knowledge into the model's weights through fine-tuning. The LLM retrieves relevant passages from an external store and uses them as context for generation. RAG became the dominant pattern for building LLM applications over proprietary or frequently-updated document collections because it is fast to set up, scales well, and keeps source material separate from the model.
Source documents are split into chunks (typically – tokens each) and converted into vector embeddings — dense numerical representations of semantic meaning — using an embedding model. These embeddings are stored in a vector database alongside the original text of each chunk.
When a user submits a query, the query is also converted into a vector embedding. The vector database computes similarity scores between the query embedding and all stored chunk embeddings (typically using cosine similarity or dot product) and returns the top- most similar chunks.
The retrieved chunks are injected into the LLM's context window as grounding material — usually formatted as: "Here are relevant passages: [chunks]. Now answer the following question: [query]." The LLM generates an answer conditioned on these passages rather than relying solely on its parametric (trained) knowledge.
RAG has no memory across sessions. Each query retrieves and synthesizes from scratch. If the system figures out that document A and document B contradict each other on a key point, that insight disappears at session end. The next query has to re-derive it. There is no accumulation — no compounding of understanding across time.
Retrieval operates on surface-text similarity between query and chunk. A chunk is a raw text passage with no semantic structure imposed on it. If the answer to a question requires combining insights from five different documents, the retrieval step must happen to return all five relevant chunks — which depends on the query phrasing closely matching the chunks' surface language. Concept-level questions (e.g., "what does the literature say about X?") are poorly served by passage-level similarity search.
The most expensive part of knowledge work — synthesizing across many sources to identify patterns, contradictions, and connections — is re-performed on every query. RAG provides no mechanism for storing the results of that synthesis permanently. A human analyst who has read 50 papers builds an internal model that improves with each paper; a RAG system does not.
The retrieval and ranking logic is opaque to the user. It is difficult to know whether important material is being systematically missed (because its chunk embeddings don't match query embeddings despite semantic relevance), whether retrieved chunks are actually the most relevant, or whether the answer is missing context that would change its meaning. Debugging a RAG pipeline requires inspecting embedding spaces, which is not accessible to most users.
Because documents are split into fixed-size or rule-based chunks, important context can be split across chunk boundaries. A passage that sets up an argument in one chunk and delivers the conclusion in the next may be retrieved in isolation, producing an incomplete or misleading grounding for generation.
| Dimension | RAG | LLM Wiki |
|---|---|---|
| Knowledge layer | Vector database of raw text chunks | Human-readable markdown concept pages |
| Per-query work | Retrieve and synthesize from scratch | Read pre-compiled, cross-linked pages |
| Accumulation | None — stateless | Persistent — each ingest compounds the structure |
| Cross-document synthesis | Re-derived every query | Pre-compiled at ingest, stored permanently |
| Auditability | Opaque retrieval pipeline | Every page is human-readable and correctable |
| Setup cost | Low (automatic indexing) | High (LLM must ingest and structure each source) |
| Best for | Large unstructured collections, lookup tasks | Multi-source synthesis, compounding research |
The two approaches are not mutually exclusive. A hybrid system could use an LLM Wiki for core conceptual knowledge and RAG for fast lookup against a large raw document corpus — consulting the wiki for context and the RAG index for specific evidence.
RAG is the current default for LLM-powered knowledge retrieval, and understanding its limitations is prerequisite to understanding why the LLM Wiki pattern represents a meaningful advance for certain use cases. The key insight is that RAG optimizes for retrieval speed and breadth at the cost of accumulation and synthesis depth. For use cases where the value lies in compounding understanding across many sources over time, RAG's statelessness is not a minor inconvenience but a fundamental architectural mismatch.