Every RAG tutorial ends at the same two steps: embed and query. You chunk your documents, store (id, embedding), run a nearest-neighbour search, and the demo works. Then you put it in front of real users and it quietly falls apart. The vector was never the hard part. What you store next to it is. A production retrieval system over real documents makes at least seven storage decisions before it is reliable, and the tutorial default silently picks the wrong side of most of them. This is the checklist: what to store with your embeddings, and the trade-off hiding behind each choice.
The uncomfortable truth is that “embed and query” is not a starting point you improve from. It is a decision you did not know you were making. The moment real scale, real permissions, and real compliance enter the picture, everything changes.
1. The Vector Alone Is Useless
A v1 that stores only (id, embedding) can find nearest neighbours and return their IDs. That is all it can do. An LLM cannot read a 768-dimensional float vector. It needs the actual text to build a prompt, and you need enough context around that text to show a user where an answer came from, delete a document cleanly, or keep two tenants apart.
The record a production system stores looks closer to this:
chunk = {
“embedding”: embed(text),
“text”: text, # the LLM reads text, not vectors
“doc_id”: “arxiv2205.14135”, # update or delete a whole document at once
“source_uri”: “https://…”, # show the user where it came from
“page”: 4,
“content_hash”: sha256(text), # skip re-embedding unchanged chunks
“embed_model”: “bge-large-v1.5”, # never mix vector spaces
“visibility”: “public”, # who is allowed to see this
}
Each field earns its place. doc_id matters because documents get revised and retracted, and without it you would scan the entire index to find every chunk belonging to one paper. content_hash solves a cost problem: re-embedding tens of thousands of chunks against a paid API costs money and latency, so a hash lets you check whether a chunk actually changed before paying to recompute it. embed_model guards a subtler bug: swap in a newer embedding model and the vector space changes entirely, so old and new vectors stop being comparable even when the dimensions match, and mixing them degrades every search with no error to warn you. visibility is what lets a single index serve public and private documents, or multiple tenants, without standing up a second one.
Think of a bare embedding as a coat-check ticket with only a number on it. The number matches fast and never lies, but it does not tell you what the coat looks like, whose it is, or whether it belongs in the building. Every field above is something written on the ticket so that once you find the match, you actually have something usable.
There is a second decision buried here. Do you copy that metadata into every chunk, or keep it once in a separate table? Copying means a read never leaves the chunk, no joins, no round trips, which matters when reads vastly outnumber writes. The cost lands the moment metadata changes: flipping one document’s visibility means updating every chunk it owns, sometimes thousands of rows. Keep the metadata normalised in one place and the trade flips. Updates become a single write, but every read now pays a join. Neither is correct in the abstract. The right answer is a function of how often your metadata changes against how often you read.
2. Keep the Big Files Out of the Hot Path
Base64-encoding a binary file inflates it by roughly a third, and embedding that string directly in a chunk record means every read — even one that only needs the vector for a similarity comparison — has to page in kilobytes of image bytes it does not need. At scale that bloats the memory footprint of in-memory index structures and slows every query, not just the ones touching an image.
Store a pointer instead:
figure = {
“embedding”: clip_embed(image),
“caption”: “Figure 2: memory hierarchy”,
“modality”: “image”,
“media_uri”: “s3://blob/2205.14135/fig2.png”, # a pointer, not the file
}
A vector index is built for fast, lightweight lookups, not for hauling heavy files around. Cramming a full image into it is like posting a couch through a letterbox: it blocks everything else trying to get through. Leave a pointer, keep the file in object storage, and the index stays fast for every query while the large blob is fetched only when it is actually needed.
3. Where Vector Similarity Structurally Cannot Help
Dense embeddings are a powerful default and a narrow one. Three common query shapes defeat them outright, and each needs a different index alongside your vectors.
Different modalities need rank fusion, not score comparison
A model like CLIP puts an image and its caption close together in one shared space, which makes a single index possible, but its text understanding is weaker than a text-only model’s because it compromises across two modalities during training. Split into per-modality indexes and you get the best model for each, but a cosine score from a text model and a cosine score from CLIP are not on the same scale, so you cannot sort both lists by raw score. Reciprocal Rank Fusion sidesteps this by throwing away the scores and fusing on rank position instead:
def rrf(rank, c=60): return 1 / (c + rank)
Rank position is comparable across systems even when the underlying numbers are not.
Relationships live in a graph, not in word overlap
Ask “what improved on FlashAttention?” and pure similarity search fails, because a paper that improves on another typically describes a different mechanism in different words. It might share only a sentence or two of vocabulary with the original. It responds to that paper by citing it, not by repeating it. Similarity search judges relatedness by comparing vocabulary, like guessing how close two scientists are by counting shared words in their papers. Two rivals who have argued for a decade sound nearly identical, while a stranger reusing a few terms looks just as close. A citation graph is the family tree: it does not read what people say, it tracks who is actually talking to whom. Store entities and citation edges next to your chunks, then retrieve in two stages. Vector search finds seed documents, and graph traversal walks one hop out to surface papers a similarity search would never return.
Exact strings need keyword search
Dense models tokenise text into subword pieces, which is exactly why they are bad at literal matches. “FlashAttention-2” might split into a piece resembling “FlashAttention” plus a generic number token, so the model represents it as roughly “something about FlashAttention” rather than as a distinct entity. The same failure hits arXiv IDs, product codes, and function names. A keyword index using BM25 scores on exact token overlap, so a literal match on “2205.14135” ranks highly precisely because it is exact. Run it alongside your dense index and fuse the results.
4. The Decisions Where “Wrong” Beats “Slow”
Most storage trade-offs are speed against cost. Two are different: get them wrong and the system is not slow, it is incorrect or non-compliant.
Filter permissions during the search, never after
A vector database fetches approximate nearest neighbours and, only if asked, applies a permission filter to what it already returned. Search the top eight candidates with no idea about permissions and all eight can legitimately be documents the user is not allowed to see, so filtering afterward returns nothing, even when plenty of public matches sit further down a list the search never surfaced. Push the permission check inside the search and the algorithm only ever considers candidates the user is allowed to see:
index.search(embed(q), k=8, filter={“visibility”: {“$in”: user_groups}})
Most production vector databases support this as a native filtered-search parameter. It is worth checking explicitly, because the difference is not a performance tweak. It is the difference between a correct result and an empty one. Post-filtering is a nightclub that lets everyone through the door and checks tickets afterward, by which point the room is full of the wrong people and the actual guests are stuck outside. Pre-filtering is a bouncer at the door.
Deletes: gone forever, or marked as gone
A hard delete removes the row and eventually its vector, which is what you want for throwaway or cache-like data, but it leaves no record when someone later asks why something is missing or who removed it. A soft delete sets a deleted_at timestamp and leaves the row in place, which means every query now carries an extra filter to avoid surfacing dead data — a small constant tax forever. For anything touching compliance, legal holds, or retraction handling, that tax is worth paying. For logs nobody will ever audit, it is pure overhead.
5. Four Layers, One Orchestrator: the PoC-to-Production Gap
Put the seven decisions together and a shape appears. A production retrieval system is not one search box, it is four cooperating layers:
Query
|
Orchestrator (fuse + rerank)
|
Vector layer: text / image / keyword indexes + permission filters
|
Graph layer: entities + citations
|
Object storage: raw files, via pointers
None of these is optional past a demo. The vector layer answers “what is related to this.” The graph layer answers “what is connected to this,” which vector similarity structurally cannot, no matter how good the embedding model gets. Object storage keeps large files off the hot path. The orchestrator’s real job is deciding, per query, which layers to consult and how to merge what comes back. Treat this as one search box instead of four systems and the architecture is where things quietly break as they scale.
This is the exact gap between a retrieval demo and a retrieval product, and it is not unique to RAG. It is where most AI builds stall. GoTeams takes a deliberately narrow position on AI: AI generates code fast, but LLMs provide snippets, not systems, and MVPs built with AI alone rarely scale. The work that separates a notebook returning plausible answers from a system returning correct, permissioned, auditable ones is orchestration, and orchestration is a human discipline, not a model capability.
That is what an AI-orchestrated engineering team is built to do. Zennify (AI and communication, 17-person team) is a working example: a contact-centre platform extended into a framework for AI assistants, with LLM orchestration, speech services, and Twilio integration behind 200+ configuration options. It is now a European market leader serving clients including Nestlé and FlixBus. The difference between that and a stalled pilot was never the model. It was the layers built around it, and the team that knew which decisions to make.
Frequently Asked Questions
What should you store with embeddings in a RAG system?
Store the source text, not just the vector, since an LLM reads text and cannot read a float vector. Add a document ID so you can update or delete a whole document at once, a source URI to show users where an answer came from, a content hash to skip re-embedding unchanged chunks, the name of the embedding model used, and a visibility field for permissions. The vector finds the match. Everything stored next to it is what makes the match usable, correct, and safe to serve.
Why does pure vector similarity search fail on some queries?
Dense embeddings match on semantic similarity, which is the wrong tool for three query shapes. Exact strings like an arXiv ID or a function name get tokenised into subword pieces and lose their identity, so they need a keyword index using BM25. Relationship questions like “what improved on X” fail because the answer cites X rather than repeating its words, so they need a citation graph. And scores from different embedding models are not comparable, so multimodal search needs rank fusion rather than raw score sorting. Production systems run these indexes together and fuse the results.
What is the difference between pre-filtering and post-filtering in vector search?
Pre-filtering applies a permission filter inside the search, so the algorithm only ever ranks candidates the user is allowed to see. Post-filtering runs the search first and removes forbidden results afterward, which can return an empty set even when valid matches exist further down the list, because the search only ever returned a small top-k that happened to be entirely private. For any system with permissions or multiple tenants, pre-filtering is not an optimisation. It is the difference between a correct result and a wrong one. Most production vector databases expose it as a filtered-search parameter.
Why do so many RAG systems work as a demo but fail in production?
Because “embed and query” is a decision, not a default, and it silently picks the wrong side of most production trade-offs. A demo has no permissions, no multi-tenancy, no compliance requirement, no exact-match queries, and no relationship queries, so a bare vector index looks sufficient. Real usage introduces all five at once. The fix is not a better model. It is the four-layer architecture of vector, graph, object storage, and orchestrator, plus the discipline to decide each storage trade-off deliberately. That orchestration work is where AI builds most often stall.
How do you decide which RAG storage trade-off is right?
There is no universally correct answer, only the trade made in the right direction for your system. Every decision is the same shape: spend effort now, at write or ingestion time, to save effort later, at query time, or take the trade the other way. Content hashing spends storage to save compute. Copying metadata spends write cost to save read latency. A citation graph spends ingestion complexity to answer queries similarity cannot. Measure your actual read-to-write ratio, your query shapes, and your compliance needs first. A setup that is obviously right at a 1000:1 read/write ratio can be just as obviously wrong for a system that re-ingests constantly.
The Bottom Line
If you are building a throwaway demo over a handful of public documents, embed and query is fine, and everything above is overhead you do not need yet. But the moment real users, real permissions, and real scale arrive, the tutorial default stops being a starting point and becomes a set of silent decisions working against you. A production retrieval system stores far more than vectors: text, identifiers, hashes, model names, permissions, pointers, graph edges, and a keyword index — each one a trade you chose on purpose. Know your read-to-write ratio, your query shapes, and your compliance needs. Then pay in the currency you can afford.
The gap between a retrieval demo and a retrieval product is orchestration, and orchestration is a human discipline. That is what an AI-orchestrated engineering team delivers: engineers who know which of these decisions to make, in which direction, for the system you are actually building, and who ship it as something that stays correct as it scales. That is the difference between a pilot that impresses and a product that lasts.
Leave a Reply