The Embedding Model You Chose in Week One
You picked an embedding model early, it worked well enough, and you never looked at it again. That's the problem.
You were moving fast. Week one of the project, you needed embeddings, you grabbed text-embedding-ada-002 or all-MiniLM-L6-v2 or whatever the tutorial used. It worked. Retrieval looked reasonable. You moved on to chunking, prompting, the interface — the things that felt more urgent.
Six months later, that same model is in production. Nobody has questioned it. Nobody has benchmarked it against your actual domain. Nobody knows if it’s the right choice or just the first choice.
That’s the embedding model problem.
Embedding models are not generic. They’re trained on specific corpora, optimized for specific similarity tasks, and they embed different kinds of text with wildly different quality. A model trained on web text does not automatically understand your internal support tickets, legal contracts, or financial reports. It’ll produce something. It just won’t produce the best retrieval for your domain.
The failure is silent. That’s what makes it dangerous. You don’t get an error. You get retrieval that’s good enough to pass a demo, mediocre enough to frustrate real users, and invisible enough to blame on the model or the prompts.
What you should have done — and can still do:
Evaluate on your actual data. Take 50-100 representative queries. Know what the correct chunks are. Run your embedding model and measure recall at k. This is not optional if you care about output quality.
Run a comparison. text-embedding-3-large, bge-large-en, e5-mistral-7b-instruct, domain-specific fine-tuned models — the landscape has changed significantly in the last year. What you picked in week one may not be the best available option today.
Know the tradeoffs. Larger models embed better but cost more and are slower. Smaller models are fast and cheap but may miss nuance in technical domains. There’s no universal winner — there’s the right model for your latency budget, your query type, and your data.
Consider fine-tuning. If you have labeled query-document pairs from real user interactions, fine-tuning an embedding model on your domain is one of the highest-leverage RAG improvements available. Most teams never try it because it sounds hard. It’s not.
The embedding model sits at the foundation of your retrieval stack. Everything above it — chunking strategy, index design, reranking, prompting — depends on the quality of the representations it produces. Shaky foundations don’t announce themselves. They just limit your ceiling.
You picked it in week one. Go back and check your work.