How do you search your own documents with embeddings, and prove that it works?

An embedding is a list of numbers that places a piece of text on a map of meaning, as this introduction explains. This project skips the theory and does the part that often gets skipped: building a labelled test, so you know if your search works before putting it in front of anybody.

The documents are a fictional community centre's handbook, Harrow Bay: 24 short passages on hours, fees, parking and so on. The questions come in three sets, each with a known right passage: 12 that share words with the document, 14 written on purpose to avoid its words, and 12 in French. The script 08_search_documents.py holds all of it. The models are free downloads, licensed Apache 2.0, and everything ran on the CPU.

Search in ten lines

import numpy as np
from sentence_transformers import SentenceTransformer

model = SentenceTransformer("sentence-transformers/all-MiniLM-L6-v2", device="cpu")
passages = ["The pool closes at 21:00 on weekdays.",
            "Households with low income can apply for a reduced rate.",
            "Only service animals may enter the building."]
vectors = model.encode(passages, normalize_embeddings=True)

question = "I can't afford the yearly price, is there help?"
scores = vectors @ model.encode([question], normalize_embeddings=True)[0]
for i in np.argsort(-scores):
    print(round(float(scores[i]), 2), passages[i])

normalize_embeddings=True brings every vector to length one, so a plain multiplication gives the cosine similarity, a closeness score that is 1 for identical directions. That is the whole of semantic search at this size. For millions of passages you would add an index, a structure that finds neighbours quickly. For a few thousand, a plain loop like this is enough.

The companion script also searches a folder of your own .txt and .md files:

python 08_search_documents.py --ask "how long do you keep lost items?" --folder my_notes/

The measured comparison

Top 1 means the right passage came first. Top 3 means it was in the first three, which matters if a person or a language model reads the results. BM25 is a classic keyword-scoring formula, written in the script in about fifteen lines.

Measured here: 24 passages, 38 questions, Apple M5 Max CPU, sentence-transformers 6.2. With so few questions, one question moves a figure by 7 or 8 points. Read the large differences only. A larger test on your own documents will tell you more than this table.

Two practical results stand out for a bilingual organization. The English model cannot serve French users: it fell to 33%. And the multilingual model gave up some English quality in exchange for working across languages.

Two results that went against expectations

Combining keyword and meaning search is often recommended. Here it did not help. Merging the two rankings with reciprocal rank fusion, which adds up scores based on position in each list, scored 62% top 1 with the first model, below the 88% for embeddings alone. Keywords were so weak on the reworded questions that they dragged good results down. Hybrid search helps most when keywords find things embeddings miss, such as part numbers and rare names, and this handbook had few of those. Test it on your data before you adopt it.

Passage size also behaved in an interesting way. The script joined the handbook into one long text and cut it blindly, as a naive pipeline would.

The "100%" for very large chunks is hollow: with two chunks, the top 3 is the whole handbook. The last column shows the actual cost. Smaller chunks did well because this handbook is made of short, self-contained passages. Long reports whose sentences depend on earlier paragraphs will behave differently. Also note that the model cuts off long input: by default, text past 256 word pieces is ignored without any error.

How do you stop it from answering when nothing matches?

Search always returns something, even for a question your documents cannot answer. Ten off-topic questions, like how to fix a faucet, show the problem. The way out is a minimum score, chosen from your own numbers.

The multilingual model needs a different setting; at 0.4 it answered 77% of good questions and 10% of off-topic ones. Scores from one model mean nothing for another, so recompute this table each time you change model.

Which row to pick depends on what a wrong answer costs. For a staff search tool, answering too much is fine, since a person looks at it. For a public assistant, prefer "I could not find this" and a link to a human. Once a language model writes answers from the passages, you have a second system to test, and this article on evaluating RAG covers the rest. If you are weighing search against training a model, the next project measures that choice directly.