[Local LLM 16] Search Documents with Embeddings and Cosine Similarity

Building a Local LLM · 16/30 · API and Document Search

Full series and learning sequence · Practice code ZIP

A question about how many days to keep backups may need to find a document passage about a retention period. Embedding search can connect different wording by representing the query and passages as vectors. Inspect the retrieved passages before using them to generate an answer.

Prepare 13 common functions and hypothetical text documents. We do not include personal information or trade secrets in the data we search for.

Preparing the embedding model

Here we turn the text into a vector with EmbeddingGemma. Reply sentences from a conversation model are not used instead of vectors. It is a configuration that examines operation with a small document, and when applied in practice, search quality is evaluated according to language and document type.

ollama pull embeddinggemma:300m
New-Item -ItemType Directory -Force .\docs

Place UTF-8 text in the docs folder. The ZIP contains virtual backup.txt and network.txt. The code below does not read the PDF directly, so writing it requires a step to extract the text first.

Creating chunks and storing vectors

Save the code as semantic_search.py. You can also use the downloaded file. Since it is a method of traversing the entire vector using the standard library, it is suitable for practice with a small number of documents.

import json
import math
from pathlib import Path
from llm_http import post

EMBED = "embeddinggemma:300m"
ROOT = Path(__file__).resolve().parent
DOCS = ROOT / "docs"
INDEX = ROOT / "index.json"

def chunks(text, size=500, overlap=80):
    if not 0 <= overlap < size:
        raise ValueError("Invalid overlap")
    for start in range(0, len(text), size - overlap):
        yield text[start:start + size]
        if start + size >= len(text):
            break

def embed(texts):
    return post("/api/embed", {
        "model": EMBED, "input": texts, "truncate": False
    })["embeddings"]

def cosine(a, b):
    if len(a) != len(b):
        raise ValueError("Embedding dimensions differ")
    den = math.sqrt(sum(x*x for x in a) * sum(y*y for y in b))
    return sum(x*y for x, y in zip(a, b)) / den if den else 0.0

def build():
    rows = []
    for path in sorted(DOCS.glob("*.txt")):
        for n, text in enumerate(chunks(path.read_text(encoding="utf-8-sig")), 1):
            if text.strip():
                rows.append({"id": f"{path.name}#{n}", "text": text})
    if not rows:
        raise ValueError("No nonempty docs/*.txt files")
    for row in rows:
        row["vector"] = embed([row["text"]])[0]
    INDEX.write_text(json.dumps({"model": EMBED, "rows": rows},
        ensure_ascii=False), encoding="utf-8")
    print(f"Indexed {len(rows)} chunks")

def search(question, top_k=3):
    saved = json.loads(INDEX.read_text(encoding="utf-8"))
    if saved["model"] != EMBED:
        raise ValueError("Rebuild index with the current embedding model")
    q = embed([question])[0]
    rows = [dict(r, score=cosine(q, r["vector"])) for r in saved["rows"]]
    return sorted(rows, key=lambda r: r["score"], reverse=True)[:top_k]

if __name__ == "__main__":
    import argparse
    parser = argparse.ArgumentParser()
    parser.add_argument("question", nargs="?")
    parser.add_argument("--build", action="store_true")
    args = parser.parse_args()
    if args.build:
        build()
    elif args.question:
        for r in search(args.question):
            print(r["id"], round(r["score"], 3), r["text"])
    else:
        parser.error("Use --build or provide a question")

.\.venv\Scripts\python.exe .\semantic_search.py --build
.\.venv\Scripts\python.exe .\semantic_search.py "백업은 며칠 보관하나요?"

Reading cosine similarity

Cosine similarity is the dot product of two vectors divided by the product of their respective lengths. Compare whether the directions of the vectors are close rather than their size. A score of 0.8 does not mean that there is an 80% chance of being correct. Even the most similar document may not answer your question.

The sample is divided into 500 characters and overlaps by 80 characters. This is a starting value to reduce the problem of sentences being cut off at boundaries, and is not the result of measuring the optimal value. Because the number of characters and the number of tokens are different, we set truncate=false to reveal that the limit was exceeded. As the data grows, we also consider dividing it based on titles, paragraphs, and tokens.

Search failure and index update

Check whether a passage containing the answer appears in the top 3 results. Do not assume the later generation step will supply information that retrieval missed. For product codes or numbers requiring exact matches, consider combining keyword search with vector retrieval.

If you change the embedding model or edit or delete the original text, the index must be recreated. index.json contains text fragments and semantic information. Be careful not to upload indexes of your work materials to public folders or source repositories.

reference material

• Ollama Embedding API
• EmbeddingGemma Model
• Guide to using Google EmbeddingGemma

Documentation and model tags checked: October 18, 2026. Primary lab environment: Windows 11 x64.

---