--- title: "6.3.1 Embeddings and vector databases" description: "Embeddings with FAISS and Chroma against BM25 and hybrid search on financial documents: measured recall, metadata filters and the cost of IVF and HNSW." url: "https://pythonparatrading.com/en/python-trading-course/level-6/embeddings-and-vector-databases/" lang: "en" image: "https://pythonparatrading.com/og/en/python-trading-course/level-6/embeddings-and-vector-databases.png" --- # 6.3.1 · Embeddings and vector databases: FAISS, Chroma, pgvector and Qdrant > A RAG system starts by finding the passages that answer the question. This lesson turns synthetic annual reports, ETF prospectuses and KIDs into vectors with MiniLM, searches them with FAISS and with Chroma, measures recall against the reference passages and compares it with BM25 and with a hybrid search. The result is not what the marketing promises: in these documents, the exact word and the metadata filters matter more than the embedding. - Course: [Level 6: Generative AI, LLMs, agents, big data and alternative data](https://pythonparatrading.com/en/python-trading-course/level-6/) - Module: 6.3 · Financial RAG - Duration: 70 min - Prerequisites: [6.1.1 · Tokenization, embeddings, financial dictionaries and FinBERT](https://pythonparatrading.com/en/python-trading-course/level-6/tokenization-embeddings-loughran-mcdonald-finbert/) ## By the end of this lesson - Compute normalized embeddings with sentence-transformers, compare them by cosine similarity and know what they capture and what they do not (polarity, language). - Index passages with FAISS and with Chroma (with your own embeddings and metadata filters) and measure recall@k against an evaluation set. - Build a BM25 baseline and a hybrid search with rank fusion (RRF), and decide with data which one to use. - Measure the trade-off between recall and cost of approximate indexes (IVF, HNSW, IVF-PQ) against exact search. - Place pgvector and Qdrant against FAISS and Chroma according to their documentation: where the data lives, filters and indexes. ## Searching by meaning A question-answering system over documents (RAG, lesson 6.3.3) has two parts: retrieving the passages that contain the answer and writing it with an LLM from them. If retrieval fails, the LLM answers without the information or makes it up. This lesson covers the first part: how to turn text into vectors, how to search them fast and, above all, how to **measure** whether what has to be found is found. ```mermaid flowchart TB A[Passages] --> B[Embeddings] B --> C[Vector index] D[Question] --> E[Same model] E --> F[Nearest neighbors] C --> F F --> G[Metadata filter] G --> H[Top-k to the LLM] ``` An **embedding** is a fixed-dimension vector (384 in all-MiniLM-L6-v2) that the model assigns to a text, trained so that similar texts end up close together. With unit-norm vectors, cosine similarity is the inner product: $\cos(u, v) = u \cdot v$ if $\lVert u \rVert = \lVert v \rVert = 1$. Lesson 6.1.1 used them as features for a classifier; here they are used to search. Before searching, it is worth seeing what they measure. The example corpus of this lesson (sentences, reports, prospectuses and questions) is in Spanish and stays as is, because the lesson measures how a model trained on English handles it; the first cell compares four Spanish sentences and one English one: ```python import numpy as np import pandas as pd from sentence_transformers import SentenceTransformer st = SentenceTransformer("sentence-transformers/all-MiniLM-L6-v2", device="cpu", local_files_only=True) frases = [ "La empresa eleva su previsión de ventas para el año", "La compañía mejora sus perspectivas de ingresos anuales", "La empresa recorta su previsión de ventas para el año", "The company raises its full-year sales guidance", "El consejo aprueba el pago de un dividendo", ] V = st.encode(frases, normalize_embeddings=True, show_progress_bar=False) print(V.shape, V.dtype, "· norm:", np.linalg.norm(V, axis=1).round(3)) sim = pd.DataFrame(V @ V.T, index=["eleva", "mejora", "recorta", "raises", "dividendo"], columns=["eleva", "mejora", "recorta", "raises", "dividendo"]) print(sim.round(2).to_string()) ``` ```output (5, 384) float32 · norm: [1. 1. 1. 1. 1.] eleva mejora recorta raises dividendo eleva 1.00 0.53 0.95 0.17 0.40 mejora 0.53 1.00 0.55 0.18 0.38 recorta 0.95 0.55 1.00 0.18 0.40 raises 0.17 0.18 0.18 1.00 0.04 dividendo 0.40 0.38 0.40 0.04 1.00 ``` The labels are the key word of each sentence: “eleva” (the company raises its sales forecast for the year), “mejora” (the company improves its annual revenue outlook), “recorta” (the company cuts its sales forecast), “raises” (the English translation of the first) and “dividendo” (the board approves a dividend payment). The result is instructive. “Eleva” and “recorta” the sales forecast, sentences of opposite meaning with almost the same words, have a similarity of **0.95**; “eleva” and its paraphrase “la compañía mejora sus perspectivas de ingresos anuales”, with different words, only 0.53. The English translation sits at 0.17: the model is not multilingual. all-MiniLM-L6-v2 was trained on English text, and with Spanish text it behaves almost like a word-fragment comparator. It is the course's model because of its size (22 million parameters, 91 MB) and license (Apache-2.0); for Spanish there are multilingual models of the same family that are not installed in the environment. The lesson measures what this one delivers, which is the usual situation: a generic model on text that is not its own. ## The corpus and its evaluation set The level 6 RAG material is 21 synthetic Markdown documents: 2023 and 2024 annual reports of four issuers, prospectuses and KIDs of five ETFs and three research notes. `rag6_pasajes.parquet` splits them into reference passages (lesson 6.3.2 covers how to split documents) and `rag6_preguntas.csv` brings 263 questions with the passages that have to be cited to answer them. Title passages (section headings, which answer nothing) and the 17 questions with no answer in the documents are discarded. ```python pas = pd.read_parquet("rag6_pasajes.parquet") docs = pd.read_parquet("rag6_documentos.parquet") preg = pd.read_csv("rag6_preguntas.csv") docs["corto"] = (docs["entidad"].str.replace(", S.A.", "") .str.replace(" UCITS ETF", "")) docs["anio"] = docs["titulo"].str.extract(r"(20\d\d)$")[0].fillna("") pas = pas.merge(docs[["id_doc", "entidad", "anio"]], on="id_doc") pas = pas[pas["tipo"] != "titulo"].reset_index(drop=True) preg = preg[preg["respondible"]].reset_index(drop=True) oro = [set(x.split(";")) for x in preg["pasajes_oro"]] print(len(docs), "documents ·", len(pas), "passages ·", len(preg), "answerable questions") print(pas["tipo"].value_counts().to_string()) ``` ```output 21 documents · 722 passages · 246 answerable questions tipo texto 431 lista 119 tabla 105 nota 67 ``` The metric is **recall@k**: of each question's gold passages, the fraction that appears among the top $k$ results, averaged over the questions. It is the one that matters in RAG, because what is not retrieved does not reach the LLM. With $k = 5$ it measures what would fit in a typical prompt. ## FAISS: exact search FAISS is a library of in-memory vector indexes. `IndexFlatIP` stores the vectors as they are and compares the query with all of them (exact search by inner product, which with normalized vectors is the cosine). With 722 passages nothing more is needed: ```python import faiss faiss.omp_set_num_threads(1) P = st.encode(list(pas["texto"]), batch_size=32, normalize_embeddings=True, show_progress_bar=False) Q = st.encode(list(preg["pregunta"]), batch_size=64, normalize_embeddings=True, show_progress_bar=False) indice = faiss.IndexFlatIP(P.shape[1]) # inner product = cosine indice.add(P) print(indice.ntotal, "vectors ·", P.nbytes // 1024, "KB in float32") ids = pas["id_pasaje"].to_numpy() def recall(rankings, k): """Fraction of the gold passages that are among the top k.""" return round(float(np.mean([len(set(ids[r[:k]]) & g) / len(g) for r, g in zip(rankings, oro)])), 3) _, denso = indice.search(Q, 20) print({f"recall@{k}": recall(denso, k) for k in (1, 5, 10)}) ``` ```output 722 vectors · 1083 KB in float32 {'recall@1': 0.022, 'recall@5': 0.114, 'recall@10': 0.265} ``` A recall@5 of 11.4% is very low, and the index is not to blame: it is exact. You have to look at what it returns: ```python i = 0 print(preg.loc[i, "pregunta"], "· gold:", preg.loc[i, "pasajes_oro"]) for j in denso[i][:4]: print(f" {ids[j]} {pas.loc[j, 'entidad'][:22]:22s} " f"{pas.loc[j, 'texto'][:52]!r}") ``` ```output ¿Cuál es el mayor mercado de Busmar Hogar por ventas en 2023 y qué peso tiene? · gold: R01-p031 R08-p076 Zaptec Capital, S.A. 'Para 2025, el grupo prevé un crecimiento de las vent' R07-p076 Zaptec Capital, S.A. 'Para 2024, el grupo prevé un crecimiento de las vent' R03-p076 Celra Semiconductores, 'Para 2024, el grupo prevé un crecimiento de las vent' R02-p076 Busmar Hogar, S.A. 'Para 2025, el grupo prevé un crecimiento de las vent' ``` The question is about the largest market of **Busmar Hogar in 2023**, and the top four results are the same passage (“Para 2025, el grupo prevé un crecimiento de las ventas…”, “For 2025, the group expects sales growth…”) in reports from Zaptec Capital, Celra Semiconductores and Busmar itself for the following year. The eight annual reports share a template, like real ones from the same sector, and a loose passage does not say which company or which year it belongs to: that information is in the document's title. No embedding can tell apart passages that say the same thing with other figures. ## BM25 and hybrid search The baseline of document search is **BM25**, a lexical model: it scores each passage by the query words it contains, with more weight on words that are rare in the corpus and a length correction. It knows nothing about synonyms, but it does not confuse “recorta” (cuts) with “eleva” (raises). **Hybrid** search combines the two rankings; the simplest and most robust way is reciprocal rank fusion (RRF, Cormack, Clarke and Büttcher, 2009), which adds $1/(k_0 + p)$ for each passage in each list, where $p$ is the passage's position in that list, with $k_0 = 60$, the value they fixed in their pilot study. It does not need the scores of the two systems to be comparable, only their positions. ```mermaid flowchart TB P[Question] --> D[Embeddings: top 20] P --> L[BM25: top 20] D --> R[RRF] L --> R R --> K[Fused top-k] ``` ```python import re import unicodedata from nltk.corpus import stopwords from rank_bm25 import BM25Okapi VACIAS = set(stopwords.words("spanish")) def tokens(t): t = unicodedata.normalize("NFKD", t.lower()) t = "".join(c for c in t if not unicodedata.combining(c)) return [w for w in re.findall(r"\w+", t) if w not in VACIAS] bm25 = BM25Okapi([tokens(t) for t in pas["texto"]]) def rank_bm25(consultas, mascaras=None): S = np.array([bm25.get_scores(tokens(c)) for c in consultas]) if mascaras is not None: S[~mascaras] = -np.inf return np.argsort(-S, axis=1, kind="stable")[:, :20] def rrf(*rankings, k0=60, n=20): """Reciprocal rank fusion: sum of 1 / (k0 + position).""" salida = [] for rs in zip(*rankings): p = {} for r in rs: for pos, j in enumerate(r): p[j] = p.get(j, 0) + 1 / (k0 + pos + 1) salida.append(sorted(p, key=lambda j: (-p[j], j))[:n]) return np.array(salida) lexico = rank_bm25(preg["pregunta"]) hibrido = rrf(denso, lexico) tabla = pd.DataFrame({m: [recall(r, k) for k in (1, 5, 10)] for m, r in [("dense", denso), ("BM25", lexico), ("hybrid", hibrido)]}, index=["recall@1", "recall@5", "recall@10"]) print(tabla.round(3).to_string()) ``` ```output dense BM25 hybrid recall@1 0.022 0.073 0.042 recall@5 0.114 0.423 0.301 recall@10 0.265 0.655 0.532 ``` BM25 retrieves 42.3% in the top 5, almost four times more than the embeddings: the questions share terms with their passages (“provisiones para litigios”, litigation provisions; “pasivos por arrendamiento”, lease liabilities; “unidad de creación”, creation unit) and the lexical model exploits them. The fusion ends up between the two (30.1%). But neither solves the underlying problem: knowing which document each passage comes from. ## Chroma: metadata and filters A **vector database** is more than an index: it stores the vectors together with the text and its **metadata** (document, issuer, date, type) and lets you filter by them in the query. Chroma is an embedded database, inside the process itself, with `where`-style filters. Following the README of the `ia` environment, the collection is created with `embedding_function=None` and given the embeddings already computed (its default function would download a model, and the course works without a network), with the cosine space in the configuration (by default it uses L2, according to its documentation): ```python import chromadb cliente = chromadb.EphemeralClient() col = cliente.create_collection( "pasajes", embedding_function=None, configuration={"hnsw": {"space": "cosine"}}) col.add(ids=list(ids), embeddings=P, documents=list(pas["texto"]), metadatas=[{"id_doc": a, "entidad": b, "anio": c} for a, b, c in pas[["id_doc", "entidad", "anio"]] .itertuples(index=False)]) def filtro(pregunta): """Entity named in the question and, if any, the most recent year.""" m = docs[[c in pregunta for c in docs["corto"]]] anios = re.findall(r"20\d\d", pregunta) if anios and (m["anio"] == max(anios)).any(): m = m[m["anio"] == max(anios)] return m def where(m): if m.empty: return None return {"id_doc": {"$in": list(m["id_doc"])}} r = col.query(query_embeddings=Q[:1], n_results=3, where=where(filtro(preg.loc[0, "pregunta"])), include=["distances"]) print(r["ids"][0], np.round(r["distances"][0], 3)) ``` ```output ['R01-p076', 'R01-p068', 'R01-p004'] [0.374 0.383 0.397] ``` The filter comes from the question itself: the entity it names and, if it cites a year, the report for that year. With the filter, the top three results for the Busmar question are already from its 2023 report (`R01`), although still not the gold passage. Chroma returns cosine distances ($1 - \cos$); they are the same FAISS would give on that subset. The same filter, as a mask, is applied to BM25: ```python pos = {p: i for i, p in enumerate(ids)} def rank_chroma(E, consultas, n=20): salida = [] for e, c in zip(E, consultas): w = where(filtro(c)) k = n if w is None else min(n, int(pas["id_doc"].isin( w["id_doc"]["$in"]).sum())) r = col.query(query_embeddings=e[None], n_results=k, where=w, include=[]) fila = [pos[x] for x in r["ids"][0]] salida.append(fila + [fila[-1]] * (n - len(fila))) return np.array(salida) mascaras = np.array([ pas["id_doc"].isin(filtro(c)["id_doc"]).to_numpy() if not filtro(c).empty else np.ones(len(pas), bool) for c in preg["pregunta"]]) denso_f = rank_chroma(Q, preg["pregunta"]) lexico_f = rank_bm25(preg["pregunta"], mascaras) hibrido_f = rrf(denso_f, lexico_f) for m, r in [("dense", denso_f), ("BM25", lexico_f), ("hybrid", hibrido_f)]: print(f"{m:8s}", {f"@{k}": recall(r, k) for k in (1, 5, 10)}) ``` ```output dense {'@1': 0.091, '@5': 0.464, '@10': 0.633} BM25 {'@1': 0.302, '@5': 0.752, '@10': 0.887} hybrid {'@1': 0.151, '@5': 0.656, '@10': 0.804} ``` The filter multiplies recall: with $k = 5$, from 11.4% to 46.4% with embeddings and from 42.3% to 75.2% with BM25. One detail remains: once the document is filtered, the company's name in the query no longer tells anything and pulls toward the passages that repeat it (“Busmar Hogar opera en el sector…”, “Busmar Hogar operates in the … sector”). It is removed: ```python def limpia(p): for c in docs["corto"].unique(): p = p.replace(c, "") return p limpias = [limpia(p) for p in preg["pregunta"]] Q2 = st.encode(limpias, batch_size=64, normalize_embeddings=True, show_progress_bar=False) denso_l = rank_chroma(Q2, preg["pregunta"]) lexico_l = rank_bm25(limpias, mascaras) hibrido_l = rrf(denso_l, lexico_l) for m, r in [("dense", denso_l), ("BM25", lexico_l), ("hybrid", hibrido_l)]: print(f"{m:8s}", {f"@{k}": recall(r, k) for k in (1, 5, 10)}) porpas = lambda r: [len(set(ids[x[:5]]) & g) / len(g) for x, g in zip(r, oro)] tipo = (pd.DataFrame({"tipo": preg["tipo_pregunta"], "dense": porpas(denso_l), "BM25": porpas(lexico_l), "hybrid": porpas(hibrido_l)}) .groupby("tipo") .agg(n=("dense", "size"), dense=("dense", "mean"), BM25=("BM25", "mean"), hybrid=("hybrid", "mean"))) print(tipo.round(2).sort_values("n", ascending=False).to_string()) ``` ```output dense {'@1': 0.307, '@5': 0.545, '@10': 0.698} BM25 {'@1': 0.546, '@5': 0.866, '@10': 0.899} hybrid {'@1': 0.468, '@5': 0.731, '@10': 0.82} n dense BM25 hybrid tipo texto 122 0.71 0.90 0.72 tabla 64 0.42 0.88 0.75 nota_pie 16 0.50 1.00 0.88 texto_y_nota 16 0.50 1.00 0.84 tabla_y_nota 13 0.19 1.00 1.00 calculo 8 0.00 0.00 0.00 comparacion 7 0.23 0.29 0.47 ``` The figure at the top summarizes the path. With the filter and the clean query, BM25 reaches 86.6% in the top 5, the embeddings 54.5% and the fusion 73.1%. By question type, BM25 retrieves every footnote and every table with a note; the embeddings fail mostly on tables (42%), where the text is figures and vertical bars. Calculation questions (“¿cuánto crecieron las ventas?”, how much did sales grow?) are found by nobody: their gold passage is the table of key figures, which does not contain the word “crecieron” (grew). Cross-document comparison questions (7) need five passages from five documents and no method gets to half. ## At scale: approximate indexes With 722 vectors, comparing against all of them is instantaneous. With millions (all the news of a decade, every EDGAR filing chunked), exact search costs one distance per vector and per query. Approximate indexes compare against a fraction: - **IVF** (inverted file): groups the vectors with k-means into `nlist` cells and, for each query, only scans the `nprobe` nearest cells. - **HNSW** (Malkov and Yashunin): a layered neighbor graph; the search descends the graph from an entry point and `efSearch` sets how many candidates it keeps. - **PQ** (product quantization): splits each vector into chunks and stores each chunk as the index of a centroid; with IVF, it saves memory as well as time. The cell builds 50,000 synthetic 384-dimensional vectors grouped into 5,000 topics (a corpus with structure, like a real one; with real embeddings the figures change, the pattern does not) and measures, for each index and parameter, what fraction of the 10 exact neighbors it retrieves and how many distances it computes per query. The number of distances is the deterministic cost measure; time depends on the machine and its load, and behaves proportionally. FAISS is pinned to one thread so that the HNSW graph is always the same. ```python rng = np.random.default_rng(0) N, dim, grupos = 50_000, 384, 5_000 centros = rng.standard_normal((grupos, dim)).astype("float32") X = centros[rng.integers(0, grupos, N)] + rng.standard_normal( (N, dim)).astype("float32") X /= np.linalg.norm(X, axis=1, keepdims=True) Xq = centros[rng.integers(0, grupos, 500)] + rng.standard_normal( (500, dim)).astype("float32") Xq /= np.linalg.norm(Xq, axis=1, keepdims=True) exacto = faiss.IndexFlatIP(dim) exacto.add(X) _, verdad = exacto.search(Xq, 10) def r10(I): return np.mean([len(set(a) & set(b)) / 10 for a, b in zip(I, verdad)]) cuantif = faiss.IndexFlatIP(dim) ivf = faiss.IndexIVFFlat(cuantif, dim, 256, faiss.METRIC_INNER_PRODUCT) ivf.cp.seed = 0 ivf.train(X[:20_000]) ivf.add(X) tam = np.array([ivf.invlists.list_size(i) for i in range(256)]) filas = [("exact", "-", 1.0, N)] for nprobe in (1, 4, 16, 64): ivf.nprobe = nprobe _, I = ivf.search(Xq, 10) _, listas = cuantif.search(Xq, nprobe) filas.append(("IVF", f"nprobe={nprobe}", r10(I), 256 + tam[listas].sum(1).mean())) hnsw = faiss.IndexHNSWFlat(dim, 16, faiss.METRIC_INNER_PRODUCT) hnsw.hnsw.efConstruction = 40 hnsw.add(X) for ef in (16, 32, 64, 128, 256): hnsw.hnsw.efSearch = ef faiss.cvar.hnsw_stats.reset() _, I = hnsw.search(Xq, 10) filas.append(("HNSW", f"efSearch={ef}", r10(I), faiss.cvar.hnsw_stats.ndis / len(Xq))) pq = faiss.IndexIVFPQ(faiss.IndexFlatIP(dim), dim, 256, 48, 8, faiss.METRIC_INNER_PRODUCT) pq.cp.seed = 0 pq.train(X[:20_000]) pq.add(X) tam_pq = np.array([pq.invlists.list_size(i) for i in range(256)]) for nprobe in (16, 64): pq.nprobe = nprobe _, I = pq.search(Xq, 10) _, listas = pq.quantizer.search(Xq, nprobe) filas.append(("IVF-PQ", f"nprobe={nprobe}", r10(I), 256 + tam_pq[listas].sum(1).mean())) ann = pd.DataFrame(filas, columns=["index", "parameter", "recall@10", "distances per query"]) ann["distances per query"] = ann["distances per query"].round() print(ann.round(3).astype({"distances per query": int}) .to_string(index=False)) print("bytes per vector: exact", dim * 4, "· IVF-PQ", pq.code_size) ``` ```output index parameter recall@10 distances per query exact - 1.000 50000 IVF nprobe=1 0.427 453 IVF nprobe=4 0.556 1041 IVF nprobe=16 0.712 3386 IVF nprobe=64 0.894 12763 HNSW efSearch=16 0.646 582 HNSW efSearch=32 0.796 993 HNSW efSearch=64 0.860 1707 HNSW efSearch=128 0.904 3067 HNSW efSearch=256 0.941 5679 IVF-PQ nprobe=16 0.658 3386 IVF-PQ nprobe=64 0.792 12763 bytes per vector: exact 1536 · IVF-PQ 48 ``` *Fig. — Recall@10 against exact search and distances computed per query for FAISS IVF and HNSW indexes on 50,000 synthetic 384-dimensional vectors.* Two curves on a log cost scale. HNSW, highlighted, goes from 64.6% with 582 distances to 94.1% with 5,679. IVF goes from 42.7% with 453 distances to 89.4% with 12,763. Exact search recovers 100% with 50,000 distances. HNSW dominates IVF on this data: with about 3,000 distances per query, HNSW retrieves 90.4% of the exact top 10 and IVF 71.2%; IVF needs about 12,800 to reach 89.4%. Neither reaches 100% without approaching the exact cost. The price of HNSW lies elsewhere: building the graph is the slowest part of the cell and the graph takes memory on top of the vectors. IVF-PQ scans the same cells as IVF (the same distances, but each one is a sum of 48 values from a table instead of a 384-dimensional product) and stores 48 bytes per vector against the 1,536 of a float32 vector (32 times less), at the cost of recall: 79.2% with 64 cells, against 89.4% for uncompressed IVF. The FAISS guide to choosing an index sums up the criterion: `IndexFlat` is the only one that guarantees exact results and is enough with few searches; if exact results are not needed and memory is not tight, HNSW (about $d \cdot 4 + M \cdot 2 \cdot 4$ bytes per vector: 1,664 here, with $M = 16$); if it is tight, product quantization. ## pgvector and Qdrant FAISS and Chroma run inside the process. In production, vectors usually live in a service: - **pgvector** adds to PostgreSQL a `vector` type (up to 2,000 indexable dimensions), the cosine distance `<=>` and HNSW indexes (by default `m = 16`, `ef_construction = 64` and `hnsw.ef_search = 40`) and IVFFlat. Its advantage is that the passages, the metadata and the permissions are in the same database and are filtered with SQL. Its documentation warns that with approximate indexes **the filter is applied after** the index scan, so a selective filter can return fewer results than requested; since version 0.8.0 there are iterative scans (`hnsw.iterative_scan`) that keep searching until the list is complete. - **Qdrant** is a dedicated server (REST on port 6333 and gRPC on 6334, with Docker locally) with *payload* filters inside the search; the Python client creates collections with their distance and queries with `query_points`. Neither of the two is run in the course (they need a server). The code, according to their documentation (accessed 2026-10-23): ```python skip # pgvector: PostgreSQL with the vector extension (not run) # SQL, once: # CREATE EXTENSION vector; # CREATE TABLE pasajes (id text PRIMARY KEY, id_doc text, # texto text, embedding vector(384)); # CREATE INDEX ON pasajes USING hnsw (embedding vector_cosine_ops); import psycopg from pgvector.psycopg import register_vector with psycopg.connect("dbname=research") as conn: register_vector(conn) for (i, doc, txt), e in zip(pas[["id_pasaje", "id_doc", "texto"]] .itertuples(index=False), P): conn.execute("INSERT INTO pasajes VALUES (%s, %s, %s, %s)", (i, doc, txt, e)) conn.execute("SET hnsw.ef_search = 100") conn.execute("SET hnsw.iterative_scan = relaxed_order") filas = conn.execute( "SELECT id, texto FROM pasajes WHERE id_doc = %s " "ORDER BY embedding <=> %s LIMIT 5", ("R01", Q[0])).fetchall() ``` ```python skip # Qdrant: self-hosted or managed server (not run in the course) from qdrant_client import QdrantClient from qdrant_client.models import (Distance, FieldCondition, Filter, MatchValue, PointStruct, VectorParams) qd = QdrantClient(url="http://localhost:6333") qd.create_collection( collection_name="pasajes", vectors_config=VectorParams(size=384, distance=Distance.COSINE)) qd.upsert(collection_name="pasajes", wait=True, points=[ PointStruct(id=k, vector=e.tolist(), payload={"id_pasaje": i, "id_doc": doc}) for k, (i, doc, e) in enumerate(zip(ids, pas["id_doc"], P))]) res = qd.query_points( collection_name="pasajes", query=Q[0].tolist(), limit=5, query_filter=Filter(must=[FieldCondition( key="id_doc", match=MatchValue(value="R01"))]), with_payload=True).points ``` | | FAISS | Chroma | pgvector | Qdrant | |---|---|---|---|---| | What it is | Index library | Embedded database (also client-server) | PostgreSQL extension | Dedicated server | | Data and metadata | Outside (managed by the user) | In the collection | In SQL tables | In the *payload* | | Filters | On your own | `where` on metadata | SQL (`WHERE`) | `Filter` in the query | | Approximate indexes | IVF, HNSW, PQ and more | HNSW | HNSW, IVFFlat | HNSW with filters | | When | Research, batches, maximum in-memory speed | Prototypes and local RAG | The data is already in Postgres | Search service with filters and scale | ## Exercises 1. Prepend to each passage the title of its document (for example, “Busmar Hogar, S.A. — Informe anual 2023”, annual report 2023) before computing its embedding and repeat the dense search **without filters**. How much does recall@5 rise? **Solution.** ```python titulos = pas["id_doc"].map(docs.set_index("id_doc")["titulo"]) P2 = st.encode(list(titulos + ". " + pas["texto"]), batch_size=32, normalize_embeddings=True, show_progress_bar=False) idx2 = faiss.IndexFlatIP(P2.shape[1]) idx2.add(P2) _, denso2 = idx2.search(Q, 20) print({f"recall@{k}": recall(denso2, k) for k in (1, 5, 10)}) ``` ```output {'recall@1': 0.199, 'recall@5': 0.515, 'recall@10': 0.583} ``` Dense recall@5 goes from 11.4% to 51.5% without any filter: more than the metadata filter with the original query (46.4%) and somewhat less than the filter with the clean query (54.5%). Putting the document's context inside the passage is the idea of contextual chunking from lesson 6.3.2: the embedding already “knows” which company and which year each passage belongs to, and it also helps when the question does not name the entity literally and an exact filter cannot be built. The two techniques combine. 2. Weight the fusion: give BM25 twice the weight of the embeddings in RRF (adding $2/(k_0 + p)$ for BM25) and measure recall@5 with the filter and the clean query. **Solution.** ```python def rrf_pesos(A, B, wa=1.0, wb=2.0, k0=60, n=20): salida = [] for ra, rb in zip(A, B): p = {} for r, w in ((ra, wa), (rb, wb)): for pos_, j in enumerate(r): p[j] = p.get(j, 0) + w / (k0 + pos_ + 1) salida.append(sorted(p, key=lambda j: (-p[j], j))[:n]) return np.array(salida) for wb in (1, 2, 4): print(f"BM25 weight {wb}: recall@5 = " f"{recall(rrf_pesos(denso_l, lexico_l, wb=wb), 5)}") ``` ```output BM25 weight 1: recall@5 = 0.731 BM25 weight 2: recall@5 = 0.81 BM25 weight 4: recall@5 = 0.813 ``` With twice the weight for BM25, the fusion rises from 73.1% to 81.0%; with four times, to 81.3%, still below the 86.6% of BM25 alone. With one retriever clearly better than the other, weighted fusion only pays off if the weak one brings passages the strong one does not find (questions phrased with other words); here that does not happen enough. The weight, moreover, is chosen on one evaluation set and checked on another: choosing it and measuring it on the same 246 questions overestimates the result. 3. An asset manager wants a search engine over 15 years of research notes (about 2 million passages), with per-team permissions and filters by date and issuer, and already stores everything in PostgreSQL. Which option would you choose and what would you measure first? **Solution.** pgvector is the natural candidate: the passages, the permissions and the metadata are already in PostgreSQL and the filters by date, issuer and team are SQL. Before deciding I would measure, with my own set of questions and their reference passages, the recall@k of dense search, of lexical search (PostgreSQL has full-text search) and of hybrid search; the recall of the HNSW index against exact search with the real filters, because a very selective filter after an approximate index can return few results (hence the iterative scans); the latency under the expected load; and the index memory with 2 million vectors of the chosen dimension. Qdrant or a separate FAISS index would make sense if vector search had to scale independently of the database. ## Self-assessment 1. With MiniLM, “La empresa eleva su previsión de ventas para el año” (“The company raises its sales forecast for the year”) and “La empresa recorta su previsión de ventas para el año” (“The company cuts its sales forecast for the year”) have a cosine similarity of 0.95, and the first one and its paraphrase “La compañía mejora sus perspectivas de ingresos anuales” (“The company improves its annual revenue outlook”), 0.53. What does it indicate? - A. That the model is broken - B. That cosine similarity is useless for text - C. That, in Spanish, this embedding mostly measures shared words and topic, not the meaning of the sentence: two opposite sentences with the same words end up closer than two equivalent sentences with different words - D. That the opposite sentences are synonyms in finance *Answer C: That, in Spanish, this embedding mostly measures shared words and topic, not the meaning of the sentence: two opposite sentences with the same words end up closer than two equivalent sentences with different words.* all-MiniLM-L6-v2 was trained on English text. In Spanish it does little more than compare word fragments. That is why searching by meaning in Spanish requires a multilingual model or, as in this lesson, combining it with lexical search. 2. Without filters, dense search retrieves 11.4% of the reference passages in the top 5. For a question about Busmar Hogar in 2023, the top results are nearly identical passages from other issuers. What is the main cause? - A. The eight annual reports share a template: many passages differ only by company and year, which do not appear in the passage - B. The FAISS index is approximate - C. The model has too few dimensions - D. The questions are too long *Answer A: The eight annual reports share a template: many passages differ only by company and year, which do not appear in the passage.* IndexFlatIP is exact. The problem is the content: “Para 2024, el grupo prevé un crecimiento de las ventas…” (“For 2024, the group expects sales growth…”) appears in four reports with other figures. Filtering by entity and year (metadata) raises dense recall@5 to 46.4% and BM25's to 75.2%. 3. After filtering and cleaning the query, recall@5 is 54.5% with embeddings, 86.6% with BM25 and 73.1% with the RRF fusion of the two. Which conclusion is correct? - A. Hybrid search is always the best - B. Embeddings are always worse than BM25 - C. RRF is implemented wrong - D. Fusion with equal weights inherits the errors of the weak retriever; here it is better to use BM25 alone, weight the fusion or switch to an embedding model that understands Spanish, and to decide it with the evaluation set *Answer D: Fusion with equal weights inherits the errors of the weak retriever; here it is better to use BM25 alone, weight the fusion or switch to an embedding model that understands Spanish, and to decide it with the evaluation set.* RRF averages rankings: if one of the two is clearly worse, the mix ends up between both. With a multilingual embedding or questions that do not share words with the passage, the result could be reversed. The decision comes from measuring recall, not from the fashionable architecture. 4. With 50,000 vectors, HNSW with efSearch = 128 retrieves 90.4% of the 10 exact neighbors computing about 3,067 distances per query, and IVF with nprobe = 64 retrieves 89.4% with about 12,763. What does an approximate index gain? - A. More precision than exact search - B. Speed: far fewer distances are computed than the 50,000 of the exact index, at the cost of losing some neighbors; the search parameter controls that trade-off - C. Always less memory - D. Nothing with fewer than a million vectors *Answer B: Speed: far fewer distances are computed than the 50,000 of the exact index, at the cost of losing some neighbors; the search parameter controls that trade-off.* HNSW and IVF store the full vectors (HNSW also stores the graph), so they do not save memory. Product quantization is what saves it: IVF-PQ stores 48 bytes per vector instead of 1,536, with less recall. 5. When does pgvector make sense compared with FAISS? - A. When the passages and their metadata already live in PostgreSQL and you need to filter with SQL, transactions and database permissions - B. When the fastest possible in-memory search is needed - C. Never: pgvector only does exact search - D. Only with more than a billion vectors *Answer A: When the passages and their metadata already live in PostgreSQL and you need to filter with SQL, transactions and database permissions.* FAISS is a library: indexes in the process's memory, with no storage or metadata filters of its own. pgvector adds a vector type and HNSW and IVFFlat indexes to PostgreSQL; its documentation warns that with approximate indexes the filter is applied after the index scan, and since version 0.8.0 it offers iterative scans. ## Key points - A normalized embedding turns a text into a unit-norm vector, and cosine similarity is their inner product. What it captures depends on the model: with all-MiniLM-L6-v2 in Spanish, “eleva” (raises) and “recorta” (cuts) the forecast end up at 0.95. - Without filters, dense search retrieves 11.4% of the reference passages in the top 5 and BM25 42.3%: with template documents, the company and the year (which are not in the passage) decide. - Metadata is half of a vector database: filtering by entity and year in Chroma and removing the entity from the query brings recall@5 to 54.5% (dense), 86.6% (BM25) and 73.1% (hybrid RRF). - Approximate indexes trade recall for cost: with 50,000 vectors, HNSW reaches 90.4% of the exact top 10 with 3,067 distances per query, IVF needs 12,763 for 89.4% and exact search, 50,000. IVF-PQ shrinks each vector from 1,536 to 48 bytes. - FAISS is an index library; Chroma, an embedded database with metadata; pgvector brings vectors to PostgreSQL and Qdrant is a dedicated server with payload filters. The choice depends on where the data lives and which filters are needed. ## Resources ### Documentation - [FAISS: guidelines to choose an index (project wiki)](https://github.com/facebookresearch/faiss/wiki/Guidelines-to-choose-an-index) · github.com - [Chroma: metadata filtering](https://docs.trychroma.com/docs/querying-collections/metadata-filtering) · docs.trychroma.com - [pgvector (GitHub): HNSW and IVFFlat indexes, filtering and iterative scans](https://github.com/pgvector/pgvector) · github.com - [Qdrant: payload filtering](https://qdrant.tech/documentation/concepts/filtering/) · qdrant.tech ### Books - [Malkov, Y. and Yashunin, D. (2016): Efficient and robust approximate nearest neighbor search using HNSW graphs. arXiv:1603.09320](https://arxiv.org/abs/1603.09320) · arxiv.org - [Cormack, G. V., Clarke, C. L. A. and Büttcher, S. (2009): Reciprocal Rank Fusion outperforms Condorcet and individual Rank Learning Methods. SIGIR](https://plg.uwaterloo.ca/~gvcormac/cormacksigir09-rrf.pdf) · plg.uwaterloo.ca ### Data - [Level 6 data: reference passages for RAG (Parquet)](https://pythonparatrading.com/curso/datos/nivel-6/rag6_pasajes.parquet) · Download · rag6_pasajes.parquet Previous lesson: [6.2.5 · Look-ahead bias in LLMs: the model that already knows the future](https://pythonparatrading.com/en/python-trading-course/level-6/look-ahead-bias-in-llms/) Next lesson: [6.3.2 · Chunking financial documents: tables, footnotes and XBRL](https://pythonparatrading.com/en/python-trading-course/level-6/chunking-financial-documents/)