--- title: "6.7.4 Data vendors and due diligence" description: "How to assess an alternative dataset before paying for it: point-in-time history versus backfill, panel biases, delays and legality (MAR and GDPR)." url: "https://pythonparatrading.com/en/python-trading-course/level-6/alternative-data-vendors-and-due-diligence/" lang: "en" image: "https://pythonparatrading.com/og/en/python-trading-course/level-6/alternative-data-vendors-and-due-diligence.png" --- # 6.7.4 · Alternative data vendors and due diligence > A data vendor sells a sample, a fact sheet and a history of several years. The sample is tested in a day; the history decides whether the backtest tells the truth. This lesson goes through what Neudata, Nasdaq Data Link and Eagle Alpha say they do and runs a complete due diligence on the course's card data: the reconstructed history inflates the R² from 0.28 to 0.39, the panel changes composition and legality is checked against the text of MAR and the GDPR. - Course: [Level 6: Generative AI, LLMs, agents, big data and alternative data](https://pythonparatrading.com/en/python-trading-course/level-6/) - Module: 6.7 · Alternative data - Duration: 60 min - Prerequisites: [1.4.4 · Universes without survivorship bias: delistings, mergers and symbol changes](https://pythonparatrading.com/en/python-trading-course/level-1/survivorship-bias-and-point-in-time-data/), [2.5.1 · Backtest biases: look-ahead, survivorship, selection and narrative](https://pythonparatrading.com/en/python-trading-course/level-2/backtest-biases/), [6.7.2 · Types of alternative data: what each one measures](https://pythonparatrading.com/en/python-trading-course/level-6/types-of-alternative-data/) ## By the end of this lesson - Place data catalogs and distributors (Neudata, Eagle Alpha, Nasdaq Data Link) in the purchase of a dataset and know which questions are theirs to answer and which are the buyer's. - Tell the data as delivered (point-in-time) apart from the history the vendor reconstructs at the time of sale, and measure the in-sample inflation the latter produces. - Detect a panel's biases in the data itself: changes in size, in demographic composition and in merchant coverage, and delivery delays. - Frame the legal questions about a dataset with the text of MAR (Regulation (EU) 596/2014) and the GDPR, without confusing due diligence with legal advice. ## Who sells and who answers Lesson 6.7.1 measured the market and 6.7.2 went through the types of data. This one starts where a real purchase starts: a portfolio manager has seen a fact sheet in a catalog, the vendor offers a free sample for a few weeks and the question is whether to pay. Between the manager and the vendor there are usually intermediaries. What the three named in the syllabus say they do, read on their own websites on 2026-10-23: | Company | What it says it does (its website, 2026-10-23) | What it does not do for the buyer | |---|---|---| | Neudata | Catalog of “7,000+ datasets” from “4,200+” vendors, dataset reports written by analysts, pricing information on “over 2,000 data products”, on-demand data sourcing and warnings about low-quality data | Test the dataset with the buyer's strategy | | Eagle Alpha | Platform and advisory: “over 2,500 alternative data products” on its home page and “over 2,600 datasets” with a taxonomy of 16 categories and 56 subcategories in its FAQs; Surveyor, a tool for managing vendors' due diligence questionnaires (DDQs) | Decide whether the use is lawful for the buyer | | Nasdaq Data Link | Cloud platform with “more than 350 trusted data sets” via API, with R, Python and Excel clients; the Python client has `get_table`, `export_table` and `get_point_in_time` to request a table “as of” a date | Guarantee that a third-party dataset has no biases | The figures are each company's about itself, unaudited, and they change between pages of the same website (Eagle Alpha gives 2,500 on one and 2,600 on another). They tell you what each one offers, not how they compare. None of the three can answer the question that matters: whether **this** dataset improves **this** strategy with the data as it would have arrived. That is the buyer's due diligence, and it has an order: ```mermaid flowchart TB A[Fact sheet and catalog] --> B[NDA and sample] B --> C[Technical tests] C --> D[Signal tests] D --> E[Legal review] E --> F[Contract] C --> X[Discard] D --> X E --> X ``` The technical tests check that the data is what it claims to be (coverage, history, deliveries, biases); the signal tests, that it adds something (lesson 6.7.5); the legal review, that it can be used. Any of the three can end the purchase. The order matters: a dataset with a reconstructed history gives brilliant and misleading signal tests, so the history is examined first. ## The dataset's fact sheet The first step is to reconstruct the fact sheet from the files, not copy it from the brochure: how many merchants, which dates, when each data point arrives. The vendor delivers two things. `alt6_tarjetas_pit.parquet` is the data **as delivered** each day (point-in-time, PIT): spending on day *t* arrives on day *t* + 3 and is preliminary. And `alt6_tarjetas_historico.parquet` is the history the vendor delivers on signing the contract, on 2023-07-03. ```python import numpy as np import pandas as pd pit = pd.read_parquet("alt6_tarjetas_pit.parquet") hist = pd.read_parquet("alt6_tarjetas_historico.parquet") mapa = pd.read_csv("alt6_comercios_verdad.csv") for nombre, df in [("PIT", pit), ("history", hist)]: print(f"{nombre:10s} {len(df):7,d} rows · " f"{df['descriptor'].nunique()} merchants · " f"{df['fecha'].min():%Y-%m-%d} to " f"{df['fecha'].max():%Y-%m-%d}") retraso = (pit["fecha_entrega"] - pit["fecha"]).dt.days print("days between the date and delivery:", retraso.unique()) print(mapa["relacion"].value_counts().to_string()) ``` ```output PIT 359,410 rows · 127 merchants · 2018-01-01 to 2025-09-30 history 254,889 rows · 127 merchants · 2018-01-01 to 2023-06-30 days between the date and delivery: [3] relacion matriz 69 marca 22 filial 18 sin_relacion 18 ``` 127 merchants: 109 belong to one of the 25 companies (the parent, a brand or a subsidiary; `matriz`, `marca`, `filial` in the data) and 18 are unrelated even though their names are similar. Here the correct mapping (`alt6_comercios_verdad.csv`) is used as if the vendor provided it; building it from scratch is the first part of lesson 6.7.5. The history ends on 2023-06-30, three days before delivery. The practical question on the fact sheet is whether the data arrives in time. A sales dataset is useful if the quarter is complete before the company reports its earnings: ```python emp = pd.read_csv("acciones6_empresas.csv") ventas = pd.read_csv("acciones6_ventas.csv", parse_dates=["fin_trimestre", "fecha_publicacion"]) ventas = ventas.merge(emp[["ticker", "id_empresa"]], on="ticker") cubiertas = mapa["id_empresa"].dropna().unique() v = ventas[ventas["id_empresa"].isin(cubiertas)] # the last day of the quarter is delivered 3 days later completo = v["fin_trimestre"] + pd.Timedelta(days=3) margen = (v["fecha_publicacion"] - completo).dt.days print(f"{len(v)} reports · margin (days): minimum {margen.min()}, " f"median {margen.median():.0f}, maximum {margen.max()}") ``` ```output 800 reports · margin (days): minimum 17, median 36, maximum 62 ``` The full quarter arrives between 17 and 62 days before earnings: there is time to compute and trade. With a real vendor, this figure is checked against the delivery dates of the sample, not the sales pitch: the advertised delays are usually those of the best day. ## Point-in-time versus reconstructed history A vendor that has been selling for years improves its product: it adds banks, fixes the merchant mapping, adjusts the weights. When a new client asks for history, the usual practice is to deliver the series **reconstructed with the current methodology** (*backfill*), not the one the clients saw back then. With price data that is a nuisance; with data that is calibrated with the figures it is meant to anticipate, it is pure look-ahead. The course's history does the three typical things: it fills in the days with the transactions that arrived late, **reweights** the old panel to the current size and composition, and **calibrates** each quarter with the sales already published. The way to see it is to put the two versions side by side. For the comparison to be clean, you need a function that returns what was known on a date: ```python def a_fecha(df, fecha): """Rows delivered up to `fecha` (inclusive).""" return df[df["fecha_entrega"] <= pd.Timestamp(fecha)] def por_trimestre(df): d = df.merge(mapa.dropna(subset=["id_empresa"]), on="descriptor") d["fin_trimestre"] = (d["fecha"].dt.to_period("Q") .dt.end_time.dt.normalize()) return d.groupby(["id_empresa", "fin_trimestre"])["gasto_usd"].sum() ejemplo = a_fecha(pit, "2021-04-02") print("last day known on 2021-04-02:", ejemplo["fecha"].max().date()) gp, gh = por_trimestre(pit), por_trimestre(hist) r = (gh / gp).dropna().rename("ratio").reset_index() r["tramo"] = np.select( [r["fin_trimestre"] < "2021-03-31", r["fin_trimestre"] == "2021-03-31", r["fin_trimestre"] <= "2023-03-31"], ["1 before the new bank", "2 mixed quarter", "3 earnings published"], "4 unpublished (2023Q2)") r.groupby("tramo")["ratio"].agg(["count", "mean", "std"]).round(3) ``` ```output last day known on 2021-04-02: 2021-03-30 count mean std tramo 1 before the new bank 300 1.431 0.029 2 mixed quarter 25 1.277 0.046 3 earnings published 200 1.053 0.066 4 unpublished (2023Q2) 25 1.067 0.001 ``` The ratio between the history and the delivered data tells what the vendor did: - **Before the new bank** the history is 43% higher: the reweighting to the current panel (which has 35% more cards) and the share of late transactions. - **2023Q2** is the only quarter in the history whose sales were not known on the delivery date. Its ratio is 1.067 across the 25 companies, with a standard deviation of 0.001: only the late-arriving transactions were added to it (6% of preliminary spending, 1 / 0.94 ≈ 1.064). - **With earnings published**, the ratio varies from one company and quarter to another (standard deviation 0.066): the history has moved toward sales that nobody knew on that date. This test does not need to know the generator's ground truth: it is enough to compare the history with the live data for the overlapping quarters, or with the delivery of a quarter that had no earnings yet. If the vendor cannot provide the archive of what it delivered each day, the comparison is made with the first months of the trial and with direct questions: when the methodology changed and whether the history is recalculated with it. Some platforms offer explicit “as of” queries. The Nasdaq Data Link Python client (version 1.0.4, MIT license) has `get_point_in_time` with three modes (`asofdate`, `from`, `between`), as well as `get_table` for paginated tables; according to its documentation, without an API key some tables return only a sample. The code, not executed (it needs a network and a key; the table is the one from the documentation's example): ```python skip import nasdaqdatalink # the key, in the NASDAQ_DATA_LINK_API_KEY environment variable tabla = nasdaqdatalink.get_table( "ZACKS/FC", paginate=True, ticker=["AAPL", "MSFT"], per_end_date={"gte": "2015-01-01"}, qopts={"columns": ["ticker", "per_end_date"]}) # the table as it was on a date vista = nasdaqdatalink.get_point_in_time( "DATABASE/CODE", interval="asofdate", date="2020-01-01") ``` What matters is the consequence for the signal. The signal in lesson 6.7.5 compares the year-over-year growth of spending per card with the growth implied by the consensus; here its R² against the sales surprise is enough, measured with each version of the data: ```python panel = pd.read_csv("alt6_tarjetas_panel.csv", parse_dates=["fecha"]) panel["fin_trimestre"] = (panel["fecha"].dt.to_period("Q") .dt.end_time.dt.normalize()) tarjetas = panel.groupby("fin_trimestre")["tarjetas_activas"].mean() T = v.set_index(["id_empresa", "fin_trimestre"]).sort_index() def interanual(s): return np.log(s).groupby(level=0).diff(4) T["g_pit"] = interanual( gp / tarjetas.reindex(gp.index.get_level_values(1)).to_numpy()) T["g_hist"] = interanual(gh) # already scaled to the current panel T["g_cons"] = (np.log(T["consenso_ventas"]) - np.log(T["ventas"].groupby(level=0).shift(4))) T["sorpresa"] = np.log(T["ventas"] / T["consenso_ventas"]) f = T.index.get_level_values(1) def r2(col, desde, hasta): d = T[(f >= desde) & (f <= hasta)].dropna(subset=[col]) x = d[col] - d["g_cons"] return len(d), x.corr(d["sorpresa"]) ** 2 for nombre, col, a, b in [ ("history, 2019Q1-2023Q1", "g_hist", "2019", "2023-03-31"), ("PIT, 2019Q1-2023Q1", "g_pit", "2019", "2023-03-31"), ("PIT after delivery, 2023Q3-", "g_pit", "2023-09-30", "2025-12-31")]: n, valor = r2(col, a, b) print(f"{nombre:30s} n = {n} R² = {valor:.3f}") verdad = pd.read_csv("alt6_verdad_trimestral.csv", parse_dates=["fin_trimestre"]) vv = verdad[verdad["ticker"].isin(v["ticker"]) & (verdad["fin_trimestre"] >= "2019")] print(f"ceiling (true measurable part) R² = " f"{vv['pred'].corr(vv['sorpresa_ventas']) ** 2:.3f}") ``` ```output history, 2019Q1-2023Q1 n = 425 R² = 0.391 PIT, 2019Q1-2023Q1 n = 425 R² = 0.276 PIT after delivery, 2023Q3- n = 225 R² = 0.277 ceiling (true measurable part) R² = 0.567 ``` This is the header figure. With the delivered history, the signal explains 39.1% of the variance of surprises between 2019 and 2023; with the data as it arrived each day, 27.6% over the same 425 company-quarters. A manager who assessed the dataset with the history would expect 39%, and after signing would see 27.7%: exactly what the live data gave before. The difference was the calibration with sales. The ceiling (0.567) is the part of the surprise that the true data measures, the maximum a perfect method could reach. The pattern is general and that is why it is a due diligence test: **a drop in performance right at the delivery date of the history** gives away a backfill, just as the drop in an LLM's accuracy at its cutoff date gives away its memory (lesson 6.2.5). ## Panel biases A card panel is a convenience sample: the cardholders of the banks or apps that share data with the vendor. Lesson 6.7.2 showed the panel jump on 2021-03-01 in daily spending. In due diligence you look at its effect on the quantity that will be used, the year-over-year growth of each quarter: ```python T["bruto"] = gp T["por_tarjeta"] = gp / tarjetas.reindex( gp.index.get_level_values(1)).to_numpy() g = pd.DataFrame({c: interanual(T[c]) for c in ["ventas", "bruto", "por_tarjeta"]}).dropna() mediana = 100 * g.groupby(level=1).median() mediana.loc["2020-12-31":"2022-06-30"].round(1) ``` ```output ventas bruto por_tarjeta fin_trimestre 2020-12-31 4.6 13.5 3.6 2021-03-31 5.4 22.5 1.0 2021-06-30 7.5 47.6 7.5 2021-09-30 6.7 48.5 8.5 2021-12-31 7.7 46.3 6.3 2022-03-31 5.5 34.0 5.3 2022-06-30 4.6 15.4 5.4 ``` *Fig. — Median year-on-year growth of 25 companies: raw card panel spend, spend per active card and sales (2019-2025).* Three quarterly lines. Raw spend, highlighted, grows between 11% and 18% for most of the period and jumps to 46-48% for four quarters from the second quarter of 2021, after a bank joins the panel on 1 March 2021 (vertical line). Spend per card and sales move together between 0% and 8.5%. For four quarters raw spending (`bruto`) grows 46-48% year over year while sales (`ventas`) grow 7-8%: a model trained on raw spending would see a consumer boom. Outside the jump, raw spending also grows about 10 points more than sales every year, because the panel gains 10% more cards a year. Normalized by active cards (`por_tarjeta`), it tracks sales. Size is corrected with a division. **Composition** is harder. With the new bank, and year after year, the panel has more young cardholders (from 30% in 2018 to 47% in 2025), and the brands they prefer gain weight in the panel without selling more. The test is to measure, company by company, how far spending per card deviates from its sales over the sample. The vendor does not publish each brand's customer base, but the generator does (`alt6_tarjetas_verdad.csv`), and it lets you see whether the deviation has a pattern: ```python clientela = pd.read_csv("alt6_tarjetas_verdad.csv", index_col="id_empresa")["clientela_joven"] deriva = 100 * (g["por_tarjeta"] - g["ventas"]).groupby(level=0).mean() deriva = pd.DataFrame({"deriva_pp": deriva, "clientela_joven": clientela}).dropna() print(deriva.sort_values("clientela_joven").iloc[[0, 1, -2, -1]] .round(2).to_string()) print("correlation:", round(deriva.corr().iloc[0, 1], 3)) b = np.polyfit(deriva["clientela_joven"], deriva["deriva_pp"], 1)[0] print(f"slope: {b:.2f} pp of annual growth per unit") ``` ```output deriva_pp clientela_joven id_empresa E011 -1.80 0.12 E117 -1.62 0.12 E120 2.04 0.95 E105 2.39 0.99 correlation: 0.982 slope: 4.70 pp of annual growth per unit ``` *Fig. — Drift of spend per card against each company's sales (average year-on-year growth, 2019-2025) by its share of young customers.* Scatter of 25 points, one per company, lined up along a highlighted rising line. Companies with older customers (young share near 0) drift by down to −1.8 points a year; those with young customers (near 1), by up to +2.4. Correlation 0.98. The drift (`deriva_pp`) has a correlation of 0.98 with the young customer share (`clientela_joven`): spending per card at a company whose customers are almost all young grows about 2 points a year more than its sales, and at one with older customers, almost 2 points less. In a sales surprise signal, that bias enters as a systematic error per company. With real data the customer base is not known, but the drift can be measured (spending against published sales, company by company), and you ask the vendor how the panel's composition changes and whether it reweights it (by age, income, region). Reweighting is legitimate if it is applied the same way live and in the history; what is not legitimate is applying it only backward. ## Survivorship and deliveries The survivorship bias of stock universes (lesson 1.4.4) has its counterpart in datasets: a vendor that only delivers history for the merchants **still active today** erases the stores that closed, and past growth looks better. The test is to count the merchants with data in each period, in both versions: ```python def vivos(df): m = df.assign(mes=df["fecha"].dt.to_period("M")) return m.groupby("mes")["descriptor"].nunique() print("PIT: merchants per month, minimum and maximum:", vivos(pit).min(), vivos(pit).max()) print("history:", vivos(hist).min(), vivos(hist).max()) primero = pit.groupby("descriptor")["fecha"].agg(["min", "max"]) print("additions after 2018-01-01:", int((primero["min"] > "2018-01-01").sum()), "· removals before 2025-09-30:", int((primero["max"] < "2025-09-30").sum())) print("PIT merchants missing from the history:", len(set(pit["descriptor"]) - set(hist["descriptor"]))) ``` ```output PIT: merchants per month, minimum and maximum: 127 127 history: 127 127 additions after 2018-01-01: 0 · removals before 2025-09-30: 0 PIT merchants missing from the history: 0 ``` No merchant disappears in this data: all 127 are there every month in both versions. The test passes, and that is also information: the synthetic panel has no merchant additions or removals, so survivorship bias cannot be measured here. With a real vendor, the norm is the opposite: stores that close, merchants that change payment processor (and descriptor) and brands that are sold. The questions are whether the history includes the merchants that were removed and whether the mapping to companies is the one from each date or today's (a brand sold in 2020 should not add to its buyer in 2018). Delay is measured the same way. It was already shown that each day arrives three days later; the data also arrives **incomplete**: the ratio of 1.067 for the 2023Q2 quarter says that preliminary spending is missing 6% that arrives later. If the vendor revises its deliveries, each version must be kept with its date (like the point-in-time fundamentals of level 5), without overwriting the previous one. ## Legality: inside information and personal data The legal review is not done by the quant analyst, but the analyst has to know what to ask and what the data does. Two EU regulations cover the essentials. What follows summarizes their text, read on 2026-10-23 in the version of the Official Journal of the EU published by the BOE[^1]; it is not legal advice. **Inside information (MAR, Regulation (EU) 596/2014).** Article 7(1)(a) defines it as “information of a precise nature, which has not been made public, relating, directly or indirectly, to one or more issuers or to one or more financial instruments” and which, if it were made public, would be likely to have a significant effect on their prices. Article 8 defines insider dealing and article 14 prohibits it, along with recommending it and unlawfully disclosing the information. For alternative data, the key text is recital 28: “Research and estimates based on publicly available data, should not per se be regarded as inside information”, with exceptions (for example, if the market routinely expects its publication and it contributes to price formation), and it asks operators to consider “the extent to which the information is non-public”. The risk is not in aggregating data from thousands of cards, but in its **origin**: a dataset containing the issuer's internal figures, passed on by an employee or by a supplier bound by confidentiality, can be inside information even if it arrives in a commercial file. **Personal data (GDPR, Regulation (EU) 2016/679).** Article 4(1) defines personal data as “any information relating to an identified or identifiable natural person”; article 4(5), pseudonymization (data that can no longer be attributed to a person “without the use of additional information”, kept separately). Recital 26 makes clear that pseudonymized data is still personal data and that the regulation does not apply to anonymous information, that which does not allow anyone to be identified by the means reasonably likely to be used. A transaction panel “anonymized” with a random identifier per cardholder is, in general, pseudonymized. The difference is not theoretical. de Montjoye and coauthors (2015) studied three months of transactions from 1.1 million people and found that four spatiotemporal points are enough to reidentify 90%. The next cell reproduces it on a small scale with simulated cardholders: 20,000 people who shop mostly at their six usual merchants, out of 400, over 90 days. ```python rng = np.random.default_rng(0) n_pers, n_com, dias = 20_000, 400, 90 habituales = rng.integers(0, n_com, size=(n_pers, 6)) quien_compra = np.repeat(np.arange(n_pers), rng.poisson(25, n_pers) + 1) comercio = habituales[quien_compra, rng.integers(0, 6, len(quien_compra))] otro = rng.random(len(comercio)) < 0.2 # purchases outside the usual ones comercio[otro] = rng.integers(0, n_com, otro.sum()) dia = rng.integers(0, dias, len(comercio)) tx = pd.DataFrame({"persona": quien_compra, "comercio": comercio, "dia": dia}).drop_duplicates() tx["punto"] = tx["comercio"] * dias + tx["dia"] personas_en = tx.groupby("punto")["persona"].apply(set).to_dict() puntos_de = tx.groupby("persona")["punto"].apply(np.array) muestra = rng.choice(n_pers, 2000, replace=False) for k in (1, 2, 3, 4): unicos = 0 for p in muestra: elegidos = rng.choice(puntos_de[p], k, replace=False) candidatos = set.intersection( *(personas_en[x] for x in elegidos)) unicos += len(candidatos) == 1 print(f"{k} known purchase(s): {unicos / 20:.1f}% unique") celdas = tx.groupby(["comercio", "dia"])["persona"].nunique() print(f"merchant-day cells with fewer than 10 people: " f"{(celdas < 10).mean():.1%}") ``` ```output 1 known purchase(s): 0.0% unique 2 known purchase(s): 96.0% unique 3 known purchase(s): 100.0% unique 4 known purchase(s): 100.0% unique merchant-day cells with fewer than 10 people: 9.9% ``` With one known purchase (the merchant and the day, for example from a receipt or a photo on social media) nobody is identified: each merchant gets about 14 people a day on average. With two, 96% of the cardholders in the sample are singled out; with three, all of them. A random identifier does not protect. What reduces the risk is to **aggregate**: the course's vendor delivers spending and number of transactions per merchant and day, not purchases by individuals. But aggregation has its own limit: in the simulation, 9.9% of merchant-day cells have fewer than 10 people, and a cell with two or three buyers is almost individual data again. That is why serious vendors suppress or group small cells, and the threshold they use is a due diligence question. For the buyer, the difference between receiving transactions with a cardholder identifier and receiving aggregates with a threshold is the difference between processing personal data and, in general, not processing it. The legal review questions come from those two regulations: ```mermaid flowchart TB A[Data origin] --> B{Non-public issuer data} B -->|yes| X[MAR risk: stop] B -->|no| C{Data on individuals} C -->|no| OK[Contract review] C -->|yes| D{Aggregated and anonymous} D -->|yes| OK D -->|no| E[GDPR legal basis] E --> F[Legal team decides] ``` In practice these are written questions to the vendor (the DDQ): where the data comes from and under which contract, whether the cardholders consented and for what, whether there is data from employees or suppliers of the issuer, how it is aggregated, what minimum threshold of people per cell is used and what happens if a cardholder exercises the right to erasure. The answers are filed with the dataset version: if they change, the review is repeated. ## The due diligence report All of the above fits in a table that can be repeated every time the vendor delivers a new version. The technical tests come from the code; the legal ones, from the vendor's answers: ```python n_in, r2_hist = r2("g_hist", "2019", "2023-03-31") _, r2_pit = r2("g_pit", "2019", "2023-03-31") informe = pd.DataFrame([ ("PIT history available", "yes (daily archive)", "ok"), ("R² inflation with the history", f"{r2_hist:.3f} versus {r2_pit:.3f}", "discard history"), ("panel size change", "+35% on 2021-03-01", "normalize"), ("composition drift", "corr. 0.98 with young customers", "correct or model"), ("merchants that disappear", "0 of 127", "ok"), ("delivery delay", "3 days; 6% arrives late", "ok"), ("margin until earnings", f"{margen.min()} days or more", "ok"), ("origin and consent", "ask the vendor", "pending"), ("minimum aggregation per cell", "ask the vendor", "pending"), ], columns=["test", "result", "decision"]) print(informe.to_string(index=False)) ``` ```output test result decision PIT history available yes (daily archive) ok R² inflation with the history 0.391 versus 0.276 discard history panel size change +35% on 2021-03-01 normalize composition drift corr. 0.98 with young customers correct or model merchants that disappear 0 of 127 ok delivery delay 3 days; 6% arrives late ok margin until earnings 17 days or more ok origin and consent ask the vendor pending minimum aggregation per cell ask the vendor pending ``` The decision in this case: the dataset is usable with the point-in-time archive, normalized by cards, with the composition drift handled and discarding the calibrated history for any backtest. With that, lesson 6.7.5 builds the signal and measures how much alpha is left. [^1]: The text was read in the versions of the OJEU published by the BOE, the Spanish Official Gazette: [MAR](https://www.boe.es/buscar/doc.php?id=DOUE-L-2014-81277) and [GDPR](https://www.boe.es/buscar/doc.php?id=DOUE-L-2016-80807), accessed 2026-10-23. EUR-Lex publishes the same text (links in the resources), in English among other languages, and the consolidated versions with later amendments; the quotations above are from the English text. ## Exercises 1. Repeat the inflation test with a different split: measure the R² of the history and of the PIT only in 2019-2020 and only in 2021Q2-2023Q1. In which period does the history inflate more? **Solution.** ```python for a, b in [("2019", "2020-12-31"), ("2021-06-30", "2023-03-31")]: _, rh = r2("g_hist", a, b) n, rp = r2("g_pit", a, b) print(f"{a} to {b}: n = {n} history {rh:.3f} PIT {rp:.3f}") ``` ```output 2019 to 2020-12-31: n = 200 history 0.437 PIT 0.330 2021-06-30 to 2023-03-31: n = 200 history 0.345 PIT 0.232 ``` The history inflates both periods by almost the same amount, about 11 points of R² (0.437 versus 0.330 in 2019-2020; 0.345 versus 0.232 in 2021Q2-2023Q1). The level of each period changes with the sample (200 company-quarters give a noisy R²), but the difference between versions is stable: it is the footprint of the calibration, which was applied to every quarter with published earnings. 2. Change the reidentification simulation so that the vendor delivers the **week** instead of the day of each purchase. How many purchases are now needed to single out the majority? **Solution.** ```python tx["punto_sem"] = tx["comercio"] * 13 + tx["dia"] // 7 personas_sem = tx.groupby("punto_sem")["persona"].apply(set).to_dict() puntos_sem = tx.groupby("persona")["punto_sem"].apply( lambda s: np.unique(s.to_numpy())) for k in (2, 3, 4): unicos = 0 for p in muestra[:500]: pts = puntos_sem[p] elegidos = rng.choice(pts, min(k, len(pts)), replace=False) cand = set.intersection(*(personas_sem[x] for x in elegidos)) unicos += len(cand) == 1 print(f"{k} purchases: {unicos / 5:.1f}% unique") ``` ```output 2 purchases: 65.4% unique 3 purchases: 98.0% unique 4 purchases: 99.6% unique ``` With the week, two purchases single out 65% (versus 96% with the day), three 98% and four almost everyone. Lowering the resolution raises the threshold without eliminating the risk, which is what de Montjoye and coauthors found with real data: even with coarse information in every dimension, anonymity is scarce. 3. A vendor offers you a dataset of “purchase intentions” obtained from surveys of purchasing managers at large retail chains, delivered two weeks before the chains report their sales. What questions would you ask before testing the signal? **Solution.** First the MAR ones: if the respondents work at the issuer or its suppliers and share unpublished internal figures (quarterly sales, orders), the dataset may contain inside information (article 7) and trading on it would be prohibited (article 14); recital 28 protects research done with public data, not research done with the company's internal data. Then the contract ones: whether the respondents have confidentiality obligations to their employer and how the vendor manages them. The GDPR ones are minor if the answers are aggregated, but you need to know whether the respondents' names are kept. And the technical ones: since when the survey has existed as it is delivered today (point-in-time) and whether the history has been “cleaned” with already published sales. ## Self-assessment 1. In July 2023 a card data vendor delivers a history going back to 2018 with which its signal explains 39% of the variance of sales surprises. With the data as delivered each day, the same calculation gives 28%. Which figure should be used to decide on the purchase? - A. 39%, because the history is cleaner and more complete - B. The average of the two - C. 28%: it is what could have been measured on each date; the history is recalibrated with sales that were not known at the time - D. Neither: the R² is no use for assessing data *Answer C: 28%: it is what could have been measured on each date; the history is recalibrated with sales that were not known at the time.* The reconstructed history includes adjustments made with later information (reweighting to the current panel and calibration with published sales). After delivery, the live data gives 28% again: the difference was look-ahead, not quality. 2. The panel's raw spending grows 47.6% year over year in the second quarter of 2021 and sales 7.5%. What is the first thing to check? - A. Whether the number of cards in the panel has changed (in the data, a new bank adds 35% more cards on 2021-03-01) - B. Whether the analyst consensus fell short - C. Whether the vendor has a rounding error - D. Whether the quarter had more business days *Answer A: Whether the number of cards in the panel has changed (in the data, a new bank adds 35% more cards on 2021-03-01).* A panel grows and changes banks or apps; without normalizing by active cards, that change looks like demand. Per card, growth is 7.5%, the same as sales. 3. Normalized by cards, the spending of companies with a young customer base grows faster than their sales year after year, and that of companies with older customers, slower. What bias is it? - A. Merchant survivorship - B. Look-ahead from the delivery date - C. Entity mapping error - D. Panel composition: the share of young cardholders rises over time, so the brands they prefer gain weight without selling more *Answer D: Panel composition: the share of young cardholders rises over time, so the brands they prefer gain weight without selling more.* In the data, the average deviation of spending per card from sales has a correlation of 0.98 with each company's share of young customers. A panel is not the population: if its composition changes, the measure changes. 4. According to recital 28 of MAR, when can research developed from public data constitute inside information? - A. Never: your own research is never inside information - B. For example, if the market routinely expects its publication and it contributes to price formation; the recital asks you to consider the extent to which the information is non-public - C. Whenever it is profitable - D. Only if it is prepared by an employee of the issuer *Answer B: For example, if the market routinely expects its publication and it contributes to price formation; the recital asks you to consider the extent to which the information is non-public.* Recital 28 says that such research should not be regarded, in itself, as inside information, with exceptions. The due diligence question is where the data comes from: a dataset containing the issuer's non-public information (for example, leaked by an employee) is a different case. 5. A vendor says its card data is “anonymized” because it replaces the cardholder's name with a random identifier. What does the GDPR say? - A. That it is anonymous data and the GDPR does not apply - B. That encrypting the identifier is enough - C. That it is pseudonymized data (it can be attributed to a person with additional information) and is still personal data; only information that does not allow anyone to be identified falls outside - D. That the GDPR only applies to health data *Answer C: That it is pseudonymized data (it can be attributed to a person with additional information) and is still personal data; only information that does not allow anyone to be identified falls outside.* Article 4(5) and recital 26 of the GDPR. With 2 purchases (merchant and day) of a person, 96% of the cardholders in the lesson's simulation are identified: a random identifier does not prevent singling out. ## Key points - Neudata, Eagle Alpha and Nasdaq Data Link cover different parts of the purchase (catalog and assessments, advisory and compliance questionnaires, API distribution); none of them replaces the buyer's tests on its own data. - The history the vendor delivers on signing is not the data that existed on each date: the course's card history is reweighted and calibrated with already published sales and takes the signal's R² from 0.28 to 0.39. After delivery, the live data goes back to 0.28. - A panel changes size (+35% cards on 2021-03-01) and composition (more young cardholders): normalize by active cards and measure each company's drift against its sales. - Survivorship and delays are checked in the data: in the course, no merchant disappears and each day arrives three days later, with 6% of transactions still to come; the full quarter is available 17 days or more before earnings. - Legality: MAR prohibits trading on inside information (articles 7, 8 and 14) and the GDPR applies to pseudonymized data. The questions to the vendor are about origin, consent and aggregation; the answers, for the legal and compliance team. ## Resources ### Documentation - [Regulation (EU) 596/2014 on market abuse (MAR), EUR-Lex](https://eur-lex.europa.eu/legal-content/EN/TXT/?uri=CELEX:32014R0596) · eur-lex.europa.eu - [Regulation (EU) 2016/679, General Data Protection Regulation (GDPR), EUR-Lex](https://eur-lex.europa.eu/legal-content/EN/TXT/?uri=CELEX:32016R0679) · eur-lex.europa.eu - [Nasdaq Data Link: Python client (get_table, get_point_in_time)](https://github.com/Nasdaq/data-link-python) · github.com - [Eagle Alpha: FAQs (taxonomy of 16 categories and 56 subcategories)](https://www.eaglealpha.com/faqs/) · eaglealpha.com ### Books - [de Montjoye, Radaelli, Singh and Pentland (2015): Unique in the shopping mall. Science 347(6221), 536-539. DOI 10.1126/science.1256297](https://doi.org/10.1126/science.1256297) · doi.org ### Data - [Level 6 data: card data as delivered (Parquet)](https://pythonparatrading.com/curso/datos/nivel-6/alt6_tarjetas_pit.parquet) · Download · alt6_tarjetas_pit.parquet - [Level 6 data: history delivered on signing (Parquet)](https://pythonparatrading.com/curso/datos/nivel-6/alt6_tarjetas_historico.parquet) · Download · alt6_tarjetas_historico.parquet Previous lesson: [6.7.3 · Responsible web scraping](https://pythonparatrading.com/en/python-trading-course/level-6/responsible-web-scraping/) Next lesson: [6.7.5 · From alternative data to a trading signal](https://pythonparatrading.com/en/python-trading-course/level-6/from-alternative-data-to-trading-signals/)