Alternative data vendors and due diligence.

Who sells and who answers

Lesson 6.7.1 measured the market and 6.7.2 went through the types of data. This one starts where a real purchase starts: a portfolio manager has seen a fact sheet in a catalog, the vendor offers a free sample for a few weeks and the question is whether to pay. Between the manager and the vendor there are usually intermediaries. What the three named in the syllabus say they do, read on their own websites on 2026-10-23:

The figures are each company's about itself, unaudited, and they change between pages of the same website (Eagle Alpha gives 2,500 on one and 2,600 on another). They tell you what each one offers, not how they compare. None of the three can answer the question that matters: whether this dataset improves this strategy with the data as it would have arrived. That is the buyer's due diligence, and it has an order:

flowchart TB
  A[Fact sheet and catalog] --> B[NDA and sample]
  B --> C[Technical tests]
  C --> D[Signal tests]
  D --> E[Legal review]
  E --> F[Contract]
  C --> X[Discard]
  D --> X
  E --> X

The technical tests check that the data is what it claims to be (coverage, history, deliveries, biases); the signal tests, that it adds something (lesson 6.7.5); the legal review, that it can be used. Any of the three can end the purchase. The order matters: a dataset with a reconstructed history gives brilliant and misleading signal tests, so the history is examined first.

The dataset's fact sheet

The first step is to reconstruct the fact sheet from the files, not copy it from the brochure: how many merchants, which dates, when each data point arrives. The vendor delivers two things. alt6_tarjetas_pit.parquet is the data as delivered each day (point-in-time, PIT): spending on day t arrives on day t + 3 and is preliminary. And alt6_tarjetas_historico.parquet is the history the vendor delivers on signing the contract, on 2023-07-03.

import numpy as np
import pandas as pd

pit = pd.read_parquet("alt6_tarjetas_pit.parquet")
hist = pd.read_parquet("alt6_tarjetas_historico.parquet")
mapa = pd.read_csv("alt6_comercios_verdad.csv")
for nombre, df in [("PIT", pit), ("history", hist)]:
    print(f"{nombre:10s} {len(df):7,d} rows · "
          f"{df['descriptor'].nunique()} merchants · "
          f"{df['fecha'].min():%Y-%m-%d} to "
          f"{df['fecha'].max():%Y-%m-%d}")
retraso = (pit["fecha_entrega"] - pit["fecha"]).dt.days
print("days between the date and delivery:", retraso.unique())
print(mapa["relacion"].value_counts().to_string())

PIT        359,410 rows · 127 merchants · 2018-01-01 to 2025-09-30
history    254,889 rows · 127 merchants · 2018-01-01 to 2023-06-30
days between the date and delivery: [3]
relacion
matriz          69
marca           22
filial          18
sin_relacion    18

127 merchants: 109 belong to one of the 25 companies (the parent, a brand or a subsidiary; matriz, marca, filial in the data) and 18 are unrelated even though their names are similar. Here the correct mapping (alt6_comercios_verdad.csv) is used as if the vendor provided it; building it from scratch is the first part of lesson 6.7.5. The history ends on 2023-06-30, three days before delivery.

The practical question on the fact sheet is whether the data arrives in time. A sales dataset is useful if the quarter is complete before the company reports its earnings:

emp = pd.read_csv("acciones6_empresas.csv")
ventas = pd.read_csv("acciones6_ventas.csv",
                     parse_dates=["fin_trimestre", "fecha_publicacion"])
ventas = ventas.merge(emp[["ticker", "id_empresa"]], on="ticker")
cubiertas = mapa["id_empresa"].dropna().unique()
v = ventas[ventas["id_empresa"].isin(cubiertas)]
# the last day of the quarter is delivered 3 days later
completo = v["fin_trimestre"] + pd.Timedelta(days=3)
margen = (v["fecha_publicacion"] - completo).dt.days
print(f"{len(v)} reports · margin (days): minimum {margen.min()}, "
      f"median {margen.median():.0f}, maximum {margen.max()}")

800 reports · margin (days): minimum 17, median 36, maximum 62

The full quarter arrives between 17 and 62 days before earnings: there is time to compute and trade. With a real vendor, this figure is checked against the delivery dates of the sample, not the sales pitch: the advertised delays are usually those of the best day.

Point-in-time versus reconstructed history

A vendor that has been selling for years improves its product: it adds banks, fixes the merchant mapping, adjusts the weights. When a new client asks for history, the usual practice is to deliver the series reconstructed with the current methodology (backfill), not the one the clients saw back then. With price data that is a nuisance; with data that is calibrated with the figures it is meant to anticipate, it is pure look-ahead.

The course's history does the three typical things: it fills in the days with the transactions that arrived late, reweights the old panel to the current size and composition, and calibrates each quarter with the sales already published. The way to see it is to put the two versions side by side. For the comparison to be clean, you need a function that returns what was known on a date:

def a_fecha(df, fecha):
    """Rows delivered up to `fecha` (inclusive)."""
    return df[df["fecha_entrega"] <= pd.Timestamp(fecha)]

def por_trimestre(df):
    d = df.merge(mapa.dropna(subset=["id_empresa"]),
                 on="descriptor")
    d["fin_trimestre"] = (d["fecha"].dt.to_period("Q")
                          .dt.end_time.dt.normalize())
    return d.groupby(["id_empresa", "fin_trimestre"])["gasto_usd"].sum()

ejemplo = a_fecha(pit, "2021-04-02")
print("last day known on 2021-04-02:",
      ejemplo["fecha"].max().date())
gp, gh = por_trimestre(pit), por_trimestre(hist)
r = (gh / gp).dropna().rename("ratio").reset_index()
r["tramo"] = np.select(
    [r["fin_trimestre"] < "2021-03-31",
     r["fin_trimestre"] == "2021-03-31",
     r["fin_trimestre"] <= "2023-03-31"],
    ["1 before the new bank", "2 mixed quarter",
     "3 earnings published"], "4 unpublished (2023Q2)")
r.groupby("tramo")["ratio"].agg(["count", "mean", "std"]).round(3)

last day known on 2021-04-02: 2021-03-30
                        count   mean    std
tramo
1 before the new bank     300  1.431  0.029
2 mixed quarter            25  1.277  0.046
3 earnings published      200  1.053  0.066
4 unpublished (2023Q2)     25  1.067  0.001

The ratio between the history and the delivered data tells what the vendor did:

• Before the new bank the history is 43% higher: the reweighting to the current panel (which has 35% more cards) and the share of late transactions.
• 2023Q2 is the only quarter in the history whose sales were not known on the delivery date. Its ratio is 1.067 across the 25 companies, with a standard deviation of 0.001: only the late-arriving transactions were added to it (6% of preliminary spending, 1 / 0.94 ≈ 1.064).
• With earnings published, the ratio varies from one company and quarter to another (standard deviation 0.066): the history has moved toward sales that nobody knew on that date.

This test does not need to know the generator's ground truth: it is enough to compare the history with the live data for the overlapping quarters, or with the delivery of a quarter that had no earnings yet. If the vendor cannot provide the archive of what it delivered each day, the comparison is made with the first months of the trial and with direct questions: when the methodology changed and whether the history is recalculated with it.

Some platforms offer explicit “as of” queries. The Nasdaq Data Link Python client (version 1.0.4, MIT license) has get_point_in_time with three modes (asofdate, from, between), as well as get_table for paginated tables; according to its documentation, without an API key some tables return only a sample. The code, not executed (it needs a network and a key; the table is the one from the documentation's example):

import nasdaqdatalink

# the key, in the NASDAQ_DATA_LINK_API_KEY environment variable
tabla = nasdaqdatalink.get_table(
    "ZACKS/FC", paginate=True, ticker=["AAPL", "MSFT"],
    per_end_date={"gte": "2015-01-01"},
    qopts={"columns": ["ticker", "per_end_date"]})
# the table as it was on a date
vista = nasdaqdatalink.get_point_in_time(
    "DATABASE/CODE", interval="asofdate", date="2020-01-01")

What matters is the consequence for the signal. The signal in lesson 6.7.5 compares the year-over-year growth of spending per card with the growth implied by the consensus; here its R² against the sales surprise is enough, measured with each version of the data:

panel = pd.read_csv("alt6_tarjetas_panel.csv", parse_dates=["fecha"])
panel["fin_trimestre"] = (panel["fecha"].dt.to_period("Q")
                          .dt.end_time.dt.normalize())
tarjetas = panel.groupby("fin_trimestre")["tarjetas_activas"].mean()
T = v.set_index(["id_empresa", "fin_trimestre"]).sort_index()

def interanual(s):
    return np.log(s).groupby(level=0).diff(4)

T["g_pit"] = interanual(
    gp / tarjetas.reindex(gp.index.get_level_values(1)).to_numpy())
T["g_hist"] = interanual(gh)  # already scaled to the current panel
T["g_cons"] = (np.log(T["consenso_ventas"])
               - np.log(T["ventas"].groupby(level=0).shift(4)))
T["sorpresa"] = np.log(T["ventas"] / T["consenso_ventas"])
f = T.index.get_level_values(1)

def r2(col, desde, hasta):
    d = T[(f >= desde) & (f <= hasta)].dropna(subset=[col])
    x = d[col] - d["g_cons"]
    return len(d), x.corr(d["sorpresa"]) ** 2

for nombre, col, a, b in [
        ("history, 2019Q1-2023Q1", "g_hist", "2019", "2023-03-31"),
        ("PIT, 2019Q1-2023Q1", "g_pit", "2019", "2023-03-31"),
        ("PIT after delivery, 2023Q3-", "g_pit", "2023-09-30",
         "2025-12-31")]:
    n, valor = r2(col, a, b)
    print(f"{nombre:30s} n = {n}  R² = {valor:.3f}")
verdad = pd.read_csv("alt6_verdad_trimestral.csv",
                     parse_dates=["fin_trimestre"])
vv = verdad[verdad["ticker"].isin(v["ticker"])
            & (verdad["fin_trimestre"] >= "2019")]
print(f"ceiling (true measurable part)  R² = "
      f"{vv['pred'].corr(vv['sorpresa_ventas']) ** 2:.3f}")

history, 2019Q1-2023Q1         n = 425  R² = 0.391
PIT, 2019Q1-2023Q1             n = 425  R² = 0.276
PIT after delivery, 2023Q3-    n = 225  R² = 0.277
ceiling (true measurable part)  R² = 0.567

This is the header figure. With the delivered history, the signal explains 39.1% of the variance of surprises between 2019 and 2023; with the data as it arrived each day, 27.6% over the same 425 company-quarters. A manager who assessed the dataset with the history would expect 39%, and after signing would see 27.7%: exactly what the live data gave before. The difference was the calibration with sales. The ceiling (0.567) is the part of the surprise that the true data measures, the maximum a perfect method could reach.

The pattern is general and that is why it is a due diligence test: a drop in performance right at the delivery date of the history gives away a backfill, just as the drop in an LLM's accuracy at its cutoff date gives away its memory (lesson 6.2.5).

Panel biases

A card panel is a convenience sample: the cardholders of the banks or apps that share data with the vendor. Lesson 6.7.2 showed the panel jump on 2021-03-01 in daily spending. In due diligence you look at its effect on the quantity that will be used, the year-over-year growth of each quarter:

T["bruto"] = gp
T["por_tarjeta"] = gp / tarjetas.reindex(
    gp.index.get_level_values(1)).to_numpy()
g = pd.DataFrame({c: interanual(T[c])
                  for c in ["ventas", "bruto", "por_tarjeta"]}).dropna()
mediana = 100 * g.groupby(level=1).median()
mediana.loc["2020-12-31":"2022-06-30"].round(1)

               ventas  bruto  por_tarjeta
fin_trimestre
2020-12-31        4.6   13.5          3.6
2021-03-31        5.4   22.5          1.0
2021-06-30        7.5   47.6          7.5
2021-09-30        6.7   48.5          8.5
2021-12-31        7.7   46.3          6.3
2022-03-31        5.5   34.0          5.3
2022-06-30        4.6   15.4          5.4

For four quarters raw spending (bruto) grows 46-48% year over year while sales (ventas) grow 7-8%: a model trained on raw spending would see a consumer boom. Outside the jump, raw spending also grows about 10 points more than sales every year, because the panel gains 10% more cards a year. Normalized by active cards (por_tarjeta), it tracks sales.

Size is corrected with a division. Composition is harder. With the new bank, and year after year, the panel has more young cardholders (from 30% in 2018 to 47% in 2025), and the brands they prefer gain weight in the panel without selling more. The test is to measure, company by company, how far spending per card deviates from its sales over the sample. The vendor does not publish each brand's customer base, but the generator does (alt6_tarjetas_verdad.csv), and it lets you see whether the deviation has a pattern:

clientela = pd.read_csv("alt6_tarjetas_verdad.csv",
                        index_col="id_empresa")["clientela_joven"]
deriva = 100 * (g["por_tarjeta"] - g["ventas"]).groupby(level=0).mean()
deriva = pd.DataFrame({"deriva_pp": deriva,
                       "clientela_joven": clientela}).dropna()
print(deriva.sort_values("clientela_joven").iloc[[0, 1, -2, -1]]
      .round(2).to_string())
print("correlation:", round(deriva.corr().iloc[0, 1], 3))
b = np.polyfit(deriva["clientela_joven"], deriva["deriva_pp"], 1)[0]
print(f"slope: {b:.2f} pp of annual growth per unit")

            deriva_pp  clientela_joven
id_empresa
E011            -1.80             0.12
E117            -1.62             0.12
E120             2.04             0.95
E105             2.39             0.99
correlation: 0.982
slope: 4.70 pp of annual growth per unit

The drift (deriva_pp) has a correlation of 0.98 with the young customer share (clientela_joven): spending per card at a company whose customers are almost all young grows about 2 points a year more than its sales, and at one with older customers, almost 2 points less. In a sales surprise signal, that bias enters as a systematic error per company. With real data the customer base is not known, but the drift can be measured (spending against published sales, company by company), and you ask the vendor how the panel's composition changes and whether it reweights it (by age, income, region). Reweighting is legitimate if it is applied the same way live and in the history; what is not legitimate is applying it only backward.

Survivorship and deliveries

The survivorship bias of stock universes (lesson 1.4.4) has its counterpart in datasets: a vendor that only delivers history for the merchants still active today erases the stores that closed, and past growth looks better. The test is to count the merchants with data in each period, in both versions:

def vivos(df):
    m = df.assign(mes=df["fecha"].dt.to_period("M"))
    return m.groupby("mes")["descriptor"].nunique()

print("PIT: merchants per month, minimum and maximum:",
      vivos(pit).min(), vivos(pit).max())
print("history:", vivos(hist).min(), vivos(hist).max())
primero = pit.groupby("descriptor")["fecha"].agg(["min", "max"])
print("additions after 2018-01-01:",
      int((primero["min"] > "2018-01-01").sum()),
      "· removals before 2025-09-30:",
      int((primero["max"] < "2025-09-30").sum()))
print("PIT merchants missing from the history:",
      len(set(pit["descriptor"]) - set(hist["descriptor"])))

PIT: merchants per month, minimum and maximum: 127 127
history: 127 127
additions after 2018-01-01: 0 · removals before 2025-09-30: 0
PIT merchants missing from the history: 0

No merchant disappears in this data: all 127 are there every month in both versions. The test passes, and that is also information: the synthetic panel has no merchant additions or removals, so survivorship bias cannot be measured here. With a real vendor, the norm is the opposite: stores that close, merchants that change payment processor (and descriptor) and brands that are sold. The questions are whether the history includes the merchants that were removed and whether the mapping to companies is the one from each date or today's (a brand sold in 2020 should not add to its buyer in 2018).

Delay is measured the same way. It was already shown that each day arrives three days later; the data also arrives incomplete: the ratio of 1.067 for the 2023Q2 quarter says that preliminary spending is missing 6% that arrives later. If the vendor revises its deliveries, each version must be kept with its date (like the point-in-time fundamentals of level 5), without overwriting the previous one.

Legality: inside information and personal data

The legal review is not done by the quant analyst, but the analyst has to know what to ask and what the data does. Two EU regulations cover the essentials. What follows summarizes their text, read on 2026-10-23 in the version of the Official Journal of the EU published by the BOE11Note 1 · The text was read in the versions of the OJEU published by the BOE, the Spanish Official Gazette: MAR and GDPR, accessed 2026-10-23. EUR-Lex publishes the same text (links in the resources), in English among other languages, and the consolidated versions with later amendments; the quotations above are from the English text.; it is not legal advice.

Inside information (MAR, Regulation (EU) 596/2014). Article 7(1)(a) defines it as “information of a precise nature, which has not been made public, relating, directly or indirectly, to one or more issuers or to one or more financial instruments” and which, if it were made public, would be likely to have a significant effect on their prices. Article 8 defines insider dealing and article 14 prohibits it, along with recommending it and unlawfully disclosing the information. For alternative data, the key text is recital 28: “Research and estimates based on publicly available data, should not per se be regarded as inside information”, with exceptions (for example, if the market routinely expects its publication and it contributes to price formation), and it asks operators to consider “the extent to which the information is non-public”. The risk is not in aggregating data from thousands of cards, but in its origin: a dataset containing the issuer's internal figures, passed on by an employee or by a supplier bound by confidentiality, can be inside information even if it arrives in a commercial file.

Personal data (GDPR, Regulation (EU) 2016/679). Article 4(1) defines personal data as “any information relating to an identified or identifiable natural person”; article 4(5), pseudonymization (data that can no longer be attributed to a person “without the use of additional information”, kept separately). Recital 26 makes clear that pseudonymized data is still personal data and that the regulation does not apply to anonymous information, that which does not allow anyone to be identified by the means reasonably likely to be used. A transaction panel “anonymized” with a random identifier per cardholder is, in general, pseudonymized.

The difference is not theoretical. de Montjoye and coauthors (2015) studied three months of transactions from 1.1 million people and found that four spatiotemporal points are enough to reidentify 90%. The next cell reproduces it on a small scale with simulated cardholders: 20,000 people who shop mostly at their six usual merchants, out of 400, over 90 days.

rng = np.random.default_rng(0)
n_pers, n_com, dias = 20_000, 400, 90
habituales = rng.integers(0, n_com, size=(n_pers, 6))
quien_compra = np.repeat(np.arange(n_pers),
                         rng.poisson(25, n_pers) + 1)
comercio = habituales[quien_compra,
                      rng.integers(0, 6, len(quien_compra))]
otro = rng.random(len(comercio)) < 0.2  # purchases outside the usual ones
comercio[otro] = rng.integers(0, n_com, otro.sum())
dia = rng.integers(0, dias, len(comercio))
tx = pd.DataFrame({"persona": quien_compra, "comercio": comercio,
                   "dia": dia}).drop_duplicates()
tx["punto"] = tx["comercio"] * dias + tx["dia"]
personas_en = tx.groupby("punto")["persona"].apply(set).to_dict()
puntos_de = tx.groupby("persona")["punto"].apply(np.array)
muestra = rng.choice(n_pers, 2000, replace=False)
for k in (1, 2, 3, 4):
    unicos = 0
    for p in muestra:
        elegidos = rng.choice(puntos_de[p], k, replace=False)
        candidatos = set.intersection(
            *(personas_en[x] for x in elegidos))
        unicos += len(candidatos) == 1
    print(f"{k} known purchase(s): {unicos / 20:.1f}% unique")
celdas = tx.groupby(["comercio", "dia"])["persona"].nunique()
print(f"merchant-day cells with fewer than 10 people: "
      f"{(celdas < 10).mean():.1%}")

1 known purchase(s): 0.0% unique
2 known purchase(s): 96.0% unique
3 known purchase(s): 100.0% unique
4 known purchase(s): 100.0% unique
merchant-day cells with fewer than 10 people: 9.9%

With one known purchase (the merchant and the day, for example from a receipt or a photo on social media) nobody is identified: each merchant gets about 14 people a day on average. With two, 96% of the cardholders in the sample are singled out; with three, all of them. A random identifier does not protect. What reduces the risk is to aggregate: the course's vendor delivers spending and number of transactions per merchant and day, not purchases by individuals. But aggregation has its own limit: in the simulation, 9.9% of merchant-day cells have fewer than 10 people, and a cell with two or three buyers is almost individual data again. That is why serious vendors suppress or group small cells, and the threshold they use is a due diligence question. For the buyer, the difference between receiving transactions with a cardholder identifier and receiving aggregates with a threshold is the difference between processing personal data and, in general, not processing it.

The legal review questions come from those two regulations:

flowchart TB
  A[Data origin] --> B{Non-public issuer data}
  B -->|yes| X[MAR risk: stop]
  B -->|no| C{Data on individuals}
  C -->|no| OK[Contract review]
  C -->|yes| D{Aggregated and anonymous}
  D -->|yes| OK
  D -->|no| E[GDPR legal basis]
  E --> F[Legal team decides]

In practice these are written questions to the vendor (the DDQ): where the data comes from and under which contract, whether the cardholders consented and for what, whether there is data from employees or suppliers of the issuer, how it is aggregated, what minimum threshold of people per cell is used and what happens if a cardholder exercises the right to erasure. The answers are filed with the dataset version: if they change, the review is repeated.

The due diligence report

All of the above fits in a table that can be repeated every time the vendor delivers a new version. The technical tests come from the code; the legal ones, from the vendor's answers:

n_in, r2_hist = r2("g_hist", "2019", "2023-03-31")
_, r2_pit = r2("g_pit", "2019", "2023-03-31")
informe = pd.DataFrame([
    ("PIT history available", "yes (daily archive)", "ok"),
    ("R² inflation with the history",
     f"{r2_hist:.3f} versus {r2_pit:.3f}", "discard history"),
    ("panel size change", "+35% on 2021-03-01",
     "normalize"),
    ("composition drift", "corr. 0.98 with young customers",
     "correct or model"),
    ("merchants that disappear", "0 of 127", "ok"),
    ("delivery delay", "3 days; 6% arrives late", "ok"),
    ("margin until earnings", f"{margen.min()} days or more", "ok"),
    ("origin and consent", "ask the vendor",
     "pending"),
    ("minimum aggregation per cell", "ask the vendor",
     "pending"),
], columns=["test", "result", "decision"])
print(informe.to_string(index=False))

                         test                          result         decision
        PIT history available             yes (daily archive)               ok
R² inflation with the history              0.391 versus 0.276  discard history
            panel size change              +35% on 2021-03-01        normalize
            composition drift corr. 0.98 with young customers correct or model
     merchants that disappear                        0 of 127               ok
               delivery delay         3 days; 6% arrives late               ok
        margin until earnings                 17 days or more               ok
           origin and consent                  ask the vendor          pending
 minimum aggregation per cell                  ask the vendor          pending

The decision in this case: the dataset is usable with the point-in-time archive, normalized by cards, with the composition drift handled and discarding the calibrated history for any backtest. With that, lesson 6.7.5 builds the signal and measures how much alpha is left.

Exercises

1. Repeat the inflation test with a different split: measure the R² of the history and of the PIT only in 2019-2020 and only in 2021Q2-2023Q1. In which period does the history inflate more?

for a, b in [("2019", "2020-12-31"), ("2021-06-30", "2023-03-31")]:
    _, rh = r2("g_hist", a, b)
    n, rp = r2("g_pit", a, b)
    print(f"{a} to {b}: n = {n}  history {rh:.3f}  PIT {rp:.3f}")

2019 to 2020-12-31: n = 200  history 0.437  PIT 0.330
2021-06-30 to 2023-03-31: n = 200  history 0.345  PIT 0.232

2. Change the reidentification simulation so that the vendor delivers the week instead of the day of each purchase. How many purchases are now needed to single out the majority?

tx["punto_sem"] = tx["comercio"] * 13 + tx["dia"] // 7
personas_sem = tx.groupby("punto_sem")["persona"].apply(set).to_dict()
puntos_sem = tx.groupby("persona")["punto_sem"].apply(
    lambda s: np.unique(s.to_numpy()))
for k in (2, 3, 4):
    unicos = 0
    for p in muestra[:500]:
        pts = puntos_sem[p]
        elegidos = rng.choice(pts, min(k, len(pts)), replace=False)
        cand = set.intersection(*(personas_sem[x] for x in elegidos))
        unicos += len(cand) == 1
    print(f"{k} purchases: {unicos / 5:.1f}% unique")

2 purchases: 65.4% unique
3 purchases: 98.0% unique
4 purchases: 99.6% unique

3. A vendor offers you a dataset of “purchase intentions” obtained from surveys of purchasing managers at large retail chains, delivered two weeks before the chains report their sales. What questions would you ask before testing the signal?