How to choose a market data provider.

Choosing a provider is choosing biases

The previous lessons in the module go through the sources: free (1.3.2), retail APIs (1.3.3), professional (1.3.4), crypto (1.3.5), brokers (1.3.6) and their licenses (1.3.7). All of them sell “market data”, but they do not sell the same thing. Two providers can give the same close for today and, at the same time, ten-year backtest results that differ widely in return, because one keeps the companies that went bankrupt and the other does not, or because one adjusts for splits and the other leaves a false 75% drop.

The useful question is not “which is the best provider?”, but “what does my system need and which defects can I not afford?”. The order of work in this lesson is the one in the figure:

flowchart TB
  U[System use case] --> R[Requirements]
  R --> E{Knockout criteria met?}
  E -- no --> X[Discarded]
  E -- yes --> P[Weighted matrix]
  P --> M[Audit sample]
  M --> D[Document]

First you write down what the system does (asset, frequency, horizon, whether it trades live). The requirements follow from that. Some are knockout criteria: if the provider does not meet them, the price does not matter. The rest are weighted. And before paying, you test a sample, because a provider's website describes what it sells, not the defects it has.

The criteria, one by one

Not all of them weigh the same in every system. For a daily equity study, delisted securities and adjustments are knockout criteria and latency hardly matters. For an intraday futures system, dividend adjustments do not exist, but the continuous series (lesson 1.4.6), latency and exact timestamps do.

The course universe as a test bench

To measure biases you need to know the truth. The course universe is a fictional 40-stock index from 2015 to 2025 with everything a cheap provider usually lacks: the prices of the 57 tickers that have been in the index (including the bankrupt, the acquired and the removed ones) and the addition and removal dates of each member.

import pandas as pd

precios = pd.read_parquet("universo_precios.parquet")
miembros = pd.read_csv("universo_miembros.csv",
                       parse_dates=["alta", "baja"])
eventos = pd.read_csv("universo_eventos.csv")

ancho = precios.pivot(index="fecha", columns="ticker",
                      values="cierre")
print(ancho.shape[1], "tickers,", len(ancho), "sessions")
eventos["evento"].value_counts()

57 tickers, 2766 sessions
evento
exclusion             7
deslistado_quiebra    4
fusion                4
cambio_simbolo        2
Name: count, dtype: int64

With the addition and removal dates you build the membership mask: True if the security was a member on that day. The portfolio for each session is decided with the members of the previous session (shift(1)), which is what was known at the open.

fechas = ancho.index
es_miembro = pd.DataFrame(False, index=fechas,
                          columns=ancho.columns)
for t, alta, baja in miembros.itertuples(index=False):
    fin = baja if pd.notna(baja) else pd.Timestamp.max
    es_miembro.loc[(fechas >= alta) & (fechas < fin), t] = True

peso = es_miembro.shift(1, fill_value=False)
rent = ancho.pct_change(fill_method=None)
es_miembro.sum(axis=1).unique()

array([40])

Exactly 40 members every day, as the index rule says.

Three providers, three returns

With this data you can simulate three providers that sell the “same” history:

• Complete (completo). It has the 57 tickers and the membership history. It is the truth of the universe.
• Without delisted securities (sin_deslistadas). It has the membership history, but only the tickers that trade today: the bankrupt, the acquired and the old tickers are missing. It is the most frequent defect in free sources.
• Current members only (miembros_actuales). Neither delisted securities nor membership history: the user takes today's 40 members and applies the whole history to them. It is what you do without noticing when you download “the index” from a current list.

cotiza_hoy = ancho.columns[ancho.iloc[-1].notna()]
actuales = miembros.loc[miembros["baja"].isna(), "ticker"]

proveedores = {
    "completo": rent.where(peso),
    "sin_deslistadas": rent[cotiza_hoy].where(peso[cotiza_hoy]),
    "miembros_actuales": rent[actuales],
}
{k: v.shape[1] for k, v in proveedores.items()}

{'completo': 57, 'sin_deslistadas': 47, 'miembros_actuales': 40}

The portfolio is equal-weighted and rebalanced every day: the index return is the average of the returns of the members the provider has. If a member has no price, it simply does not enter the average, which is what any code that does not check coverage would do.

def anual(r_dia):
    curva = (1 + r_dia.iloc[1:].fillna(0)).cumprod()
    anos = (curva.index[-1] - curva.index[0]).days / 365.25
    return curva, curva.iloc[-1] ** (1 / anos) - 1

curvas, tabla = {}, {}
for nombre, r in proveedores.items():
    curvas[nombre], cagr = anual(r.mean(axis=1))
    tabla[nombre] = {"final": curvas[nombre].iloc[-1],
                     "anual_pct": 100 * cagr}
pd.DataFrame(tabla).T.round(2)

                   final  anual_pct
completo            1.74       5.18
sin_deslistadas     2.21       7.49
miembros_actuales   3.13      10.94

The figure at the top compares the three providers. The same index, the same rule, the same eleven years. With the complete provider, 1 dollar becomes 1.74 (5.18% per year). Without the delisted securities, 2.21 (7.49%). With the current members, 3.13 (10.94%): more than twice the annual return. None of that is an edge of the system: it is the provider.

The reason is the one explained in lesson 1.4.4. Securities leave the index after falling (because of market capitalization, because of bankruptcy) and enter it after rising. A provider without delisted securities erases the losers of the past; using the current members also puts into the past the winners that were not there yet.

It is not enough for the provider to say it has delisted securities. You have to check that it has the ones that matter for your universe: the members of the index at the start of the history.

ini = miembros.loc[miembros["alta"] == "2015-01-02", "ticker"]
for nombre, r in proveedores.items():
    tiene = ini.isin(r.columns).sum()
    print(f"{nombre:18} {tiene} of {len(ini)}")

completo           40 of 40
sin_deslistadas    30 of 40
miembros_actuales  23 of 40

A current-members-only provider has 23 of the 40 securities the index started with: 42% of the 2015 universe has disappeared from its data or was never there. The provider without delisted securities has 30: it is missing the ones that went bankrupt, the acquired ones and the two old tickers.

Identifiers: the ticker is not the company

Those two old tickers are another criterion. There are two ticker changes in the universe: ZXUV is renamed ZXMA in 2018 and ZXLU becomes ZXBI in 2022. They are the same company, but the data stores them as two series:

cambios = eventos[eventos["evento"] == "cambio_simbolo"]
for _, ev in cambios.iterrows():
    for t in (ev["ticker"], ev["ticker_nuevo"]):
        s = ancho[t].dropna()
        print(f"{t}: {s.index[0]:%Y-%m-%d} to "
              f"{s.index[-1]:%Y-%m-%d}")

ZXUV: 2015-01-02 to 2018-06-01
ZXMA: 2018-06-04 to 2025-12-31
ZXLU: 2015-01-02 to 2022-01-07
ZXBI: 2022-01-10 to 2025-12-31

A provider that indexes by ticker and does not link the two stretches gives ZXMA a history that starts in 2018: a company that seems to have gone public that year. Serious providers maintain a permanent identifier (their own or a public standard) that does not change with the ticker, and a ticker table with validity dates. When evaluating them, ask for an example of a ticker change and see how they return it.

Corporate adjustments: know what you are given

The second knockout criterion for equities is adjustments. A provider can give raw prices (the ones that were traded), prices adjusted for splits, adjusted for splits and dividends, or all three. None of them is dangerous: what is dangerous is not knowing which one it is. NOVA's daily series is unadjusted; if it is taken as if it were adjusted:

nova = pd.read_csv("nova_diario_sin_ajustar.csv",
                   parse_dates=["fecha"], index_col="fecha")
r_nova = nova["cierre"].pct_change()
(100 * r_nova).nsmallest(3).round(2)

fecha
2021-08-23   -75.19
2021-06-02   -12.21
2025-03-26   -12.10
Name: cierre, dtype: float64

A 75.19% drop in one day, with no bankruptcy or news. The audit consists of cross-checking the large jumps against the corporate events:

ev_nova = pd.read_csv("nova_eventos.csv", parse_dates=["fecha_ex"])

def saltos_con_evento(r, eventos, umbral=0.20):
    grandes = r[r.abs() > umbral]
    return pd.DataFrame({
        "rent_pct": (100 * grandes).round(2),
        "evento": grandes.index.isin(eventos["fecha_ex"]),
    })

saltos_con_evento(r_nova, ev_nova)

            rent_pct  evento
fecha
2021-08-02     23.94   False
2021-08-23    -75.19    True

Of the two moves of more than 20%, only the one on 2021-08-23 falls on an event date: the 4-for-1 split. The other, a 23.94% rise on 2021-08-02, does not match any event: it is a market move and has to stay. A filter that deleted every large jump would also remove that one, which is real. If the previous close is divided by the split ratio, the jump disappears:

dia = pd.Timestamp("2021-08-23")
previo = nova["cierre"].shift(1).loc[dia]
print(f"Previous close {previo:.2f}, close {nova.loc[dia, 'cierre']}")
print(f"Actual return: "
      f"{nova.loc[dia, 'cierre'] / (previo / 4) - 1:.2%}")

Previous close 240.00, close 59.55
Actual return: -0.75%

The full adjustment (splits and dividends, multiplicative method) is lesson 1.4.3. To choose a provider it is enough to know three things: whether it adjusts, by which method, and whether it also provides the raw prices and the event table so you can redo the adjustment. A provider that only gives adjusted prices forces you to trust its adjustment; one that gives raw prices and events lets you check it.

Depth: regimes, not rows

Many free or cheap plans limit history to a few years. Two years of daily data are about 500 sessions, which seems like a lot. The problem is that they may be a single type of market:

pit = proveedores["completo"].mean(axis=1).iloc[1:]
por_ano = (1 + pit).groupby(pit.index.year).prod() - 1
(100 * por_ano).round(1).to_frame("rent_pct").T

fecha     2015  2016  2017  2018  2019  2020  2021  2022  2023  2024  2025
rent_pct -24.6  18.7  10.5  -9.7  30.1  -9.7  49.3 -29.9  14.2  34.7   3.2

With only 2024 and 2025 you would see two positive years (+34.7% and +3.2%) and no bear year. The full history has four negative years, among them 2015 (−24.6%) and 2022 (−29.9%). A system that has only been seen in two bull years has not been tested in a downturn. For a daily system, useful depth is the one that covers several regimes: bull, bear, sideways, low and high interest rates. For an intraday system, the number of sessions matters less than the variety of conditions (news days, expirations, half-day sessions).

The decision matrix

With clear criteria, the choice can be written in pandas. Three generic profiles, inspired by the categories in the module (they are not specific providers and the scores are illustrative):

• A: unofficial free source, end of day, no delisted securities, personal license.
• B: paid retail API, delisted securities and adjustments, monthly fee.
• C: professional provider, broad coverage and tick data, high usage-based price.

Scores from 1 to 5 per criterion and two different use cases, each with its weights. First the knockout criteria, then the weighted sum:

notas = pd.DataFrame({
    "cobertura":    [3, 4, 5],
    "profundidad":  [3, 4, 5],
    "deslistadas":  [1, 4, 5],
    "ajustes":      [3, 4, 4],
    "intradia":     [1, 3, 5],
    "licencia":     [1, 3, 4],
    "coste":        [5, 4, 1],
}, index=["A", "B", "C"])

usos = pd.DataFrame({
    "diario_acciones":  [2, 3, 3, 3, 0, 1, 4],
    "intradia_futuros": [2, 2, 0, 0, 4, 1, 1],
}, index=notas.columns)
pesos = usos / usos.sum()

eliminatorio = {"diario_acciones": ("deslistadas", 3),
                "intradia_futuros": ("intradia", 4)}

def puntuar(uso):
    criterio, minimo = eliminatorio[uso]
    validos = notas[notas[criterio] >= minimo]
    return (validos @ pesos[uso]).round(2)

pd.DataFrame({u: puntuar(u) for u in usos})

   diario_acciones  intradia_futuros
B             3.94               NaN
C             3.75               4.5

For the daily equity system, A is out before anything is added up: without delisted securities there is no valid study, however cheap it is. Between B and C, B wins (3.94 versus 3.75): C is better at almost everything, but cost weighs a lot and intraday does not count. For the intraday futures system, A and B are out because of the intraday data criterion (NaN in the table) and C wins alone, with 4.5. With the same providers, the choice changes with the use case; that is why the weights are written down before looking at the prices, not after.

The matrix is not an algorithm for deciding but a way of making priorities explicit. If, after seeing the result, you feel like changing the weights, it is worth asking whether the weight was wrong or whether you simply do not like the result.

Audit a sample before paying

Most providers let you try them: a free plan, a trial period or a sample. That sample goes through an automatic audit. With the course universe as an example:

def auditar(r_dia, universo_inicial, cotiza_hoy):
    tiene = r_dia.columns
    perdidos = sorted(set(universo_inicial) - set(tiene))
    muertos = set(tiene) - set(cotiza_hoy)
    return {
        "tickers": len(tiene),
        "faltan_del_inicio": len(perdidos),
        "deslistados_incluidos": len(muertos),
        "primer_dia": f"{r_dia.index[0]:%Y-%m-%d}",
        "saltos_75pct": int((r_dia.abs() > 0.70).sum().sum()),
    }

pd.DataFrame({k: auditar(v, ini, cotiza_hoy)
              for k, v in proveedores.items()})

                         completo sin_deslistadas miembros_actuales
tickers                        57              47                40
faltan_del_inicio               0              10                17
deslistados_incluidos          10               0                 0
primer_dia             2015-01-02      2015-01-02        2015-01-02
saltos_75pct                    0               0                 0

Each row is a question asked of the sample: how many securities from the initial universe are missing, whether there is any that no longer trades (if there is none, the database has no delisted securities; in the complete one there are 10, the 8 bankrupt or acquired ones and the 2 old tickers), when it starts, and whether there are impossible jumps. Here no provider has 70% jumps because universo_precios already comes adjusted; the bankrupt ones fall to near zero over several sessions, not all at once. The full list for any provider:

And one last recommendation: store, next to each dataset, which provider it comes from, with which parameters it was requested and when. If a year from now the provider corrects its history, you will know why your backtest has changed (lesson 1.4.8).

Exercises

1. Repeat the annual return of the “without delisted securities” provider, also removing the securities that enter the index after 2015 (the IPOs and the replacements). Does it go up or down relative to 7.49%? Why?

viejos = [t for t in cotiza_hoy if t in set(ini)]
r_viejos = rent[viejos].where(peso[viejos])
_, cagr = anual(r_viejos.mean(axis=1))
print(len(viejos), "tickers;", f"{cagr:.2%} per year")

30 tickers; 6.05% per year

2. In the decision matrix, raise the weight of cost in the diario_acciones use case from 4 to 8. Does the winner change? What happens if you also remove the knockout criterion?

usos.loc["coste", "diario_acciones"] = 8
pesos = usos / usos.sum()
print(puntuar("diario_acciones"))
print((notas @ pesos["diario_acciones"]).round(2))

B    3.95
C    3.20
dtype: float64
A    3.40
B    3.95
C    3.20
dtype: float64

3. Write the requirements and the knockout criteria of a system that trades a European ETF daily with closing data, and say which three audit checks you would run first.