Professional market data providers.

What changes in the professional tier

The APIs in lesson 1.3.3 sell bars, trades and, in the higher plans, the best quote. A professional provider sells something else: the direct feeds of each exchange, with the full order-by-order book (L3, lesson 1.1.3), the venue's own timestamps, reference data (instrument definitions, corporate actions, indices) and decades of history. And it sells them under other terms: licenses from each exchange, annual contracts and prices that are often not published.

flowchart TB
  B[Exchanges and venues] --> F[Direct feeds]
  B --> S[Consolidated tape]
  F --> P[Provider]
  S --> P
  P --> N[Normalized data]
  N --> T[Your system]

The provider's job is to capture each feed (each exchange has its own protocol), normalize it to a common format, store it with reliable timestamps and serve it through an API or in files. That is what you pay for: not the data itself, which comes from the exchange, but the capture, the normalization, the history and the license.

Databento: exchange feeds with published prices

Databento is the most accessible case in the tier, because it publishes prices and full documentation. Its datasets are named after the source feed; three of the main ones11Note 1 · Databento: GLBX.MDP3, XNAS.ITCH and OPRA.PILLAR, accessed on October 21, 2026.:

• GLBX.MDP3: CME Globex (futures and options on futures from CME, CBOT, NYMEX and COMEX). Coverage starts on June 6, 2010, with a caveat given by its own documentation: CME introduced the MDP 3.0 protocol in May 2017; before that, the data comes from the previous protocol (FIX/FAST), and until May 21, 2017 it has no capture timestamps.
• XNAS.ITCH: Nasdaq TotalView-ITCH, Nasdaq's full book, since May 1, 2018.
• OPRA.PILLAR: US equity options (trades and consolidated best price from all options exchanges), since April 1, 2013.

For US stocks, its home page mentions 15 exchanges and 30 ATSs, with proprietary L3 feeds and the consolidated tapes. And the pricing model separates historical data and real time. According to its pricing page, as of October 21, 202622Note 2 · Databento: Pricing, accessed on October 21, 2026.:

• Historical by usage: you pay per gigabyte, and the size that counts is that of the data uncompressed in its binary encoding. It can also be used with any subscription.
• Real time by plan: Standard, 199 USD a month with monthly billing; Plus and Unlimited, with an annual contract and 1,750 and 4,500 USD a month in licenses. The page shows these plans with CME coverage; the amounts may vary with the dataset chosen.
• An initial credit of 125 USD on signing up, which expires after six months.

Billing historical data by size changes the way you work: before downloading, you calculate how much space the request takes. Its Python client has methods for that (it needs a network and a key):

import os

import databento as db             # uv add databento

client = db.Historical(os.environ["DATABENTO_API_KEY"])
peticion = dict(dataset="GLBX.MDP3", symbols=["ESZ5"],
                schema="trades", start="2025-10-01", end="2025-10-02")

print(client.metadata.get_billable_size(**peticion), "bytes")
print(client.metadata.get_cost(**peticion), "USD")

datos = client.timeseries.get_range(**peticion)   # this one is billed
df = datos.to_df()

get_billable_size returns the uncompressed billable bytes and get_cost, the cost in dollars of that request; timeseries.get_range downloads and returns an object that is converted to a DataFrame with to_df()33Note 3 · Databento: timeseries.get_range, metadata.get_cost and metadata.get_billable_size.. Good practice is to ask first and download afterwards.

Schemas: what decides the size

The same market can be requested at different levels of detail. Databento calls those formats schemas (more than fifteen) and groups them by level: L0 (bars, statistics), L1 (best price), L2 (depth by levels) and L3 (order by order)44Note 4 · Databento: What's a schema?. The main ones, with the size of each record in its DBN binary format (version 3)55Note 5 · Sizes checked in the source code of the DBN format (record.rs, DBN version 3) and 16-byte header in Databento Binary Encoding.:

All records start with the same 16-byte header (type, instrument, event timestamp). Order by order (mbo) is the most detailed, but each message is small; mbp-10 repeats the ten levels on every change and that is why it is the heaviest per record. A NOVA trade record can be written the same way, with a numpy structured type, to see where those numbers come from:

import numpy as np
import pandas as pd

registro_op = np.dtype([
    ("ts", "<u8"),        # nanoseconds since 1970
    ("precio", "<i8"),    # price as an integer (fixed scale)
    ("tamano", "<u4"),
    ("agresor", "u1"), ("centro", "u1"), ("condicion", "u1"),
])
print(registro_op.itemsize, "bytes per trade")

op = pd.read_parquet("nova_operaciones.parquet")
csv = len(op.to_csv(index=False).encode())
print(f"CSV: {csv / len(op):.1f} bytes per trade")

23 bytes per trade
CSV: 65.9 bytes per trade

23 bytes per trade in our own binary format, versus almost 66 in CSV. The course's Parquet takes even less, because it compresses: columns with few distinct values (venue, condition) and prices on a grid of cents compress very well.

from pathlib import Path

filas = {}
for nombre in ["nova_operaciones", "nova_cotizaciones",
               "nova_libro_l2", "nova_libro_l3"]:
    df = pd.read_parquet(f"{nombre}.parquet")
    filas[nombre] = {
        "filas": len(df),
        "csv": len(df.to_csv(index=False).encode()) / len(df),
        "parquet": Path(f"{nombre}.parquet").stat().st_size / len(df),
    }
pd.DataFrame(filas).T.round(1)

                      filas   csv  parquet
nova_operaciones    50946.0  65.9      4.2
nova_cotizaciones  129247.0  55.4      6.5
nova_libro_l2       44126.0  45.4      3.6
nova_libro_l3        7241.0  64.0      6.2

Between 3.6 and 6.5 bytes per row in Parquet: 8 to 16 times less than in CSV. Synthetic data compresses better than real data (more regular prices), so with market data the gain is smaller, but it is still large. That is why the billable size (uncompressed) and the size on disk are different things: a provider that charges by uncompressed size does not charge less because the data compresses well, and one that delivers compressed files delivers them much smaller than what you pay for. The Databento record takes 48 bytes because it carries more fields (the common header, the receipt timestamp, the sequence number, flags). Prices are stored as fixed-scale integers (for example, in billionths of a dollar) and not as floats: that way there are no rounding errors (lesson 0.3.1) and price comparison is exact.

How much space a year of ticks takes

With the sizes per record, the volume of a request is estimated by counting messages. The NOVA session has trades for the whole day, quotes for the whole day and L3 messages for only five minutes (from 10:00 to 10:05 New York time). To estimate the L3 messages of the whole day, we use the ratio between messages and executions in that window:

l3 = pd.read_parquet("nova_libro_l3.parquet")
foto = (l3["ts"] == l3["ts"].iloc[0]).sum()     # initial snapshot
mensajes = len(l3) - foto
ejecuciones = (l3["evento"] == "ejecucion").sum()
por_ejecucion = mensajes / ejecuciones
print(mensajes, "messages,", ejecuciones, "executions")
print(f"{por_ejecucion:.2f} messages per execution")
l3["evento"].iloc[foto:].value_counts()

6883 messages, 746 executions
9.23 messages per execution
evento
alta            3085
cancelacion     2842
ejecucion        746
modificacion     210
Name: count, dtype: int64

For each execution there are 9.23 messages in total: mostly adds (3,085) and cancellations (2,842) of orders that never get executed. That is normal in a book with market makers, who move their orders continuously (lesson 1.1.3). With that ratio and the day's trades on the venues with a book:

cot = pd.read_parquet("nova_cotizaciones.parquet")
con_libro = op["centro"].isin(["PRINCIPAL", "MTF_A", "MTF_B"]).sum()
mbo_dia = con_libro * por_ejecucion

estim = pd.DataFrame({
    "mensajes": {
        "mbo": mbo_dia,
        "mbp-10": mbo_dia,                    # upper bound
        "mbp-1": len(cot) + len(op),
        "trades": len(op),
        "ohlcv-1s": op["ts"].dt.floor("s").nunique(),
        "ohlcv-1m": 390,
    },
    "bytes": {"mbo": 56, "mbp-10": 368, "mbp-1": 80, "trades": 48,
              "ohlcv-1s": 56, "ohlcv-1m": 56},
})
estim["mb_dia"] = estim["mensajes"] * estim["bytes"] / 1e6
estim.round({"mensajes": 0, "mb_dia": 2})

          mensajes  bytes  mb_dia
mbo       393475.0     56   22.03
mbp-10    393475.0    368  144.80
mbp-1     180193.0     80   14.42
trades     50946.0     48    2.45
ohlcv-1s   17908.0     56    1.00
ohlcv-1m     390.0     56    0.02

The assumptions, to make clear what has been estimated and what has not:

• mbo: the L3 messages of the window, scaled by the day's 42,646 trades on venues with a book. The window is in mid-morning; at the open and the close there is more activity, so it may fall short.
• mbp-10: as an upper bound, one record per L3 message (every change within the top ten levels generates one).
• mbp-1: one record per change in the best price (nova_cotizaciones) plus one per trade.
• ohlcv-1s: one bar per second with at least one trade (17,908 of the session's 23,400 seconds); ohlcv-1m, the 390 minute bars of the regular session.

With this, scaling is multiplying. One year (252 sessions) of 100 stocks like NOVA:

sesiones, valores = 252, 100
estim["gb_ano"] = estim["mb_dia"] * sesiones * valores / 1e3
estim[["mb_dia", "gb_ano"]].round(1).sort_values("gb_ano")

          mb_dia  gb_ano
ohlcv-1m     0.0     0.6
ohlcv-1s     1.0    25.3
trades       2.4    61.6
mbp-1       14.4   363.3
mbo         22.0   555.3
mbp-10     144.8  3648.9

The figure in the header plots it on a logarithmic scale. From half a gigabyte for minute bars to more than 3.6 TB for ten levels: four orders of magnitude depending on the schema chosen. These are figures for a fictitious stock with a specific level of activity; a very liquid real stock can have many more messages, and an illiquid one, many fewer. The method is what counts: count messages in a sample, multiply by the record size and by the period, and compare with what the system really needs.

Nasdaq Data Link, formerly Quandl

For years Quandl was the reference data platform for research in Python; today quandl.com redirects to data.nasdaq.com and its client's own repository explains that the platform has become Nasdaq Data Link66Note 6 · Nasdaq Data Link and the README of the quandl-python client.. It works as a catalog: each provider publishes its datasets (tables or time series) and they are paid for separately. A widely used paid example in equity research is Sharadar's Core US fundamentals (table SHARADAR/SF1).

# the key is read automatically from the NASDAQ_DATA_LINK_API_KEY variable
import nasdaqdatalink                # uv add nasdaq-data-link

fund = nasdaqdatalink.get_table("SHARADAR/SF1", ticker="AAPL")

What matters when evaluating a Data Link dataset is not the platform but the provider of that dataset: what it covers, since when, whether it includes delisted companies and whether the fundamentals are point-in-time (lessons 1.3.8 and 1.5.1).

LSEG, Bloomberg and ICE: industry literacy

These three are not contracted for a personal system, but they come up in any conversation in the industry and it is worth knowing what each one is.

• LSEG. Thomson Reuters' data division was renamed Refinitiv in July 2018 and LSEG (London Stock Exchange Group) completed its acquisition in January 2021. The Eikon terminal was retired on June 30, 2025 and its replacement is LSEG Workspace. The current Python library is lseg-data, version 2 of the former refinitiv-data77Note 7 · Thomson Reuters: announcement of the Refinitiv name (July 27, 2018), LSEG: history, Eikon retirement and LSEG Data Library for Python..
• Bloomberg. The Terminal is the desktop product for data, news and analytics. For programs there is the API (BLPAPI), whose Python package is installed from Bloomberg's repository, not from PyPI; the Terminal uses the desktop API and the server products are different: B-PIPE, the real-time market data feed, and Data License, reference data, pricing and historical data in batches88Note 8 · Bloomberg Professional: Terminal, API Library, B-PIPE and Data License..
• ICE (Intercontinental Exchange) has its own data business: fixed income pricing and reference data, the ICE Consolidated Feed (content from more than 600 sources) and the ICE Data Indices indices99Note 9 · ICE: Fixed Income & Data Services, ICE Consolidated Feed and Index Solutions..

flowchart TB
  Q{What do you need?} -- ticks and book --> D[Databento]
  Q -- fundamentals --> N[Nasdaq Data Link]
  Q -- everything and news --> T[Terminal]
  Q -- fixed income --> I[ICE or LSEG]

The diagram is a simplification: the big ones cover almost everything, and the real choice depends on the contract the firm already has. For an individual or a small team, the usual approach is to combine a feed provider with published prices for the market being traded and a source of fundamentals.

Exercises

1. Repeat the mbo estimate assuming that the open and the close have twice as many messages per execution as the mid-morning window: apply 2 × por_ejecucion to the trades of the first and the last half hour. How much does the daily volume rise?

hora = op["ts"].dt.tz_convert("America/New_York").dt.strftime("%H:%M")
extremo = (hora < "10:00") | (hora >= "15:30")
libro = op["centro"].isin(["PRINCIPAL", "MTF_A", "MTF_B"])
n_ext = (libro & extremo).sum()
n_resto = (libro & ~extremo).sum()
mbo2 = n_ext * 2 * por_ejecucion + n_resto * por_ejecucion
print(n_ext, "trades at the edges, out of", libro.sum())
print(f"mbo: {mbo_dia:,.0f} -> {mbo2:,.0f} messages "
      f"(+{mbo2 / mbo_dia - 1:.0%})")

10785 trades at the edges, out of 42646
mbo: 393,475 -> 492,983 messages (+25%)

2. With the estim table, calculate for how many stocks one year of mbp-1 fits in 1 TB, and for how many of trades.

por_valor_ano = estim["mb_dia"] * sesiones / 1e3      # GB
(1000 / por_valor_ano[["mbp-1", "trades"]]).round(0)

mbp-1      275.0
trades    1623.0
Name: mb_dia, dtype: float64

3. A friend wants to “download the entire CME book since 2010” to research futures. Given what Databento's documentation says about that dataset, what would you recommend checking first?