Types of market data: EOD, bars, ticks, order book and reference.

A ladder of aggregation

When people talk about “market data” they mean very different things: the closing price of each day, a five-minute candle, each trade, each change in the best price or each order that enters and leaves the book. They are not alternative sources but rungs of the same ladder. At the bottom are the messages emitted by the matching engine; everything else is an aggregation of them.

flowchart TB
  M[L3 messages] --> B[Rebuilt book]
  B --> L2[L2 book]
  L2 --> L1[L1 quotes]
  M --> T[Trades]
  T --> BA[Intraday bars]
  BA --> D[Daily data]
  R[Reference] -.-> D

• L3 messages (market by order): every order that arrives, is modified, is canceled or is executed, with its identifier. This is what the exchanges' order-by-order protocols publish, such as Nasdaq's ITCH11Note 1 · Nasdaq TotalView-ITCH 5.0 (link in the resources): messages for adding, executing, canceling, deleting and replacing orders, with nanosecond timestamps since midnight..
• L2 book (market by price): the book aggregated by price level, with the total size and the number of orders at each level (lesson 1.1.3).
• L1 quotes (top of book): only the best bid and the best ask, with their sizes.
• Trades (also called ticks): each execution, with price, size, venue and conditions.
• Intraday bars: trades aggregated over intervals (1 minute, 5 minutes, 1 hour) as open, high, low, close and volume (OHLCV).
• Daily data (EOD, end of day): one bar per session.
• Reference data: what describes the instruments (specifications, corporate actions, symbols, universes). It is not aggregated from the messages, but it is needed to interpret and adjust them; that is why the arrow is dashed.

The rule of the ladder is asymmetric: from one rung you can go down to the next (aggregate), but not up. Trades build bars; bars do not give back the trades.

Six rungs of NOVA

The level's data has an example of each rung for the same fictitious stock, NOVA, and for the same session, June 2, 2025, output of a simulated order-by-order book (lesson 1.1.1). Starting from the top, the daily data:

import numpy as np
import pandas as pd

diario = pd.read_csv("nova_diario_sin_ajustar.csv",
                     parse_dates=["fecha"]).set_index("fecha")
print(len(diario), "sessions;", diario.index.min().date(), "→",
      diario.index.max().date())
diario.loc["2025-05-30":"2025-06-03"]

2011 sessions; 2018-01-02 → 2025-12-31
            apertura  maximo  minimo  cierre  volumen
fecha
2025-05-30     52.58   53.98   52.01   53.13  6106070
2025-06-02     53.35   53.93   52.55   53.12  8888714
2025-06-03     53.05   53.99   52.74   53.00  8044956

One row per session, eight years in 2,011 rows. It answers questions from days to years: trends, daily volatility, portfolios rebalanced at the close. It is the cheapest data to obtain and to store, and the one with the most adjustment pitfalls (lesson 1.4.3: this one is unadjusted, with the 2021 split in plain sight).

barras = pd.read_parquet("nova_barras_5min.parquet").set_index("ts")
sesion = barras.loc["2025-06-02"]
print(len(barras), "bars in", barras.index.normalize().nunique(),
      "sessions;", len(sesion), "on June 2")
sesion.head(3)

38940 bars in 502 sessions; 78 on June 2
                           apertura  maximo  minimo  cierre  volumen
ts
2025-06-02 13:30:00+00:00     53.35   53.48   53.28   53.46   285081
2025-06-02 13:35:00+00:00     53.46   53.48   53.22   53.24   220651
2025-06-02 13:40:00+00:00     53.24   53.47   53.23   53.35   187320

78 bars per normal session (six and a half hours in five-minute slices). They answer questions from minutes to days: intraday patterns, seasonality within the session, execution on a schedule. The timestamp is the start of the bar, in UTC: the first one, 13:30 UTC, is 9:30 in New York.

op = pd.read_parquet("nova_operaciones.parquet")
cot = pd.read_parquet("nova_cotizaciones.parquet")
print(len(op), "trades;", len(cot), "quote changes")
print(op.iloc[1:3].to_string())
print(cot.iloc[1:3].to_string())

50946 trades; 129247 quote changes
                                ts  precio  tamano agresor     centro condicion
1 2025-06-02 13:30:00.301051+00:00   53.34     100   venta  PRINCIPAL   regular
2 2025-06-02 13:30:00.301052+00:00   53.34     100   venta  PRINCIPAL   regular
                                ts    bid    ask  tam_bid  tam_ask
1 2025-06-02 13:30:00.006643+00:00  53.34  53.36     1600      500
2 2025-06-02 13:30:00.075521+00:00  53.34  53.36     1600      800

Trades and quotes are the “tick-by-tick” data. Trades say what was traded, where and at what price; quotes, at what price one could trade at each instant. Together they make it possible to measure the effective bid-ask spread and slippage (lesson 1.1.5), classify buys and sells or build alternative bars (lesson 1.4.5).

l2 = pd.read_parquet("nova_libro_l2.parquet")
l3 = pd.read_parquet("nova_libro_l3.parquet")
print(len(l2), "L2 rows (snapshots every 10 s, 10 levels per side)")
print(l2[l2["ts"] == l2["ts"].iloc[0]].head(4).to_string())
print(len(l3), "L3 messages in 5 minutes;",
      l3["evento"].value_counts().to_dict())

44126 L2 rows (snapshots every 10 s, 10 levels per side)
                         ts lado  nivel  precio  tamano  ordenes
0 2025-06-02 13:30:00+00:00  bid      1   53.34    1200        3
1 2025-06-02 13:30:00+00:00  bid      2   53.33     400        2
2 2025-06-02 13:30:00+00:00  bid      3   53.32    1100        3
3 2025-06-02 13:30:00+00:00  bid      4   53.31    1100        2
7241 L3 messages in 5 minutes; {'alta': 3443, 'cancelacion': 2842, 'ejecucion': 746, 'modificacion': 210}

L2 says how much liquidity there was at each level; L3, who (which order) was providing it and at what position in the queue it stood. In five minutes of NOVA there are 7,241 messages, and most are not trades: they are adds and cancellations of orders that never execute. That noise is the raw material of microstructure studies (queues, priority, iceberg orders) and is almost never needed for a low-frequency system.

How much space each rung takes

The difference between rungs is orders of magnitude. To measure it for one session, we count the rows and the memory they take in pandas. The L3 messages are only available for five minutes; multiplied by 78 they give an estimate for the full session:

def mb(df):
    return df.memory_usage(deep=True).sum() / 1e6

dia = diario.loc[["2025-06-02"]]
l3_bytes_fila = l3.memory_usage(deep=True).sum() / len(l3)
tamanos = pd.DataFrame({
    "filas": [len(dia), len(sesion), len(op), len(cot), len(l2),
              len(l3) * 78],
    "MB": [mb(dia), mb(sesion), mb(op), mb(cot), mb(l2),
           l3_bytes_fila * len(l3) * 78 / 1e6],
}, index=["daily", "5-min bars", "trades", "quotes",
          "L2 book (10 s)", "L3 messages (estimated)"])
tamanos["filas"] = tamanos["filas"].astype(int)
tamanos.round(4)

                          filas       MB
daily                         1   0.0000
5-min bars                   78   0.0037
trades                    50946   3.4860
quotes                   129247   5.1700
L2 book (10 s)            44126   2.2506
L3 messages (estimated)  564798  33.0479

The figure in the header plots the table on a logarithmic scale. From the daily bar to L3 there are almost six orders of magnitude: one row versus more than half a million. And that is for a fictitious stock of average liquidity in a single book. Multiplied by thousands of stocks and years of history, the difference decides the infrastructure: ten years of daily data for a whole market fit on a laptop; the L3 messages of a single exchange for one year do not (lesson 1.4.7). It also decides the price: providers charge by granularity, historical depth and number of exchanges (lessons 1.3.3 and 1.3.4).

sesiones, valores = 252, 3_000
anual_gb = tamanos["MB"] * sesiones * valores / 1e3
anual_gb.round(3).rename("GB per year (3,000 stocks)")

daily                          0.036
5-min bars                     2.830
trades                      2635.384
quotes                      3908.529
L2 book (10 s)              1701.422
L3 messages (estimated)    24984.211
Name: GB per year (3,000 stocks), dtype: float64

The calculation is crude, because NOVA is not representative of a whole market and on-disk formats compress much more than pandas in memory (Parquet, lesson 1.4.7). But the order of magnitude is what matters: with daily data, one year of 3,000 stocks is a few tens of megabytes; with trades and quotes, several terabytes; with L3, tens of terabytes.

Thirty seconds of NOVA

The ladder shows clearly over a short interval: from 14:00:00 to 14:00:30 UTC (10:00 in New York), how many events each granularity records:

ini = pd.Timestamp("2025-06-02 14:00:00", tz="UTC")
fin = ini + pd.Timedelta("30s")
en = lambda df: df[(df["ts"] >= ini) & (df["ts"] < fin)]
print("L3 messages:  ", len(en(l3[l3["ts"] > ini])))
print("quotes:       ", len(en(cot)))
print("trades:       ", len(en(op)))
print("L2 snapshots: ", en(l2)["ts"].nunique())
print("5-min bars:    1 (in progress)")
precios = en(cot)[["bid", "ask"]].drop_duplicates()
print("distinct bid/ask pairs:", precios.values.tolist())

L3 messages:   656
quotes:        145
trades:        79
L2 snapshots:  3
5-min bars:    1 (in progress)
distinct bid/ask pairs: [[53.08, 53.09]]

In half a minute there are 656 order messages, 145 changes in the best bid or ask (almost all of them in size: in those 30 seconds the price does not move from 53.08/53.09) and 79 trades; L2 takes three snapshots and the five-minute bar that contains them all has not even closed yet. The figure accumulates the events second by second. (The first 358 rows of L3, the initial snapshot of the book at exactly 14:00:00, are excluded from the count.)

The asymmetry of the ladder can be checked in the other direction: the trades rebuild the bars and the bars rebuild the daily data. They must match the aggregated files:

ohlcv = {"precio": ["first", "max", "min", "last"], "tamano": "sum"}
# the closing auction (20:00:00 UTC) goes in the last bar
cierre = op["condicion"] == "subasta_cierre"
ts_barra = op["ts"].mask(cierre, op["ts"] - pd.Timedelta("1us"))
mias = op.set_index(ts_barra).resample("5min").agg(ohlcv)
mias.columns = ["apertura", "maximo", "minimo", "cierre", "volumen"]
print(len(mias), "bars; equal:", np.allclose(mias, sesion))

dia_mio = pd.Series({"apertura": sesion["apertura"].iloc[0],
                     "maximo": sesion["maximo"].max(),
                     "minimo": sesion["minimo"].min(),
                     "cierre": sesion["cierre"].iloc[-1],
                     "volumen": sesion["volumen"].sum()})
print("Day equal:", np.allclose(dia_mio, dia.iloc[0]))

78 bars; equal: True
Day equal: True

The 78 bars come out exactly from the trades, and the daily data from the bars. There is one detail: the closing auction crosses at 16:00:00 New York time (20:00:00 UTC), right on the edge, and a plain resample would open a 79th bar at 20:00 just for it. The provider of these data counts it in the 15:55 bar, and to match it you have to do the same (here, by moving its timestamp back one microsecond when grouping). It is the kind of convention you need to know for two sources to agree.

Consolidated tape or a single venue

In the US, all exchanges and venues send their trades and quotes to a consolidator, which publishes the consolidated tape: for stocks listed on the NYSE and on other regional exchanges, the CTS (trades) and CQS (quotes) systems of the Consolidated Tape Association22Note 2 · Consolidated Tape Association, overview page (link in the resources).. A provider can give the consolidated tape or the proprietary feed of a single exchange, and the numbers change a lot, as seen in lesson 1.1.1:

def barra_dia(t):
    return pd.Series({
        "apertura": t["precio"].iloc[0], "maximo": t["precio"].max(),
        "minimo": t["precio"].min(), "cierre": t["precio"].iloc[-1],
        "volumen": t["tamano"].sum(), "operaciones": len(t),
        "vwap": (t["precio"] * t["tamano"]).sum() / t["tamano"].sum()})

pd.DataFrame({"consolidated": barra_dia(op),
              "primary only": barra_dia(
                  op[op["centro"] == "PRINCIPAL"])}).round(3)

             consolidated  primary only
apertura           53.350        53.350
maximo             53.930        53.930
minimo             52.550        52.550
cierre             53.120        53.120
volumen       8888714.000   3564476.000
operaciones     50946.000     25776.000
vwap               53.319        53.291

With the tape of a single venue, the day's volume is 40% of the real one, there are half as many trades and the VWAP drops from 53.319 to 53.291. Open and close match because they are set by the primary venue's auctions; the high and the low also match that day, but they do not have to: the session's extreme price can be crossed on another venue. Before comparing volumes across sources, or computing a VWAP to measure an execution, you need to know which tape is being used.

Timestamps and time zones

Every row of market data carries a timestamp, and every timestamp has a convention. Three questions must always be asked:

1. Which event is the timestamp from? The matching engine's (when the event happens) or the receipt one (when whoever records the data receives it). Professional providers usually give both: Databento, for example, publishes ts_event and ts_recv, and warns that the market's and the receiver's clocks are not necessarily synchronized33Note 3 · Databento, “Common fields, enums and types” (link in the resources)..
2. Start or end of the interval? In bars, a 14:00 timestamp can be the start (14:00–14:05) or the end (13:55–14:00). Confusing them is looking into the future.
3. Which time zone? UTC, the exchange's time or that of the computer of whoever downloaded the data. Dates without a time (daily data) are the session date, not an instant.

The first matters when venues are combined: each feed arrives with its own latency, and sorting by the receipt timestamp changes the order of the events. A simulation shows it with assumed latencies of a few hundred microseconds, different for each venue:

rng = np.random.default_rng(7)
media_us = {"PRINCIPAL": 150, "MTF_A": 400, "MTF_B": 600,
            "OSCURO": 900, "INTERNALIZADOR": 1_200}
lat = rng.exponential(op["centro"].map(media_us).to_numpy())
op["ts_recv"] = op["ts"] + pd.to_timedelta(lat.round(), unit="us")
orden_evento = op.sort_values("ts", kind="stable").index
orden_recv = op.sort_values("ts_recv", kind="stable").index
cambia = (orden_evento != orden_recv).mean()
print(f"Trades in a different position when sorted by receipt: "
      f"{cambia:.1%}")

Trades in a different position when sorted by receipt: 28.7%

With latencies of this size, 28.7% of the trades change position in the sequence. For a low-frequency system it is irrelevant; for aligning trades with quotes to the microsecond (lesson 1.1.1) or for measuring who gets ahead of whom, it is the difference between a correct study and a false one.

The second and third questions affect any intraday system:

b = sesion.head(2).copy()
b["fin"] = b.index + pd.Timedelta("5min")
b.index = b.index.tz_convert("America/New_York")
b[["fin", "apertura", "cierre"]].assign(
    fin=lambda x: x["fin"].dt.tz_convert("America/New_York"))

                                                fin  apertura  cierre
ts
2025-06-02 09:30:00-04:00 2025-06-02 09:35:00-04:00     53.35   53.46
2025-06-02 09:35:00-04:00 2025-06-02 09:40:00-04:00     53.46   53.24

The bar that starts at 9:30 New York time ends at 9:35, and its close is not known until then. A signal that uses that close can only be executed from 9:35 onward. If the provider labeled bars by their end, the same bar would carry the 9:35 timestamp; mixing two sources with different conventions shifts one of them by five minutes. And when converting from UTC to local time, daylight saving time moves the UTC time of the open (13:30 in summer, 14:30 in winter; lesson 0.4.4): that is why it is best to store in UTC and convert only to display or to filter by time of day.

Reference data

The last type has no price, but without it prices cannot be understood. Three examples from the level:

spec = pd.read_csv("fix_especificaciones.csv")
eventos = pd.read_csv("nova_eventos.csv", parse_dates=["fecha_ex"])
miembros = pd.read_csv("universo_miembros.csv",
                       parse_dates=["alta", "baja"])
print(spec[["multiplicador", "tick", "valor_tick", "meses"]]
      .to_string(index=False))
print(eventos[eventos["tipo"] != "dividendo"].to_string(index=False))
print(f"{len(miembros)} tickers have been in the index; "
      f"{miembros['baja'].isna().sum()} remain today")

 multiplicador  tick  valor_tick   meses
            50  0.25        12.5 H,M,U,Z
  fecha_ex                     tipo  ratio  importe fecha_pago
2021-08-23                    split    4.0      NaN 2021-08-20
2023-12-14 dividendo_extraordinario    NaN      1.5 2024-01-04
57 tickers have been in the index; 40 remain today

• Contract specifications: multiplier, tick size, expiry months and settlement rules of a future (lesson 1.2.3). Without the multiplier, a 10-point move cannot be converted into money.
• Corporate actions: splits, dividends, name changes, mergers. They are used to adjust the series (lesson 1.4.3). NOVA's 4-for-1 split in 2021 is in this table; without it, the daily price appears to fall 75% in one day.
• Universes: which stocks made up an index on each date. 57 tickers have been in an index that always has 40 members; using only today's 40 is survivorship bias (lesson 1.4.4).

Reference data takes little space, changes little and nobody looks at it until it fails. And when it fails, the error gives no warning: it produces a plausible and false backtest.

Which data for which question

The practical rule is to use the highest rung (the most aggregated) that answers the question. Each finer rung multiplies the cost, the volume of data and the ways of getting it wrong.

flowchart TB
  P{Horizon?} -- days or more --> D[Daily data]
  P -- minutes or hours --> B[Intraday bars]
  P -- seconds --> T[Trades and L1]
  T --> C{Queues, priority?}
  C -- yes --> L3[L3 messages]
  C -- no --> L2[L2 book]

A daily system like those of levels 2 and 3 lives on daily data, properly adjusted and free of survivorship bias. An intraday one needs bars and, to estimate costs, trades and quotes. Only microstructure research or high-frequency execution justify the full book.

Exercises

1. Build 1-minute bars with NOVA's trades from June 2 and count how many the session has. How many would there be with 30-minute bars? Where does the closing auction fall in each case?

for regla in ("1min", "30min"):
    b = op.set_index("ts")["precio"].resample(regla).ohlc()
    print(regla, "→", len(b), "intervals;", "last:",
          b.index[-1].strftime("%H:%M"), "UTC")

1min → 391 intervals; last: 20:00 UTC
30min → 14 intervals; last: 20:00 UTC

2. Using the trades, compute what percentage of NOVA's volume is traded in the first and in the last half hour of the session, and compare it with the share it would have if volume were uniform.

v = op.set_index("ts")["tamano"].resample("30min").sum()
cuota = 100 * v / v.sum()
print(f"First half hour: {cuota.iloc[0]:.1f} %")
print(f"Last half hour (with the auction): "
      f"{cuota.iloc[-2:].sum():.1f} %")
print(f"Uniform: {100 / 13:.1f} % per half hour")

First half hour: 13.5 %
Last half hour (with the auction): 20.1 %
Uniform: 7.7 % per half hour

3. A provider offers “tick-by-tick data” for a stock at a price. What questions would you ask before buying it?