--- title: "1.3.1 Types of market data: from EOD to L3" description: "Daily data, bars, trades, quotes, L2 and L3 order book and reference data: what each one answers, how much space it takes and how one derives from another." url: "https://pythonparatrading.com/en/python-trading-course/level-1/types-of-market-data/" lang: "en" image: "https://pythonparatrading.com/og/en/python-trading-course/level-1/types-of-market-data.png" --- # 1.3.1 · Types of market data: EOD, bars, ticks, order book and reference > The same trading day can be stored in one row or in half a million. This lesson walks the full ladder with one session of the fictitious stock NOVA, from the order-by-order message to the daily bar, measures how much space each rung takes and reviews the pitfalls of tape, timestamps and time zones that come with each one. - Course: [Level 1: Financial data, markets and analysis](https://pythonparatrading.com/en/python-trading-course/level-1/) - Module: 1.3 · Market data sources - Duration: 55 min - Prerequisites: [1.1.1 · Market participants and trading venues](https://pythonparatrading.com/en/python-trading-course/level-1/market-participants-and-trading-venues/), [1.1.3 · Order book: bid, ask, spread and depth](https://pythonparatrading.com/en/python-trading-course/level-1/order-book-bid-ask-spread-depth/) ## By the end of this lesson - Distinguish daily data, intraday bars, trades, quotes, L2 order book, L3 messages and reference data, and which question each one answers. - Measure with pandas how many rows and how much memory one session takes at each granularity. - Rebuild 5-minute bars and the daily bar from the trades, and check that they match the aggregated data. - Recognize the problems of consolidated tape versus a single venue, of timestamps (event versus receipt, bar start versus bar end) and of time zones. ## A ladder of aggregation When people talk about “market data” they mean very different things: the closing price of each day, a five-minute candle, each trade, each change in the best price or each order that enters and leaves the book. They are not alternative sources but rungs of the same ladder. At the bottom are the **messages** emitted by the matching engine; everything else is an aggregation of them. ```mermaid flowchart TB M[L3 messages] --> B[Rebuilt book] B --> L2[L2 book] L2 --> L1[L1 quotes] M --> T[Trades] T --> BA[Intraday bars] BA --> D[Daily data] R[Reference] -.-> D ``` - **L3 messages** (*market by order*): every order that arrives, is modified, is canceled or is executed, with its identifier. This is what the exchanges' order-by-order protocols publish, such as Nasdaq's ITCH[^1]. - **L2 book** (*market by price*): the book aggregated by price level, with the total size and the number of orders at each level (lesson 1.1.3). - **L1 quotes** (*top of book*): only the best bid and the best ask, with their sizes. - **Trades** (also called *ticks*): each execution, with price, size, venue and conditions. - **Intraday bars**: trades aggregated over intervals (1 minute, 5 minutes, 1 hour) as open, high, low, close and volume (OHLCV). - **Daily data** (EOD, *end of day*): one bar per session. - **Reference data**: what describes the instruments (specifications, corporate actions, symbols, universes). It is not aggregated from the messages, but it is needed to interpret and adjust them; that is why the arrow is dashed. The rule of the ladder is asymmetric: from one rung you can go down to the next (aggregate), but not up. Trades build bars; bars do not give back the trades. [^1]: Nasdaq TotalView-ITCH 5.0 (link in the resources): messages for adding, executing, canceling, deleting and replacing orders, with nanosecond timestamps since midnight. ## Six rungs of NOVA The level's data has an example of each rung for the same fictitious stock, NOVA, and for the same session, June 2, 2025, output of a simulated order-by-order book (lesson 1.1.1). Starting from the top, the daily data: ```python import numpy as np import pandas as pd diario = pd.read_csv("nova_diario_sin_ajustar.csv", parse_dates=["fecha"]).set_index("fecha") print(len(diario), "sessions;", diario.index.min().date(), "→", diario.index.max().date()) diario.loc["2025-05-30":"2025-06-03"] ``` ```output 2011 sessions; 2018-01-02 → 2025-12-31 apertura maximo minimo cierre volumen fecha 2025-05-30 52.58 53.98 52.01 53.13 6106070 2025-06-02 53.35 53.93 52.55 53.12 8888714 2025-06-03 53.05 53.99 52.74 53.00 8044956 ``` One row per session, eight years in 2,011 rows. It answers questions from days to years: trends, daily volatility, portfolios rebalanced at the close. It is the cheapest data to obtain and to store, and the one with the most adjustment pitfalls (lesson 1.4.3: this one is **unadjusted**, with the 2021 split in plain sight). ```python barras = pd.read_parquet("nova_barras_5min.parquet").set_index("ts") sesion = barras.loc["2025-06-02"] print(len(barras), "bars in", barras.index.normalize().nunique(), "sessions;", len(sesion), "on June 2") sesion.head(3) ``` ```output 38940 bars in 502 sessions; 78 on June 2 apertura maximo minimo cierre volumen ts 2025-06-02 13:30:00+00:00 53.35 53.48 53.28 53.46 285081 2025-06-02 13:35:00+00:00 53.46 53.48 53.22 53.24 220651 2025-06-02 13:40:00+00:00 53.24 53.47 53.23 53.35 187320 ``` 78 bars per normal session (six and a half hours in five-minute slices). They answer questions from minutes to days: intraday patterns, seasonality within the session, execution on a schedule. The timestamp is the **start** of the bar, in UTC: the first one, 13:30 UTC, is 9:30 in New York. ```python op = pd.read_parquet("nova_operaciones.parquet") cot = pd.read_parquet("nova_cotizaciones.parquet") print(len(op), "trades;", len(cot), "quote changes") print(op.iloc[1:3].to_string()) print(cot.iloc[1:3].to_string()) ``` ```output 50946 trades; 129247 quote changes ts precio tamano agresor centro condicion 1 2025-06-02 13:30:00.301051+00:00 53.34 100 venta PRINCIPAL regular 2 2025-06-02 13:30:00.301052+00:00 53.34 100 venta PRINCIPAL regular ts bid ask tam_bid tam_ask 1 2025-06-02 13:30:00.006643+00:00 53.34 53.36 1600 500 2 2025-06-02 13:30:00.075521+00:00 53.34 53.36 1600 800 ``` Trades and quotes are the “tick-by-tick” data. Trades say what was traded, where and at what price; quotes, at what price one could trade at each instant. Together they make it possible to measure the effective bid-ask spread and slippage (lesson 1.1.5), classify buys and sells or build alternative bars (lesson 1.4.5). ```python l2 = pd.read_parquet("nova_libro_l2.parquet") l3 = pd.read_parquet("nova_libro_l3.parquet") print(len(l2), "L2 rows (snapshots every 10 s, 10 levels per side)") print(l2[l2["ts"] == l2["ts"].iloc[0]].head(4).to_string()) print(len(l3), "L3 messages in 5 minutes;", l3["evento"].value_counts().to_dict()) ``` ```output 44126 L2 rows (snapshots every 10 s, 10 levels per side) ts lado nivel precio tamano ordenes 0 2025-06-02 13:30:00+00:00 bid 1 53.34 1200 3 1 2025-06-02 13:30:00+00:00 bid 2 53.33 400 2 2 2025-06-02 13:30:00+00:00 bid 3 53.32 1100 3 3 2025-06-02 13:30:00+00:00 bid 4 53.31 1100 2 7241 L3 messages in 5 minutes; {'alta': 3443, 'cancelacion': 2842, 'ejecucion': 746, 'modificacion': 210} ``` L2 says how much liquidity there was at each level; L3, who (which order) was providing it and at what position in the queue it stood. In five minutes of NOVA there are 7,241 messages, and most are not trades: they are adds and cancellations of orders that never execute. That noise is the raw material of microstructure studies (queues, priority, iceberg orders) and is almost never needed for a low-frequency system. | Type | One row is | Answers | Typical use | |---|---|---|---| | Daily (EOD) | One session | Trend, volatility, portfolios | Daily backtests, factor research | | Intraday bars | One interval | Intraday patterns, schedules | Intraday systems, session seasonality | | Trades | One execution | What was traded, where, how much | VWAP, volume bars, costs | | Quotes (L1) | A change in the best price | At what price one could trade | Bid-ask spread, slippage, alignment | | L2 book | One level at one instant | How much liquidity there was | Market impact, depth | | L3 messages | One event of one order | Queues and priority | Microstructure, execution simulation | | Reference | One instrument or event | What it is and what happened to it | Adjustments, universes, specifications | ## How much space each rung takes The difference between rungs is orders of magnitude. To measure it for one session, we count the rows and the memory they take in pandas. The L3 messages are only available for five minutes; multiplied by 78 they give an estimate for the full session: ```python def mb(df): return df.memory_usage(deep=True).sum() / 1e6 dia = diario.loc[["2025-06-02"]] l3_bytes_fila = l3.memory_usage(deep=True).sum() / len(l3) tamanos = pd.DataFrame({ "filas": [len(dia), len(sesion), len(op), len(cot), len(l2), len(l3) * 78], "MB": [mb(dia), mb(sesion), mb(op), mb(cot), mb(l2), l3_bytes_fila * len(l3) * 78 / 1e6], }, index=["daily", "5-min bars", "trades", "quotes", "L2 book (10 s)", "L3 messages (estimated)"]) tamanos["filas"] = tamanos["filas"].astype(int) tamanos.round(4) ``` ```output filas MB daily 1 0.0000 5-min bars 78 0.0037 trades 50946 3.4860 quotes 129247 5.1700 L2 book (10 s) 44126 2.2506 L3 messages (estimated) 564798 33.0479 ``` The figure in the header plots the table on a logarithmic scale. From the daily bar to L3 there are almost six orders of magnitude: one row versus more than half a million. And that is for a fictitious stock of average liquidity in a single book. Multiplied by thousands of stocks and years of history, the difference decides the infrastructure: ten years of daily data for a whole market fit on a laptop; the L3 messages of a single exchange for one year do not (lesson 1.4.7). It also decides the price: providers charge by granularity, historical depth and number of exchanges (lessons 1.3.3 and 1.3.4). ```python sesiones, valores = 252, 3_000 anual_gb = tamanos["MB"] * sesiones * valores / 1e3 anual_gb.round(3).rename("GB per year (3,000 stocks)") ``` ```output daily 0.036 5-min bars 2.830 trades 2635.384 quotes 3908.529 L2 book (10 s) 1701.422 L3 messages (estimated) 24984.211 Name: GB per year (3,000 stocks), dtype: float64 ``` The calculation is crude, because NOVA is not representative of a whole market and on-disk formats compress much more than pandas in memory (Parquet, lesson 1.4.7). But the order of magnitude is what matters: with daily data, one year of 3,000 stocks is a few tens of megabytes; with trades and quotes, several terabytes; with L3, tens of terabytes. ## Thirty seconds of NOVA The ladder shows clearly over a short interval: from 14:00:00 to 14:00:30 UTC (10:00 in New York), how many events each granularity records: ```python ini = pd.Timestamp("2025-06-02 14:00:00", tz="UTC") fin = ini + pd.Timedelta("30s") en = lambda df: df[(df["ts"] >= ini) & (df["ts"] < fin)] print("L3 messages: ", len(en(l3[l3["ts"] > ini]))) print("quotes: ", len(en(cot))) print("trades: ", len(en(op))) print("L2 snapshots: ", en(l2)["ts"].nunique()) print("5-min bars: 1 (in progress)") precios = en(cot)[["bid", "ask"]].drop_duplicates() print("distinct bid/ask pairs:", precios.values.tolist()) ``` ```output L3 messages: 656 quotes: 145 trades: 79 L2 snapshots: 3 5-min bars: 1 (in progress) distinct bid/ask pairs: [[53.08, 53.09]] ``` *Fig. — Thirty seconds of NOVA, 14:00:00 to 14:00:30 UTC on 2 June 2025: cumulative events at each granularity, from order-by-order messages to the 5-minute bar.* Three step lines accumulating events second by second on a square-root scale: L3 messages, highlighted, reach 656 in 30 seconds; L1 quote changes, 145; trades, 79. Labels on the right add 3 L2 snapshots and no closed 5-minute bar. In half a minute there are 656 order messages, 145 changes in the best bid or ask (almost all of them in size: in those 30 seconds the price does not move from 53.08/53.09) and 79 trades; L2 takes three snapshots and the five-minute bar that contains them all has not even closed yet. The figure accumulates the events second by second. (The first 358 rows of L3, the initial snapshot of the book at exactly 14:00:00, are excluded from the count.) The asymmetry of the ladder can be checked in the other direction: the trades rebuild the bars and the bars rebuild the daily data. They must match the aggregated files: ```python ohlcv = {"precio": ["first", "max", "min", "last"], "tamano": "sum"} # the closing auction (20:00:00 UTC) goes in the last bar cierre = op["condicion"] == "subasta_cierre" ts_barra = op["ts"].mask(cierre, op["ts"] - pd.Timedelta("1us")) mias = op.set_index(ts_barra).resample("5min").agg(ohlcv) mias.columns = ["apertura", "maximo", "minimo", "cierre", "volumen"] print(len(mias), "bars; equal:", np.allclose(mias, sesion)) dia_mio = pd.Series({"apertura": sesion["apertura"].iloc[0], "maximo": sesion["maximo"].max(), "minimo": sesion["minimo"].min(), "cierre": sesion["cierre"].iloc[-1], "volumen": sesion["volumen"].sum()}) print("Day equal:", np.allclose(dia_mio, dia.iloc[0])) ``` ```output 78 bars; equal: True Day equal: True ``` The 78 bars come out exactly from the trades, and the daily data from the bars. There is one detail: the closing auction crosses at 16:00:00 New York time (20:00:00 UTC), right on the edge, and a plain `resample` would open a 79th bar at 20:00 just for it. The provider of these data counts it in the 15:55 bar, and to match it you have to do the same (here, by moving its timestamp back one microsecond when grouping). It is the kind of convention you need to know for two sources to agree. ## Consolidated tape or a single venue In the US, all exchanges and venues send their trades and quotes to a consolidator, which publishes the **consolidated tape**: for stocks listed on the NYSE and on other regional exchanges, the CTS (trades) and CQS (quotes) systems of the Consolidated Tape Association[^2]. A provider can give the consolidated tape or the proprietary feed of a single exchange, and the numbers change a lot, as seen in lesson 1.1.1: [^2]: Consolidated Tape Association, overview page (link in the resources). ```python def barra_dia(t): return pd.Series({ "apertura": t["precio"].iloc[0], "maximo": t["precio"].max(), "minimo": t["precio"].min(), "cierre": t["precio"].iloc[-1], "volumen": t["tamano"].sum(), "operaciones": len(t), "vwap": (t["precio"] * t["tamano"]).sum() / t["tamano"].sum()}) pd.DataFrame({"consolidated": barra_dia(op), "primary only": barra_dia( op[op["centro"] == "PRINCIPAL"])}).round(3) ``` ```output consolidated primary only apertura 53.350 53.350 maximo 53.930 53.930 minimo 52.550 52.550 cierre 53.120 53.120 volumen 8888714.000 3564476.000 operaciones 50946.000 25776.000 vwap 53.319 53.291 ``` With the tape of a single venue, the day's volume is 40% of the real one, there are half as many trades and the VWAP drops from 53.319 to 53.291. Open and close match because they are set by the primary venue's auctions; the high and the low also match that day, but they do not have to: the session's extreme price can be crossed on another venue. Before comparing volumes across sources, or computing a VWAP to measure an execution, you need to know which tape is being used. ## Timestamps and time zones Every row of market data carries a timestamp, and every timestamp has a convention. Three questions must always be asked: 1. **Which event is the timestamp from?** The matching engine's (when the event happens) or the receipt one (when whoever records the data receives it). Professional providers usually give both: Databento, for example, publishes `ts_event` and `ts_recv`, and warns that the market's and the receiver's clocks are not necessarily synchronized[^3]. 2. **Start or end of the interval?** In bars, a 14:00 timestamp can be the start (14:00–14:05) or the end (13:55–14:00). Confusing them is looking into the future. 3. **Which time zone?** UTC, the exchange's time or that of the computer of whoever downloaded the data. Dates without a time (daily data) are the session date, not an instant. [^3]: Databento, “Common fields, enums and types” (link in the resources). The first matters when venues are combined: each feed arrives with its own latency, and sorting by the receipt timestamp changes the order of the events. A simulation shows it with assumed latencies of a few hundred microseconds, different for each venue: ```python rng = np.random.default_rng(7) media_us = {"PRINCIPAL": 150, "MTF_A": 400, "MTF_B": 600, "OSCURO": 900, "INTERNALIZADOR": 1_200} lat = rng.exponential(op["centro"].map(media_us).to_numpy()) op["ts_recv"] = op["ts"] + pd.to_timedelta(lat.round(), unit="us") orden_evento = op.sort_values("ts", kind="stable").index orden_recv = op.sort_values("ts_recv", kind="stable").index cambia = (orden_evento != orden_recv).mean() print(f"Trades in a different position when sorted by receipt: " f"{cambia:.1%}") ``` ```output Trades in a different position when sorted by receipt: 28.7% ``` With latencies of this size, 28.7% of the trades change position in the sequence. For a low-frequency system it is irrelevant; for aligning trades with quotes to the microsecond (lesson 1.1.1) or for measuring who gets ahead of whom, it is the difference between a correct study and a false one. The second and third questions affect any intraday system: ```python b = sesion.head(2).copy() b["fin"] = b.index + pd.Timedelta("5min") b.index = b.index.tz_convert("America/New_York") b[["fin", "apertura", "cierre"]].assign( fin=lambda x: x["fin"].dt.tz_convert("America/New_York")) ``` ```output fin apertura cierre ts 2025-06-02 09:30:00-04:00 2025-06-02 09:35:00-04:00 53.35 53.46 2025-06-02 09:35:00-04:00 2025-06-02 09:40:00-04:00 53.46 53.24 ``` *Fig. — The same 5-minute bar under two timestamp conventions: NOVA's first bar on 2 June 2025 runs from 9:30 to 9:35 New York time; its close is only known at 9:35, even though the file labels it 13:30 UTC (schematic).* Timeline from 9:25 to 9:50 New York time with three 5-minute bars (9:30, 9:35 and 9:40). Two vertical lines mark the start of the first bar, 13:30 UTC, its timestamp if labelled by start, and its end, 13:35 UTC, its timestamp if labelled by end. A highlighted triangle at 9:35 marks when the close of the 9:30 bar is known. The bar that starts at 9:30 New York time ends at 9:35, and its close is not known until then. A signal that uses that close can only be executed from 9:35 onward. If the provider labeled bars by their end, the same bar would carry the 9:35 timestamp; mixing two sources with different conventions shifts one of them by five minutes. And when converting from UTC to local time, daylight saving time moves the UTC time of the open (13:30 in summer, 14:30 in winter; lesson 0.4.4): that is why it is best to store in UTC and convert only to display or to filter by time of day. ## Reference data The last type has no price, but without it prices cannot be understood. Three examples from the level: ```python spec = pd.read_csv("fix_especificaciones.csv") eventos = pd.read_csv("nova_eventos.csv", parse_dates=["fecha_ex"]) miembros = pd.read_csv("universo_miembros.csv", parse_dates=["alta", "baja"]) print(spec[["multiplicador", "tick", "valor_tick", "meses"]] .to_string(index=False)) print(eventos[eventos["tipo"] != "dividendo"].to_string(index=False)) print(f"{len(miembros)} tickers have been in the index; " f"{miembros['baja'].isna().sum()} remain today") ``` ```output multiplicador tick valor_tick meses 50 0.25 12.5 H,M,U,Z fecha_ex tipo ratio importe fecha_pago 2021-08-23 split 4.0 NaN 2021-08-20 2023-12-14 dividendo_extraordinario NaN 1.5 2024-01-04 57 tickers have been in the index; 40 remain today ``` - **Contract specifications**: multiplier, tick size, expiry months and settlement rules of a future (lesson 1.2.3). Without the multiplier, a 10-point move cannot be converted into money. - **Corporate actions**: splits, dividends, name changes, mergers. They are used to adjust the series (lesson 1.4.3). NOVA's 4-for-1 split in 2021 is in this table; without it, the daily price appears to fall 75% in one day. - **Universes**: which stocks made up an index on each date. 57 tickers have been in an index that always has 40 members; using only today's 40 is survivorship bias (lesson 1.4.4). Reference data takes little space, changes little and nobody looks at it until it fails. And when it fails, the error gives no warning: it produces a plausible and false backtest. ## Which data for which question The practical rule is to use the highest rung (the most aggregated) that answers the question. Each finer rung multiplies the cost, the volume of data and the ways of getting it wrong. ```mermaid flowchart TB P{Horizon?} -- days or more --> D[Daily data] P -- minutes or hours --> B[Intraday bars] P -- seconds --> T[Trades and L1] T --> C{Queues, priority?} C -- yes --> L3[L3 messages] C -- no --> L2[L2 book] ``` A daily system like those of levels 2 and 3 lives on daily data, properly adjusted and free of survivorship bias. An intraday one needs bars and, to estimate costs, trades and quotes. Only microstructure research or high-frequency execution justify the full book. ## Exercises 1. Build **1-minute** bars with NOVA's trades from June 2 and count how many the session has. How many would there be with 30-minute bars? Where does the closing auction fall in each case? **Solution.** ```python for regla in ("1min", "30min"): b = op.set_index("ts")["precio"].resample(regla).ohlc() print(regla, "→", len(b), "intervals;", "last:", b.index[-1].strftime("%H:%M"), "UTC") ``` ```output 1min → 391 intervals; last: 20:00 UTC 30min → 14 intervals; last: 20:00 UTC ``` With `resample` on the trades, the last interval is the one that contains the 20:00:00 UTC closing auction: 391 one-minute intervals (390 of the continuous session plus one at 20:00 with only the auction) and 14 of 30 minutes (13 plus the auction one). To get the 390 bars of the session you have to decide where the auction goes, as in the reconstruction of the 5-minute bars. 2. Using the trades, compute what percentage of NOVA's volume is traded in the first and in the last half hour of the session, and compare it with the share it would have if volume were uniform. **Solution.** ```python v = op.set_index("ts")["tamano"].resample("30min").sum() cuota = 100 * v / v.sum() print(f"First half hour: {cuota.iloc[0]:.1f} %") print(f"Last half hour (with the auction): " f"{cuota.iloc[-2:].sum():.1f} %") print(f"Uniform: {100 / 13:.1f} % per half hour") ``` ```output First half hour: 13.5 % Last half hour (with the auction): 20.1 % Uniform: 7.7 % per half hour ``` The first half hour concentrates 13.5% of the volume and the last one, with the closing auction, 20.1%, versus the 7.7% they would get with uniform volume. It is the U-shaped profile seen in almost every stock (lessons 1.1.2 and 1.6.3): that is why execution algorithms spread orders following a volume profile and not in equal parts. 3. A provider offers “tick-by-tick data” for a stock at a price. What questions would you ask before buying it? **Solution.** What exactly it contains: only trades, or quotes as well; whether it is L1, L2 or L3. Which tape: consolidated or from a single exchange, and which venues it includes (dark pools, internalizers, off-hours trades). Which timestamps: event, receipt or both, with what precision and in which time zone. Which trade conditions it includes (auctions, blocks, corrected or canceled trades). Whether the data is adjusted for corporate actions and how the corporate actions are delivered. How far back the history goes and whether it includes delisted stocks. And the license: what you are allowed to do with it (lesson 1.3.7). ## Self-assessment 1. You want to study whether a daily moving average crossover strategy works on US stocks since 2005. What type of data do you need? - A. L3 messages from every exchange - B. Daily (EOD) data with corporate action adjustments and a universe free of survivorship bias - C. Quotes with the best bid and ask - D. L2 snapshots every second *Answer B: Daily (EOD) data with corporate action adjustments and a universe free of survivorship bias.* A rule that decides once a day on the close only needs daily OHLCV. The critical part is not the granularity but the adjustments for splits and dividends and a universe that includes the stocks that stopped trading (lessons 1.4.3 and 1.4.4). Finer data would only add cost. 2. What is the difference between L2 and L3 data? - A. None - B. L2 is trades and L3 is quotes - C. L3 is delayed L2 - D. L2 aggregates the book by price level (total size and number of orders); L3 gives each individual order with its identifier and its adds, cancellations and executions *Answer D: L2 aggregates the book by price level (total size and number of orders); L3 gives each individual order with its identifier and its adds, cancellations and executions.* With L3 (market by order) you can rebuild the book and see the queue at each price; with L2 (market by price), only the total of each level. L2 can be derived from L3, but not the other way around. 3. A 5-minute bar is stamped 14:00. Your provider labels bars by their start. When is its close known? - A. At 14:05, when the bar ends - B. At 14:00 - C. At 13:55 - D. At the close of the session *Answer A: At 14:05, when the bar ends.* A bar labeled by its start contains what happens from 14:00 to 14:05; its close only exists at 14:05. Using it as if it were available at 14:00 is looking five minutes into the future. Each provider chooses its convention, and you have to check it. 4. You compare NOVA's daily volume from two providers: one gives 3.56 million shares and the other 8.89 million. What is the most likely explanation? - A. One of the two is wrong - B. One includes dividends - C. One uses only the trades of the primary venue and the other the consolidated tape of all venues - D. One uses daylight saving time *Answer C: One uses only the trades of the primary venue and the other the consolidated tape of all venues.* In the NOVA session, the primary venue accounts for 3,564,476 shares and the consolidated tape for 8,888,714. Before comparing volumes, VWAP or highs and lows, you need to know which tape each source uses. 5. What is reference data? - A. The official closing prices - B. The information that describes instruments and their life: contract specifications, corporate actions, calendars, symbols and universe membership - C. The data of a benchmark index - D. Data delayed by 15 minutes *Answer B: The information that describes instruments and their life: contract specifications, corporate actions, calendars, symbols and universe membership.* Without multiplier, tick size, expiries, splits, symbol changes or index entry and exit dates, prices cannot be interpreted or adjusted. Reference data takes little space, but an error in it spoils everything else. ## Key points - Market data forms a ladder: order-by-order messages (L3) → book by levels (L2) → best price (L1, quotes) and trades → bars → daily data. Each rung is derived from the previous one, never the other way around. - For NOVA, one session is 1 row of daily data, 78 rows of 5-minute bars, 50,946 trades, 129,247 quotes and on the order of half a million L3 messages. - Granularity is chosen by the question: a daily rule does not need ticks; a study of execution costs needs trades and quotes; one of queues and priority, L3. - Consolidated tape and a single venue give different volumes, highs, lows and VWAP. For NOVA, the primary venue accounts for only 40% of the volume. - Every timestamp has its convention: event or receipt, bar start or bar end, UTC or local time. Getting it wrong is a source of look-ahead bias. - Reference data (specifications, corporate actions, universes) takes little space and conditions everything. ## Resources ### Research - [Limit Order Book - Aggregated Market Liquidity](https://pythonparatrading.com/en/limit-order-book-and-market-liquidity/) · Research-0008 · 11 min - [Big Data Management. 1 - Introduction to Big Data for Trading with ArcticDB](https://pythonparatrading.com/en/big-data-for-trading/) · Research-0019 · 15 min ### Documentation - [Databento: data schemas (MBO, MBP-10, MBP-1, TBBO, trades, OHLCV, definitions)](https://databento.com/docs/schemas-and-data-formats) · databento.com - [Databento: common fields and timestamps (ts_event, ts_recv)](https://databento.com/docs/standards-and-conventions/common-fields-enums-types) · databento.com - [Nasdaq TotalView-ITCH 5.0: order-by-order protocol specification](https://www.nasdaqtrader.com/content/technicalsupport/specifications/dataproducts/NQTVITCHspecification.pdf) · nasdaqtrader.com - [Consolidated Tape Association (CTA): the US consolidated tape](https://www.ctaplan.com/index) · ctaplan.com ### Data - [Level 1 data: NOVA 5-minute bars, 2024-2025 (Parquet)](https://pythonparatrading.com/curso/datos/nivel-1/nova_barras_5min.parquet) · Download · nova_barras_5min.parquet Previous lesson: [1.2.7 · Fixed income and interest rates; CFDs and leveraged products](https://pythonparatrading.com/en/python-trading-course/level-1/fixed-income-rates-and-leveraged-products/) Next lesson: [1.3.2 · Free market data sources](https://pythonparatrading.com/en/python-trading-course/level-1/free-market-data-sources/)