When data becomes “big”: ticks, order books, options and universes.

“Big” is a relationship, not a size

A 10 GB file is small for a server with 1 TB of memory and huge for a 16 GB laptop. And a 2 GB table may not fit in that laptop if it has to be merged with another, sorted and grouped. The useful definition for a quant is operational: data are big when the computation you need does not fit comfortably in the memory of the machine you have. From there come the three questions of this lesson: how many records there are, how many bytes each one takes in the representation you use and how much extra memory each operation asks for.

Wes McKinney, the creator of pandas, summed it up in 2017 in a rule of thumb: with pandas you should have 5 to 10 times the size of the data in RAM; for a 10 GB dataset, 64 to 128 GB. The rule predates pandas 3 and Arrow as the default format, but the mechanism behind it is still there and is measured below: data take more space in memory than in the file, and each operation creates intermediate copies.

Lesson 1.3.1 measured one session of the fictitious stock NOVA at each granularity, from the daily bar to the L3 book messages: from 1 row to more than half a million. Here we go from one stock to a market.

flowchart TB
  B[Bars] --> T[Trades]
  T --> Q[L1 quotes]
  Q --> L2[L2 book]
  L2 --> L3[L3 book]

• Bars (daily or minute): one record per instrument and period, with open, high, low, close and volume.
• Trades: one record per execution.
• L1 quotes: the best bid and the best ask, one record per change.
• L2 book: depth aggregated by price level, as snapshots or as per-level updates.
• L3 book (market by order): every order and every change to every order; with it, the full book is reconstructed.

Each step to the right multiplies the number of records. Options multiply it again: each underlying has hundreds or thousands of series (expirations by strikes by type) and each one is quoted on several exchanges.

A 36-byte message

Nasdaq's TotalView-ITCH 5.0 protocol disseminates the L3 book of its market as a sequence of binary messages. The specification fixes every field: unsigned integers in network byte order (big endian), ASCII text padded with spaces, the time in nanoseconds since midnight in 6 bytes and prices as integers with 4 implied decimals. The message for a new order (Add Order, type A) takes 36 bytes:

The first cell generates a synthetic session of 2 million trades in 500 fictitious securities (T000 to T499, with the activity concentrated in the first ones, as in a real market) and writes it in that binary format. NumPy reads the binary without copying it with a structured type that reproduces the table; since NumPy has no 6-byte integers, the time is read as a 2-byte integer and a 4-byte one. The data are synthetic: they are not market data.

import gc
import os
import tracemalloc

import numpy as np
import pandas as pd
import pyarrow as pa
import pyarrow.parquet as pq

rng = np.random.default_rng(661)
N = 2_000_000
VALORES = np.array([f"T{i:03d}" for i in range(500)])
peso = 1 / np.arange(1, 501) ** 1.1
i_valor = rng.choice(500, N, p=peso / peso.sum())
ns = np.sort(rng.integers(34_200, 57_600, N) * 10**9      # 9:30-16:00
             + rng.integers(0, 10**9, N))
precio = np.round(rng.gamma(2.0, 40.0, 500)[i_valor]
                  * np.exp(rng.normal(0, 0.002, N)), 2)
acciones = rng.geometric(0.02, N).astype(np.int32)
lado = rng.choice(np.array(["B", "S"]), N)

ADD_ORDER = np.dtype([
    ("tipo", "S1"), ("locate", ">u2"), ("tracking", ">u2"),
    ("ts_alto", ">u2"), ("ts_bajo", ">u4"), ("referencia", ">u8"),
    ("lado", "S1"), ("acciones", ">u4"), ("valor", "S8"),
    ("precio", ">u4")])
msg = np.zeros(N, ADD_ORDER)
msg["tipo"], msg["locate"] = b"A", i_valor + 1
msg["ts_alto"], msg["ts_bajo"] = ns >> 32, ns & 0xFFFFFFFF
msg["referencia"] = np.arange(1, N + 1)
msg["lado"] = lado.astype("S1")
msg["acciones"] = acciones
msg["valor"] = np.char.ljust(VALORES[i_valor], 8).astype("S8")
msg["precio"] = np.round(precio * 10_000).astype(np.uint32)
binario = msg.tobytes()

leido = np.frombuffer(binario, ADD_ORDER)      # no copy
m = leido[0]
hora = (int(m["ts_alto"]) << 32) | int(m["ts_bajo"])
print(f"{ADD_ORDER.itemsize} bytes per message; "
      f"{len(binario) / 2**20:.1f} MiB in total")
print(m["valor"].decode().strip(), m["lado"].decode(),
      int(m["acciones"]), int(m["precio"]) / 10_000,
      pd.Timedelta(hora, "ns"))

36 bytes per message; 68.7 MiB in total
T001 S 55 36.52 0 days 09:30:00.018014726

Two million messages take 68.7 MiB, exactly 36 bytes each, and reading them costs nothing: np.frombuffer interprets the bytes in place, without copying or converting them. The first one is a sale of 55 shares of T001 at 36.52 at 9:30:00.018. It is the most compact format in this lesson and the least convenient: the integers are in network order, the symbol carries spaces, the time is split and the price is an integer. To analyze it, it has to be converted, and the conversion has a price in memory.

The same trade in pandas, Arrow and Parquet

From the binary to an analysis table: time as a timestamp, security, price, shares, execution venue and side. Five representations of the same data are measured. In pandas, memory_usage(deep=True), which with Python objects counts the size of each object as if it were not shared with any other cell: it is an upper bound. In Arrow, nbytes. In Parquet, the size of the file compressed with zstd, which is deleted afterwards.

mercado = rng.choice(np.array(list("NQPZKVJX")), N)
base = pd.Timestamp("2026-10-22")
columnas = {"ts": base + pd.to_timedelta(ns, "ns"),
            "valor": VALORES[i_valor], "precio": precio,
            "acciones": acciones, "mercado": mercado,
            "lado": lado}
TEXTOS = ["valor", "mercado", "lado"]

con_str = pd.DataFrame(columnas)            # pandas 3: str type
con_objetos = con_str.astype({c: object for c in TEXTOS})
compacto = con_str.astype({c: "category" for c in TEXTOS})
tabla = pa.Table.from_pandas(compacto, preserve_index=False)
pq.write_table(tabla, "operaciones.parquet", compression="zstd")

bytes_fila = {
    "ITCH binary (Add Order)": ADD_ORDER.itemsize,
    "pandas, Python objects":
        con_objetos.memory_usage(deep=True).sum() / N,
    "pandas, str type (Arrow)":
        con_str.memory_usage(deep=True).sum() / N,
    "pandas, categories": compacto.memory_usage(deep=True).sum() / N,
    "Arrow, dictionaries": tabla.nbytes / N,
    "Parquet zstd (disk)": os.path.getsize("operaciones.parquet") / N,
}
os.remove("operaciones.parquet")
for nombre, b in bytes_fila.items():
    print(f"{nombre:<27} {b:6.1f} B/row   "
          f"{b * N / 2**20:6.1f} MiB")
print(compacto.dtypes.astype(str).to_dict())

ITCH binary (Add Order)       36.0 B/row     68.7 MiB
pandas, Python objects       173.0 B/row    330.0 MiB
pandas, str type (Arrow)      50.0 B/row     95.4 MiB
pandas, categories            24.0 B/row     45.8 MiB
Arrow, dictionaries           24.0 B/row     45.8 MiB
Parquet zstd (disk)           10.0 B/row     19.1 MiB
{'ts': 'datetime64[ns]', 'valor': 'category', 'precio': 'float64', 'acciones': 'int32', 'mercado': 'category', 'lado': 'category'}

The same session takes from 10 to 173 bytes per trade depending on how it is stored:

• With Python objects (the object of pandas 2), each string is an object with its header (more than 40 bytes for a four-letter string) and the column stores an 8-byte pointer to each one: 173 bytes per trade, almost five times the binary message. The figure is an upper bound, because memory_usage(deep=True) counts each object even if several cells share the same one, but it gives the order of magnitude.
• With pandas 3's str type, which stores the strings in an Arrow array (the characters one after another plus an 8-byte offset per cell), it drops to 50.
• With categories, each repeated string is stored once and each row carries a small code: 2 bytes for 500 securities, 1 for 8 venues or 2 sides. That leaves 24 bytes per row, less than the binary, because only the fields that are analyzed are stored: 8 for the time, 8 for the price, 4 for the shares and 4 for the three codes.
• Arrow with dictionaries (the translation of categories) takes the same, 24 bytes.
• Parquet with zstd drops to 10 bytes on disk: it compresses well because the times are sorted and the codes repeat. But it is a disk format, not a memory format: to work with the data they have to be decompressed.

Going from objects to categories divides memory by seven without losing information. It is the first lever, and almost always the cheapest.

What each operation asks for

A table fitting in memory does not mean you can work with it. Each pandas operation creates intermediate results: a sort index, the group keys, the merged table before returning it. The standard library's tracemalloc module records the memory allocations of Python and NumPy (not those of Arrow, which has its own allocator; that is why it is measured with the categories table, which is all in NumPy) and gives the memory peak since it was started. The cell measures four common operations on the first million trades, with the peak expressed as a multiple of the size of the table:

muestra = compacto.iloc[:1_000_000].copy()
tam = muestra.memory_usage(deep=True).sum()
sectores = pd.DataFrame({"valor": VALORES,
                         "sector": np.arange(500) % 11})
OPERACIONES = {
    "sort by security and time":
        lambda d: d.sort_values(["valor", "ts"]),
    "VWAP per security":
        lambda d: ((d["precio"] * d["acciones"])
                   .groupby(d["valor"], observed=True).sum()
                   / d.groupby("valor", observed=True)
                   ["acciones"].sum()),
    "1-minute bars per security":
        lambda d: d.groupby(["valor", d["ts"].dt.floor("min")],
                            observed=True)["precio"]
                   .agg(["first", "max", "min", "last"]),
    "merge with the sector":
        lambda d: d.merge(sectores.astype({"valor": "category"}),
                          on="valor"),
}

def pico(funcion, datos):
    gc.collect()
    tracemalloc.start()
    resultado = funcion(datos)
    maximo = tracemalloc.get_traced_memory()[1]
    tracemalloc.stop()
    del resultado
    return maximo

picos = {k: pico(f, muestra) / tam for k, f in OPERACIONES.items()}
print(f"table: {tam / 2**20:.1f} MiB")
for nombre, veces in picos.items():
    print(f"{nombre:<30} peak {veces:4.1f} times the table")

table: 22.9 MiB
sort by security and time      peak  2.8 times the table
VWAP per security              peak  1.3 times the table
1-minute bars per security     peak  3.2 times the table
merge with the sector          peak  5.9 times the table

The table of one million trades takes 22.9 MiB, and no operation settles for that. Sorting by security and time asks for 2.8 times the size of the table, on top of the table: the sort index and the sorted copy. The VWAP per security, which reduces a million rows to 500, asks for 1.3. The one-minute bars, with a new group key (the truncated time), 3.2. And the merge with a table of 500 sectors, the most innocent in appearance, 5.9 times: pandas builds the indexes of the two tables and the complete result table before returning it.

These are the figures for this table and this version of pandas, not universal constants, but they convey the idea behind McKinney's rule: an analysis chains several of these operations and keeps several intermediate results, so the peak of a typical working notebook is several times the size of the data.

How much each market takes up

With the bytes per row and the measured peaks, the number of records is what is missing. The figures in this section come from publications of the systems themselves (consulted on October 23, 2026) or are declared assumptions:

Two caveats. The CTA figures are those of the peak day of each quarter, not the average, and they only cover tapes A and B (stocks listed on NYSE and on exchanges other than Nasdaq; Nasdaq stocks go through another SIP, UTP's). And the OPRA figure is the capacity the system is sized for, not the traffic of a normal day. Both serve to size an infrastructure, which is what they are published for.

The bytes per row of quotes and bars are measured the same way as those of trades, with one-row tables with their columns in compact types:

def bytes_por_fila(**tipos):
    vacia = pd.DataFrame({c: pd.Series([0], dtype=t)
                          for c, t in tipos.items()})
    return vacia.memory_usage(index=False).sum()

B_OPERACION = bytes_fila["pandas, categories"]
B_COTIZACION = bytes_por_fila(
    ts="datetime64[ns]", valor="int16", compra="float64",
    venta="float64", tam_compra="int32", tam_venta="int32",
    mercado="int8")
B_BARRA = bytes_por_fila(
    ts="datetime64[ns]", valor="int32", apertura="float64",
    maximo="float64", minimo="float64", cierre="float64",
    volumen="int64")
print(f"bytes per row: trade {B_OPERACION:.0f}, quote "
      f"{B_COTIZACION}, bar {B_BARRA}")

CONJUNTOS = {                      # (records, bytes per record)
    "CTA trades, peak day": (113.9e6, B_OPERACION),
    "CTA trades, 252 peak days": (113.9e6 * 252, B_OPERACION),
    "CTA quotes, peak day": (2_761e6, B_COTIZACION),
    "OPRA, one day at capacity": (311e9, B_COTIZACION),
    "1-min bars, 5,000 stocks, 1 year":
        (5_000 * 390 * 252, B_BARRA),
    "daily bars, 50,000 securities, 20 years":
        (50_000 * 252 * 20, B_BARRA),
}
pico_union = 1 + round(picos["merge with the sector"], 1)   # 1 + 5.9
tamanos = pd.DataFrame(
    [(k, n / 1e6, n * b / 1e9, n * b * pico_union / 1e9)
     for k, (n, b) in CONJUNTOS.items()],
    columns=["dataset", "millions", "GB", "GB when merging"]
).set_index("dataset")
tamanos.round(1)

bytes per row: trade 24, quote 35, bar 52
                                         millions       GB  GB when merging
dataset
CTA trades, peak day                        113.9      2.7             18.9
CTA trades, 252 peak days                 28702.8    689.0           4753.8
CTA quotes, peak day                       2761.0     96.6            666.8
OPRA, one day at capacity                311000.0  10885.0          75106.5
1-min bars, 5,000 stocks, 1 year            491.4     25.6            176.3
daily bars, 50,000 securities, 20 years     252.0     13.1             90.4

With 24 bytes per trade, 35 per quote and 52 per bar:

• The peak trade day of tapes A and B (113.9 million) takes 2.7 GB in memory: it fits in a laptop, but merging it with another table already asks for 19 GB, more than a 16 GB one has. A year of days like that, 689 GB, fits in none.
• The peak quote day (2,761 million) takes 96.6 GB: it no longer fits in a laptop, but it does in a workstation or a server with 128 or 256 GB, and with the merge it asks for 670 GB.
• The L3 book of a single exchange, Nasdaq, takes 7.9 to 12.6 GB per session in its sample files, already compressed with gzip; decompressed and turned into a table, several times more. At 36 bytes per message, 12.6 GB compressed is hundreds of millions of messages in one day.
• OPRA at its projected capacity, 311 billion messages in one day, would be almost 11 TB in memory. There is no machine with that much memory; not even a laptop's disk can hold it.
• The bars of a large universe are moderate: 25.6 GB for the one-minute bars of 5,000 stocks over a year and 13.1 GB for the daily bars of 50,000 securities over 20 years. A one-machine problem, not a cluster problem.

The figure at the top sums it up on a logarithmic scale, against the memory of a 16 GB laptop and of a large 1 TB server.

On a linear scale, the OPRA bar would be about 4,000 times as long as the one for the trade day. The jump between what a quant works with using bars or a day's trades and what the full options book requires is three to four orders of magnitude. That is why “big” means different things on each desk: for a team that researches with daily bars, a global universe fits in a laptop; for one that models options microstructure, no computer is enough and the data are filtered (underlyings, expirations, time windows) before reaching memory.

A year of OPRA at its capacity, in a single memory allocation, stops being a pandas problem: the operating system cannot provide it. NumPy computes the size and requests it; Linux, with its default memory policy, refuses before touching a byte:

un_ano = int(311e9 * 252 * B_COTIZACION)
np.empty(un_ano, dtype=np.uint8)

numpy._core._exceptions._ArrayMemoryError: Unable to allocate 2.44 PiB for an array with shape (2743020000000000,) and data type uint8

What to do when it does not fit

The first step is not a new tool, but to reduce. Almost no analysis needs all the columns, all the rows and maximum precision at the same time:

If that is not enough, the order of the following lessons is the order in which it pays to scale:

flowchart TB
  A[Fits with room to spare?] -->|yes| P[pandas or Polars]
  A -->|no| R[Reduce]
  R --> B[Fits now?]
  B -->|yes| P
  B -->|no| S[Local streaming]
  S --> C[Fits on one disk?]
  C -->|yes| S2[Polars or DuckDB]
  C -->|no| D[Distribute]

• On one machine, out of core (lesson 6.6.2): partitioned Parquet, Arrow, Polars in lazy and streaming mode and DuckDB with a memory limit process in chunks datasets much larger than RAM. A CTA peak quote day (about 97 GB in memory) is a problem for one machine with a good disk, not for a cluster.
• Distribute (lesson 6.6.3): Dask, Spark or Ray, when the computation does not fit even on one machine's disk or has to be split so that it finishes in time.
• Store for querying (lesson 6.6.4): time series databases and ArcticDB, when the data arrive every day and are queried by date ranges and securities.
• Cloud and lakehouse (lesson 6.6.5): when the volume or the team justifies it.

Exercises

1. Compute how much one day of L2 book snapshots with 10 levels per side takes (price as float64 and size as int32 per level, plus time and security), every 100 milliseconds, for the lesson's 500 securities during the 6.5 hours of the session. Does it fit in 16 GB with the merge peak?

niveles, lados = 10, 2
b_foto = 8 + 2 + niveles * lados * (8 + 4)     # ts, security, levels
fotos = 500 * 6.5 * 3600 * 10                   # every 100 ms
gb = fotos * b_foto / 1e9
print(f"{fotos:,.0f} snapshots of {b_foto} bytes: {gb:.1f} GB; "
      f"with the merge peak, {gb * pico_union:.1f} GB")

117,000,000 snapshots of 250 bytes: 29.2 GB; with the merge peak, 201.8 GB

2. Measure the peak of sorting the table with Python objects (the first million rows) and compare it with that of the categories table. How much memory does it ask for in total, table included?

muestra_obj = con_objetos.iloc[:1_000_000].copy()
tam_obj = muestra_obj.memory_usage(deep=True).sum()
p_obj = pico(OPERACIONES["sort by security and time"], muestra_obj)
p_cat = picos["sort by security and time"] * tam
for nombre, t_, p_ in [("objects", tam_obj, p_obj),
                       ("categories", tam, p_cat)]:
    print(f"{nombre:<11} table {t_ / 2**20:6.1f} MiB, peak "
          f"{p_ / 2**20:5.1f} MiB, total "
          f"{(t_ + p_) / 2**20:6.1f} MiB")

objects     table  165.0 MiB, peak  63.5 MiB, total  228.5 MiB
categories  table   22.9 MiB, peak  63.5 MiB, total   86.4 MiB

3. In Parquet, a trade takes about 10 bytes. Does that mean that a CTA peak trade day in Parquet (about 1.1 GB) can be analyzed with 2 GB of RAM?