--- title: "6.6.1 When trading data becomes “big”" description: "How much ticks, L2/L3 books, OPRA and global universes take up, with CTA, Nasdaq and OPRA figures, and pandas and Arrow memory measured on generated data." url: "https://pythonparatrading.com/en/python-trading-course/level-6/when-financial-data-becomes-big-data/" lang: "en" image: "https://pythonparatrading.com/og/en/python-trading-course/level-6/when-financial-data-becomes-big-data.png" --- # 6.6.1 · When data becomes “big”: ticks, order books, options and universes > “Big data” is not a size, it is a relationship: data are big when they do not fit comfortably in the memory of the machine for what has to be done with them. This lesson puts figures on that relationship. It starts from a 36-byte message of Nasdaq's ITCH protocol, measures how much the same trade takes up in pandas, in Arrow and in Parquet, how much extra memory sorting, grouping or merging asks for, and with the statistics published by CTA, Nasdaq and OPRA it computes which market datasets fit in a laptop and which do not fit in any machine. - Course: [Level 6: Generative AI, LLMs, agents, big data and alternative data](https://pythonparatrading.com/en/python-trading-course/level-6/) - Module: 6.6 · Big data for trading - Duration: 60 min - Prerequisites: [1.3.1 · Types of market data: EOD, bars, ticks, order book and reference](https://pythonparatrading.com/en/python-trading-course/level-1/types-of-market-data/), [1.4.7 · Storage: CSV, Parquet, partitioning, DuckDB, TimescaleDB and ArcticDB](https://pythonparatrading.com/en/python-trading-course/level-1/storing-market-data-parquet-duckdb-timeseries/), [0.5.4 · Polars: lazy versus eager API, expressions, performance and when to pick it over pandas](https://pythonparatrading.com/en/python-trading-course/level-0/polars-lazy-api-and-expressions/) ## By the end of this lesson - Compute the in-memory size of a market dataset from the number of records and the bytes per row of each representation. - Measure the memory of a table in pandas with strings, with the `str` type, with categories, in Arrow and in Parquet, and explain where each difference comes from. - Measure the memory peak of common operations (sorting, grouping, merging, resampling) and use it to estimate when a computation stops fitting. - Place tick data, L2 and L3 books, OPRA and global universes relative to the memory of a machine with figures from primary sources and declared assumptions. - Decide between reducing, processing in chunks, using an out-of-core engine or distributing, depending on how far the computation exceeds memory. ## “Big” is a relationship, not a size A 10 GB file is small for a server with 1 TB of memory and huge for a 16 GB laptop. And a 2 GB table may not fit in that laptop if it has to be merged with another, sorted and grouped. The useful definition for a quant is operational: **data are big when the computation you need does not fit comfortably in the memory of the machine you have**. From there come the three questions of this lesson: how many records there are, how many bytes each one takes in the representation you use and how much extra memory each operation asks for. Wes McKinney, the creator of pandas, summed it up in 2017 in a rule of thumb: with pandas you should have **5 to 10 times** the size of the data in RAM; for a 10 GB dataset, 64 to 128 GB. The rule predates pandas 3 and Arrow as the default format, but the mechanism behind it is still there and is measured below: data take more space in memory than in the file, and each operation creates intermediate copies. Lesson 1.3.1 measured one session of the fictitious stock NOVA at each granularity, from the daily bar to the L3 book messages: from 1 row to more than half a million. Here we go from one stock to a market. ```mermaid flowchart TB B[Bars] --> T[Trades] T --> Q[L1 quotes] Q --> L2[L2 book] L2 --> L3[L3 book] ``` - **Bars** (daily or minute): one record per instrument and period, with open, high, low, close and volume. - **Trades**: one record per execution. - **L1 quotes**: the best bid and the best ask, one record per change. - **L2 book**: depth aggregated by price level, as snapshots or as per-level updates. - **L3 book** (*market by order*): every order and every change to every order; with it, the full book is reconstructed. Each step to the right multiplies the number of records. Options multiply it again: each underlying has hundreds or thousands of series (expirations by strikes by type) and each one is quoted on several exchanges. ## A 36-byte message Nasdaq's TotalView-ITCH 5.0 protocol disseminates the L3 book of its market as a sequence of binary messages. The specification fixes every field: unsigned integers in network byte order (*big endian*), ASCII text padded with spaces, the time in **nanoseconds since midnight** in 6 bytes and prices as integers with 4 implied decimals. The message for a new order (*Add Order*, type `A`) takes 36 bytes: | Field | Offset | Bytes | Content | |---|---|---|---| | Message Type | 0 | 1 | `A` | | Stock Locate | 1 | 2 | Code of the security in the session | | Tracking Number | 3 | 2 | Nasdaq internal number | | Timestamp | 5 | 6 | Nanoseconds since midnight | | Order Reference Number | 11 | 8 | Order identifier | | Buy/Sell Indicator | 19 | 1 | `B` or `S` | | Shares | 20 | 4 | Shares | | Stock | 24 | 8 | Symbol, padded with spaces | | Price | 32 | 4 | Price with 4 implied decimals | The first cell generates a synthetic session of 2 million trades in 500 fictitious securities (`T000` to `T499`, with the activity concentrated in the first ones, as in a real market) and writes it in that binary format. NumPy reads the binary without copying it with a structured type that reproduces the table; since NumPy has no 6-byte integers, the time is read as a 2-byte integer and a 4-byte one. The data are synthetic: they are not market data. ```python import gc import os import tracemalloc import numpy as np import pandas as pd import pyarrow as pa import pyarrow.parquet as pq rng = np.random.default_rng(661) N = 2_000_000 VALORES = np.array([f"T{i:03d}" for i in range(500)]) peso = 1 / np.arange(1, 501) ** 1.1 i_valor = rng.choice(500, N, p=peso / peso.sum()) ns = np.sort(rng.integers(34_200, 57_600, N) * 10**9 # 9:30-16:00 + rng.integers(0, 10**9, N)) precio = np.round(rng.gamma(2.0, 40.0, 500)[i_valor] * np.exp(rng.normal(0, 0.002, N)), 2) acciones = rng.geometric(0.02, N).astype(np.int32) lado = rng.choice(np.array(["B", "S"]), N) ADD_ORDER = np.dtype([ ("tipo", "S1"), ("locate", ">u2"), ("tracking", ">u2"), ("ts_alto", ">u2"), ("ts_bajo", ">u4"), ("referencia", ">u8"), ("lado", "S1"), ("acciones", ">u4"), ("valor", "S8"), ("precio", ">u4")]) msg = np.zeros(N, ADD_ORDER) msg["tipo"], msg["locate"] = b"A", i_valor + 1 msg["ts_alto"], msg["ts_bajo"] = ns >> 32, ns & 0xFFFFFFFF msg["referencia"] = np.arange(1, N + 1) msg["lado"] = lado.astype("S1") msg["acciones"] = acciones msg["valor"] = np.char.ljust(VALORES[i_valor], 8).astype("S8") msg["precio"] = np.round(precio * 10_000).astype(np.uint32) binario = msg.tobytes() leido = np.frombuffer(binario, ADD_ORDER) # no copy m = leido[0] hora = (int(m["ts_alto"]) << 32) | int(m["ts_bajo"]) print(f"{ADD_ORDER.itemsize} bytes per message; " f"{len(binario) / 2**20:.1f} MiB in total") print(m["valor"].decode().strip(), m["lado"].decode(), int(m["acciones"]), int(m["precio"]) / 10_000, pd.Timedelta(hora, "ns")) ``` ```output 36 bytes per message; 68.7 MiB in total T001 S 55 36.52 0 days 09:30:00.018014726 ``` Two million messages take 68.7 MiB, exactly 36 bytes each, and reading them costs nothing: `np.frombuffer` interprets the bytes in place, without copying or converting them. The first one is a sale of 55 shares of `T001` at 36.52 at 9:30:00.018. It is the most compact format in this lesson and the least convenient: the integers are in network order, the symbol carries spaces, the time is split and the price is an integer. To analyze it, it has to be converted, and the conversion has a price in memory. ## The same trade in pandas, Arrow and Parquet From the binary to an analysis table: time as a timestamp, security, price, shares, execution venue and side. Five representations of the same data are measured. In pandas, `memory_usage(deep=True)`, which with Python objects counts the size of each object as if it were not shared with any other cell: it is an upper bound. In Arrow, `nbytes`. In Parquet, the size of the file compressed with zstd, which is deleted afterwards. ```python mercado = rng.choice(np.array(list("NQPZKVJX")), N) base = pd.Timestamp("2026-10-22") columnas = {"ts": base + pd.to_timedelta(ns, "ns"), "valor": VALORES[i_valor], "precio": precio, "acciones": acciones, "mercado": mercado, "lado": lado} TEXTOS = ["valor", "mercado", "lado"] con_str = pd.DataFrame(columnas) # pandas 3: str type con_objetos = con_str.astype({c: object for c in TEXTOS}) compacto = con_str.astype({c: "category" for c in TEXTOS}) tabla = pa.Table.from_pandas(compacto, preserve_index=False) pq.write_table(tabla, "operaciones.parquet", compression="zstd") bytes_fila = { "ITCH binary (Add Order)": ADD_ORDER.itemsize, "pandas, Python objects": con_objetos.memory_usage(deep=True).sum() / N, "pandas, str type (Arrow)": con_str.memory_usage(deep=True).sum() / N, "pandas, categories": compacto.memory_usage(deep=True).sum() / N, "Arrow, dictionaries": tabla.nbytes / N, "Parquet zstd (disk)": os.path.getsize("operaciones.parquet") / N, } os.remove("operaciones.parquet") for nombre, b in bytes_fila.items(): print(f"{nombre:<27} {b:6.1f} B/row " f"{b * N / 2**20:6.1f} MiB") print(compacto.dtypes.astype(str).to_dict()) ``` ```output ITCH binary (Add Order) 36.0 B/row 68.7 MiB pandas, Python objects 173.0 B/row 330.0 MiB pandas, str type (Arrow) 50.0 B/row 95.4 MiB pandas, categories 24.0 B/row 45.8 MiB Arrow, dictionaries 24.0 B/row 45.8 MiB Parquet zstd (disk) 10.0 B/row 19.1 MiB {'ts': 'datetime64[ns]', 'valor': 'category', 'precio': 'float64', 'acciones': 'int32', 'mercado': 'category', 'lado': 'category'} ``` *Fig. — Bytes per trade for the same synthetic session of 2 million trades in six representations, from the binary message to the Parquet file.* Horizontal bars: ITCH binary 36 bytes; pandas with Python objects 173 (highlighted); pandas with the str dtype 50; pandas with categoricals 24; Arrow with dictionaries 24; Parquet with zstd on disk 10. The same session takes from 10 to 173 bytes per trade depending on how it is stored: - With **Python objects** (the `object` of pandas 2), each string is an object with its header (more than 40 bytes for a four-letter string) and the column stores an 8-byte pointer to each one: 173 bytes per trade, almost five times the binary message. The figure is an upper bound, because `memory_usage(deep=True)` counts each object even if several cells share the same one, but it gives the order of magnitude. - With pandas 3's **`str` type**, which stores the strings in an Arrow array (the characters one after another plus an 8-byte offset per cell), it drops to 50. - With **categories**, each repeated string is stored once and each row carries a small code: 2 bytes for 500 securities, 1 for 8 venues or 2 sides. That leaves 24 bytes per row, less than the binary, because only the fields that are analyzed are stored: 8 for the time, 8 for the price, 4 for the shares and 4 for the three codes. - **Arrow** with dictionaries (the translation of categories) takes the same, 24 bytes. - **Parquet** with zstd drops to 10 bytes on disk: it compresses well because the times are sorted and the codes repeat. But it is a disk format, not a memory format: to work with the data they have to be decompressed. Going from objects to categories divides memory by seven without losing information. It is the first lever, and almost always the cheapest. ## What each operation asks for A table fitting in memory does not mean you can work with it. Each pandas operation creates intermediate results: a sort index, the group keys, the merged table before returning it. The standard library's `tracemalloc` module records the memory allocations of Python and NumPy (not those of Arrow, which has its own allocator; that is why it is measured with the categories table, which is all in NumPy) and gives the memory **peak** since it was started. The cell measures four common operations on the first million trades, with the peak expressed as a multiple of the size of the table: ```python muestra = compacto.iloc[:1_000_000].copy() tam = muestra.memory_usage(deep=True).sum() sectores = pd.DataFrame({"valor": VALORES, "sector": np.arange(500) % 11}) OPERACIONES = { "sort by security and time": lambda d: d.sort_values(["valor", "ts"]), "VWAP per security": lambda d: ((d["precio"] * d["acciones"]) .groupby(d["valor"], observed=True).sum() / d.groupby("valor", observed=True) ["acciones"].sum()), "1-minute bars per security": lambda d: d.groupby(["valor", d["ts"].dt.floor("min")], observed=True)["precio"] .agg(["first", "max", "min", "last"]), "merge with the sector": lambda d: d.merge(sectores.astype({"valor": "category"}), on="valor"), } def pico(funcion, datos): gc.collect() tracemalloc.start() resultado = funcion(datos) maximo = tracemalloc.get_traced_memory()[1] tracemalloc.stop() del resultado return maximo picos = {k: pico(f, muestra) / tam for k, f in OPERACIONES.items()} print(f"table: {tam / 2**20:.1f} MiB") for nombre, veces in picos.items(): print(f"{nombre:<30} peak {veces:4.1f} times the table") ``` ```output table: 22.9 MiB sort by security and time peak 2.8 times the table VWAP per security peak 1.3 times the table 1-minute bars per security peak 3.2 times the table merge with the sector peak 5.9 times the table ``` *Fig. — Peak memory of four pandas operations on one million trades with categoricals, in multiples of the table size (on top of the table).* Horizontal bars: sort by security and time 2.8 times; VWAP by security 1.3; 1-minute bars by security 3.2; join with the sector 5.9 (highlighted). A line marks 1 time, the table size. The table of one million trades takes 22.9 MiB, and no operation settles for that. Sorting by security and time asks for 2.8 times the size of the table, on top of the table: the sort index and the sorted copy. The VWAP per security, which reduces a million rows to 500, asks for 1.3. The one-minute bars, with a new group key (the truncated time), 3.2. And the merge with a table of 500 sectors, the most innocent in appearance, 5.9 times: pandas builds the indexes of the two tables and the complete result table before returning it. These are the figures for this table and this version of pandas, not universal constants, but they convey the idea behind McKinney's rule: an analysis chains several of these operations and keeps several intermediate results, so the peak of a typical working notebook is several times the size of the data. ## How much each market takes up With the bytes per row and the measured peaks, the number of records is what is missing. The figures in this section come from publications of the systems themselves (consulted on October 23, 2026) or are declared assumptions: | Dataset | Records | Source | |---|---|---| | Trades on US tapes A and B | 113.9 million on the peak day of Q3 2026 | CTA SIP metrics (CTS) | | Quotes on tapes A and B | 2,761 million on the peak day of Q3 2026 | CTA SIP metrics (CQS) | | US options (OPRA) | 311 billion messages a day: projected capacity for July 2026 (253 billion for October 2025) | SIAC notice of September 15, 2025 | | Nasdaq L3 book (TotalView-ITCH) | Sample files for the sessions of December 8 to 12, 2025: 7.9 to 12.6 GB per session, compressed with gzip | Nasdaq sample server | | 1-minute bars | 5,000 stocks × 390 minutes × 252 sessions | Assumption | | Daily bars of a global universe | 50,000 instruments × 252 sessions × 20 years | Assumption | Two caveats. The CTA figures are those of the **peak day** of each quarter, not the average, and they only cover tapes A and B (stocks listed on NYSE and on exchanges other than Nasdaq; Nasdaq stocks go through another SIP, UTP's). And the OPRA figure is the **capacity** the system is sized for, not the traffic of a normal day. Both serve to size an infrastructure, which is what they are published for. The bytes per row of quotes and bars are measured the same way as those of trades, with one-row tables with their columns in compact types: ```python def bytes_por_fila(**tipos): vacia = pd.DataFrame({c: pd.Series([0], dtype=t) for c, t in tipos.items()}) return vacia.memory_usage(index=False).sum() B_OPERACION = bytes_fila["pandas, categories"] B_COTIZACION = bytes_por_fila( ts="datetime64[ns]", valor="int16", compra="float64", venta="float64", tam_compra="int32", tam_venta="int32", mercado="int8") B_BARRA = bytes_por_fila( ts="datetime64[ns]", valor="int32", apertura="float64", maximo="float64", minimo="float64", cierre="float64", volumen="int64") print(f"bytes per row: trade {B_OPERACION:.0f}, quote " f"{B_COTIZACION}, bar {B_BARRA}") CONJUNTOS = { # (records, bytes per record) "CTA trades, peak day": (113.9e6, B_OPERACION), "CTA trades, 252 peak days": (113.9e6 * 252, B_OPERACION), "CTA quotes, peak day": (2_761e6, B_COTIZACION), "OPRA, one day at capacity": (311e9, B_COTIZACION), "1-min bars, 5,000 stocks, 1 year": (5_000 * 390 * 252, B_BARRA), "daily bars, 50,000 securities, 20 years": (50_000 * 252 * 20, B_BARRA), } pico_union = 1 + round(picos["merge with the sector"], 1) # 1 + 5.9 tamanos = pd.DataFrame( [(k, n / 1e6, n * b / 1e9, n * b * pico_union / 1e9) for k, (n, b) in CONJUNTOS.items()], columns=["dataset", "millions", "GB", "GB when merging"] ).set_index("dataset") tamanos.round(1) ``` ```output bytes per row: trade 24, quote 35, bar 52 millions GB GB when merging dataset CTA trades, peak day 113.9 2.7 18.9 CTA trades, 252 peak days 28702.8 689.0 4753.8 CTA quotes, peak day 2761.0 96.6 666.8 OPRA, one day at capacity 311000.0 10885.0 75106.5 1-min bars, 5,000 stocks, 1 year 491.4 25.6 176.3 daily bars, 50,000 securities, 20 years 252.0 13.1 90.4 ``` With 24 bytes per trade, 35 per quote and 52 per bar: - The **peak trade day** of tapes A and B (113.9 million) takes 2.7 GB in memory: it fits in a laptop, but merging it with another table already asks for 19 GB, more than a 16 GB one has. A year of days like that, 689 GB, fits in none. - The **peak quote day** (2,761 million) takes 96.6 GB: it no longer fits in a laptop, but it does in a workstation or a server with 128 or 256 GB, and with the merge it asks for 670 GB. - The **L3 book of a single exchange**, Nasdaq, takes 7.9 to 12.6 GB per session in its sample files, already compressed with gzip; decompressed and turned into a table, several times more. At 36 bytes per message, 12.6 GB compressed is hundreds of millions of messages in one day. - **OPRA** at its projected capacity, 311 billion messages in one day, would be almost 11 TB in memory. There is no machine with that much memory; not even a laptop's disk can hold it. - The **bars** of a large universe are moderate: 25.6 GB for the one-minute bars of 5,000 stocks over a year and 13.1 GB for the daily bars of 50,000 securities over 20 years. A one-machine problem, not a cluster problem. The figure at the top sums it up on a logarithmic scale, against the memory of a 16 GB laptop and of a large 1 TB server. On a linear scale, the OPRA bar would be about 4,000 times as long as the one for the trade day. The jump between what a quant works with using bars or a day's trades and what the full options book requires is three to four orders of magnitude. That is why “big” means different things on each desk: for a team that researches with daily bars, a global universe fits in a laptop; for one that models options microstructure, no computer is enough and the data are filtered (underlyings, expirations, time windows) before reaching memory. A year of OPRA at its capacity, in a single memory allocation, stops being a pandas problem: the operating system cannot provide it. NumPy computes the size and requests it; Linux, with its default memory policy, refuses before touching a byte: ```python raises un_ano = int(311e9 * 252 * B_COTIZACION) np.empty(un_ano, dtype=np.uint8) ``` ```output numpy._core._exceptions._ArrayMemoryError: Unable to allocate 2.44 PiB for an array with shape (2743020000000000,) and data type uint8 ``` ## What to do when it does not fit The first step is not a new tool, but to **reduce**. Almost no analysis needs all the columns, all the rows and maximum precision at the same time: | Lever | Example | Measured or computed effect | |---|---|---| | Compact types | Repeated strings as categories | 170 to 24 bytes per trade (divide by 7) | | Columns | Read only time, security, price and shares | 24 to 22 bytes per trade | | Rows | One session, one venue, the securities in the universe | Proportional | | Aggregate early | From trades to 1-minute bars | 2 million rows to a few tens of thousands | | On-disk format | Parquet with zstd instead of CSV | 10 bytes per trade, and only the requested columns are read | If that is not enough, the order of the following lessons is the order in which it pays to scale: ```mermaid flowchart TB A[Fits with room to spare?] -->|yes| P[pandas or Polars] A -->|no| R[Reduce] R --> B[Fits now?] B -->|yes| P B -->|no| S[Local streaming] S --> C[Fits on one disk?] C -->|yes| S2[Polars or DuckDB] C -->|no| D[Distribute] ``` - **On one machine, out of core** (lesson 6.6.2): partitioned Parquet, Arrow, Polars in lazy and streaming mode and DuckDB with a memory limit process in chunks datasets much larger than RAM. A CTA peak quote day (about 97 GB in memory) is a problem for one machine with a good disk, not for a cluster. - **Distribute** (lesson 6.6.3): Dask, Spark or Ray, when the computation does not fit even on one machine's disk or has to be split so that it finishes in time. - **Store for querying** (lesson 6.6.4): time series databases and ArcticDB, when the data arrive every day and are queried by date ranges and securities. - **Cloud and lakehouse** (lesson 6.6.5): when the volume or the team justifies it. ## Exercises 1. Compute how much one day of L2 book snapshots with 10 levels per side takes (price as `float64` and size as `int32` per level, plus time and security), every 100 milliseconds, for the lesson's 500 securities during the 6.5 hours of the session. Does it fit in 16 GB with the merge peak? **Solution.** ```python niveles, lados = 10, 2 b_foto = 8 + 2 + niveles * lados * (8 + 4) # ts, security, levels fotos = 500 * 6.5 * 3600 * 10 # every 100 ms gb = fotos * b_foto / 1e9 print(f"{fotos:,.0f} snapshots of {b_foto} bytes: {gb:.1f} GB; " f"with the merge peak, {gb * pico_union:.1f} GB") ``` ```output 117,000,000 snapshots of 250 bytes: 29.2 GB; with the merge peak, 201.8 GB ``` Each snapshot takes 250 bytes (8 for the time, 2 for the security and 20 levels of 12 bytes) and a day has 117 million snapshots: 29.2 GB in memory and 201.8 GB with the merge peak. It does not fit in 16 GB. Options: fewer levels (the best five divide the size almost by two), lower frequency (one snapshot per second divides it by ten), only the securities in the strategy's universe, or storing the book changes instead of full snapshots. 2. Measure the peak of sorting the table with Python objects (the first million rows) and compare it with that of the categories table. How much memory does it ask for in total, table included? **Solution.** ```python muestra_obj = con_objetos.iloc[:1_000_000].copy() tam_obj = muestra_obj.memory_usage(deep=True).sum() p_obj = pico(OPERACIONES["sort by security and time"], muestra_obj) p_cat = picos["sort by security and time"] * tam for nombre, t_, p_ in [("objects", tam_obj, p_obj), ("categories", tam, p_cat)]: print(f"{nombre:<11} table {t_ / 2**20:6.1f} MiB, peak " f"{p_ / 2**20:5.1f} MiB, total " f"{(t_ + p_) / 2**20:6.1f} MiB") ``` ```output objects table 165.0 MiB, peak 63.5 MiB, total 228.5 MiB categories table 22.9 MiB, peak 63.5 MiB, total 86.4 MiB ``` The peak of sorting is the same, 63.5 MiB, in both tables: it is dominated by the integer arrays pandas uses to compute the order, which do not depend on how the strings are stored. What changes is the table: 165.0 MiB with objects against 22.9 with categories. In total, 228.5 against 86.4 MiB: 2.6 times more memory to do the same thing. 3. In Parquet, a trade takes about 10 bytes. Does that mean that a CTA peak trade day in Parquet (about 1.1 GB) can be analyzed with 2 GB of RAM? **Solution.** No. The 10 bytes per trade are on disk and compressed. When the whole file is read, Parquet is decompressed into Arrow or pandas and goes back to taking its 24 bytes per row: 2.7 GB, more than there is. What Parquet does allow is not to read it whole: only the necessary columns, only the row groups that pass a filter, and a streaming engine (Polars, DuckDB) that aggregates in chunks and never holds the complete table in memory. With that, a VWAP per security for the whole day fits in much less than 2 GB. That is the topic of lesson 6.6.2. ## Self-assessment 1. An Add Order message of Nasdaq TotalView-ITCH 5.0 takes 36 bytes. How much does the same information take in a pandas DataFrame with the text stored as Python objects? - A. The same: pandas stores the data as they arrive - B. Less, because pandas compresses - C. Several times more: each string is a Python object with its header and an 8-byte pointer, and integers and prices are widened to 8 bytes; with categories it drops below the binary - D. Exactly double *Answer C: Several times more: each string is a Python object with its header and an 8-byte pointer, and integers and prices are widened to 8 bytes; with categories it drops below the binary.* In the lesson, the table with Python objects counts about 170 bytes per trade according to `memory_usage(deep=True)`, pandas 3's `str` type (Arrow underneath) 50 and categories 24, less than the message's 36 because only the fields that are used are stored. 2. A table takes 10 GB in memory. Is a 16 GB laptop enough to work with it in pandas? - A. No: sorting or merging it asks for between one and almost six times its size in addition (as measured in the lesson), and McKinney's rule, 5 to 10 times the data size in RAM, points to 64–128 GB - B. Yes, there are 6 GB to spare - C. Yes, if the browser is closed - D. It depends only on the processor *Answer A: No: sorting or merging it asks for between one and almost six times its size in addition (as measured in the lesson), and McKinney's rule, 5 to 10 times the data size in RAM, points to 64–128 GB.* What counts is the operation's peak, not the size of the table. In the lesson, a merge asked for 5.9 times the size of the table on top of the table, and a grouping by minute, 3.2. 3. According to the capacity projection published in September 2025, how many messages a day must OPRA be able to disseminate in July 2026? - A. 311 million - B. 3.11 billion - C. 311 billion (it is the capacity the system is sized for, not the average traffic) - D. 31 trillion *Answer C: 311 billion (it is the capacity the system is sized for, not the average traffic).* SIAC's notice of September 15, 2025 projects 253 billion daily messages for October 2025 and 311 billion for July 2026. At 35 bytes per message, one day at that capacity is almost 11 TB. 4. What is the first thing to try when a computation does not fit in memory? - A. Set up a Spark cluster - B. Reduce: read only the necessary columns and rows, use compact types (categories, small integers) and aggregate early; then, process in chunks or with an out-of-core engine on the same machine - C. Buy more RAM - D. Move to the cloud *Answer B: Reduce: read only the necessary columns and rows, use compact types (categories, small integers) and aggregate early; then, process in chunks or with an out-of-core engine on the same machine.* Reducing costs little and is often enough: in the lesson, turning strings into categories divides memory by seven. Lessons 6.6.2 and 6.6.3 cover what comes next: columnar formats and streaming engines, and then distribution. 5. CTA's metrics give 113.9 million trades on the peak day of the third quarter of 2026. What is that figure good for and what is it not good for? - A. It is the average daily volume: it can be used to budget a year of storage by multiplying it by 252 - B. It is the US total, tapes A, B and C - C. It is the maximum capacity of the system - D. It is the busiest day of the quarter on tapes A and B: it serves to size the memory and bandwidth needed on the worst day, and multiplied by 252 it gives an upper bound, not an estimate *Answer D: It is the busiest day of the quarter on tapes A and B: it serves to size the memory and bandwidth needed on the worst day, and multiplied by 252 it gives an upper bound, not an estimate.* The CTA document gives the daily peak of each quarter, only for tapes A and B (Nasdaq stocks go through the UTP SIP). Sizing with the peak is the prudent choice for memory; for a year of storage, the average is needed. ## Key points - Data are “big” relative to the memory of a machine and to what has to be done with them: Wes McKinney's (2017) rule of thumb for pandas is to have 5 to 10 times the size of the data in RAM. - The same trade takes 36 bytes in an ITCH message, about 170 in pandas with Python objects, 50 with the `str` type, 24 with categories or in Arrow and about 10 in Parquet with zstd. - Operations ask for extra memory: measured with `tracemalloc`, sorting asks for 2.8 times the size of the table, grouping into one-minute bars 3.2 and merging 5.9. - With published figures: the peak trade day of CTA's tapes A and B (113.9 million, 2.7 GB) fits in a laptop's memory; the quote day (2,761 million, about 97 GB) does not; one day of OPRA at its projected capacity (311 billion messages) is almost 11 TB and fits in no machine. - Before scaling, reduce: columns, rows, types and early aggregation. Then, in this order: columnar formats and out-of-core engines on one machine (6.6.2), distribution (6.6.3), time series databases (6.6.4) and the cloud (6.6.5). ## Resources ### Research - [Big Data Management 1: an introduction to big data for trading with ArcticDB](https://pythonparatrading.com/en/big-data-for-trading/) · Research-0019 · 15 min ### Documentation - [Nasdaq TotalView-ITCH 5.0: specification](https://www.nasdaqtrader.com/content/technicalsupport/specifications/dataproducts/NQTVITCHspecification.pdf) · nasdaqtrader.com - [Nasdaq: TotalView-ITCH sample files](https://emi.nasdaq.com/ITCH/Nasdaq%20ITCH/) · emi.nasdaq.com - [CTA Plan: SIP metrics for tapes A and B (Q3 2026)](https://www.ctaplan.com/publicdocs/ctaplan/CTA_SIP_Metrics_3Q_2026.pdf) · ctaplan.com - [SIAC / OPRA: revised capacity projections (September 15, 2025)](https://cdn.opraplan.com/documents/notices/OPRA_Capacity_Projections_Update_0925.pdf) · cdn.opraplan.com - [McKinney, W. (2017): Apache Arrow and the “10 Things I Hate About pandas”](https://wesmckinney.com/blog/apache-arrow-pandas-internals/) · wesmckinney.com - [Python: tracemalloc](https://docs.python.org/3/library/tracemalloc.html) · docs.python.org Previous lesson: [6.5.3 · Risks of AI-generated code](https://pythonparatrading.com/en/python-trading-course/level-6/risks-of-ai-generated-code/) Next lesson: [6.6.2 · Parquet, Arrow, Polars streaming and out-of-core DuckDB](https://pythonparatrading.com/en/python-trading-course/level-6/parquet-arrow-polars-streaming-duckdb/)