--- title: "6.6.5 Cloud storage, lakehouse and GPU" description: "S3 and GCS measured in requests, the small-files problem, a Delta- or Iceberg-style transaction log and the GPU with RAPIDS and cuDF." url: "https://pythonparatrading.com/en/python-trading-course/level-6/cloud-storage-lakehouse-and-gpu/" lang: "en" image: "https://pythonparatrading.com/og/en/python-trading-course/level-6/cloud-storage-lakehouse-and-gpu.png" --- # 6.6.5 · Cloud storage, lakehouse (Delta, Iceberg) and GPU (RAPIDS, cuDF) > In the cloud, every read is a request that is billed and takes time. This lesson stores two million ticks in three ways in a simulated bucket and counts the requests and their cost, builds a transaction log with JSON and Parquet to understand what Delta Lake and Iceberg add (atomic commits, time travel, compaction, vacuum) and places the GPU with RAPIDS, cuDF and Polars as background knowledge, with its requirements. - Course: [Level 6: Generative AI, LLMs, agents, big data and alternative data](https://pythonparatrading.com/en/python-trading-course/level-6/) - Module: 6.6 · Big data for trading - Duration: 60 min - Prerequisites: [6.6.2 · Parquet, Arrow, Polars streaming and out-of-core DuckDB](https://pythonparatrading.com/en/python-trading-course/level-6/parquet-arrow-polars-streaming-duckdb/), [6.6.4 · Time series databases: TimescaleDB, QuestDB, ClickHouse, ArcticDB and kdb+](https://pythonparatrading.com/en/python-trading-course/level-6/time-series-databases-and-arcticdb/) ## By the end of this lesson - Explain how object storage (S3, GCS) differs from a disk: flat keys, whole objects, range reads, consistency and cost per request. - Measure how many requests and bytes a query costs depending on the lake design and recognize the small-files problem. - Build a minimal transaction log with atomic commits, time travel, compaction, optimistic concurrency and vacuum, and relate it to Delta Lake and Iceberg. - Write the Delta Lake (delta-rs) and PyIceberg code for the same operations from their documentation. - Decide when a GPU (RAPIDS, cuDF, Polars on GPU) pays off for tabular data and what to check first. ## Object storage: S3 and GCS A data lake in the cloud does not live on a disk, but in **object storage**: Amazon S3, Google Cloud Storage (GCS), Azure Blob Storage or their S3-compatible equivalents. From Python they look like folders (Polars, DuckDB and PyArrow read `s3://cubo/lago/` just as they read `lago/`), but they work differently, and those differences explain half of the design decisions of a lake: - **There are no folders.** An S3 bucket is flat: every object has a key (`lago/fecha=2025-03-14/parte-0.parquet`) and the “folders” are shared prefixes that the console presents as such[^1]. Listing means requesting the keys with a prefix: `ListObjectsV2` returns at most 1,000 per request[^1]. - **The whole object is written.** You cannot modify a piece or append at the end: you upload (`PUT`) a complete new object. Renaming means copying and deleting the original[^1]. - **It is read by ranges.** A `GET` with the `Range` header downloads an interval of bytes[^1]; that is how a Parquet reader reads the footer and then only the column chunks it needs (lesson 6.6.2). - **Strong consistency.** Since December 2020, S3 guarantees that a read after a write or a delete sees the result in all regions; GCS also offers strong global read-after-write consistency[^2]. - **Every request is billed.** On the S3 pricing page, in S3 Standard, `GET` requests cost 0.0004 USD per 1,000 and `PUT`, `COPY`, `POST` and `LIST`, 0.005 USD per 1,000 (Northern Virginia, accessed on October 23, 2026)[^3]. Storage and data transfer are charged separately. ```mermaid flowchart LR C[Python client] -- LIST prefix --> B[(S3 bucket)] C -- GET Range: footer --> B C -- GET Range: columns --> B C -- PUT whole object --> B ``` The course environment has no `s3fs` or any library that imitates S3, and the lesson does not use the network. To measure anyway, the first cell defines a “bucket” that is a local folder seen through `fsspec` (the same layer `s3fs` uses for S3) and that counts every object written as a `PUT` and every read as a range `GET`. It does not reproduce S3's latency or prices, but it does reproduce the number of requests a reader would make, which is what costs money and time in the cloud. ```python import math import numpy as np import pandas as pd import pyarrow as pa import pyarrow.compute as pc import pyarrow.dataset as ds import pyarrow.parquet as pq import os from fsspec.implementations.local import LocalFileSystem from pyarrow.fs import FSSpecHandler, PyFileSystem class AlmacenObjetos(LocalFileSystem): """Imitates an S3 bucket in a folder: counts PUT and GET.""" def reiniciar(self): self.put = self.get = self.bytes_get = 0 def _open(self, path, mode="rb", **kw): f = super()._open(path, mode, **kw) if "w" in mode: self.put += 1 # one whole object per PUT return f leer = f.read def read(n=-1): datos = leer(n) self.get += 1 # every read, one GET self.bytes_get += len(datos) return datos f.read = read return f CUBO = os.path.abspath("cubo") # the “bucket”: a folder s3 = AlmacenObjetos() s3.reiniciar() cubo = PyFileSystem(FSSpecHandler(s3)) print(type(cubo).__name__, "on", type(s3).__name__) ``` ```output PyFileSystem on AlmacenObjetos ``` Real code against S3 only changes the path and the credentials (which are not written in the code: they are taken from the environment or the AWS profile). With `s3fs` installed, the three tools from lesson 6.6.2 read like this: ```python skip title="lago_en_s3.py" import duckdb import polars as pl import pyarrow.dataset as ds ticks = pl.scan_parquet("s3://mi-cubo/lago/ticks/**/*.parquet", hive_partitioning=True) # credentials: environment lago = ds.dataset("s3://mi-cubo/lago/ticks/", format="parquet", partitioning="hive") con = duckdb.connect() con.sql("INSTALL httpfs; LOAD httpfs;") con.sql("CREATE SECRET (TYPE s3, PROVIDER credential_chain)") con.sql("SELECT count(*) FROM read_parquet(" "'s3://mi-cubo/lago/ticks/**/*.parquet')") ``` The test data: two million synthetic ticks for 100 stocks over the 20 sessions of March 2025, stored in three ways: a single object, one object per day and one object per day and symbol. ```python import exchange_calendars as xcals tickers = sorted(pd.read_csv("panel5_activos.csv")["ticker"])[:100] cal = xcals.get_calendar("XNYS") sesiones = cal.sessions_in_range("2025-03-03", "2025-03-28") rng = np.random.default_rng(665) partes = [] for s in sesiones: n = rng.poisson(1000, len(tickers)) abre = cal.session_open(s).value // 1000 partes.append(pd.DataFrame({ "fecha": s.date(), "simbolo": np.repeat(tickers, n), "ts": pd.to_datetime( rng.integers(abre, abre + 23_400_000_000, n.sum()), unit="us", utc=True), "precio": rng.lognormal(4, 0.5, n.sum()).round(2), "tamano": rng.geometric(0.02, n.sum()).astype("int32")})) ticks = pd.concat(partes).sort_values(["fecha", "simbolo", "ts"]) tabla = pa.Table.from_pandas(ticks, preserve_index=False) formato = ds.ParquetFileFormat() opciones = formato.make_write_options(compression="zstd") def escribir(ruta, particiones): s3.reiniciar() ds.write_dataset(tabla, ruta, filesystem=cubo, format=formato, file_options=opciones, partitioning=particiones, partitioning_flavor="hive" if particiones else None, basename_template="parte-{i}.parquet", max_rows_per_group=131_072) claves = s3.find(ruta) return {"objetos": len(claves), "PUT": s3.put, "MB": round(sum(s3.size(k) for k in claves) / 1e6, 1)} disenos = {"one file": (f"{CUBO}/uno", None), "by day": (f"{CUBO}/dia", ["fecha"]), "by day and symbol": (f"{CUBO}/dia_simbolo", ["fecha", "simbolo"])} escritura = pd.DataFrame([escribir(*v) for v in disenos.values()], index=list(disenos)) print(f"{len(ticks):,} ticks, {len(sesiones)} sessions," f" {len(tickers)} symbols") escritura ``` ```output 1,999,891 ticks, 20 sessions, 100 symbols objetos PUT MB one file 1 1 22.1 by day 20 20 22.3 by day and symbol 2000 2000 27.9 ``` Writing costs one `PUT` per object: 1, 20 or 2,000. And the 2,000 small objects take up 26% more space (27.9 MB versus 22.1): each one carries its own footer and compresses worse, as seen in lesson 1.4.7. Now, the same pair of queries on the three designs: the BAGO ticks for one day and the whole `precio` column. ```python GET_1000, PUT_LIST_1000 = 0.0004, 0.005 # USD, S3 Standard def consultar(ruta, particiones, filtro, columnas): s3.reiniciar() esquema = pa.schema([tabla.schema.field(c) for c in particiones or []]) lago = ds.dataset(ruta, filesystem=cubo, format="parquet", partitioning=ds.partitioning( esquema, flavor="hive") if particiones else None) claves = len(s3.find(ruta)) for frag in lago.get_fragments(filtro): # footers that are needed frag.ensure_complete_metadata() t = lago.to_table( # reads in a fixed order columns=columnas, filter=filtro, use_threads=False, fragment_scan_options=ds.ParquetFragmentScanOptions( pre_buffer=False)) listados = math.ceil(claves / 1000) # 1,000 keys per LIST return {"filas": t.num_rows, "LIST": listados, "GET": s3.get, "MB": round(s3.bytes_get / 1e6, 2), "USD_por_millón": round((s3.get * GET_1000 + listados * PUT_LIST_1000) * 1000, 2)} un_dia = ((pc.field("fecha") == sesiones[9].date()) & (pc.field("simbolo") == "BAGO")) filas = {} for k, (ruta, part) in disenos.items(): filas[(k, "BAGO, one day")] = consultar(ruta, part, un_dia, ["ts", "precio"]) filas[(k, "all, one column")] = consultar(ruta, part, None, ["precio"]) pd.DataFrame(list(filas.values()), index=pd.MultiIndex.from_tuples(filas)) ``` ```output filas LIST GET MB USD_por_millón one file BAGO, one day 1016 1 10 0.79 9.0 all, one column 1999891 1 64 5.75 30.6 by day BAGO, one day 1016 1 5 0.31 7.0 all, one column 1999891 1 102 7.22 45.8 by day and symbol BAGO, one day 1016 2 4 0.04 11.6 all, one column 1999891 2 4061 36.90 1634.4 ``` The figure at the top plots the cost column. What the numbers say: - **For the point query**, all three designs are cheap: between 4 and 10 requests. The single file needs more `GET` requests (10) because it has to read the footer and the chunks of several columns in the candidate row groups; the by-day-and-symbol design reads only 0.04 MB, but needs 2 `LIST` requests (2,000 keys, 1,000 per request), which on S3 cost more than `GET` requests. - **For the full read of a column**, small objects are a disaster: 4,061 `GET` requests and 36.90 MB, more than the whole lake (27.9 MB). With 14 KB objects, the first footer read (64 KB) already brings the whole object, and then the reader requests the column again. A million reads like that would cost about 1,634 USD in requests alone, versus 31 USD with a single object. - **The by-day design** is the reasonable compromise here: cheap on both queries. With real data of hundreds of gigabytes, the equivalent is partitioning by date with objects of hundreds of megabytes, not kilobytes. Time follows the same logic, even more strongly: every request to S3 has a latency of tens of milliseconds, much higher than that of a local disk, so thousands of small sequential requests can take minutes even if they download only a few megabytes. Readers compensate by issuing requests in parallel, but the number of requests is still what you have to watch. ## Lakehouse: a transaction log on top of the lake A lake of files has three shortcomings that a database does not: there are no **transactions** (if a process rewrites ten files and fails on the sixth, the lake is left half done), there are no **versions** (correcting a day erases what was there) and two writers at the same time can overwrite each other. **Lakehouse** formats (Delta Lake, Apache Iceberg, Apache Hudi) add a **transaction log** on top of the Parquet files: an ordered list of *commits*, each one with the files it adds and the ones it removes. The table, at a given version, is the set of live files after applying the commits up to it. ```mermaid flowchart LR R[_registro/] --> V0[000000.json: add x15] R --> V1[000001.json: add x5] R --> V2[000002.json: remove, add] R --> V3[000003.json: compact] V3 --> D[datos/*.parquet] ``` Data files are never modified: correcting means writing a new file and removing the old one in the same commit (*copy-on-write*). And the commit is **atomic**: the file `000004.json` either exists in full or does not exist. To understand the idea without depending on any library, the following cells build a minimal log with JSON and Parquet files. It is not Delta or Iceberg, but it has their pieces: numbered commits, writing with `open(..., "x")` (create only if it does not exist: the local equivalent of a conditional write), reading any version and cleaning up old files. ```python import json import os from pathlib import Path RAIZ = Path("tabla_ticks") (RAIZ / "_registro").mkdir(parents=True) (RAIZ / "datos").mkdir() def confirmar(version, acciones): """Writes commit `version` only if it does not exist (put-if-absent).""" ruta = RAIZ / "_registro" / f"{version:06d}.json" with open(ruta, "x") as f: # "x": fails if it already exists json.dump(acciones, f) def ficheros_vivos(version): vivos = set() for v in range(version + 1): ruta = RAIZ / "_registro" / f"{v:06d}.json" for a in json.loads(ruta.read_text()): if a["op"] == "add": vivos.add(a["f"]) elif a["op"] == "remove": vivos.discard(a["f"]) return sorted(vivos) def ultima_version(): return len(list((RAIZ / "_registro").glob("*.json"))) - 1 def leer(version=None): v = ultima_version() if version is None else version return pa.concat_tables( pq.read_table(RAIZ / "datos" / f) for f in ficheros_vivos(v)) def escribir_fichero(df, nombre): pq.write_table(pa.Table.from_pandas(df, preserve_index=False), RAIZ / "datos" / nombre, compression="zstd") return {"op": "add", "f": nombre, "filas": len(df)} dias = [d for d, _ in ticks.groupby("fecha")] # version 0: the first 15 sessions, one file per day confirmar(0, [escribir_fichero(ticks[ticks["fecha"] == d], f"{d}-a.parquet") for d in dias[:15]]) # version 1: the next 5 sessions arrive confirmar(1, [escribir_fichero(ticks[ticks["fecha"] == d], f"{d}-a.parquet") for d in dias[15:]]) print("version", ultima_version(), "|", len(ficheros_vivos(1)), "live files |", f"{leer().num_rows:,} rows") ``` ```output version 1 | 20 live files | 1,999,891 rows ``` Two commits: the first adds the first 15 sessions (one file per day) and the second, the next 5. The table at version 1 has 20 live files and the 1,999,891 ticks. Now, two operations that would be risky in a lake of loose files: the vendor corrects March 14 (it cancels the single-share prints, which in this data are 2,058 rows) and then the 20 daily files are compacted into one. ```python # version 2: the vendor corrects March 14 (copy-on-write) d14 = ticks[ticks["fecha"] == dias[9]].copy() d14 = d14[d14["tamano"] > 1] # cancels the single-share prints confirmar(2, [{"op": "remove", "f": f"{dias[9]}-a.parquet"}, escribir_fichero(d14, f"{dias[9]}-b.parquet")]) # version 3: compact the 20 daily files into one todo = leer().to_pandas() confirmar(3, [{"op": "remove", "f": f} for f in ficheros_vivos(2)] + [escribir_fichero(todo, "compacto-0.parquet")]) historia = pd.DataFrame([ {"versión": v, "ficheros_vivos": len(ficheros_vivos(v)), "filas": leer(v).num_rows, "filas_14_marzo": pc.sum(pc.equal( leer(v)["fecha"], pa.scalar(dias[9]))).as_py() or 0} for v in range(4)]) en_disco = len(list((RAIZ / "datos").iterdir())) print(en_disco, "files on disk") historia ``` ```output 22 files on disk versión ficheros_vivos filas filas_14_marzo 0 0 15 1499553 99833 1 1 20 1999891 99833 2 2 20 1997833 97775 3 3 1 1997833 97775 ``` *Fig. — Data files on disk and live files at each version of the lesson's transaction log.* Wide bars for files on disk and narrow bars for live files. Version 0: 15 and 15. Version 1: 20 and 20. Version 2: 21 on disk and 20 live (the correction retires one). Version 3: 22 on disk and 1 live after compaction (highlighted). After vacuum: 1 and 1. Every version remains readable (*time travel*): version 0 has 15 files and 1,499,553 rows; version 1, 20 files and 1,999,891; version 2 replaces the March 14 file and drops to 97,775 rows that day; version 3 has a single file with the same 1,997,833 rows as version 2. A reader that started reading version 2 while the compaction was running sees nothing odd: its 20 files are still on disk, because compaction only adds a new one and records in the log that the old ones no longer count. On disk there are 22 files (the 20 originals, the correction and the compacted one) for 1 live file. Atomicity also solves concurrent writers. If two processes read version 3 and both try to write version 4, only one can create `000004.json`; the second gets an error and must reread the log, check whether its change is still valid and try again with version 5 (**optimistic concurrency**): ```python raises version_leida = ultima_version() # two writers read version 3 confirmar(version_leida + 1, [{"op": "nota", "texto": "writer A"}]) confirmar(version_leida + 1, # the second one arrives late [{"op": "nota", "texto": "writer B"}]) ``` ```output FileExistsError: [Errno 17] File exists: 'tabla_ticks/_registro/000004.json' ``` On a local disk, `open(..., "x")` is atomic. In object storage you need its equivalent: S3 supports conditional writes with the `If-None-Match` header, which prevent overwriting an object that already exists[^4], and Iceberg delegates the version change to a catalog that atomically swaps the pointer to the current metadata file[^5]. What nobody cleans up are the files that no version uses any longer. **Vacuum** deletes them, and with them the ability to travel to the versions that used them: ```python def vacuum(desde_version): """Deletes the data that no version >= desde_version uses.""" usados = set() for v in range(desde_version, ultima_version() + 1): usados |= set(ficheros_vivos(v)) borrados = 0 for f in (RAIZ / "datos").iterdir(): if f.name not in usados: f.unlink() borrados += 1 return borrados print("deleted:", vacuum(desde_version=3), "| remaining:", len(list((RAIZ / "datos").iterdir()))) try: leer(1) except FileNotFoundError: print("version 1 can no longer be read") print(f"{leer().num_rows:,} rows in the latest version") ``` ```output deleted: 21 | remaining: 1 version 1 can no longer be read 1,997,833 rows in the latest version ``` It deletes 21 files and keeps the compacted one. From that moment on, version 1 can no longer be read: the log describes it, but its data no longer exists. That is why real formats impose a retention period (for example, not deleting anything that stopped being used less than a week ago) and why it pays to decide beforehand how much history you need to be able to reproduce, as with ArcticDB versions in lesson 6.6.4. ## Delta Lake and Iceberg **Delta Lake** stores the log in `_delta_log/`: one JSON file per commit (`00000000000000000042.json`) with actions like those in the cell, plus periodic **checkpoints** in Parquet that summarize the state so that not all the JSON files have to be read from the beginning[^6]. **Apache Iceberg** uses a hierarchy of metadata: a table metadata file with its **snapshots**; each snapshot, a manifest list (since version 4 of the format, a root manifest) and each manifest, a list of data files with their statistics; a catalog stores the pointer to the current metadata file and each commit swaps it atomically[^5]. Neither of the two is installed in the course environment (`deltalake` and `pyiceberg` were not added), so the code is not run. With `deltalake` (delta-rs, the Rust implementation with a Python API), the operations from the previous section are: ```python skip title="delta_ticks.py" from deltalake import DeltaTable, write_deltalake write_deltalake("s3://mi-cubo/delta/ticks", ticks_v0, partition_by=["fecha"]) # version 0 write_deltalake("s3://mi-cubo/delta/ticks", ticks_nuevos, mode="append") # version 1 dt = DeltaTable("s3://mi-cubo/delta/ticks") dt.load_as_version(0) # time travel v0 = dt.to_pyarrow_table() dt = DeltaTable("s3://mi-cubo/delta/ticks") dt.optimize.compact() # compact small files dt.vacuum(retention_hours=168, dry_run=False) ``` And with PyIceberg, which needs a configured catalog (REST, Glue, SQL…): ```python skip title="iceberg_ticks.py" from pyiceberg.catalog import load_catalog catalogo = load_catalog("produccion") # in .pyiceberg.yaml tabla = catalogo.load_table("mercado.ticks") bago = tabla.scan(row_filter="simbolo == 'BAGO'", selected_fields=("ts", "precio")).to_arrow() for s in tabla.snapshots(): print(s.snapshot_id, s.timestamp_ms) antes = tabla.scan(snapshot_id=tabla.snapshots()[0].snapshot_id) ``` The choice between the two is usually dictated by the platform and the engines already in use; the most visible design difference is that Iceberg makes the catalog explicit, while Delta keeps all the state next to the data. For a solo researcher with a Parquet lake, neither is essential; they start to pay off when several processes write to the same lake or you need to reproduce the state of the data as of a date. ## GPU: RAPIDS, cuDF and Polars GPUs, designed for graphics and today for training networks, also work for tables: thousands of cores apply the same operation to millions of rows at once. **RAPIDS** is NVIDIA's set of libraries for that, and **cuDF** its GPU `DataFrame`. Its `cudf.pandas` mode speeds up pandas code without changing it: the operations cuDF knows how to do go to the GPU and the rest automatically fall back to pandas on the CPU[^7]. ```python skip title="cudf_pandas.py" # In Jupyter: %load_ext cudf.pandas before importing pandas # As a script: python -m cudf.pandas barras_diarias.py import pandas as pd ticks = pd.read_parquet("ticks/") # on the GPU if possible diarias = ticks.groupby(["simbolo", "fecha"]).agg( volumen=("tamano", "sum"), maximo=("precio", "max")) ``` Polars has a GPU engine (in open beta, built on cuDF) that is requested with `collect(engine="gpu")`; if an operation is not supported, the whole query runs on the CPU, unless the opposite is requested with `pl.GPUEngine(raise_on_fail=True)`[^8]. The requirements, according to the RAPIDS and Polars documentation: an NVIDIA GPU with Volta architecture or later (compute capability 7.0 or higher), CUDA 12 or 13 and Linux or WSL2[^7][^8]. Without any of that, Polars does not even attempt the query: ```python import polars as pl consulta = pl.LazyFrame(ticks).group_by("simbolo").agg( pl.col("tamano").sum()) try: consulta.collect(engine="gpu") except ModuleNotFoundError as e: print(str(e).splitlines()[0]) print(consulta.collect().height, "groups on CPU") ``` ```output GPU engine requested, but required package 'cudf_polars' could not be imported. 100 groups on CPU ``` Without the `cudf_polars` package the request fails with a clear message; the same query on the CPU returns the 100 groups. On a machine with a GPU and `pip install polars[gpu]`, the code would not change. When a GPU pays off for tabular data: aggregations, joins and sorts over tens or hundreds of millions of rows that fit in the card's memory, repeated many times (a feature sweep, a vectorized backtest over many assets). When it does not: data that does not fit in GPU memory (much smaller than a server's RAM), queries that read little and compute less (the bottleneck is the disk or the network, not computation) and code with many operations the GPU does not support, which go back and forth to the CPU copying data over the bus. As with the clusters in lesson 6.6.3, you first have to measure where the time goes. ## Costs and decisions | Need | Option | What it costs | |---|---|---| | Lake shared by a team, serverless | Parquet on S3 or GCS, partitioned by date | Requests and storage; watch out for small files | | Concurrent writes, corrections, versions | Delta Lake or Iceberg on top of the lake | A log and, with Iceberg, a catalog; periodic compaction and vacuum | | Per-symbol series with versions from Python | ArcticDB on S3 (lesson 6.6.4) | BSL license; its own data model | | Repeated massive aggregations | GPU with cuDF or Polars on GPU | NVIDIA hardware, card memory, partial operation support | | Solo research, data on one disk | Local Parquet, Polars and DuckDB (lesson 6.6.2) | Nothing new | [^1]: AWS, “What is Amazon S3?”, “Organizing objects using prefixes”, “Copying objects” (renaming), `GetObject` (Range) and `ListObjectsV2`: and , accessed on October 23, 2026. [^2]: AWS, announcement of December 1, 2020 on strong consistency in S3: ; Google Cloud, “Consistency”: . Accessed on October 23, 2026. [^3]: AWS, “Amazon S3 pricing”: , accessed on October 23, 2026. Prices change and depend on the region and the storage class. [^4]: AWS, “Conditional writes”: , accessed on October 23, 2026. [^5]: Apache Iceberg, format specification (metadata, snapshots, manifests and optimistic concurrency): , accessed on October 23, 2026. [^6]: Delta Lake, transaction protocol: ; delta-rs: . Accessed on October 23, 2026. [^7]: NVIDIA RAPIDS, `cudf.pandas` and installation guide: and , accessed on October 23, 2026. [^8]: Polars, “GPU Support”: , accessed on October 23, 2026. ## Exercises 1. A backtest requests the ticks for each of the 100 symbols in each of the 20 sessions (2,000 point queries like “BAGO, one day”). With the lesson's request table, how much does it cost in S3 requests with each design? And if it were run every night for a year? **Solution.** ```python tabla_p = pd.DataFrame(list(filas.values()), index=pd.MultiIndex.from_tuples(filas)) puntual = tabla_p.xs("BAGO, one day", level=1) coste = (puntual["GET"] * GET_1000 + puntual["LIST"] * PUT_LIST_1000) \ / 1000 * 2000 pd.DataFrame({"USD_backtest": coste.round(4), "USD_año": (coste * 252).round(2)}) ``` ```output USD_backtest USD_año one file 0.0180 4.54 by day 0.0140 3.53 by day and symbol 0.0232 5.85 ``` Cents: between 0.014 and 0.023 USD per backtest and between 3.5 and 5.9 USD a year running it every session. For point queries, S3 requests are cheap with any reasonable design; `LIST` requests weigh more than `GET` requests, and that is why the 2,000-object design, which needs two per query, is the most expensive even though it downloads less. The real cost appears in full reads with small objects (the lesson's second query) and, outside this calculation, in data transfer and in time: 2,000 sequential queries with tens of milliseconds of latency per request add up to minutes. A backtest that reads the same data every night should also keep a local copy or read in batches. 2. Write a function that, reading only the log (without opening any Parquet file), returns for each version how many files it adds and removes and how many rows it adds. **Solution.** ```python def historial(): filas_h = [] for v in range(ultima_version() + 1): ruta = RAIZ / "_registro" / f"{v:06d}.json" acciones = json.loads(ruta.read_text()) filas_h.append({ "versión": v, "añade": sum(a["op"] == "add" for a in acciones), "retira": sum(a["op"] == "remove" for a in acciones), "filas_añadidas": sum(a.get("filas", 0) for a in acciones if a["op"] == "add")}) return pd.DataFrame(filas_h) historial() ``` ```output versión añade retira filas_añadidas 0 0 15 0 1499553 1 1 5 0 500338 2 2 1 1 97775 3 3 1 20 1997833 4 4 0 0 0 ``` Five versions: version 4 is writer A's note, which neither adds nor removes anything. The history comes entirely from the log, without opening any Parquet file; real formats store much more in each action (size, column statistics, number of rows) so that a reader can discard files without opening them, as the Parquet footer does inside a file. The last cell deletes the table and the simulated bucket. ```python import shutil shutil.rmtree(RAIZ) s3.rm(CUBO, recursive=True) print(sorted(p.name for p in Path(".").glob("tabla_*")), s3.exists(CUBO)) ``` ```output [] False ``` 3. A team processes 300 million rows of 1-minute bars (about 20 GB in memory) in pandas every night to compute 40 features. It takes two hours. Someone proposes buying a GPU. What would you check first? **Solution.** First, where the two hours go: measure how much is reading, how much conversion and how much computation, and the peak memory of each step (lesson 6.6.1). If most of it is reading and converting data (CSV, `object` columns, poorly chosen types), the GPU does not help: what helps is well-partitioned Parquet and reading only the columns needed (lesson 6.6.2). Second, whether the computation can be vectorized: per-row Python loops or custom functions with `apply` do not go to the GPU, and in `cudf.pandas` they would fall back to the CPU copying data. Third, whether it fits: 20 GB in pandas can be quite a bit less in Arrow, but you have to check that it fits in the card's memory along with its intermediate structures. Fourth, the cheap alternative: the same computation in lazy Polars or DuckDB on the CPU usually speeds up a pandas process a lot without new hardware. Only if, after all that, the bottleneck is computation on data that fits on the GPU is it worth trying one, for example by renting one by the hour in the cloud before buying it, and measuring with the real data. ## Self-assessment 1. Reading the whole `precio` column costs 64 `GET` requests with a single object and 4,061 with 2,000 objects of 14 KB, which also download 36.9 MB from a 27.9 MB lake. Why? - A. Because S3 compresses small objects worse - B. Because every object requires at least one read of the footer and another of the column, and with 14 KB objects the footer read (64 KB) already brings the whole object: many requests and repeated bytes - C. Because small objects have no statistics - D. Because `LIST` returns only 1,000 keys *Answer B: Because every object requires at least one read of the footer and another of the column, and with 14 KB objects the footer read (64 KB) already brings the whole object: many requests and repeated bytes.* A million reads like that would cost about 1,634 USD in requests versus 31 USD with one object. This is the small-files problem, and the solution is to compact into objects of hundreds of megabytes. 2. What guarantees that two writers that read version 3 of the table do not both publish a different version 4? - A. That each one writes its data files into different folders - B. An operating system lock for the whole write - C. That the reader picks the most recent version - D. That the commit is created only if it does not exist (`open(..., "x")` locally, a conditional write on S3, an atomic swap of the pointer in the Iceberg catalog): the second one gets an error, rereads and retries *Answer D: That the commit is created only if it does not exist (`open(..., "x")` locally, a conditional write on S3, an atomic swap of the pointer in the Iceberg catalog): the second one gets an error, rereads and retries.* This is optimistic concurrency: nothing is locked while the data is written; only the final step, publishing the commit, is atomic. The second writer gets `FileExistsError` and must check whether its change is still valid. 3. After compaction (version 3) there are 22 files on disk and 1 live. After `vacuum(desde_version=3)`, what is lost? - A. Nothing: the log keeps the versions - B. The latest version - C. The ability to read versions 0 to 2: the log describes them, but their data files have been deleted - D. The compaction, which is undone *Answer C: The ability to read versions 0 to 2: the log describes them, but their data files have been deleted.* Vacuum deletes the files that no retained version uses. That is why real formats impose a retention period: time travel only reaches as far back as the data does. 4. A Polars query with `collect(engine="gpu")` includes an operation the GPU engine does not support. What happens by default? - A. The whole query runs on the CPU with the normal engine, unless `GPUEngine(raise_on_fail=True)` is requested - B. It always fails - C. It runs approximately on the GPU - D. Polars installs cuDF automatically *Answer A: The whole query runs on the CPU with the normal engine, unless `GPUEngine(raise_on_fail=True)` is requested.* According to the Polars documentation, the GPU engine (in open beta) transparently falls back to the CPU. Without the `cudf_polars` package installed, however, the request fails with `ModuleNotFoundError`, as in the lesson. 5. What does the lesson's cost table count and what does it leave out? - A. The entire cost of S3 - B. Only `GET` and `LIST` requests at S3 Standard prices; storage, data transfer and latency are left out - C. Only storage - D. The cost of the GPU *Answer B: Only `GET` and `LIST` requests at S3 Standard prices; storage, data transfer and latency are left out.* The S3 pricing page charges separately for storage, requests and transfer. The lesson measures requests because they are what changes with the lake design; the latency of each request also turns thousands of small requests into minutes. ## Key points - Object storage is flat, is written in whole objects, is read by ranges, is strongly consistent (S3 since December 2020) and charges for every request: 0.0004 USD per 1,000 GET and 0.005 USD per 1,000 PUT or LIST in S3 Standard. - With the same two million ticks, reading a whole column costs 64 GET with one object, 102 with one per day and 4,061 with 2,000 small objects, which also download more bytes than the whole lake: partition by date with large objects and compact. - A lakehouse adds a transaction log on top of the Parquet files: atomic commits that add and remove files, copy-on-write for corrections, time travel, optimistic concurrency and vacuum, which deletes the files and with them the old versions. - Delta Lake stores one JSON per commit and checkpoints in `_delta_log/`; Iceberg, metadata, snapshots and manifests with a catalog that atomically swaps the pointer to the current version. - The GPU (RAPIDS, `cudf.pandas`, Polars with `engine="gpu"`) speeds up massive aggregations that fit in its memory; it requires NVIDIA Volta or later, CUDA 12 or 13 and Linux or WSL2, and before buying one you have to measure whether the bottleneck is computation. ## Resources ### Documentation - [Amazon S3: user guide (consistency, prefixes, ranges)](https://docs.aws.amazon.com/AmazonS3/latest/userguide/Welcome.html) · docs.aws.amazon.com - [Amazon S3: pricing](https://aws.amazon.com/s3/pricing/) · aws.amazon.com - [Delta Lake: transaction log protocol](https://github.com/delta-io/delta/blob/master/PROTOCOL.md) · github.com - [Apache Iceberg: table specification](https://iceberg.apache.org/spec/) · iceberg.apache.org - [PyIceberg: Python API](https://py.iceberg.apache.org/api/) · py.iceberg.apache.org - [NVIDIA RAPIDS: cudf.pandas](https://docs.rapids.ai/api/cudf/stable/cudf_pandas/) · docs.rapids.ai - [Polars: GPU support](https://docs.pola.rs/user-guide/gpu-support/) · docs.pola.rs Previous lesson: [6.6.4 · Time series databases: TimescaleDB, QuestDB, ClickHouse, ArcticDB and kdb+](https://pythonparatrading.com/en/python-trading-course/level-6/time-series-databases-and-arcticdb/) Next lesson: [6.7.1 · Overview and size of the alternative data market](https://pythonparatrading.com/en/python-trading-course/level-6/alternative-data-market-overview/)