Polars: lazy versus eager API, expressions, performance and when to pick it over pandas.

Another way to work with tables

pandas was born in 2008 on top of NumPy and keeps decisions from that time: an index on every table, operations that run one by one in the order they are written and, for the most part, a single thread. Polars, written in Rust and using the Apache Arrow in-memory format, starts from different ideas:

The course uses Polars 2.0, released in October 202611Note 1 · Polars 2.0.0 on PyPI (pypi.org/project/polars/2.0.0), released on October 6, 2026; accessed on October 21, 2026. (version pinned in curso-code/uv.lock). The functions that appear in this lesson already existed in the 1.x series: what you learn here also applies to somewhat older code.

import polars as pl

df = pl.read_csv("sint_diario.csv", try_parse_dates=True)
print(pl.__version__, df.shape)
print(df.schema)
df.select("fecha", "cierre", "volumen").head(3)

2.0.0 (1303, 6)
Schema({'fecha': Date, 'apertura': Float64, 'maximo': Float64, 'minimo': Float64, 'cierre': Float64, 'volumen': Int64})
shape: (3, 3)
┌────────────┬────────┬─────────┐
│ fecha      ┆ cierre ┆ volumen │
│ ---        ┆ ---    ┆ ---     │
│ date       ┆ f64    ┆ i64     │
╞════════════╪════════╪═════════╡
│ 2021-01-04 ┆ 99.14  ┆ 2322609 │
│ 2021-01-05 ┆ 99.25  ┆ 1517954 │
│ 2021-01-06 ┆ 97.38  ┆ 3245658 │
└────────────┴────────┴─────────┘

try_parse_dates=True recognizes the date column, which becomes of type Date (a day with no time). There is no index: the table has six columns, and fecha is one of them. Rows are selected by position or with filters, never by label.

Expressions and contexts

The central piece of Polars is the expression: a description of a computation on columns that has not run yet. pl.col("cierre") is "the cierre column", and pl.col("cierre").pct_change() is "the change of the cierre column relative to the previous row". An expression is evaluated in a context:

r = pl.col("cierre").pct_change()
print(r)

barras = df.with_columns(
    r=r,
    rango=(pl.col("maximo") - pl.col("minimo")) / pl.col("cierre"),
)
barras.select("fecha", "cierre", "r", "rango").head(3)

col("cierre").pct_change([dyn int: 1])
shape: (3, 4)
┌────────────┬────────┬───────────┬──────────┐
│ fecha      ┆ cierre ┆ r         ┆ rango    │
│ ---        ┆ ---    ┆ ---       ┆ ---      │
│ date       ┆ f64    ┆ f64       ┆ f64      │
╞════════════╪════════╪═══════════╪══════════╡
│ 2021-01-04 ┆ 99.14  ┆ null      ┆ 0.015836 │
│ 2021-01-05 ┆ 99.25  ┆ 0.00111   ┆ 0.011688 │
│ 2021-01-06 ┆ 97.38  ┆ -0.018841 ┆ 0.019203 │
└────────────┴────────┴───────────┴──────────┘

Printed, r is only the description of the computation. It can be stored in a variable and reused in several queries: it is the same idea as pandas 3.0's pd.col() (lesson 0.5.2). The first return comes out null, the Polars missing value, which exists in the same way in columns of numbers, text and dates.

Inside a context, expressions are evaluated in parallel and each one produces a column. That is why each result needs its own name: two expressions on r without alias would raise a duplicate name error.

caidas = barras.filter(
    (pl.col("r") < -0.03)
    & (pl.col("volumen") > pl.col("volumen").median())
)
print(caidas.height, "sharp drops on high volume")

barras.select(
    media=pl.col("r").mean(),
    vol_anual=pl.col("r").std() * 252 ** 0.5,
    sesiones=pl.col("r").count(),
)

10 sharp drops on high volume
shape: (1, 3)
┌──────────┬───────────┬──────────┐
│ media    ┆ vol_anual ┆ sesiones │
│ ---      ┆ ---       ┆ ---      │
│ f64      ┆ f64       ┆ u32      │
╞══════════╪═══════════╪══════════╡
│ 0.000342 ┆ 0.180962  ┆ 1302     │
└──────────┴───────────┴──────────┘

The ten sessions are the same as in lesson 0.5.3, and the volume median is computed inside the filter without auxiliary variables. count() does not count null values: 1,302 returns from 1,303 closes.

group_by and windows

group_by works as in pandas, with the key also written as an expression. The result of a group_by does not keep the order of the groups (they are computed in parallel): for a readable table, sort it with sort or ask for maintain_order=True.

por_ano = (
    barras.group_by(pl.col("fecha").dt.year().alias("ano"))
    .agg(
        n=pl.col("r").count(),
        anual=(1 + pl.col("r")).product() - 1,
        vol=pl.col("r").std() * 252 ** 0.5,
    )
    .sort("ano")
)
por_ano.with_columns(pl.col("anual", "vol").round(4))

shape: (5, 4)
┌──────┬─────┬────────┬────────┐
│ ano  ┆ n   ┆ anual  ┆ vol    │
│ ---  ┆ --- ┆ ---    ┆ ---    │
│ i32  ┆ u32 ┆ f64    ┆ f64    │
╞══════╪═════╪════════╪════════╡
│ 2021 ┆ 259 ┆ 0.152  ┆ 0.1599 │
│ 2022 ┆ 260 ┆ -0.085 ┆ 0.2483 │
│ 2023 ┆ 260 ┆ 0.1421 ┆ 0.1495 │
│ 2024 ┆ 262 ┆ 0.1432 ┆ 0.1499 │
│ 2025 ┆ 261 ┆ 0.0424 ┆ 0.1792 │
└──────┴─────┴────────┴────────┘

The figures match the pandas ones in lesson 0.5.3: −8.5 % in 2022 with a volatility of 24.8 %. pl.col accepts several names at once, and the expression is applied to each column.

Rolling and cumulative windows are methods of expressions. The 21-session volatility and the drawdown, in a single context:

riesgo = barras.with_columns(
    vol_21=pl.col("r").rolling_std(21) * 252 ** 0.5,
    dd=pl.col("cierre") / pl.col("cierre").cum_max() - 1,
)
riesgo.select(
    vol_max=pl.col("vol_21").max(),
    sin_ventana=pl.col("vol_21").null_count(),
    peor_dd=pl.col("dd").min(),
    fecha_fondo=pl.col("fecha").get(pl.col("dd").arg_min()),
)

shape: (1, 4)
┌──────────┬─────────────┬───────────┬─────────────┐
│ vol_max  ┆ sin_ventana ┆ peor_dd   ┆ fecha_fondo │
│ ---      ┆ ---         ┆ ---       ┆ ---         │
│ f64      ┆ u32         ┆ f64       ┆ date        │
╞══════════╪═════════════╪═══════════╪═════════════╡
│ 0.339454 ┆ 21          ┆ -0.250278 ┆ 2022-10-20  │
└──────────┴─────────────┴───────────┴─────────────┘

Maximum volatility of 33.9 %, 21 values without a full window and a 25 % drawdown with its trough on October 20, 2022: the same numbers as in pandas. Polars has no index, so "the date of the minimum" is written as an expression: the date at the position of the drawdown minimum.

Per-group computations that keep all rows use over, the equivalent of SQL window functions (lesson 0.5.5). In the trade log, the weight of each trade in the result of its symbol:

ops = pl.read_csv("operaciones_sint.csv", try_parse_dates=True)
signo = pl.when(pl.col("lado") == "largo").then(1).otherwise(-1)
pnl = (signo * (pl.col("precio_salida") - pl.col("precio_entrada"))
       * pl.col("cantidad") - pl.col("comision"))
ops = ops.with_columns(pnl=pnl).with_columns(
    n_simbolo=pl.len().over("simbolo"),
    total_simbolo=pl.col("pnl").sum().over("simbolo"),
)
ops.select("id", "simbolo", "pnl", "n_simbolo", "total_simbolo").head(4)

shape: (4, 5)
┌─────┬─────────┬──────────┬───────────┬───────────────┐
│ id  ┆ simbolo ┆ pnl      ┆ n_simbolo ┆ total_simbolo │
│ --- ┆ ---     ┆ ---      ┆ ---       ┆ ---           │
│ i64 ┆ str     ┆ f64      ┆ u32       ┆ f64           │
╞═════╪═════════╪══════════╪═══════════╪═══════════════╡
│ 1   ┆ ORO     ┆ -3123.56 ┆ 3         ┆ -3237.96      │
│ 2   ┆ SINT    ┆ 7.43     ┆ 30        ┆ 4405.08       │
│ 3   ┆ SINT    ┆ -31.5    ┆ 30        ┆ 4405.08       │
│ 4   ┆ SINT    ┆ -306.72  ┆ 30        ┆ 4405.08       │
└─────┴─────────┴──────────┴───────────┴───────────────┘

pl.when(...).then(...).otherwise(...) is the np.where of expressions. With over("simbolo"), the sum is computed within each symbol and repeated on all its rows: trade 1, on ORO, sees the total of the three ORO trades. With group_by there would be one row per symbol; with over, all 40 are kept.

Lazy versus eager

Everything above is eager: each with_columns or filter runs immediately and returns a DataFrame. The lazy API changes the order: scan_parquet (or scan_csv, or df.lazy()) returns a LazyFrame, which only accumulates the query. Nothing is read or computed until collect(), and before running, the optimizer rewrites the whole plan:

flowchart LR
  A[scan_parquet] --> B[filter]
  B --> C[with_columns]
  C --> D[collect]

consulta = (
    pl.scan_parquet("sint_diario.parquet")
    .filter(pl.col("fecha").dt.year() == 2022)
    .with_columns(r=pl.col("cierre").pct_change())
    .select("fecha", "r")
)
print(type(consulta).__name__)
print(consulta.explain())

LazyFrame
simple π 2/2 ["fecha", "r"]
   WITH_COLUMNS:
   [col("cierre").pct_change([dyn int: 1]).alias("r")]
    Parquet SCAN [sint_diario.parquet]
    PROJECT 2/6 COLUMNS
    SELECTION: (col("fecha").dt.year() == 2022)
    ESTIMATED ROWS: 1303

The plan is read from bottom to top. The optimizer has done two things that are not possible in eager mode:

• Projection pushdown (PROJECT 2/6 COLUMNS): the query only uses fecha and cierre, so the Parquet reader does not load the other four columns from disk.
• Predicate pushdown (SELECTION): the year condition is applied while the file is read, and Parquet can skip whole blocks that do not meet the condition thanks to the statistics it stores for each one.

With a 50 KB file you will not notice. With years of one-minute bars for hundreds of symbols, the difference is between reading a few hundred megabytes and reading tens of gigabytes. This plan, however, has a problem that is not about performance:

antes = consulta.collect()
despues = (
    pl.scan_parquet("sint_diario.parquet")
    .with_columns(r=pl.col("cierre").pct_change())
    .filter(pl.col("fecha").dt.year() == 2022)
    .select("fecha", "r")
    .collect()
)
print(antes.head(2))
print(despues.head(2))

shape: (2, 2)
┌─────────────────────┬───────────┐
│ fecha               ┆ r         │
│ ---                 ┆ ---       │
│ datetime[μs]        ┆ f64       │
╞═════════════════════╪═══════════╡
│ 2022-01-03 00:00:00 ┆ null      │
│ 2022-01-04 00:00:00 ┆ -0.017069 │
└─────────────────────┴───────────┘
shape: (2, 2)
┌─────────────────────┬───────────┐
│ fecha               ┆ r         │
│ ---                 ┆ ---       │
│ datetime[μs]        ┆ f64       │
╞═════════════════════╪═══════════╡
│ 2022-01-03 00:00:00 ┆ 0.000263  │
│ 2022-01-04 00:00:00 ┆ -0.017069 │
└─────────────────────┴───────────┘

In the first query, the filter is applied before pct_change: January 3, 2022 no longer has the December 31 close and its return comes out null. In the second, the return is computed with the whole series and filtered afterwards: January 3 has its 0.03 %. The optimizer does not change the order in this case, because pct_change depends on the previous row and the result would be different; writing the correct order is the responsibility of whoever writes the query. The mistake in the opposite direction is more serious: any computation that looks backwards (shift, windows, cumulative maxima) must be done with the full history before trimming the period of a backtest.

When should you use each mode? Eager is convenient for exploring in a notebook: each step shows its result. Lazy pays off as soon as the data comes from large files or the query has several steps, and it is the natural mode for scripts that process data periodically (the daily bar download, the nightly signal recalculation). A lazy query ends in collect(), which returns a DataFrame, or in sink_parquet("salida.parquet"), which writes the result to disk without gathering it all in memory. And df.lazy() turns an already loaded DataFrame into a LazyFrame, to take advantage of the optimizer in a long chain of operations.

fecha appears as datetime[μs] and not as Date: the data's Parquet file was saved by pandas, which writes dates with a time. Reading the same data from two sources and comparing the types is a check that prevents joins that do not match.

Performance without a stopwatch

This lesson gives no timings: they depend on the machine, the version and the data size, and those of a 1,303-row file say nothing. What can be stated is why Polars is usually faster with large data, and when it stops being so:

• Expressions run in Rust, on several threads and on contiguous Arrow columns, without a Python object per value.
• In lazy mode, the optimizer removes unnecessary columns and rows before reading them and reuses common subexpressions.
• With the streaming engine, a lazy query processes files larger than memory in chunks.
• A Python function inside an expression (map_elements) hands the computation back to Python row by row and cancels almost all the advantages. If an operation does not exist as an expression, there is usually a combination of expressions that does it.

Polars and pandas together

Most projects use both. Conversions are direct, with two details: pandas needs an index and date types change resolution.

pdf = df.to_pandas().set_index("fecha")
print(type(pdf).__name__, pdf.index.dtype, pdf.shape)
de_vuelta = pl.from_pandas(pdf, include_index=True)
print(de_vuelta.schema["fecha"], de_vuelta.shape)

DataFrame datetime64[ms] (1303, 5)
Datetime(time_unit='ms', time_zone=None) (1303, 6)

The Polars Date reaches pandas as datetime64[ms] and comes back as Datetime, not as Date. If the rest of the code compares dates across tables, it is worth normalizing the type in a single place (for example, with pl.col("fecha").cast(pl.Date)). The conversion uses Arrow under the hood and, for numeric types, it is fast; it is not free with large tables, so it is done once, at the boundary between the two parts of the code.

Lesson 0.5.5 adds a third option for the same queries: SQL on Parquet with DuckDB.

Exercises

1. With group_by_dynamic("fecha", every="1mo"), compute the last close of each month for SINT and its monthly return. Compare the label of each month with that of pandas resample("ME") (lesson 0.5.3). What risk does that label carry?

mensual = (
    df.group_by_dynamic("fecha", every="1mo")
    .agg(pl.col("cierre").last())
    .with_columns(r=pl.col("cierre").pct_change())
)
print(mensual.height)
mensual.head(3)

60
shape: (3, 3)
┌────────────┬────────┬──────────┐
│ fecha      ┆ cierre ┆ r        │
│ ---        ┆ ---    ┆ ---      │
│ date       ┆ f64    ┆ f64      │
╞════════════╪════════╪══════════╡
│ 2021-01-01 ┆ 95.99  ┆ null     │
│ 2021-02-01 ┆ 97.65  ┆ 0.017293 │
│ 2021-03-01 ┆ 103.27 ┆ 0.057552 │
└────────────┴────────┴──────────┘

2. Write a lazy query on sint_diario.parquet that returns the annual return of each year (compounded from the daily returns, computed with the whole series) and check with explain() how many columns are read from the file.

anual = (
    pl.scan_parquet("sint_diario.parquet")
    .with_columns(r=pl.col("cierre").pct_change())
    .group_by(pl.col("fecha").dt.year().alias("ano"))
    .agg(anual=(1 + pl.col("r")).product() - 1)
    .sort("ano")
)
print("PROJECT 2/6" in anual.explain())
anual.collect().with_columns(pl.col("anual").round(4))

True
shape: (5, 2)
┌──────┬────────┐
│ ano  ┆ anual  │
│ ---  ┆ ---    │
│ i32  ┆ f64    │
╞══════╪════════╡
│ 2021 ┆ 0.152  │
│ 2022 ┆ -0.085 │
│ 2023 ┆ 0.1421 │
│ 2024 ┆ 0.1432 │
│ 2025 ┆ 0.0424 │
└──────┴────────┘

3. With ops (which already has pnl), compute for each trade the difference between its result and the mean of the trades on the same symbol, without losing rows. Show the three TECH trades.

(
    ops.with_columns(
        frente_media=pl.col("pnl")
        - pl.col("pnl").mean().over("simbolo"),
    )
    .filter(pl.col("simbolo") == "TECH")
    .select("id", "pnl", "frente_media")
)

shape: (3, 3)
┌─────┬─────────┬──────────────┐
│ id  ┆ pnl     ┆ frente_media │
│ --- ┆ ---     ┆ ---          │
│ i64 ┆ f64     ┆ f64          │
╞═════╪═════════╪══════════════╡
│ 10  ┆ 239.14  ┆ 250.813333   │
│ 18  ┆ -47.0   ┆ -35.326667   │
│ 38  ┆ -227.16 ┆ -215.486667  │
└─────┴─────────┴──────────────┘