LEAN/QuantConnect and Zipline Reloaded; bt for asset allocation.

Three engines, three philosophies

Lessons 2.4.1 to 2.4.5 went from the simplest (a vectorized backtest) to the most complete (NautilusTrader). The three engines in this lesson fill three other gaps:

Licenses and dates come from PyPI and GitHub as of the lesson date (links in the resources). All three are maintained: Zipline Reloaded by Stefan Jansen and the community since Quantopian closed, LEAN by QuantConnect and bt by its author (pmorissette on GitHub) and contributors.

This lesson runs in a separate environment: Zipline Reloaded 3.2.0 requires pandas < 3, and the rest of the course uses pandas 3. The environment (curso-code/entornos/zipline, with pandas 2.3.3 and numpy 2.4.6) is created with uv sync --project curso-code/entornos/zipline; LEAN and bt are not installed, and their code goes in blocks that are not run.

Inside Zipline

flowchart TB
  B[Bundle] --> DP[DataPortal]
  B --> PE[Pipeline]
  PE --> BT[Before the session]
  DP --> AL[Algorithm]
  BT --> AL
  AL --> BL[Blotter]
  BL --> SL[Slippage]
  BL --> CM[Commission]
  SL --> MT[Metrics]
  CM --> MT
  MT --> PF[perf DataFrame]

• Bundle. The data, ingested once into a local directory (ZIPLINE_ROOT): daily or minute bars in a columnar format, a SQLite database of assets (symbol, lifetime dates, exchange) and another of adjustments (splits, mergers, dividends).
• DataPortal. Serves the algorithm the prices of each moment, adjusted with what was known that day.
• Pipeline. Computes factors over the whole universe every day, before the open, with windows adjusted point in time. Its result reaches before_trading_start.
• Algorithm. Functions that Zipline calls: initialize (once), before_trading_start (every day), handle_data (every bar) and the ones scheduled with schedule_function.
• Blotter. Receives the orders and executes them on the next bar, with the slippage model (which can cap participation in volume and leave orders partially filled) and the commission model.
• Metrics. At the end, a DataFrame with one row per session: portfolio value, return, exposure, positions, orders and transactions.

A custom bundle with the level 2 data

Zipline's built-in bundles download data from the internet. For the course, a custom bundle with an ingest function writes the 120 stocks of the level 2 universe (synthetic, with companies that go public, go bankrupt or are acquired) from the Parquet files: unadjusted prices, and separately the splits and the dividends, so that Zipline applies the adjustments itself.

import os
import tempfile
import warnings

import numpy as np
import pandas as pd

# zipline 3.2.0 with pandas 2.3 emits internal FutureWarnings (in
# asset_writer.py and benchmark_source.py): kept out of the output
warnings.filterwarnings("ignore", category=FutureWarning,
                        module="zipline")
os.environ.setdefault("ZIPLINE_ROOT",
                      tempfile.mkdtemp(prefix="zipline-"))

from zipline.data.bundles import ingest, load, register

precios = pd.read_parquet("universo2_precios.parquet")
eventos = pd.read_csv("universo2_eventos.csv", parse_dates=["fecha"])
divs = pd.read_csv("universo2_dividendos.csv",
                   parse_dates=["fecha_anuncio", "fecha_ex",
                                "fecha_pago"])
simbolos = sorted(precios["ticker"].unique())
sid = {s: i for i, s in enumerate(simbolos)}
vida = precios.groupby("ticker")["fecha"].agg(["min", "max"])


def ingest_universo(environ, asset_db_writer, minute_bar_writer,
                    daily_bar_writer, adjustment_writer, calendar,
                    start_session, end_session, cache, show_progress,
                    output_dir):
    def barras():
        for s, g in precios.groupby("ticker"):
            c = g.set_index("fecha")["cierre"]
            # there is only a close: open, high and low = close
            yield sid[s], pd.DataFrame({
                "open": c, "high": c, "low": c, "close": c,
                "volume": g.set_index("fecha")["volumen"]
                .astype(float)})
    daily_bar_writer.write(barras(), show_progress=False)
    asset_db_writer.write(
        equities=pd.DataFrame({
            "symbol": simbolos, "exchange": "CURSO",
            "start_date": vida.loc[simbolos, "min"].values,
            "end_date": vida.loc[simbolos, "max"].values,
            "auto_close_date": (vida.loc[simbolos, "max"]
                                + pd.Timedelta(days=1)).values}),
        exchanges=pd.DataFrame({"exchange": ["CURSO"],
                                "canonical_name": ["CURSO"],
                                "country_code": ["US"]}))
    sp = eventos[eventos["evento"] == "split"]
    adjustment_writer.write(
        splits=pd.DataFrame({
            "sid": sp["ticker"].map(sid),
            "effective_date": sp["fecha"],
            "ratio": 1 / sp["ratio"]}),        # 2-for-1 -> 0.5
        dividends=pd.DataFrame({
            "sid": divs["ticker"].map(sid),
            "ex_date": divs["fecha_ex"],
            "declared_date": divs["fecha_anuncio"],
            "record_date": divs["fecha_ex"],
            "pay_date": divs["fecha_pago"],
            "amount": divs["importe"]}))


sesiones = pd.DatetimeIndex(sorted(precios["fecha"].unique()))
register("universo2", ingest_universo, calendar_name="XNYS",
         start_session=sesiones[0], end_session=sesiones[-1]);
ingest("universo2", show_progress=False)
n_splits = (eventos["evento"] == "split").sum()
print(len(simbolos), "stocks ·", n_splits, "splits ·",
      len(divs), "dividends")

120 stocks · 9 splits · 7029 dividends

Three details that cost hours if you do not know them (they are in the README of the course environment):

• country_code="US" in the exchanges table. Without it, Pipeline finds no assets for its default domain, the US one.
• The ratio of a split is the factor by which earlier prices are multiplied: a 2-for-1 split is 0.5.
• A single register per name and the ; at the end of the line: register returns the function, and without the semicolon the environment would print its memory address.

auto_close_date is the day on which Zipline closes the positions in a stock that stops trading, at the last known price. For a bankruptcy, that price is between 0.30 and 1.50 USD (level 2 README): almost the −100% of the delisting return, but not exactly.

The bundle stores the adjustments and Zipline applies them when needed. You can check it by reading the bundle:

datos = load("universo2")
qxxc = datos.asset_finder.lookup_symbol("QXXC", as_of_date=None)
print(qxxc, "·", qxxc.start_date.date(), "to", qxxc.end_date.date())
splits = datos.adjustment_reader.get_adjustments_for_sid(
    "splits", qxxc.sid)
print("splits:", [(pd.Timestamp(f).date(), r) for f, r in splits])
qt = datos.asset_finder.lookup_symbol("QXQT", as_of_date=None)
print(qt, "· stops trading:", qt.end_date.date(),
      "· auto close:", qt.auto_close_date.date())

Equity(102 [QXXC]) · 2006-01-03 to 2025-12-31
splits: [(datetime.date(2007, 8, 29), 0.5)]
Equity(79 [QXQT]) · stops trading: 2008-10-17 · auto close: 2008-10-18

Pipeline: the momentum factor

Pipeline computes a factor for all stocks every day with a data window adjusted to that date: if QXXC had a 2-for-1 split on August 29, 2007, the window for a day in September 2007 has the earlier prices divided by two, and the window for a day in July does not. That way there is no look-ahead in the adjustments.

The factor is the 12-1 momentum from lesson 2.3.1: the return over about twelve months skipping the last one. In the level 2 universe it has a planted premium: according to the data README, the expected rank IC is ≈ +0.03, the spread between quintiles ≈ 8% a year, and there are crashes in April-June 2009 and 2020, when the losers rebound.

from zipline.pipeline import CustomFactor, Pipeline
from zipline.pipeline.data import USEquityPricing
from zipline.pipeline.factors import AverageDollarVolume


class Momentum121(CustomFactor):
    """Return from t-252 to t-21 sessions."""
    inputs = [USEquityPricing.close]
    window_length = 253

    def compute(self, today, assets, out, close):
        out[:] = close[-22] / close[0] - 1


def hacer_pipeline():
    # the 100 most traded by dollar volume (21 sessions)
    universo = AverageDollarVolume(window_length=21).top(100)
    mom = Momentum121(mask=universo)
    return Pipeline(
        columns={"mom": mom, "quintil": mom.quantiles(5)},
        screen=universo)

close arrives as a matrix of 253 rows (days) by as many columns as there are stocks: compute writes one number per stock into out. The universe is set every day with the dollar volume of the last 21 sessions, so it includes the stocks that will later go bankrupt and excludes the ones that have not yet gone public: the bundle has the whole history, including that of the delisted stocks, and that is why there is no survivorship bias (lesson 1.4.4).

The strategy, with and without costs

On the first trading day of each month, the strategy buys the winner quintile and sells short the loser quintile, with 50% of the capital in each leg and equal weights, and closes the stocks that leave the extremes. With daily data the time in time_rules does not matter: the order is sent that day and executed at the close of the next session. Two versions:

• Without costs: NoCommission and NoSlippage.
• With costs: PerShare(cost=0.005, min_trade_cost=1.0), the per-share fee of Alcor from lesson 2.4.7, and FixedBasisPointsSlippage(basis_points=5, volume_limit=0.1): 5 bp against you on each fill and, at most, 10% of the day's volume; whatever does not fit stays pending for the next day (a partial fill, lesson 2.4.8).

from zipline import run_algorithm
from zipline.api import (attach_pipeline, date_rules,
                         order_target_percent, pipeline_output, record,
                         schedule_function, set_commission,
                         set_slippage, time_rules)
from zipline.finance import commission, slippage


def hacer_algoritmo(costes):
    def initialize(context):
        attach_pipeline(hacer_pipeline(), "factores")
        schedule_function(rebalanceo, date_rules.month_start(),
                          time_rules.market_open(minutes=30))
        if costes:
            set_commission(commission.PerShare(
                cost=0.005, min_trade_cost=1.0))
            set_slippage(slippage.FixedBasisPointsSlippage(
                basis_points=5, volume_limit=0.1))
        else:
            set_commission(commission.NoCommission())
            set_slippage(slippage.NoSlippage())

    def before_trading_start(context, data):
        context.f = pipeline_output("factores")

    def rebalanceo(context, data):
        f = context.f
        largos = f.index[f["quintil"] == 4]
        cortos = f.index[f["quintil"] == 0]
        for a in context.portfolio.positions:
            if (a not in largos and a not in cortos
                    and data.can_trade(a)):
                order_target_percent(a, 0)
        for a in largos:
            if data.can_trade(a):
                order_target_percent(a, 0.5 / len(largos))
        for a in cortos:
            if data.can_trade(a):
                order_target_percent(a, -0.5 / len(cortos))
        record(largos=len(largos), cortos=len(cortos))

    return dict(initialize=initialize,
                before_trading_start=before_trading_start)

set_benchmark is not called: without it, Zipline 3.2.0 does not compare against any index. (Careful: set_benchmark(False) does not disable the benchmark; it takes the asset with sid 0, here QXAC, which stops trading in 2007, and the backtest fails.) The two runs, from 2015 to 2025 so that the chapter takes no more than a minute and a half:

resultados = {}
for nombre, costes in [("bruto", False), ("neto", True)]:
    resultados[nombre] = run_algorithm(
        start=pd.Timestamp("2015-01-02"),
        end=pd.Timestamp("2025-12-31"),
        capital_base=1_000_000, data_frequency="daily",
        bundle="universo2", **hacer_algoritmo(costes))
perf = resultados["neto"]
print(perf.shape)
perf[["portfolio_value", "long_exposure", "short_exposure",
      "largos", "cortos"]].iloc[[30, -1]].round(0)

(2766, 39)
                           portfolio_value  long_exposure  short_exposure  largos  cortos
2015-02-17 21:00:00+00:00         999410.0       502812.0       -504200.0      20      20
2025-12-31 21:00:00+00:00        1471150.0       774988.0       -753217.0      20      20

perf has one row per session. In February 2015 the portfolio holds about twenty positions per side, half a million long and half a million short; the orders, transactions and positions columns store, day by day, lists of dictionaries with each order, fill and position.

Results

def resumen(p):
    r = p["returns"]
    años = len(r) / 252
    total = p["portfolio_value"].iloc[-1] / 1_000_000 - 1
    curva = p["portfolio_value"]
    return {
        "total_pct": 100 * total,
        "anual_pct": 100 * ((1 + total) ** (1 / años) - 1),
        "vol_pct": 100 * r.std() * np.sqrt(252),
        "sharpe": r.mean() / r.std() * np.sqrt(252),
        "max_dd_pct": 100 * (curva / curva.cummax() - 1).min(),
    }

tabla = pd.DataFrame({k: resumen(v) for k, v in resultados.items()})
trans = [t for dia in perf["transactions"] for t in dia]
# the commission lives in the orders, cumulative: the last version
ordenes = {o["id"]: o for dia in perf["orders"] for o in dia}
comision = sum(o["commission"] or 0 for o in ordenes.values())
print(len(trans), "transactions ·", len(ordenes), "orders ·",
      f"commissions {comision:,.0f} USD")
tabla.round(2)

6408 transactions · 6406 orders · commissions 18,056 USD
            bruto   neto
total_pct   54.40  47.12
anual_pct    4.04   3.58
vol_pct      6.11   6.11
sharpe       0.68   0.61
max_dd_pct -19.08 -19.45

The header figure shows the two curves and this one, the result of each year. Three readings:

• The planted premium shows up, with the shape of momentum: good years (+17.7% in 2019, +24.7% in 2021), flat years and a sharp drop in 2020 (−13.3%). The spread between quintiles that the README states (≈ 8% a year) is split here into two legs of 50% of the capital, so half of it is expected per year, minus the crashes: 3.6% a year net, with a volatility of 6.1% and a Sharpe of 0.61.
• Costs take the difference between the two curves: 1.54 versus 1.47 million, about 73,000 USD. The commission, 18,056 USD over 6,406 orders, is only a quarter of it; the rest is the 5 bp of slippage per side on each rebalance, which shows up in the price of each transaction and not in a separate column. Each year costs subtract between 0.4 and 0.6 points.
• Costs are missing. Zipline does not charge for borrowing the stocks sold short nor pay interest on the cash from the sales, and it does not prevent short selling stocks with no availability. With the data in costes2_prestamo.csv, lesson 2.4.7 measures how much that changes the result.

The data is synthetic and the premium is planted on purpose: the backtest shows that the engine and the factor recover it, not that momentum works in a real market.

LEAN and QuantConnect

LEAN is the open-source engine (Apache-2.0) behind QuantConnect. It is written in C# and accepts algorithms in C# or in Python; the same algorithm runs in backtest, paper and live with the brokers it integrates. There are two ways to use it: in the QuantConnect cloud, with its data, or locally with the LEAN CLI.

flowchart TB
  CLI[lean CLI] --> DK[Docker container]
  DK --> EN[LEAN engine]
  DT[Local data] --> EN
  EN --> AM[Algorithm Manager]
  AM --> QA[QCAlgorithm]
  QA --> TH[Orders and portfolio]
  TH --> BR[Broker or simulator]
  EN --> RS[Results]

According to its documentation, the commands that run the engine locally (lean backtest, lean optimize, lean research) do so inside the quantconnect/lean Docker image, which contains the engine and the QuantConnect libraries; without Docker installed and running, it cannot be done. This course does not run it (Docker is not in its environment), so the code has no output.

pip install --upgrade lean          # the CLI is a Python package
lean init                           # creates the working folder and the data folder
lean create-project "Momentum sectores" --language python
lean backtest "Momentum sectores"   # downloads the image the first time

The algorithm is a subclass of QCAlgorithm. This is the same 12-1 momentum on nine real US sector ETFs (in the QuantConnect cloud its data is used; locally you need to have it in LEAN's format), with the lowercase Python API that the current documentation uses:

from AlgorithmImports import *


class MomentumSectores(QCAlgorithm):

    def initialize(self) -> None:
        self.set_start_date(2015, 1, 1)
        self.set_end_date(2025, 12, 31)
        self.set_cash(1_000_000)
        tickers = ["XLB", "XLE", "XLF", "XLI", "XLK",
                   "XLP", "XLU", "XLV", "XLY"]
        self._simbolos = []
        for t in tickers:
            sec = self.add_equity(t, Resolution.DAILY)
            sec.set_fee_model(InteractiveBrokersFeeModel())
            sec.set_slippage_model(VolumeShareSlippageModel())
            self._simbolos.append(sec.symbol)
        self.schedule.on(
            self.date_rules.month_start("XLK"),
            self.time_rules.after_market_open("XLK", 30),
            self._rebalanceo)

    def _rebalanceo(self) -> None:
        hist = self.history(self._simbolos, 253, Resolution.DAILY)
        cierres = hist.close.unstack(0).dropna(axis=1)
        if len(cierres) < 253:
            return
        mom = cierres.iloc[-22] / cierres.iloc[0] - 1
        ganadores = mom.nlargest(3).index
        for s in self._simbolos:
            peso = 1 / 3 if s in ganadores else 0
            self.set_holdings(s, peso)

The pieces map to Zipline's: initialize configures, add_equity registers the asset and its resolution, set_fee_model and set_slippage_model set the asset's cost models (InteractiveBrokersFeeModel models that broker's fee schedule), schedule.on schedules the rebalance and history provides the price window. lean backtest stores the results in a dated folder inside backtests/ in the project. Unlike Zipline, going live requires no rewrite: lean live deploy launches the same algorithm against a broker, again in a Docker container with the quantconnect/lean image. In exchange, the engine is a large system in another language, and debugging what happens inside requires reading C#.

bt for asset allocation

bt solves a narrower problem: portfolios that are rebalanced periodically with selection and weighting rules. A strategy is a stack of algos, small functions that run in order at each date and stop as soon as one returns False (for example, RunMonthly on a day that is not a rebalance day). The one below is a naive risk parity (inverse-volatility weights) across five level 2 continuous futures, one per asset class:

import bt
import pandas as pd

fut = pd.read_parquet("futuros2_continuos.parquet")
precios = (fut.pivot(index="fecha", columns="mercado",
                     values="cierre_ajustado")
           [["IDA", "BOC", "ENA", "MEA", "DIA"]].dropna())

paridad = bt.Strategy("paridad_riesgo", [
    bt.algos.RunMonthly(),
    bt.algos.SelectAll(),
    bt.algos.WeighInvVol(lookback=pd.DateOffset(months=3)),
    bt.algos.Rebalance(),
])
equiponderada = bt.Strategy("equiponderada", [
    bt.algos.RunMonthly(),
    bt.algos.SelectAll(),
    bt.algos.WeighEqually(),
    bt.algos.Rebalance(),
])
coste = lambda cantidad, precio: 0.0005 * abs(cantidad * precio)
res = bt.run(bt.Backtest(paridad, precios, commissions=coste),
             bt.Backtest(equiponderada, precios, commissions=coste))
res.display()

bt.run returns a results object with statistics (display), curves and the weights at each date. The pieces are reusable and can be nested: a strategy can itself be an asset of another, which allows portfolios of strategies. What bt lacks is what Zipline, LEAN or NautilusTrader have: orders, an order book, latency, a path to live trading. Nor does it model the futures roll or their margins: here it is given adjusted continuous prices (lesson 1.4.6) as if they were spot assets. For comparing allocation rules it is convenient and fast; for execution, it is not enough.

Which one to choose

flowchart TB
  P{What is being tested} --> F[Factors, stocks]
  P --> A[Allocation]
  P --> R[Going live]
  F --> ZL[Zipline Reloaded]
  A --> BTB[bt or custom engine]
  R --> NL{Which language}
  NL -- Python and Rust --> NT[NautilusTrader]
  NL -- C# or Python --> LE[LEAN]

No engine fixes a bad hypothesis or a bias in the data. What changes between them is which errors they make hard (Zipline: adjustments and look-ahead; NautilusTrader and LEAN: the gap between simulation and live) and which assumptions they leave to the user (in all three: securities borrowing, the queue and market impact).

Exercises

1. With perf (the version with costs), compute the turnover: how many shares are bought and sold in total, the average commission per transaction and what share of the average capital turns over each year.

t = pd.DataFrame(trans)
efectivo = (t["amount"].abs() * t["price"]).sum()
años = len(perf) / 252
print(f"shares traded: {t['amount'].abs().sum():,.0f}")
print(f"average commission per order: {comision / len(ordenes):.2f} USD")
print(f"turnover: {efectivo / años / 1_000_000:.1f} times"
      " the initial capital per year")

shares traded: 3,061,155
average commission per order: 2.82 USD
turnover: 8.0 times the initial capital per year

2. The level 2 README says that in April, May and June 2020 the momentum coefficient changes sign (the losers rebound). Check in perf what happened in those months in the version with costs.

mensual = (1 + perf["returns"]).resample("ME").prod() - 1
(100 * mensual.loc["2020-02":"2020-08"]).round(2)

2020-02-29 00:00:00+00:00    1.61
2020-03-31 00:00:00+00:00   -5.08
2020-04-30 00:00:00+00:00   -1.85
2020-05-31 00:00:00+00:00    0.21
2020-06-30 00:00:00+00:00   -2.89
2020-07-31 00:00:00+00:00   -2.58
2020-08-31 00:00:00+00:00    2.20
Freq: ME, Name: returns, dtype: float64

3. You want to take this same strategy to LEAN with real US stock data. List three things from the Zipline backtest that you would have to redo or check.