In algorithmic trading there are countless options for carrying out your analyses or backtests, but we are going to use the best quant stack to build a professional research and backtesting pipeline.
What Was Quantopian?
Quantopian was an extremely ambitious project that set out to build a profitable crowd-sourced hedge fund following this philosophy:
from zipline.api import order_target_percent, record, symbol, schedule_function
from zipline import run_algorithm
from zipline.utils.events import date_rules, time_rules
import pandas as pd
def initialize(context):
"""Called once at the start of the backtest."""
context.spy = symbol("SPY")
context.ief = symbol("IEF")
schedule_function(rebalance,
date_rules.month_start(),
time_rules.market_open())
def rebalance(context, data):
"""Monthly rebalance: 60/40 portfolio."""
order_target_percent(context.spy, 0.60)
order_target_percent(context.ief, 0.40)
record(spy_weight=0.60, ief_weight=0.40)
# Run the backtest
start = pd.Timestamp("2020-01-01", tz="utc")
end = pd.Timestamp("2024-01-01", tz="utc")
results = run_algorithm(start=start, end=end,
initialize=initialize,
capital_base=10000,
bundle="quantopian-quandl")- Open Source: All the tools, as well as the underlying codebases, are free (which is the reason that, in 2023, it is in better shape than ever)
- Professional Infrastructure: Replicating proprietary software and deploying it in production, making the process more accessible and democratic, and simplifying the researcher's task down to pure research
- Education and Community: Training the community with high-quality content, covering both their own tools and finance or programming in general. They also built a highly reputable community that shed light on topics which had previously been very opaque and considered proprietary knowledge of the funds. (It was easy to spot Wall Street quants chiming in on a random thread).
This perfect mix steadily gained value and attracted funding
- https://www.institutionalinvestor.com/article/2bsvz8k1h4id41opnf4lc/portfolio/steven-cohens-crowdsourced-quant-experiment-gets-real
- https://www.reuters.com/article/hedgefunds-cohen-quantopian-idUSKCN1071XI
The ultimate goal was to create a hedge fund based on the community's ideas and generate returns.
They brought together all the great minds of quantitative finance in a healthy community, and even ran an annual in-person event whose excellent recordings can be found on YouTube, for example:
All this mix of variables had created an ecosystem where everything was perfect for the user. But perhaps not so much for the company…
The Great Fall of the Quantopian Empire
In the final phase of Quantopian, those of us researchers who worked actively on the platform had access to fundamental data from FactSet, a data catalog with a wealth of alternative real-time data, and they also opened the door to custom ingest bundle procedures, which let you process your own data and have it served back to you inside the pipeline.
The range of possibilities they opened up was staggering, and this is where the end began.
Quantopian cut the ability to take algorithms to production in a simulated-capital account overnight. They only let us work with real accounts. After speaking with them, I could summarize their justification as:
"We are spending X millions a year to build the best community ever imagined, and the returns of the models that make it to evaluation are pitiful."
Over time, they returned all the funds under client management and shut down the website completely. The founder went straight to Robinhood to become CTO and lead its entire technology side, leaving all of Quantopian's code open and suggesting that everyone make backups, because he was going to take it down for good. Actions that, in hindsight and after seeing Robinhood's business model, make a great deal more sense.
Especially now that we can look back on this event with perspective: whenever someone had real alpha, they left Quantopian and deployed it somewhere else. In short, on Quantopian everyone was working at zero cost—from research and modeling to backtesting and live deployment. When they found a source of alpha, they escaped Quantopian, creating a massive leak in its business model. The ultimate goal was to build a decentralized hedge fund that fused the alphas of all participants. But the reality was that once people had the alpha in their hands, they walked away… And that cannibalization of the business model ended up dying of success…
Back to 2023
Quantopian was fully abandoned in 2020, and the code is no longer officially maintained. Thanks to the existing community—a very broad community—people have taken one of the following paths to work around Quantopian's great shutdown.
- Find an alternative, such as QuantRocket or Quantopian.
- Maintain Quantopian so it remains compatible.
Both options are perfectly valid, but we are going to focus on the option of maintaining Quantopian in 2023. By choosing this solution we gain the following advantages:
- Total control: Over the platform, the backtest engine, absolutely everything—meaning there are no limits to deploying any kind of strategy
- Data Management: We are fully responsible for the data (and for the entire pipeline in general)
- Advanced Reporting. Beyond being able to do research and backtesting, we can also perform factor analysis to rigorously evaluate the validity of a model.
Choosing to maintain Quantopian gives us a solid foundation on which to build our projects and create technology with intellectual property.
Our Quant Stack
Now we will explain all the necessary libraries. Building a complete, robust, and stable pipeline is not a simple process, and we will need different tools to collaborate on each part of it. We are going to present the complete stack needed for quantitative trading, algorithmic trading, financial research, asset allocation, and so on…
Process Pipeline
An example of a Process Pipeline, known as the ML4T Workflow, shows what a minimal pipeline would look like to trade a Machine Learning-based model in a sensible way. Within many parts of the chain there are secondary pipelines, but broadly speaking the main pipeline follows a scheme very similar to this in a production environment. In the Research phase it is very similar, just omitting some non-critical parts of the process.
The Process Pipeline, or the "value chain," is the entire journey a piece of data takes from the moment it is generated until a signal is processed. This whole process could be compared to industrial production, where through the ingestion of inputs (in industry, raw materials) and the execution of various processes (in industry, the different production stages), outputs are obtained in the form of orders (which in industry would be the finished product).
We should also add position management—that is, verifying that the position behaves within the expected limits, and validating in real time the tests that confirm the model is still alive and that we are not caught in a random process with no edge whatsoever.
This has been a very superficial overview of some of the basic parts of the process pipeline, which we will gradually unpack. Step by step we will go deeper into the labyrinth to discover its limits and its possibilities.
Toolbox
The toolbox consists of the most relevant tools needed for each stage of the process. Beyond the usual pandas, numpy, matplotlib, scipy, and so on, we use other, more specialized libraries. With the exception of ArcticDB, whose owner is the Man Group asset manager—specialists in hedge funds—the rest come from the late Quantopian, and in 2023 they are more agile than ever!
- ArcticDB: Financial Datalake, a NoSQL database for every kind of data. Schemaless, versioned, cloud-by-definition via S3. Everything data-related is managed through this component.
- zipline-reloaded: Quantopian's backtester and research engine. A few small adjustments to the library's code are needed to make it functional in 2023.
- alphalens-reloaded: Factor research analysis. It automates the generation of the most relevant estimators and ratios for evaluating a factor against a set of assets.
- pyfolio-reloaded: Reporting tool. It automates the generation of reports on backtests, both at the single-asset level and at the portfolio level, for all kinds of assets.
- empyrical-reloaded: A tool that computes all the statistical ratios needed to evaluate the performance of strategies.
ArcticDB
With ArcticDB we are going to build our Datalake in the cloud. We can access and download data in dataframe format in record time, thanks to its petabyte-scale dataframe, making it a valid option for storing every kind of data, from order books to unaggregated tick data or monthly data.
Being NoSQL, it can store every kind of information and be extremely efficient at handling and processing all of it. Moreover, ArcticDB is a serverless solution, removing the need to maintain a server to sustain a data structure and pushing all the computational load onto the client making the request.
How Does ArcticDB Store Data?
It uses a key-value structure with the following layout.
Zipline
Zipline is a Python library used to backtest systems with the event-driven technique. It was developed for research, backtesting, and live trading on Quantopian. It is now maintained by the community, driven by Stefan Jansen.
With Zipline, we can backtest anything we can program. What's more, Andreas Clenow did all the work to adapt it to futures and released it, and there are other mods—for example, to backtest options, energy derivatives (even using solar and wind generation forecasts in the theoretical mix). The only limit is human imagination.
Being an industry standard, many pipelines and proprietary software packages build on the foundation of Zipline, and the classic internal scheme of how Zipline works is the following:
This scheme is modular and adapts to your needs; it also lets you program new blocks and add them to the pipeline as if they were Lego pieces, giving you the versatility to develop any strategy imaginable. For example, we use ArcticDB to retrieve our data, and with just a few lines we created a platform that interconnects our datalake with the backtester, feeding it gigabytes of quality data for research and backtesting in an extremely fast and easy-to-maintain way. We will build the Datalake right here, on QuantArmy.
In upcoming installments we will go deeper into Zipline; for now, having a superficial understanding of the libraries is more than enough.
Alphalens
Alphalens automates the pipeline needed to analyze factors going forward. It computes all the factors across all the requested periods and analyzes the results—a fast and robust way to model an edge.
We define a Factor as the model used to explain (from a statistical definition) an asset's returns (usually, though any other target can be set). Through the factor, we assign a value to each asset in the universe so we can run a pipeline that, in an automated fashion, performs a broad-spectrum analysis, reducing the researcher's task to simply verifying results.
The classic factors tend to be:
- Momentums, various estimates
- Mean reversions, various estimates
- Volatility, various estimates
- Fundamental style factors: market capitalization, sector, and so on…
- Technical: Based on a criterion, such as volatility bands
- Seasonal: Based on temporal windows
- Sentiment: Based on analysis models or risk perception
- Fourth generation: Models based on artificial intelligence black boxes
- Fifth generation: With fully self-learning models (current)
And through this kind of analysis, we will be able to answer the following hypotheses:
- Is the factor alive (is it predictive? has it been?)
- How predictive the factor is
- Is it consistent across the universe? or only on certain assets
- How many assets should be included; should there be shorts?
- What is the best holding period?
- What would the turnover be as a function of the period
- How the factor would behave at different levels of volatility, of the universe's market cap, of beta, and so on…
Through the scientific method and trial and error, we will generate hypotheses that we must validate mathematically in order to move the process forward.
Pyfolio and Empyrical
By combining Pyfolio and Empyrical, we create a final reporting pipeline with a high degree of customization that gives us two options:
- Use the preset templates, which are quicker to access
- Generate custom templates to produce fully personalized reports (even branded)
We will use the preset templates and also venture into how to create custom templates.
And to tell the two libraries apart: Pyfolio is the predefined template that generates the reports, and Empyrical is the library that, using the backtest data, produces the ratios, statistics, estimators, and so on that Pyfolio needs to generate the report.
Summary
With this set of tools, we build our quant stack, which will form part of our pipeline. This pipeline will let us research and backtest all kinds of models in a solid and secure way.
Coming Up
In the upcoming installments:
- Zipline, Backtesting in Python 2 : My First Data Research
- Zipline, Backtesting in Python 3: My First Pipeline
- Zipline, Backtesting in Python 4: My First Backtest
- Zipline, Backtesting in Python 5: My First Portfolio Report
- Riskfolio, Risk Management in Python 6: My First Risk Management Model in Python
Other Recommended Posts
