Background
Within the field of financial engineering, we use a wide collection of tools and libraries that were never designed to be used together. Consequently, we need to package all the necessary libraries and avoid conflicts with other projects or enable portability.
If we add that team members do not use the same operating systems, etc., a blend of configurations is generated, and we need the development environment to be homogeneous for all team members. Consequently, we would also need to include the operating system and additional configurations within the package.
To solve this, we are going to create a Docker image, a container system that allows us to virtualize specific applications instead of an entire operating system.
The plan is to create a Docker container with the operating system, the Python environment and package manager, and the necessary tools for quant research.
Once the container is created and compiled, only a single command line will be needed to have the environment ready to access in less than 10 seconds.
What's inside the stack?
Inside our Docker image, we will have a global configuration that will be used across all projects, and another configuration with a more project-specific setup.
For the global configuration, we will use Debian as the operating system. Why? Because it is stable and requires very little maintenance. Alpine is not an option because our goal is not to save resources, and any other operating system is automatically ruled out.
To manage environments and packages, we use miniconda, which provides a Python environment ready to run, along with other necessary libraries such as environment add-ons, JupyterLab itself, etc.
Within the project-specific part, we will install the necessary packages and customizations. In this case, we will create a simple initial configuration where we will use zipline-reloaded and pyfolio-reloaded.
We will create a "Hello World" within zipline, where once the entire environment is deployed, we run a basic backtest to verify the project works. Anyway, the project is under constant maintenance and improvements, since I personally use this same environment (in more advanced versions; here we release somewhat older methods).
Stack components summary:
- Debian GNU Linux
- Miniconda3
- JupyterLab
- Configuration files and JupyterLab configuration libraries
- Configuration files, optimizations, and libraries to work with zipline.
- Extras: Files such as research examples, code optimizations, a local datalake, or any other information needed for the stack.
Final Advantage
Once the image is compiled, we will only need to run the command
docker compose up -d
to have the environment up and running.
Creating a stack
First, we need to create the project structure, which consists of:
Two main folders. In the folder called DockerSource, we have everything related to building the image.
- .env: Private environment data to connect to the datalake, etc.
- Dockerfile: Docker image build instructions, where the specific steps that are automated up to the image creation are defined
- extension.py: Zipline optimizations
- overrides.json: JupyterLab optimizations
- req.txt: Detailed list of packages needed to install in the environment via pip.
- qa_datalake.py: Datalake ingestion code into zipline.
Inside the second folder, we include the so-called workdirs, or working directories, which are the directories where the initial information that the researcher will interact with is stored.
In this case, we have a series of folders that group notebooks or documentation by themes that have been or will be relevant at some point.
Building the image
1 - Clone the repository
Download the latest version of the code from the repository.
git clone https://github.com/quantarmyz/quantstack.gitCreate environment file .env
Create a .env file inside the DockerSource folder like:
/quantstack/DockerSource/.env
File format:
ENDPOINT=TuEndPoint.COM
DB=TuBucket
ACCESS_KEY=TuAccessKey
SECRET_KEY=TuSecretKey
Building the image
From the terminal, in the /quantstack path, run:
docker compose build --no-cache
Optional Docker Compose Configuration
Edit /quantstack/docker-compose.yml as needed (configuring the token is recommended).
Start the stack
Run the following command for debug mode:
docker compose upTo run in the background:
docker compose up -dAccessing JupyterLab
If no modifications have been made, the address is:
localhost:8888Password:
testingAdding a Data Feed to Zipline
From JupyterLab, open a terminal and run:
conda activate base
zipline ingest -b qa_datalakeThis will connect to the datalake, download the data, and add it to the Zipline database.
Backtesting Example
In the image, you can see how we launched the default backtesting template in the qastack.
Video
To make things easier, I recorded a video where I compile the image and run it. In less than 9 minutes, the stack can be up and running.
Full Repository
quantarmyz
