Datalake.

What it is

QUANTARMY Datalake (version 5.1) is the code to build and manage a market data lake based on ArcticDB and several data sources.

The problem it addresses is fragmentation. Prices, fundamentals, macro and alternative data end up in different places and formats. The datalake centralises them in a single NoSQL store and serves them in the format researchers already use: pandas DataFrames.

Storage is an S3 bucket. There is no database server: the Python client itself sends the queries to S3. ArcticDB, by Man Group, versions every symbol: a new write does not delete the previous version.

What's included

The repository is small. The qadatalake package has three files:

• cli.py: menu-driven management CLI (version V.5.1.0-predator) with three options: list the symbols of every library in the datalake, run the Alpha Vantage macro routine, and exit.
• tasks/routine.py: connection to ArcticDB over S3 and listing of libraries and symbols.
• data/alphavantage.py: downloads the US unemployment and inflation series from Alpha Vantage and stores them in the alphavantage library (UNEMPLOYMENT.MACRO.US and INFLATION.MACRO.US). It updates at most every 15 days.

The connection reads four variables from a .env file: ENDPOINT, DB, ACCESS_KEY and SECRET_KEY. Quantstack uses the same ones.

Price ingestion is not in the repository. The Big Data Management series builds it step by step: connecting to ArcticDB, managing libraries, a macroeconomic calendar, and a Nasdaq datalake with prices, dividends, splits and metadata from EODHD.

Installation and first run

The README does not document installation and the repository has no setup.py or requirements.txt. The dependencies are the ones the code imports: arcticdb, pandas, requests and python-dotenv.

git clone https://github.com/quantarmyz/datalake
cd datalake/qadatalake
pip install arcticdb pandas requests python-dotenv

The CLI looks for .env in the folder it is launched from:

ENDPOINT=your-endpoint.com
DB=your-bucket
ACCESS_KEY=your-access-key
SECRET_KEY=your-secret-key

python cli.py

From a notebook, the connection is the same one the repository code uses:

from os import getenv
from dotenv import load_dotenv
from arcticdb import Arctic

load_dotenv('.env')
ac = Arctic(f"s3s://{getenv('ENDPOINT')}:{getenv('DB')}?access={getenv('ACCESS_KEY')}&secret={getenv('SECRET_KEY')}")
ac.list_libraries()

How it fits

The Datalake is the first piece of the lab chain: it stores the data. Quantstack reads it with its qa_datalake.py connector, which queries ArcticDB with the same four environment variables and loads prices, dividends and splits into a Zipline bundle. Part 4 of the Big Data Management series explains that step. Backtest results end up in the QAStats report.