Background
At this point, we have our datalake fully deployed, with data from EODHD. We can access the data from pandas for all kinds of analysis and studies in JupyterLab.
Now we are going to extend the functionality of our stack by programming an algorithm that performs the following function;
BACKTESTER INGEST DATA ->
REQUEST FROM DATALAKE ->
DOWNLOAD - TRANSFORM ->
INGEST IN BACKTESTEROr in other words, we are going to create an algorithm that, when the backtester requests the information, will query arcticdb, download it, transform it to use the zipline format, and add it to the backtester.
In this case, we will work only with a very small list of American stocks, but we will explain how to create universes and filter assets based on their characteristics.
What is Zipline
Zipline is the backtester used by the Quantopian project, integrable with its tool stack. If you want to learn more about Zipline, we recommend:
jcxQA
jcxQA
In these articles, you will find all the theory and practice needed to take your first steps with Zipline. But how do you add custom data for alternative assets, or simply stock data from your usual data broker? Well, that is the problem we will solve in this article.
Introduction to Zipline's Architecture
Since Zipline is an Open Source project, all developers can modify the code according to their needs, but it also provides the advantage of understanding, in depth, how its internal engine works.
In the following image we see how it manages the calls.
The data we have stored in our datalake must be requested and transformed into a format that zipline accepts, so that subsequently zipline automatically adjusts it via bcolz_storage.
About bcolz storage
For those who don't know what bcolz was:
bcolz provides columnar, chunked data containers that can be compressed either in-memory and on-disk. Column storage allows for efficiently querying tables, as well as for cheap column addition and removal. It is based on NumPy, and uses it as the standard data container to communicate with bcolz objects, but it also comes with support for import/export facilities to/from HDF5/PyTables tables and pandas dataframes.
In an early stage of algorithmic trading for Python, solutions like the current arcticdb were nothing but dreams, since they were technologically unfeasible; data management libraries with the lowest possible latency were needed, and bcolz solved the requirement in a panda-friendly way, which led to its "massive adoption" in many projects.
Basically, a bundle is a universe of assets, where we must add all the necessary symbols for its composition, as well as a loading mechanism for them.
In our particular case, we are going to select a small universe which will be:
SPY, QQQ, DIA, TLT, GLD
Universes can be declared explicitly, as we will do in our case, or they can be created through parameters.
Universes created through parameters refer to the fact that, to determine the final tickers we will use, we could filter by the company's launch year to avoid overly new companies, filter them by market capitalization, or any other requirement, such as outperforming an index the previous week, or belonging to a specific sector.
How to Add Datalake Data to Zipline
The full code can be found at the end of the post, for registered users. Here we will comment on certain relevant parts, which is where the "magic" really happens.
In the following snippet, we query the .env file of our environment, which provides the credentials to access the datalake, and assign them to variables, to later be able to create our connection.
Our pointer is the variable assigned as ac, which contains an object with all the necessary information to make the connection with the datalake, and will be the entry point for any interaction with the arcticdb datalake located in the cloud.
warnings.filterwarnings('ignore')
dotenv_path = '/root/.env'
load_dotenv(dotenv_path)
endpoint = getenv("ENDPOINT")
db = getenv("DB")
access_key = getenv("ACCESS_KEY")
secret_key = getenv("SECRET_KEY")
ac = Arctic(f's3s://{endpoint}:{db}?access={access_key}&secret={secret_key}') my_cal = get_calendar('NYSE')
prices = ac.get_library('prices.etfs.us.stable')
for sid, symbol in enumerate(symbols):
print('[QA DATALAKE CARNIVORE ] ||| Loading {}...'.format(symbol))
df = prices.read(symbol).data
df = df['2010':]
start_date = df.index[0]
end_date = df.index[-1]
sessions = my_cal.sessions_in_range(start_date, end_date)
df = df[df.index.isin(sessions)]
df = df.reindex(sessions.tz_localize(None))[start_date:end_date] #tz_localize(None)
df.fillna(method='ffill', inplace=True)
df.dropna(inplace=True)
ac_date = end_date + pd.Timedelta(days=1)
metadata.loc[sid] = start_date, end_date, ac_date, symbol, 'QAX'The following snippet is part of a loop, where the asset's parameters are assigned, such as its calendar, and then the datalake is queried to read the information from 2010 onward, and transform it so that it can later be ingested by zipline, reindexing based on market trading sessions (to avoid issues with half-days and holidays).
We will repeat the process also with splits and dividends, in addition to always adding unadjusted data, where we let zipline's own engine, with the provided information, perform the relevant price adjustments internally in its backtests, thus having control of the calculation at all stages of the process, and not depending on third parties such as the data broker.
Conclusions
Once we reach this point, we have a datalake with financial information from EODHD, and an integration with zipline.
The next natural step is to create a complete environment, where we use everything built so far. A professional development environment, based on Docker container technology, which facilitates deployment and maintenance tasks for the researcher. Simplifying its execution, and fully customizable by the user.
This environment will be taken as a basis for future developments, and it is assumed that all readers are familiar with it.
