Big Data Management 3 - Building a Datalake on the Nasdaq with ArcticDB.

Jesús Cuesta 3 de abr. de 2025 5 min read

In this article, we will explore the creation of an algorithm to manage financial data in our datalake, using ArcticDB as the backend and the End Of Day Historical (EODH) data provider. This algorithm will meet the following requirements:

ArcticDB architecture — ArcticDB as central datalake

Requirements

Universal Pandas-style format.
ArcticDB as the backend.
Automatic database management within the datalake.
Metadata management.

Data Used

The information stored in the datalake will be used as the data source for the backtesting engine. We will use EODH as our data broker due to its affordability and customer service. Their quality data sources and the variety of information they provide make it a solid choice.

EODHD purchases data packs from exchanges and redistributes them among its clients; it is not 100% dedicated to data collection, but this is an advantage, since it is more likely for a small data broker to have a failure in data collection, but for NASDAQ to do so is more difficult, because if it happened, more people are using this data professionally, and the error would be fixed.

Where does EODHD get its data?

Quoting what they say on their website:

For end-of-day (EOD) and delayed data, we have direct contracts with several exchanges for their corresponding data streams. We get USA data from Nasdaq Cloud API.

The UK data is derived directly from the London Stock Exchange Group (LSE), with more than 22,000 tickers in total.

We get European markets' data from Cboe Europe Equities (Cboe) which covers more than 60 exchanges from Western and Eastern European markets.

For Australian feeds, we have a direct agreement with the Australian Stock Exchange (ASX). Delayed and End of Day NEO Exchange Data for NEO and TSX/TSXV powered by Quotemedia.com.

Data Types

Within the data offered by the data broker, we find an infinite number of categories, but basically we could classify them into these major groups.

Descriptive Data

Descriptive data is a subset of fundamental data, where the most relevant ones are considered, in order to establish the characteristics of the company being dealt with.

This data includes the exchange, currency, IPO date, sector, subsector, number of employees, market cap, etc.

It is used to create universes from ticker metadata. For example, creating a universe with stocks that have an IPO after 1960 and belong to the financial sector. Or any other combination of characteristics.

Screenshot from Big Data Management 3 - Building a Datal — Descriptive Data (Subset of fundamental data)

Price Data

OHLCV (Open, High, Low, Close, Volume) data will be fundamental for our analysis. It is the basic way in which price information is distributed, and the most common.

Corporate Events Data

Corporate events include information exogenous to the price, but necessary to understand unadjusted data. It is recommended to perform data adjustment within the backtester itself, and never use adjusted data.

In the first case, we see splits, and in the second, dividends.

Within dividends, we see that there are different dates, but the following image represents the dividend cycle.

Fundamental Data

Information obtained from companies' financial statements, including accounting ratios, sectors and competitors, etc. In this case, the information is a current snapshot. But the database could be adjusted to store these values for each day, in order to build models based on fundamental data.

Alternative Data

Alternative data includes information such as news, market sentiment, supply chain relationships, among others, data from which alpha can be discovered, given that including them is the current trend, although traditionally they have not been used.

Alternative data refers to all information correlated in some way with the current object of study, and whose inclusion in models yields new ways to extract alpha from the asset. We could speak of news, sales forecasts, or any type of information that, once modeled, can create or add alpha to a specific model.

Algorithm to Feed the Datalake

We are going to design an algorithm to manage our databases in the datalake and transform the broker's data into our datalake format, generating prices and metadata. The algorithm structure will be similar to the following:

Collect symbols
Collect data from those symbols in the data broker
Transform that information into a Pandas-like format
Send that information to our datalake via arcticdb

all_tickers = recopile_symbols_by_filter_results
for each symbol in all_tickers:
    download, transform, and send price data to datalake
    download, transform, and send dividend and split data to datalake
    download, transform, and send metadata to datalake

Managing Databases within the Datalake

A datalake is composed of many libraries (databases). To work in our datalake, we need to have some databases created, in order to organize all the information dumped into the datalake.

We will implement a simple mechanism to manage databases in the datalake using ArcticDB. The following code checks the datalake's consistency:

dataz = ['prices.stocks.us.stable', 'divs.stocks.us.stable', 'splits.stocks.us.stable', 'meta.stable']

for d in dataz:
    if d not in ac.list_libraries():
        ac.create_library(d)
        print('[DATA CHECK]:', d.upper(), 'CREATING NEW DATASTORAGE')
        print('DATA STORAGE CREATED!')
    else:
        print('[DATA CHECK]:', d.upper(), 'IS IN THE DATALAKE')
        print('Listing Symbols:', ac.get_library(d).list_symbols())

print('[QA] - DATALAKE v2 carnivore READY :) :) :)')

This code ensures that the necessary databases exist in the datalake and creates them if they don't.

Ticker Filtering

Ticker filtering can be done in several ways, either by querying metadata or manually. In this example, we filter stocks belonging to the Nasdaq Stock Exchange:

# Pseudocódigo
ts = pd.DataFrame(call.get_exchange_symbols('US'))
ts = ts[ts['Exchange'] == 'NASDAQ']
ts = ts[ts['Type'] == 'Common Stock']
all_tickers = ts['Code'].tolist()
tickers_in_lake = prices.list_symbols()
all_tickers = [valor + ".US" for valor in all_tickers]
tickers = [ticker for ticker in all_tickers if ticker not in tickers_in_lake]

This process avoids downloading duplicate tickers and saves time and API calls.

Final Conclusions

Once we have some tickers in our datalake, the next step is to build a professional working environment based on technologies like Docker, JupyterLab, Zipline, and Linux to perform research and backtesting tasks.

The amount of information available today is invaluable! ArcticDB solves all the problems related to the infrastructure for storing and querying large-scale dataframes with negligible latency, allowing work even on very small timeframes. This article provides a solid foundation for implementing a datalake compatible with tools like Zipline in the future.

To download the full code, you must be registered!