Lesson 1.5.2 · 60 min
Macro data, economic calendars, estimates and transcripts
A macro figure is published, revised and revised again: the GDP you see today is not the one that moved the market on the day it came out. This lesson works with first releases and their revisions, the calendar of when each figure becomes known (in UTC) and the surprise versus consensus, which is what really moves prices.
- Module
- 1.5 · Fundamental and macro data
- Duration
- 60 min
- Prerequisites
- 0.4.4 · Files (CSV, JSON, YAML, TOML) and dates with time zones, 1.5.1 · Financial statements, ratios, XBRL and SEC EDGAR
- Author
- Jesús Cuesta
- Updated
Level 1 lessons
1.1 · How markets work
1.2 · Asset classes
- 1.2.1 Stocks: splits, dividends and corporate actions
- 1.2.2 ETFs: replication, tracking error, leveraged ETFs and UCITS
- 1.2.3 Futures: specifications, margin, roll and contango
- 1.2.4 Forex: pairs, pips, lots, rollover and CFDs
- 1.2.5 Options: calls, puts, intrinsic value and time value
- 1.2.6 Crypto: spot, perpetuals, funding and custody
- 1.2.7 Fixed income and interest rates; CFDs and leveraged products
1.3 · Market data sources
- 1.3.1 Types of market data: EOD, bars, ticks, order book and reference
- 1.3.2 Free market data sources
- 1.3.3 Retail and prosumer market data APIs
- 1.3.4 Professional market data providers
- 1.3.5 Crypto and on-chain data
- 1.3.6 Brokers as market data sources
- 1.3.7 Market data licensing and terms of use
- 1.3.8 How to choose a market data provider
1.4 · Financial data engineering
- 1.4.1 Downloading over REST and websockets: pagination, rate limits and retries
- 1.4.2 Data cleaning: missing values, bad ticks, gaps and time zones
- 1.4.3 Adjusting for splits and dividends: adjusted and unadjusted prices
- 1.4.4 Universes without survivorship bias: delistings, mergers and symbol changes
- 1.4.5 Building bars: time, tick, volume, dollar and imbalance
- 1.4.6 Continuous futures series: roll rules, difference adjustment and ratio adjustment
- 1.4.7 Storage: CSV, Parquet, partitioning, DuckDB, TimescaleDB and ArcticDB
- 1.4.8 Reproducible pipeline: download, validation, incremental updates and quality tests
1.5 · Fundamental and macro data
1.6 · Financial exploratory analysis
1.7 · Programmed technical analysis
One figure, several versions
Macro statistics are published before they are complete. A statistics agency gives a first estimate of GDP with part of the quarter's information and corrects it when the missing surveys, tax data or trade balances come in. CPI usually has a flash estimate and a final figure; unemployment is revised the following month and in the annual survey revision. Each version is a vintage: the series as it was known on a date.
Mermaid source
flowchart LR A[Advance] --> S[Second estimate] S --> T[Third estimate] T --> R[Annual revision]
For a trading system this has two consequences. The first is the one from lesson 1.5.1 under another name: if a backtest uses today's series, it uses figures that did not exist when it simulates the decision. The second is more subtle: the market does not react to the figure, but to the difference between the figure and what was expected. Both are solved the same way, with a knowledge date per figure.
The data in this lesson are fictional series from Valdoria, a made-up country with a release calendar similar to
that of the US, and from Brenia, one with a European calendar. macro_vintages.csv holds every release of four
series with its date:
import pandas as pd
mv = pd.read_csv("macro_vintages.csv",
parse_dates=["fecha_publicacion"])
print(mv.groupby(["serie", "vintage"]).size().unstack(fill_value=0))
mv[(mv["serie"] == "pib_trimestral")
& mv["periodo"].isin(["2019T4", "2020T1"])]vintage 1 2 3 4
serie
ipc_interanual 83 83 0 0
paro 83 82 19 0
pib_trimestral 27 27 27 14
tipo_oficial 84 0 0 0
serie periodo fecha_publicacion valor vintage
362 pib_trimestral 2019T4 2020-01-30 -0.3 1
363 pib_trimestral 2019T4 2020-02-27 2.3 2
364 pib_trimestral 2019T4 2020-03-26 1.7 3
365 pib_trimestral 2020T1 2020-04-30 -3.8 1
366 pib_trimestral 2020T1 2020-05-28 -5.0 2
367 pib_trimestral 2020T1 2020-06-25 -4.6 3
368 pib_trimestral 2020T1 2021-07-29 -4.5 4CPI has two versions per month (flash and final); unemployment, two or three; GDP, three estimates and, in half of the quarters, an annual revision; the policy rate is not revised. The fourth quarter of 2019 is the extreme example: the first estimate, on January 30, 2020, said −0.3% (annualized quarterly rate); a month later, +2.3%; the third, +1.7%.
First release versus latest version
How much is each series revised? Comparing the first version of each period with the latest:
def primera_y_ultima(mv, serie):
s = mv[mv["serie"] == serie].sort_values("vintage")
g = s.groupby("periodo")["valor"]
return pd.DataFrame({"primera": g.first(), "ultima": g.last()})
filas = {}
for serie in ["ipc_interanual", "paro", "pib_trimestral"]:
pu = primera_y_ultima(mv, serie)
rev = pu["ultima"] - pu["primera"]
filas[serie] = {"periodos": len(pu),
"revisados_%": 100 * (rev.abs() > 1e-9).mean(),
"rev_media_abs": rev.abs().mean(),
"rev_max_abs": rev.abs().max()}
pd.DataFrame(filas).T.round(2) periodos revisados_% rev_media_abs rev_max_abs
ipc_interanual 83.0 61.45 0.07 0.3
paro 83.0 54.22 0.06 0.2
pib_trimestral 27.0 81.48 0.49 2.0CPI and unemployment move little: tenths of a point, in just over half of the months (61% and 54%). GDP is revised in 81.5% of the quarters, with a mean revision of almost half a point and a maximum of 2 points, the one for the fourth quarter of 2019. The figure at the top puts the two versions of each quarter side by side.
Half a point of annualized GDP is the difference between “the economy is slowing down” and “the economy is doing fine”. And in some quarters the sign changes:
pib = primera_y_ultima(mv, "pib_trimestral")
signo = (pib["primera"] < 0) != (pib["ultima"] < 0)
pib[signo] primera ultima
periodo
2019T4 -0.3 1.7With the first estimate, the fourth quarter of 2019 was a contraction. A rule such as “reduce exposure if last quarter's GDP is negative” would have acted in February 2020 on that figure. With today's series, that signal does not exist. Neither of the two is “the right one”: to study the economy, the latest; to simulate a decision, the one that was known.
The series as it was known: vintages and ALFRED
Rebuilding the series in force on a date means filtering by release date and keeping the latest version of each
period. That is the idea behind ALFRED (Archival FRED), the vintage archive of the Federal Reserve Bank of St.
Louis: by default FRED returns the series as it stands today; ALFRED lets you request it as it stood on a past
date1Note 1 · FRED API documentation: fred/series/observations (parameters realtime_start, realtime_end,
vintage_dates, output_type; 32-character api_key) and “Real-Time Periods”: by default the real-time period is
today; ALFRED lets you retrieve what was known on a past date..
def conocida_en(mv, serie, fecha):
"""The series as it was known at the close of `fecha`."""
s = mv[(mv["serie"] == serie)
& (mv["fecha_publicacion"] <= fecha)]
return (s.sort_values("vintage")
.groupby("periodo")["valor"].last())
fechas = ["2020-02-15", "2020-06-30", "2025-12-31"]
cortes = {f: conocida_en(mv, "pib_trimestral", f) for f in fechas}
pd.DataFrame(cortes).loc["2019T3":"2020T2"] 2020-02-15 2020-06-30 2025-12-31
periodo
2019T3 2.8 2.8 3.0
2019T4 -0.3 1.7 1.7
2020T1 NaN -4.6 -4.5
2020T2 NaN NaN -29.4Each column is a vintage. On February 15, 2020 nobody knew anything about the first quarter of 2020 and the fourth quarter of 2019 was −0.3; by the end of June the fourth quarter of 2019 was already +1.7 and the first quarter of 2020 had been revised twice. The full table, periods in rows and vintage dates in columns, is the “revision triangle” that economists use to study the quality of first estimates.
To use a macro series as a variable in a daily backtest, the join is the same as with fundamentals: a merge_asof by
release date, with each day's version. Comparing it with the naive join (the latest version of each quarter, from the
day the quarter ends):
sesiones = pd.read_csv("fix_indice_diario.csv",
parse_dates=["fecha"])[["fecha"]]
p = mv[mv["serie"] == "pib_trimestral"].copy()
# end of each quarter: 2019T4 -> 2019-12-31
p["periodo_fin"] = (pd.PeriodIndex(p["periodo"].str.replace(
"T", "Q"), freq="Q").end_time.normalize())
pit = pd.merge_asof(
sesiones, p.sort_values("fecha_publicacion")
[["fecha_publicacion", "periodo", "valor"]],
left_on="fecha", right_on="fecha_publicacion",
allow_exact_matches=False)
ult = p.sort_values("vintage").groupby("periodo").last()
ingenuo = pd.merge_asof(
sesiones, ult.reset_index().sort_values("periodo_fin")
[["periodo_fin", "periodo", "valor"]],
left_on="fecha", right_on="periodo_fin")
c = pit.merge(ingenuo, on="fecha", suffixes=("_pit", "_ing"))
c = c.dropna(subset=["valor_pit", "valor_ing"])
print(f"Sessions: {len(c)}")
print("Different quarter:",
f"{(c['periodo_pit'] != c['periodo_ing']).mean():.1%}")
print("Different value:",
f"{(c['valor_pit'] != c['valor_ing']).mean():.1%}")
negativo_distinto = (c["valor_pit"] < 0) != (c["valor_ing"] < 0)
print(f"Different sign: {negativo_distinto.mean():.1%}")Sessions: 1681
Different quarter: 36.6%
Different value: 89.5%
Different sign: 9.8%In 89.5% of the sessions the naive join uses a value different from the one that was known and in 36.6%, even a different quarter: one that had not been released yet. In one in ten (9.8%) not even the sign of growth matches. A macro model trained this way learns a relationship between the economy and the market that no investor could have exploited.
The economic calendar: when each figure becomes known
An economic calendar lists, for each release, the date and time, the consensus (the mean or median of the analyst
forecasts surveyed by the calendar provider), the previous value and the released one. calendario_economico.csv is
the 2025 calendar for Valdoria and Brenia:
ce = pd.read_csv("calendario_economico.csv",
parse_dates=["fecha_hora"])
print(ce["fecha_hora"].dtype, len(ce), "releases")
ce[["fecha_hora", "evento", "pais", "consenso",
"publicado"]].head(6)datetime64[us, UTC] 123 releases
fecha_hora evento pais consenso publicado
0 2025-01-02 09:00:00+00:00 PMI manufacturero Brenia 47.70 47.70
1 2025-01-02 15:00:00+00:00 PMI manufacturero Valdoria 51.30 51.00
2 2025-01-03 13:30:00+00:00 Tasa de paro Valdoria 4.30 4.20
3 2025-01-13 13:30:00+00:00 IPC interanual (definitivo) Valdoria 2.70 2.40
4 2025-01-16 13:30:00+00:00 Ventas minoristas, mensual Valdoria 0.70 0.60
5 2025-01-29 19:00:00+00:00 Decisión de tipos Valdoria 4.35 4.35The times are in UTC, as they should be stored. Before using it, it is worth checking that it matches the series: the released value of each Valdoria CPI flash estimate must be vintage 1 of that month, and the “previous” value, the latest known version of the prior month.
av = ce[(ce["pais"] == "Valdoria")
& (ce["evento"] == "IPC interanual (avance)")].copy()
av["fecha"] = av["fecha_hora"].dt.tz_convert(
"America/New_York").dt.tz_localize(None).dt.normalize()
ipc1 = mv[(mv["serie"] == "ipc_interanual")
& (mv["vintage"] == 1)]
cmp = av.merge(ipc1, left_on="fecha",
right_on="fecha_publicacion")
print(len(av), "flash estimates;", len(cmp), "with their vintage;",
"equal:", (cmp["publicado"] == cmp["valor"]).all())11 flash estimates; 11 with their vintage; equal: TrueAll eleven releases match. It is a cheap check worth automating in a pipeline (lesson 1.4.8): a calendar from one provider and a series from another can disagree on the vintage, the unit or the day.
The surprise: figure minus consensus
An asset's price already discounts what the market expects. What arrives with the release is the unexpected part:
To compare events with different units and volatilities, the surprise is normalized by dividing it by the standard deviation of that event's surprises (standardized surprise). In a real study that deviation is estimated with the past only, with a window that does not include the release being evaluated; here, with one year of data, the full year's deviation is used to describe the distribution.
ce["sorpresa"] = ce["publicado"] - ce["consenso"]
corto = ce["evento"].str.replace(
"PIB trimestral, tasa anualizada", "PIB")
ce["evento_pais"] = corto + " · " + ce["pais"]
res = ce.groupby("evento_pais")["sorpresa"].agg(
n="size", media="mean", desv="std",
exactas=lambda s: (s.abs() < 1e-9).mean())
res.sort_values("desv", ascending=False).round(2) n media desv exactas
evento_pais
Confianza del consumidor · Valdoria 12 -0.67 1.27 0.00
PIB (segunda estimación) · Valdoria 4 -0.03 0.51 0.00
PMI manufacturero · Valdoria 12 -0.07 0.46 0.00
PIB (avance) · Valdoria 4 -0.47 0.43 0.00
PMI manufacturero · Brenia 12 -0.09 0.43 0.17
Ventas minoristas, mensual · Valdoria 12 -0.07 0.31 0.33
PIB (tercera estimación) · Valdoria 4 -0.35 0.31 0.00
IPC interanual (avance) · Valdoria 11 0.08 0.22 0.09
IPC interanual (definitivo) · Valdoria 12 -0.02 0.16 0.50
Tasa de paro · Valdoria 12 -0.07 0.15 0.50
IPC interanual (avance) · Brenia 12 -0.01 0.08 0.42
Decisión de tipos · Brenia 8 0.00 0.00 1.00
Decisión de tipos · Valdoria 8 0.00 0.00 1.00Rate decisions never surprise in this year: the consensus gets all sixteen right, as expected from central banks that prepare the market before deciding. Unemployment and final CPI are right half the time: they move by tenths and, for the final figure, the flash estimate is already out. The most dispersed surprises are those of consumer confidence (1.27 index points), GDP and the PMIs. Three of the four surprises in the GDP advance estimate were negative (mean of −0.47 points), but with four data points that is not a consensus bias, it is an anecdote. The deviations are in different units (index points, percentages): that is why they have to be normalized. An event without surprises barely moves the price; one with large, frequent surprises is the one that deserves an event study (level 2).
desv = ce.groupby("evento_pais")["sorpresa"].transform("std")
ce["z"] = ce["sorpresa"] / desv
# for unemployment, more is worse: flip the sign
ce.loc[ce["evento"] == "Tasa de paro", "z"] *= -1
top = ce.dropna(subset=["z"]).reindex(
ce["z"].abs().sort_values(ascending=False).index)
top[["fecha_hora", "evento_pais", "consenso",
"publicado", "z"]].head(5).round({"z": 2}) fecha_hora evento_pais consenso publicado z
54 2025-06-06 12:30:00+00:00 Tasa de paro · Valdoria 4.7 4.3 2.67
50 2025-05-30 14:00:00+00:00 Confianza del consumidor · Valdoria 95.3 92.3 -2.36
46 2025-05-16 12:30:00+00:00 Ventas minoristas, mensual · Valdoria 0.7 -0.0 -2.25
37 2025-04-24 12:30:00+00:00 PIB (avance) · Valdoria -0.3 -1.2 -2.07
27 2025-03-27 12:30:00+00:00 PIB (tercera estimación) · Valdoria 3.0 2.4 -1.93With the standardized surprise and the economic sign (for unemployment, less is better), releases of different indicators can be put on the same scale and ranked by how unexpected they were.
Times in UTC, markets in local time
Valdoria releases its data at 08:30 New York time and decides on rates at 14:00; Brenia releases at 10:00 or 11:00 Madrid time and decides at 14:15. In UTC, those times change with daylight saving time, and not at the same moment: the US switches on the second Sunday of March and the first Sunday of November; the EU, on the last Sunday of March and of October (lesson 0.4.4).
ny = ce["fecha_hora"].dt.tz_convert("America/New_York")
mad = ce["fecha_hora"].dt.tz_convert("Europe/Madrid")
ce["hora_local"] = ny.dt.strftime("%H:%M").where(
ce["pais"] == "Valdoria", mad.dt.strftime("%H:%M"))
ce["hora_utc"] = ce["fecha_hora"].dt.strftime("%H:%M")
print(pd.crosstab([ce["pais"], ce["hora_local"]], ce["hora_utc"]))
# 2025 clock changes according to the time zone database (tzdata)
dias = pd.date_range("2025-01-01", "2025-12-31", freq="D",
tz="UTC") + pd.Timedelta(hours=12)
for zona in ["America/New_York", "Europe/Madrid"]:
off = dias.tz_convert(zona).map(lambda t: t.utcoffset())
cambio = dias[1:][off[1:] != off[:-1]]
print(zona, [str(d.date()) for d in cambio])hora_utc 08:00 09:00 10:00 12:15 12:30 13:15 13:30 14:00 15:00 18:00 19:00
pais hora_local
Brenia 10:00 7 5 0 0 0 0 0 0 0 0 0
11:00 0 7 5 0 0 0 0 0 0 0 0
14:15 0 0 0 4 0 4 0 0 0 0 0
Valdoria 08:30 0 0 0 0 39 0 20 0 0 0 0
10:00 0 0 0 0 0 0 0 15 9 0 0
14:00 0 0 0 0 0 0 0 0 0 6 2
America/New_York ['2025-03-09', '2025-11-02']
Europe/Madrid ['2025-03-30', '2025-10-26']08:30 in New York is 12:30 UTC during daylight saving time and 13:30 outside it. A filter written as “do not trade between 13:25 and 13:45 UTC” is right in winter and an hour late from March to November. According to the time zone database, in 2025 New York switched on March 9 and November 2, and Madrid on March 30 and October 26. Between those dates, New York and Madrid were five hours apart instead of six: two windows a year in which a rule like “Madrid time minus six” fails.
Mermaid source
flowchart TB
C[UTC calendar] --> L[Local time]
L --> S{Market open?}
S -- yes --> B[Event bar]
S -- no --> N[Next session]
B --> V[Reaction window]
N --> VFor a reaction study, each release has to be joined to the first bar in which the market can react. With NOVA's
5-minute bars (09:30 to 16:00 New York session) and a forward merge_asof:
barras = pd.read_parquet("nova_barras_5min.parquet")
ev = ce[ce["pais"] == "Valdoria"].sort_values("fecha_hora")
m = pd.merge_asof(ev, barras[["ts"]], left_on="fecha_hora",
right_on="ts", direction="forward")
m["espera_min"] = (m["ts"] - m["fecha_hora"]).dt.total_seconds() / 60
print(m.groupby("hora_local")["espera_min"].median())
largas = m[m["espera_min"] > 120]
largas[["fecha_hora", "ts", "espera_min"]]hora_local
08:30 60.0
10:00 0.0
14:00 0.0
Name: espera_min, dtype: float64
fecha_hora ts espera_min
11 2025-02-17 13:30:00+00:00 2025-02-18 14:30:00+00:00 1500.0
47 2025-07-04 12:30:00+00:00 2025-07-07 13:30:00+00:00 4380.0
61 2025-09-01 14:00:00+00:00 2025-09-02 13:30:00+00:00 1410.0
81 2025-11-27 13:30:00+00:00 2025-11-28 14:30:00+00:00 1500.0
89 2025-12-25 13:30:00+00:00 2025-12-26 14:30:00+00:00 1500.0The 08:30 releases arrive while the stock market is closed: the first bar that can react is the opening bar, an hour later, and whatever happened in between is in the opening gap (and in futures and currencies, which do trade). The 10:00 releases and the 14:00 decision fall within the session. And there are five releases on days when the NYSE is closed (in this fictional calendar, February 17, July 4, September 1, November 27 and December 25): their first bar is in the next session, one to three days later. A study that measured the reaction “in the 09:30 bar on the day of the release” would mix those five with sessions without news.
The APIs: FRED, ECB and INE
FRED, the ECB and INE (Spain's National Statistics Institute), three public sources of macro data for the US, the euro area and Spain, have open APIs. All of them need network access, so the code is not executed.
FRED (Federal Reserve Bank of St. Louis) requires a 32-character key (api_key). The observations endpoint
accepts the real-time period (realtime_start, realtime_end), a list of vintage dates (vintage_dates) and the
output type (output_type: 1, by real-time period; 2, all observations by vintage; 3, only new and revised ones; 4,
only the first release)1:
import os
import requests
URL = "https://api.stlouisfed.org/fred/series/observations"
base = {"series_id": "GDP", "api_key": os.environ["FRED_API_KEY"],
"file_type": "json"}
hoy = requests.get(URL, params=base, timeout=30).json()
primeras = requests.get(URL, params=base | {"output_type": 4},
timeout=30).json()
en_2020 = requests.get(URL, params=base | {
"realtime_start": "2020-02-15", "realtime_end": "2020-02-15"},
timeout=30).json()
# each observation: realtime_start, realtime_end, date, valueThe response carries an observations list in which each row has date (the period), value (as text; "." when
missing) and the real-time interval during which that figure was in force. It is exactly the structure of
macro_vintages.csv, with two date columns instead of one.
ECB (Data Portal): URLs follow the SDMX syntax data-api.ecb.europa.eu/service/data/<flow>/<key>, with no API key.
format=csvdata or jsondata choose the format; startPeriod and endPeriod, the range; updatedAfter, only what
changed since a date (useful for updating a local database), and includeHistory=true also returns earlier versions of
the data2Note 2 · ECB Data Portal, “Data”: entry point data-api.ecb.europa.eu/service/, formats csvdata and jsondata,
parameters startPeriod, endPeriod, updatedAfter and includeHistory (earlier versions of the data).:
import io
url = ("https://data-api.ecb.europa.eu/service/data/"
"EXR/D.USD.EUR.SP00.A")
r = requests.get(url, params={"format": "csvdata",
"startPeriod": "2025-01-01"},
timeout=30)
eurusd = pd.read_csv(io.StringIO(r.text))
# SDMX columns: KEY, FREQ, CURRENCY, ..., TIME_PERIOD, OBS_VALUEINE: the JSON API has the form servicios.ine.es/wstempus/js/{language}/{function}/{input}; the DATOS_SERIE
function returns the data of a series by its code and the nult parameter limits the response to the latest
periods3Note 3 · INE, “Referencia de la API” (API reference): structure
https://servicios.ine.es/wstempus/js/{language}/{function}/{input}, function DATOS_SERIE and parameter nult.. The date of each data point (Fecha) comes as milliseconds since 1970, and there is a time zone trap
here. The following sample response has the API's structure with a made-up code and values:
import json
ej = json.loads("""{"COD": "SERIE_EJEMPLO",
"Nombre": "Example series. Annual change.",
"Data": [
{"Fecha": 1761951600000, "FK_Periodo": 11, "Anyo": 2025,
"Valor": 3.1},
{"Fecha": 1764543600000, "FK_Periodo": 12, "Anyo": 2025,
"Valor": 2.8}]}""")
d = pd.DataFrame(ej["Data"])
d["utc"] = pd.to_datetime(d["Fecha"], unit="ms", utc=True)
d["madrid"] = d["utc"].dt.tz_convert("Europe/Madrid")
d["mes_mal"] = d["utc"].dt.strftime("%Y-%m")
d["mes_bien"] = d["madrid"].dt.strftime("%Y-%m")
d[["FK_Periodo", "utc", "mes_mal", "mes_bien", "Valor"]] FK_Periodo utc mes_mal mes_bien Valor
0 11 2025-10-31 23:00:00+00:00 2025-10 2025-11 3.1
1 12 2025-11-30 23:00:00+00:00 2025-11 2025-12 2.8Read in UTC, both data points fall on the last day of the previous month: November's looks like October's. The
timestamp is midnight in Madrid on the first day of the period, which in UTC is 23:00 (or 22:00 in summer) on the day
before. Converting to Europe/Madrid before extracting the month, or using Anyo and FK_Periodo, solves it. The
real request:
url = ("https://servicios.ine.es/wstempus/js/ES/DATOS_SERIE/"
"<series code>")
r = requests.get(url, params={"nult": 12}, timeout=30)
datos = r.json()["Data"]None of these APIs says when each figure was published in the default response: FRED needs the real-time period, the
ECB includeHistory, and for INE you have to check its release calendar. If you store macro data for backtests, also
store the day you downloaded it: your own database then becomes a vintage archive.
Analyst estimates and transcripts
Analyst estimates (earnings per share, revenue, recommendations) are the corporate version of the macro consensus: each analyst publishes a forecast, the provider computes the mean or median, and the earnings surprise is the reported earnings minus that consensus. They have the same date problems: each estimate gets revised, a quarter's consensus changes day by day until the release, and a dataset without the date of each revision is no good for simulation. Before paying for an estimates database (lesson 1.3.4), check whether it stores each forecast with its date or only the consensus at the time of download.
Transcripts of earnings presentations (earnings calls) are text: what management says about the quarter and the outlook, and the analysts' questions. They are the input to the language models of level 6: tone, changes in guidance, words that anticipate problems.
Mermaid source
flowchart LR T[Transcript] --> M[Timestamp] M --> X[Clean text] X --> F[NLP signal]
The principle is the same as in this whole section: a transcript counts from the moment the call ends, not from the end of the quarter or from the 10-Q filing. Lessons 6.1.2 and 6.2.3 build signals on filings and transcripts with that timestamp.
What this changes in a backtest
- Use vintages. If the source only gives the current series, store your downloads with their date or use ALFRED. For a series without a vintage archive, delay each data point to its real release date, even if you keep using the final version, and say so.
- Trade the surprise, not the level. The CPI level is in prices long before it is released; what is new is the difference from the consensus.
- UTC for storage, local time for trading. Convert with
zoneinfoevery time; never with fixed offsets. - Holidays and closed markets. A release while the market is closed is reflected at the next open or in the markets that do trade (futures, currencies). Decide which one you measure.
Exercises
- For unemployment, compute the revision between vintage 1 and 2 and between 2 and 3 (when it exists). Which of the two revisions is larger in mean absolute terms?
Solution
paro = mv[mv["serie"] == "paro"].pivot(
index="periodo", columns="vintage", values="valor")
r12 = (paro[2] - paro[1]).abs()
r23 = (paro[3] - paro[2]).abs()
print(f"v1 -> v2: mean {r12.mean():.3f}, n = {r12.count()}")
print(f"v2 -> v3: mean {r23.mean():.3f}, n = {r23.count()}")v1 -> v2: mean 0.062, n = 82
v2 -> v3: mean 0.100, n = 19The annual revision (vintage 3) only exists in the months that change, so its mean is computed over few months and is larger: when the annual revision touches a month, it moves it more than the monthly revision. For a model with unemployment data, vintage 3 arrives more than a year after the period: if it is used from the start, it brings in information from a year later.
- For each Valdoria release, compute the New York local time and check that all the 08:30 ones fall outside the NYSE session and that the rate decision falls within it.
Solution
val = ce[ce["pais"] == "Valdoria"].copy()
local = val["fecha_hora"].dt.tz_convert("America/New_York")
minuto = local.dt.hour * 60 + local.dt.minute
val["en_sesion"] = minuto.between(9 * 60 + 30, 16 * 60 - 1)
val.groupby("hora_local")["en_sesion"].mean()hora_local
08:30 0.0
10:00 1.0
14:00 1.0
Name: en_sesion, dtype: float64All the 08:30 releases are outside the cash session and the 10:00 and 14:00 ones, inside. The check does not look at
holidays: for that you need the exchange calendar (exchange_calendars, lesson 1.1.2), which is what the merge_asof
with NOVA's real bars did.
- A provider offers you a history of consensus figures with a single row per release (the final consensus). What study can you do with it and what study can you not?
Solution
With the final consensus and the released figure you can measure the surprise and study the price reaction in the minutes or days after the release: it is an event study, and the final consensus is what the market expected right before. You cannot study how the expectation formed (revisions of the consensus in the preceding weeks) or build a signal that trades before the release with the consensus at that moment, because that version is not there: using the final consensus for a decision days earlier would be look-ahead.
Footnotes
-
FRED API documentation:
fred/series/observations(parametersrealtime_start,realtime_end,vintage_dates,output_type; 32-characterapi_key) and “Real-Time Periods”: by default the real-time period is today; ALFRED lets you retrieve what was known on a past date. ↩ ↩2 -
ECB Data Portal, “Data”: entry point
data-api.ecb.europa.eu/service/, formatscsvdataandjsondata, parametersstartPeriod,endPeriod,updatedAfterandincludeHistory(earlier versions of the data). ↩ -
INE, “Referencia de la API” (API reference): structure
https://servicios.ine.es/wstempus/js/{language}/{function}/{input}, functionDATOS_SERIEand parameternult. ↩
Each answer is folded under its question
Self-assessment
The first estimate of a quarter's GDP was −0.3% and the latest version is +1.7%. Which figure should a backtest use to make a decision on the day of the first release?
- A−0.3%, the one known that day
- B+1.7%, because it is the correct one
- CThe average of the two
- DNeither: GDP is useless for trading
Show answer
Answer A. −0.3%, the one known that day
The market reacted to −0.3%. The latest version did not exist that day: using it is look-ahead. That is why ALFRED exists: it keeps every version with its date.
In the FRED API, which parameter lets you request the data as it stood on a past date?
- A
file_type - B
realtime_startandrealtime_end(orvintage_dates) - C
series_id - D
frequency
Show answer
Answer B.
realtime_startandrealtime_end(orvintage_dates)The real-time period says what was known and when. By default it is today: FRED returns the current version. With a past period or with
vintage_dates, it returns what was known back then;output_type=4returns only the first release.- A
A Valdoria figure is released at 08:30 New York time. At what UTC time does it appear in the calendar?
- AAlways at 13:30
- BAt 08:30
- CAt 12:30 during US daylight saving time and at 13:30 outside it
- DAt 14:30
Show answer
Answer C. At 12:30 during US daylight saving time and at 13:30 outside it
New York is at UTC−4 in summer and UTC−5 in winter. Storing the calendar in UTC is right, but a fixed rule such as “13:30 UTC” gets the time wrong for about seven months of the year.
The unemployment consensus was 4.7% and 4.3% is released. What is the surprise and what sign does it have for the economy?
- A+0.4 points: worse than expected
- B4.3%: the surprise is the figure
- C0: the consensus is always right
- D−0.4 points: less unemployment than expected, a positive surprise for the economy
Show answer
Answer D. −0.4 points: less unemployment than expected, a positive surprise for the economy
Surprise = released − consensus = 4.3 − 4.7 = −0.4. For unemployment, a negative sign is good news; to compare events it helps to give each one its economic sign and normalize it by the dispersion of its surprises.
Why does an earnings call transcript need its own timestamp in an NLP dataset?
- ABecause the text changes over time
- BTo sort them alphabetically
- CBecause the information reaches the market during the call, not at the quarter's end or when the 10-Q is filed: it is the date from which a signal can use it
- DIt does not need one
Show answer
Answer C. Because the information reaches the market during the call, not at the quarter's end or when the 10-Q is filed: it is the date from which a signal can use it
It is the same principle as with fundamentals and macro: each data point counts from the moment it becomes known. A sentiment signal built on the transcript can only trade after the call (level 6).
Key points
- Macro data gets revised: in this data, 81% of GDP quarters change between the first and the latest version, with a mean absolute revision of 0.49 points.
- A backtest must use the vintage known on each date. FRED returns the current version by default; ALFRED and the real-time parameters return the earlier ones.
- What moves the price is the surprise versus consensus, not the figure. It has to be normalized by event and given its economic sign.
- Store the calendar in UTC and convert it to the market's local time with zoneinfo: the UTC time of the same release changes with daylight saving time, and holidays push the reaction to the next session.
- Analyst estimates and transcripts are dated data, like financial statements: they count from the moment they are published.