Lesson 1.5.2 · 60 min

Macro data, economic calendars, estimates and transcripts

A macro figure is published, revised and revised again: the GDP you see today is not the one that moved the market on the day it came out. This lesson works with first releases and their revisions, the calendar of when each figure becomes known (in UTC) and the surprise versus consensus, which is what really moves prices.

Fig. — Valdoria quarterly GDP, 2019–2025: first estimate and latest vintage of each quarter; the bar is the revision
02/02
Module
1.5 · Fundamental and macro data
Duration
60 min
Updated
Level 1 lessons
  1. 1.1 · How markets work

    1. 1.1.1 Market participants and trading venues
    2. 1.1.2 Sessions, auctions, holidays and calendars
    3. 1.1.3 Order book: bid, ask, spread and depth
    4. 1.1.4 Order types
    5. 1.1.5 Costs, settlement, leverage and margin
  2. 1.2 · Asset classes

    1. 1.2.1 Stocks: splits, dividends and corporate actions
    2. 1.2.2 ETFs: replication, tracking error, leveraged ETFs and UCITS
    3. 1.2.3 Futures: specifications, margin, roll and contango
    4. 1.2.4 Forex: pairs, pips, lots, rollover and CFDs
    5. 1.2.5 Options: calls, puts, intrinsic value and time value
    6. 1.2.6 Crypto: spot, perpetuals, funding and custody
    7. 1.2.7 Fixed income and interest rates; CFDs and leveraged products
  3. 1.3 · Market data sources

    1. 1.3.1 Types of market data: EOD, bars, ticks, order book and reference
    2. 1.3.2 Free market data sources
    3. 1.3.3 Retail and prosumer market data APIs
    4. 1.3.4 Professional market data providers
    5. 1.3.5 Crypto and on-chain data
    6. 1.3.6 Brokers as market data sources
    7. 1.3.7 Market data licensing and terms of use
    8. 1.3.8 How to choose a market data provider
  4. 1.4 · Financial data engineering

    1. 1.4.1 Downloading over REST and websockets: pagination, rate limits and retries
    2. 1.4.2 Data cleaning: missing values, bad ticks, gaps and time zones
    3. 1.4.3 Adjusting for splits and dividends: adjusted and unadjusted prices
    4. 1.4.4 Universes without survivorship bias: delistings, mergers and symbol changes
    5. 1.4.5 Building bars: time, tick, volume, dollar and imbalance
    6. 1.4.6 Continuous futures series: roll rules, difference adjustment and ratio adjustment
    7. 1.4.7 Storage: CSV, Parquet, partitioning, DuckDB, TimescaleDB and ArcticDB
    8. 1.4.8 Reproducible pipeline: download, validation, incremental updates and quality tests
  5. 1.5 · Fundamental and macro data

    1. 1.5.1 Financial statements, ratios, XBRL and SEC EDGAR
    2. 1.5.2 Macro data, economic calendars, estimates and transcripts
  6. 1.6 · Financial exploratory analysis

    1. 1.6.1 Returns, volatility, rolling correlations and drawdown
    2. 1.6.2 Stylized facts: fat tails, clustering, autocorrelation and leverage
    3. 1.6.3 Seasonality: day of the week, month, turn of the month and intraday sessions
  7. 1.7 · Programmed technical analysis

    1. 1.7.1 Technical indicators in Python
    2. 1.7.2 Your own implementation versus TA-Lib: validating results
    3. 1.7.3 Candlestick patterns and support and resistance
    4. 1.7.4 Multi-timeframe analysis without look-ahead

One figure, several versions

Macro statistics are published before they are complete. A statistics agency gives a first estimate of GDP with part of the quarter's information and corrects it when the missing surveys, tax data or trade balances come in. CPI usually has a flash estimate and a final figure; unemployment is revised the following month and in the annual survey revision. Each version is a vintage: the series as it was known on a date.

AdvanceSecond estimateThird estimateAnnual revision
Mermaid source
flowchart LR
  A[Advance] --> S[Second estimate]
  S --> T[Third estimate]
  T --> R[Annual revision]

For a trading system this has two consequences. The first is the one from lesson 1.5.1 under another name: if a backtest uses today's series, it uses figures that did not exist when it simulates the decision. The second is more subtle: the market does not react to the figure, but to the difference between the figure and what was expected. Both are solved the same way, with a knowledge date per figure.

The data in this lesson are fictional series from Valdoria, a made-up country with a release calendar similar to that of the US, and from Brenia, one with a European calendar. macro_vintages.csv holds every release of four series with its date:

IN [01]python
import pandas as pd

mv = pd.read_csv("macro_vintages.csv",
                 parse_dates=["fecha_publicacion"])
print(mv.groupby(["serie", "vintage"]).size().unstack(fill_value=0))
mv[(mv["serie"] == "pib_trimestral")
   & mv["periodo"].isin(["2019T4", "2020T1"])]
OUT [01]Macro series with their vintages
vintage          1   2   3   4
serie
ipc_interanual  83  83   0   0
paro            83  82  19   0
pib_trimestral  27  27  27  14
tipo_oficial    84   0   0   0
              serie periodo fecha_publicacion  valor  vintage
362  pib_trimestral  2019T4        2020-01-30   -0.3        1
363  pib_trimestral  2019T4        2020-02-27    2.3        2
364  pib_trimestral  2019T4        2020-03-26    1.7        3
365  pib_trimestral  2020T1        2020-04-30   -3.8        1
366  pib_trimestral  2020T1        2020-05-28   -5.0        2
367  pib_trimestral  2020T1        2020-06-25   -4.6        3
368  pib_trimestral  2020T1        2021-07-29   -4.5        4

CPI has two versions per month (flash and final); unemployment, two or three; GDP, three estimates and, in half of the quarters, an annual revision; the policy rate is not revised. The fourth quarter of 2019 is the extreme example: the first estimate, on January 30, 2020, said −0.3% (annualized quarterly rate); a month later, +2.3%; the third, +1.7%.

First release versus latest version

How much is each series revised? Comparing the first version of each period with the latest:

IN [02]python
def primera_y_ultima(mv, serie):
    s = mv[mv["serie"] == serie].sort_values("vintage")
    g = s.groupby("periodo")["valor"]
    return pd.DataFrame({"primera": g.first(), "ultima": g.last()})

filas = {}
for serie in ["ipc_interanual", "paro", "pib_trimestral"]:
    pu = primera_y_ultima(mv, serie)
    rev = pu["ultima"] - pu["primera"]
    filas[serie] = {"periodos": len(pu),
                    "revisados_%": 100 * (rev.abs() > 1e-9).mean(),
                    "rev_media_abs": rev.abs().mean(),
                    "rev_max_abs": rev.abs().max()}
pd.DataFrame(filas).T.round(2)
OUT [02]Size of the revisions
                periodos  revisados_%  rev_media_abs  rev_max_abs
ipc_interanual      83.0        61.45           0.07          0.3
paro                83.0        54.22           0.06          0.2
pib_trimestral      27.0        81.48           0.49          2.0

CPI and unemployment move little: tenths of a point, in just over half of the months (61% and 54%). GDP is revised in 81.5% of the quarters, with a mean revision of almost half a point and a maximum of 2 points, the one for the fourth quarter of 2019. The figure at the top puts the two versions of each quarter side by side.

Half a point of annualized GDP is the difference between “the economy is slowing down” and “the economy is doing fine”. And in some quarters the sign changes:

IN [03]python
pib = primera_y_ultima(mv, "pib_trimestral")
signo = (pib["primera"] < 0) != (pib["ultima"] < 0)
pib[signo]
OUT [03]Quarters that change sign
         primera  ultima
periodo
2019T4      -0.3     1.7

With the first estimate, the fourth quarter of 2019 was a contraction. A rule such as “reduce exposure if last quarter's GDP is negative” would have acted in February 2020 on that figure. With today's series, that signal does not exist. Neither of the two is “the right one”: to study the economy, the latest; to simulate a decision, the one that was known.

The series as it was known: vintages and ALFRED

Rebuilding the series in force on a date means filtering by release date and keeping the latest version of each period. That is the idea behind ALFRED (Archival FRED), the vintage archive of the Federal Reserve Bank of St. Louis: by default FRED returns the series as it stands today; ALFRED lets you request it as it stood on a past date1Note 1 · FRED API documentation: fred/series/observations (parameters realtime_start, realtime_end, vintage_dates, output_type; 32-character api_key) and “Real-Time Periods”: by default the real-time period is today; ALFRED lets you retrieve what was known on a past date..

IN [04]python
def conocida_en(mv, serie, fecha):
    """The series as it was known at the close of `fecha`."""
    s = mv[(mv["serie"] == serie)
           & (mv["fecha_publicacion"] <= fecha)]
    return (s.sort_values("vintage")
            .groupby("periodo")["valor"].last())

fechas = ["2020-02-15", "2020-06-30", "2025-12-31"]
cortes = {f: conocida_en(mv, "pib_trimestral", f) for f in fechas}
pd.DataFrame(cortes).loc["2019T3":"2020T2"]
OUT [04]The series in force on a date
         2020-02-15  2020-06-30  2025-12-31
periodo
2019T3          2.8         2.8         3.0
2019T4         -0.3         1.7         1.7
2020T1          NaN        -4.6        -4.5
2020T2          NaN         NaN       -29.4

Each column is a vintage. On February 15, 2020 nobody knew anything about the first quarter of 2020 and the fourth quarter of 2019 was −0.3; by the end of June the fourth quarter of 2019 was already +1.7 and the first quarter of 2020 had been revised twice. The full table, periods in rows and vintage dates in columns, is the “revision triangle” that economists use to study the quality of first estimates.

To use a macro series as a variable in a daily backtest, the join is the same as with fundamentals: a merge_asof by release date, with each day's version. Comparing it with the naive join (the latest version of each quarter, from the day the quarter ends):

IN [05]python
sesiones = pd.read_csv("fix_indice_diario.csv",
                       parse_dates=["fecha"])[["fecha"]]
p = mv[mv["serie"] == "pib_trimestral"].copy()
# end of each quarter: 2019T4 -> 2019-12-31
p["periodo_fin"] = (pd.PeriodIndex(p["periodo"].str.replace(
    "T", "Q"), freq="Q").end_time.normalize())

pit = pd.merge_asof(
    sesiones, p.sort_values("fecha_publicacion")
    [["fecha_publicacion", "periodo", "valor"]],
    left_on="fecha", right_on="fecha_publicacion",
    allow_exact_matches=False)
ult = p.sort_values("vintage").groupby("periodo").last()
ingenuo = pd.merge_asof(
    sesiones, ult.reset_index().sort_values("periodo_fin")
    [["periodo_fin", "periodo", "valor"]],
    left_on="fecha", right_on="periodo_fin")

c = pit.merge(ingenuo, on="fecha", suffixes=("_pit", "_ing"))
c = c.dropna(subset=["valor_pit", "valor_ing"])
print(f"Sessions: {len(c)}")
print("Different quarter:",
      f"{(c['periodo_pit'] != c['periodo_ing']).mean():.1%}")
print("Different value:",
      f"{(c['valor_pit'] != c['valor_ing']).mean():.1%}")
negativo_distinto = (c["valor_pit"] < 0) != (c["valor_ing"] < 0)
print(f"Different sign: {negativo_distinto.mean():.1%}")
OUT [05]Point-in-time versus naive GDP, session by session
Sessions: 1681
Different quarter: 36.6%
Different value: 89.5%
Different sign: 9.8%

In 89.5% of the sessions the naive join uses a value different from the one that was known and in 36.6%, even a different quarter: one that had not been released yet. In one in ten (9.8%) not even the sign of growth matches. A macro model trained this way learns a relationship between the economy and the market that no investor could have exploited.

The economic calendar: when each figure becomes known

An economic calendar lists, for each release, the date and time, the consensus (the mean or median of the analyst forecasts surveyed by the calendar provider), the previous value and the released one. calendario_economico.csv is the 2025 calendar for Valdoria and Brenia:

IN [06]python
ce = pd.read_csv("calendario_economico.csv",
                 parse_dates=["fecha_hora"])
print(ce["fecha_hora"].dtype, len(ce), "releases")
ce[["fecha_hora", "evento", "pais", "consenso",
    "publicado"]].head(6)
OUT [06]2025 economic calendar
datetime64[us, UTC] 123 releases
                 fecha_hora                       evento      pais  consenso  publicado
0 2025-01-02 09:00:00+00:00            PMI manufacturero    Brenia     47.70      47.70
1 2025-01-02 15:00:00+00:00            PMI manufacturero  Valdoria     51.30      51.00
2 2025-01-03 13:30:00+00:00                 Tasa de paro  Valdoria      4.30       4.20
3 2025-01-13 13:30:00+00:00  IPC interanual (definitivo)  Valdoria      2.70       2.40
4 2025-01-16 13:30:00+00:00   Ventas minoristas, mensual  Valdoria      0.70       0.60
5 2025-01-29 19:00:00+00:00            Decisión de tipos  Valdoria      4.35       4.35

The times are in UTC, as they should be stored. Before using it, it is worth checking that it matches the series: the released value of each Valdoria CPI flash estimate must be vintage 1 of that month, and the “previous” value, the latest known version of the prior month.

IN [07]python
av = ce[(ce["pais"] == "Valdoria")
        & (ce["evento"] == "IPC interanual (avance)")].copy()
av["fecha"] = av["fecha_hora"].dt.tz_convert(
    "America/New_York").dt.tz_localize(None).dt.normalize()
ipc1 = mv[(mv["serie"] == "ipc_interanual")
          & (mv["vintage"] == 1)]
cmp = av.merge(ipc1, left_on="fecha",
               right_on="fecha_publicacion")
print(len(av), "flash estimates;", len(cmp), "with their vintage;",
      "equal:", (cmp["publicado"] == cmp["valor"]).all())
OUT [07]The calendar versus the vintages
11 flash estimates; 11 with their vintage; equal: True

All eleven releases match. It is a cheap check worth automating in a pipeline (lesson 1.4.8): a calendar from one provider and a series from another can disagree on the vintage, the unit or the day.

The surprise: figure minus consensus

An asset's price already discounts what the market expects. What arrives with the release is the unexpected part:

surprise=released−consensus\text{surprise} = \text{released} - \text{consensus}

To compare events with different units and volatilities, the surprise is normalized by dividing it by the standard deviation of that event's surprises (standardized surprise). In a real study that deviation is estimated with the past only, with a window that does not include the release being evaluated; here, with one year of data, the full year's deviation is used to describe the distribution.

IN [08]python
ce["sorpresa"] = ce["publicado"] - ce["consenso"]
corto = ce["evento"].str.replace(
    "PIB trimestral, tasa anualizada", "PIB")
ce["evento_pais"] = corto + " · " + ce["pais"]
res = ce.groupby("evento_pais")["sorpresa"].agg(
    n="size", media="mean", desv="std",
    exactas=lambda s: (s.abs() < 1e-9).mean())
res.sort_values("desv", ascending=False).round(2)
OUT [08]Surprises by event
                                         n  media  desv  exactas
evento_pais
Confianza del consumidor · Valdoria     12  -0.67  1.27     0.00
PIB (segunda estimación) · Valdoria      4  -0.03  0.51     0.00
PMI manufacturero · Valdoria            12  -0.07  0.46     0.00
PIB (avance) · Valdoria                  4  -0.47  0.43     0.00
PMI manufacturero · Brenia              12  -0.09  0.43     0.17
Ventas minoristas, mensual · Valdoria   12  -0.07  0.31     0.33
PIB (tercera estimación) · Valdoria      4  -0.35  0.31     0.00
IPC interanual (avance) · Valdoria      11   0.08  0.22     0.09
IPC interanual (definitivo) · Valdoria  12  -0.02  0.16     0.50
Tasa de paro · Valdoria                 12  -0.07  0.15     0.50
IPC interanual (avance) · Brenia        12  -0.01  0.08     0.42
Decisión de tipos · Brenia               8   0.00  0.00     1.00
Decisión de tipos · Valdoria             8   0.00  0.00     1.00
Fig. — 2025 surprises by event (actual − consensus) divided by the standard deviation of that event's surprises; unemployment with the sign flipped (less unemployment = positive surprise)

Rate decisions never surprise in this year: the consensus gets all sixteen right, as expected from central banks that prepare the market before deciding. Unemployment and final CPI are right half the time: they move by tenths and, for the final figure, the flash estimate is already out. The most dispersed surprises are those of consumer confidence (1.27 index points), GDP and the PMIs. Three of the four surprises in the GDP advance estimate were negative (mean of −0.47 points), but with four data points that is not a consensus bias, it is an anecdote. The deviations are in different units (index points, percentages): that is why they have to be normalized. An event without surprises barely moves the price; one with large, frequent surprises is the one that deserves an event study (level 2).

IN [09]python
desv = ce.groupby("evento_pais")["sorpresa"].transform("std")
ce["z"] = ce["sorpresa"] / desv
# for unemployment, more is worse: flip the sign
ce.loc[ce["evento"] == "Tasa de paro", "z"] *= -1
top = ce.dropna(subset=["z"]).reindex(
    ce["z"].abs().sort_values(ascending=False).index)
top[["fecha_hora", "evento_pais", "consenso",
     "publicado", "z"]].head(5).round({"z": 2})
OUT [09]Standardized surprise and economic sign
                  fecha_hora                            evento_pais  consenso  publicado     z
54 2025-06-06 12:30:00+00:00                Tasa de paro · Valdoria       4.7        4.3  2.67
50 2025-05-30 14:00:00+00:00    Confianza del consumidor · Valdoria      95.3       92.3 -2.36
46 2025-05-16 12:30:00+00:00  Ventas minoristas, mensual · Valdoria       0.7       -0.0 -2.25
37 2025-04-24 12:30:00+00:00                PIB (avance) · Valdoria      -0.3       -1.2 -2.07
27 2025-03-27 12:30:00+00:00    PIB (tercera estimación) · Valdoria       3.0        2.4 -1.93

With the standardized surprise and the economic sign (for unemployment, less is better), releases of different indicators can be put on the same scale and ranked by how unexpected they were.

Times in UTC, markets in local time

Valdoria releases its data at 08:30 New York time and decides on rates at 14:00; Brenia releases at 10:00 or 11:00 Madrid time and decides at 14:15. In UTC, those times change with daylight saving time, and not at the same moment: the US switches on the second Sunday of March and the first Sunday of November; the EU, on the last Sunday of March and of October (lesson 0.4.4).

IN [10]python
ny = ce["fecha_hora"].dt.tz_convert("America/New_York")
mad = ce["fecha_hora"].dt.tz_convert("Europe/Madrid")
ce["hora_local"] = ny.dt.strftime("%H:%M").where(
    ce["pais"] == "Valdoria", mad.dt.strftime("%H:%M"))
ce["hora_utc"] = ce["fecha_hora"].dt.strftime("%H:%M")
print(pd.crosstab([ce["pais"], ce["hora_local"]], ce["hora_utc"]))

# 2025 clock changes according to the time zone database (tzdata)
dias = pd.date_range("2025-01-01", "2025-12-31", freq="D",
                     tz="UTC") + pd.Timedelta(hours=12)
for zona in ["America/New_York", "Europe/Madrid"]:
    off = dias.tz_convert(zona).map(lambda t: t.utcoffset())
    cambio = dias[1:][off[1:] != off[:-1]]
    print(zona, [str(d.date()) for d in cambio])
OUT [10]The same local time, two UTC times
hora_utc             08:00  09:00  10:00  12:15  12:30  13:15  13:30  14:00  15:00  18:00  19:00
pais     hora_local
Brenia   10:00           7      5      0      0      0      0      0      0      0      0      0
         11:00           0      7      5      0      0      0      0      0      0      0      0
         14:15           0      0      0      4      0      4      0      0      0      0      0
Valdoria 08:30           0      0      0      0     39      0     20      0      0      0      0
         10:00           0      0      0      0      0      0      0     15      9      0      0
         14:00           0      0      0      0      0      0      0      0      0      6      2
America/New_York ['2025-03-09', '2025-11-02']
Europe/Madrid ['2025-03-30', '2025-10-26']
Fig. — UTC time of the 2025 releases at 08:30 New York (Valdoria) and 10:00 Madrid (Brenia); in the bands, the two daylight-saving schedules do not overlap

08:30 in New York is 12:30 UTC during daylight saving time and 13:30 outside it. A filter written as “do not trade between 13:25 and 13:45 UTC” is right in winter and an hour late from March to November. According to the time zone database, in 2025 New York switched on March 9 and November 2, and Madrid on March 30 and October 26. Between those dates, New York and Madrid were five hours apart instead of six: two windows a year in which a rule like “Madrid time minus six” fails.

yesnoUTC calendarLocal timeMarket open?Event barNext sessionReaction window
Mermaid source
flowchart TB
  C[UTC calendar] --> L[Local time]
  L --> S{Market open?}
  S -- yes --> B[Event bar]
  S -- no --> N[Next session]
  B --> V[Reaction window]
  N --> V

For a reaction study, each release has to be joined to the first bar in which the market can react. With NOVA's 5-minute bars (09:30 to 16:00 New York session) and a forward merge_asof:

IN [11]python
barras = pd.read_parquet("nova_barras_5min.parquet")
ev = ce[ce["pais"] == "Valdoria"].sort_values("fecha_hora")
m = pd.merge_asof(ev, barras[["ts"]], left_on="fecha_hora",
                  right_on="ts", direction="forward")
m["espera_min"] = (m["ts"] - m["fecha_hora"]).dt.total_seconds() / 60
print(m.groupby("hora_local")["espera_min"].median())
largas = m[m["espera_min"] > 120]
largas[["fecha_hora", "ts", "espera_min"]]
OUT [11]First bar after each Valdoria release
hora_local
08:30    60.0
10:00     0.0
14:00     0.0
Name: espera_min, dtype: float64
                  fecha_hora                        ts  espera_min
11 2025-02-17 13:30:00+00:00 2025-02-18 14:30:00+00:00      1500.0
47 2025-07-04 12:30:00+00:00 2025-07-07 13:30:00+00:00      4380.0
61 2025-09-01 14:00:00+00:00 2025-09-02 13:30:00+00:00      1410.0
81 2025-11-27 13:30:00+00:00 2025-11-28 14:30:00+00:00      1500.0
89 2025-12-25 13:30:00+00:00 2025-12-26 14:30:00+00:00      1500.0

The 08:30 releases arrive while the stock market is closed: the first bar that can react is the opening bar, an hour later, and whatever happened in between is in the opening gap (and in futures and currencies, which do trade). The 10:00 releases and the 14:00 decision fall within the session. And there are five releases on days when the NYSE is closed (in this fictional calendar, February 17, July 4, September 1, November 27 and December 25): their first bar is in the next session, one to three days later. A study that measured the reaction “in the 09:30 bar on the day of the release” would mix those five with sessions without news.

The APIs: FRED, ECB and INE

FRED, the ECB and INE (Spain's National Statistics Institute), three public sources of macro data for the US, the euro area and Spain, have open APIs. All of them need network access, so the code is not executed.

FRED (Federal Reserve Bank of St. Louis) requires a 32-character key (api_key). The observations endpoint accepts the real-time period (realtime_start, realtime_end), a list of vintage dates (vintage_dates) and the output type (output_type: 1, by real-time period; 2, all observations by vintage; 3, only new and revised ones; 4, only the first release)1:

IN [12]python
import os
import requests

URL = "https://api.stlouisfed.org/fred/series/observations"
base = {"series_id": "GDP", "api_key": os.environ["FRED_API_KEY"],
        "file_type": "json"}

hoy = requests.get(URL, params=base, timeout=30).json()
primeras = requests.get(URL, params=base | {"output_type": 4},
                        timeout=30).json()
en_2020 = requests.get(URL, params=base | {
    "realtime_start": "2020-02-15", "realtime_end": "2020-02-15"},
    timeout=30).json()
# each observation: realtime_start, realtime_end, date, value

The response carries an observations list in which each row has date (the period), value (as text; "." when missing) and the real-time interval during which that figure was in force. It is exactly the structure of macro_vintages.csv, with two date columns instead of one.

ECB (Data Portal): URLs follow the SDMX syntax data-api.ecb.europa.eu/service/data/<flow>/<key>, with no API key. format=csvdata or jsondata choose the format; startPeriod and endPeriod, the range; updatedAfter, only what changed since a date (useful for updating a local database), and includeHistory=true also returns earlier versions of the data2Note 2 · ECB Data Portal, “Data”: entry point data-api.ecb.europa.eu/service/, formats csvdata and jsondata, parameters startPeriod, endPeriod, updatedAfter and includeHistory (earlier versions of the data).:

IN [13]python
import io

url = ("https://data-api.ecb.europa.eu/service/data/"
       "EXR/D.USD.EUR.SP00.A")
r = requests.get(url, params={"format": "csvdata",
                              "startPeriod": "2025-01-01"},
                 timeout=30)
eurusd = pd.read_csv(io.StringIO(r.text))
# SDMX columns: KEY, FREQ, CURRENCY, ..., TIME_PERIOD, OBS_VALUE

INE: the JSON API has the form servicios.ine.es/wstempus/js/{language}/{function}/{input}; the DATOS_SERIE function returns the data of a series by its code and the nult parameter limits the response to the latest periods3Note 3 · INE, “Referencia de la API” (API reference): structure https://servicios.ine.es/wstempus/js/{language}/{function}/{input}, function DATOS_SERIE and parameter nult.. The date of each data point (Fecha) comes as milliseconds since 1970, and there is a time zone trap here. The following sample response has the API's structure with a made-up code and values:

IN [14]python
import json

ej = json.loads("""{"COD": "SERIE_EJEMPLO",
 "Nombre": "Example series. Annual change.",
 "Data": [
  {"Fecha": 1761951600000, "FK_Periodo": 11, "Anyo": 2025,
   "Valor": 3.1},
  {"Fecha": 1764543600000, "FK_Periodo": 12, "Anyo": 2025,
   "Valor": 2.8}]}""")
d = pd.DataFrame(ej["Data"])
d["utc"] = pd.to_datetime(d["Fecha"], unit="ms", utc=True)
d["madrid"] = d["utc"].dt.tz_convert("Europe/Madrid")
d["mes_mal"] = d["utc"].dt.strftime("%Y-%m")
d["mes_bien"] = d["madrid"].dt.strftime("%Y-%m")
d[["FK_Periodo", "utc", "mes_mal", "mes_bien", "Valor"]]
OUT [14]INE: milliseconds that are midnight in Madrid
   FK_Periodo                       utc  mes_mal mes_bien  Valor
0          11 2025-10-31 23:00:00+00:00  2025-10  2025-11    3.1
1          12 2025-11-30 23:00:00+00:00  2025-11  2025-12    2.8

Read in UTC, both data points fall on the last day of the previous month: November's looks like October's. The timestamp is midnight in Madrid on the first day of the period, which in UTC is 23:00 (or 22:00 in summer) on the day before. Converting to Europe/Madrid before extracting the month, or using Anyo and FK_Periodo, solves it. The real request:

IN [15]python
url = ("https://servicios.ine.es/wstempus/js/ES/DATOS_SERIE/"
       "<series code>")
r = requests.get(url, params={"nult": 12}, timeout=30)
datos = r.json()["Data"]

None of these APIs says when each figure was published in the default response: FRED needs the real-time period, the ECB includeHistory, and for INE you have to check its release calendar. If you store macro data for backtests, also store the day you downloaded it: your own database then becomes a vintage archive.

Analyst estimates and transcripts

Analyst estimates (earnings per share, revenue, recommendations) are the corporate version of the macro consensus: each analyst publishes a forecast, the provider computes the mean or median, and the earnings surprise is the reported earnings minus that consensus. They have the same date problems: each estimate gets revised, a quarter's consensus changes day by day until the release, and a dataset without the date of each revision is no good for simulation. Before paying for an estimates database (lesson 1.3.4), check whether it stores each forecast with its date or only the consensus at the time of download.

Transcripts of earnings presentations (earnings calls) are text: what management says about the quarter and the outlook, and the analysts' questions. They are the input to the language models of level 6: tone, changes in guidance, words that anticipate problems.

TranscriptTimestampClean textNLP signal
Mermaid source
flowchart LR
  T[Transcript] --> M[Timestamp]
  M --> X[Clean text]
  X --> F[NLP signal]

The principle is the same as in this whole section: a transcript counts from the moment the call ends, not from the end of the quarter or from the 10-Q filing. Lessons 6.1.2 and 6.2.3 build signals on filings and transcripts with that timestamp.

What this changes in a backtest

  • Use vintages. If the source only gives the current series, store your downloads with their date or use ALFRED. For a series without a vintage archive, delay each data point to its real release date, even if you keep using the final version, and say so.
  • Trade the surprise, not the level. The CPI level is in prices long before it is released; what is new is the difference from the consensus.
  • UTC for storage, local time for trading. Convert with zoneinfo every time; never with fixed offsets.
  • Holidays and closed markets. A release while the market is closed is reflected at the next open or in the markets that do trade (futures, currencies). Decide which one you measure.

Exercises

  1. For unemployment, compute the revision between vintage 1 and 2 and between 2 and 3 (when it exists). Which of the two revisions is larger in mean absolute terms?
Solution
IN [16]python
paro = mv[mv["serie"] == "paro"].pivot(
    index="periodo", columns="vintage", values="valor")
r12 = (paro[2] - paro[1]).abs()
r23 = (paro[3] - paro[2]).abs()
print(f"v1 -> v2: mean {r12.mean():.3f}, n = {r12.count()}")
print(f"v2 -> v3: mean {r23.mean():.3f}, n = {r23.count()}")
OUT [16]Unemployment revisions
v1 -> v2: mean 0.062, n = 82
v2 -> v3: mean 0.100, n = 19

The annual revision (vintage 3) only exists in the months that change, so its mean is computed over few months and is larger: when the annual revision touches a month, it moves it more than the monthly revision. For a model with unemployment data, vintage 3 arrives more than a year after the period: if it is used from the start, it brings in information from a year later.

  1. For each Valdoria release, compute the New York local time and check that all the 08:30 ones fall outside the NYSE session and that the rate decision falls within it.
Solution
IN [17]python
val = ce[ce["pais"] == "Valdoria"].copy()
local = val["fecha_hora"].dt.tz_convert("America/New_York")
minuto = local.dt.hour * 60 + local.dt.minute
val["en_sesion"] = minuto.between(9 * 60 + 30, 16 * 60 - 1)
val.groupby("hora_local")["en_sesion"].mean()
OUT [17]Inside and outside the session
hora_local
08:30    0.0
10:00    1.0
14:00    1.0
Name: en_sesion, dtype: float64

All the 08:30 releases are outside the cash session and the 10:00 and 14:00 ones, inside. The check does not look at holidays: for that you need the exchange calendar (exchange_calendars, lesson 1.1.2), which is what the merge_asof with NOVA's real bars did.

  1. A provider offers you a history of consensus figures with a single row per release (the final consensus). What study can you do with it and what study can you not?
Solution

With the final consensus and the released figure you can measure the surprise and study the price reaction in the minutes or days after the release: it is an event study, and the final consensus is what the market expected right before. You cannot study how the expectation formed (revisions of the consensus in the preceding weeks) or build a signal that trades before the release with the consensus at that moment, because that version is not there: using the final consensus for a decision days earlier would be look-ahead.

Footnotes

  1. FRED API documentation: fred/series/observations (parameters realtime_start, realtime_end, vintage_dates, output_type; 32-character api_key) and “Real-Time Periods”: by default the real-time period is today; ALFRED lets you retrieve what was known on a past date. ↩ ↩2

  2. ECB Data Portal, “Data”: entry point data-api.ecb.europa.eu/service/, formats csvdata and jsondata, parameters startPeriod, endPeriod, updatedAfter and includeHistory (earlier versions of the data). ↩

  3. INE, “Referencia de la API” (API reference): structure https://servicios.ine.es/wstempus/js/{language}/{function}/{input}, function DATOS_SERIE and parameter nult. ↩

Each answer is folded under its question

Self-assessment

  1. The first estimate of a quarter's GDP was −0.3% and the latest version is +1.7%. Which figure should a backtest use to make a decision on the day of the first release?

    • A−0.3%, the one known that day
    • B+1.7%, because it is the correct one
    • CThe average of the two
    • DNeither: GDP is useless for trading
    Show answer

    Answer A. −0.3%, the one known that day

    The market reacted to −0.3%. The latest version did not exist that day: using it is look-ahead. That is why ALFRED exists: it keeps every version with its date.

  2. In the FRED API, which parameter lets you request the data as it stood on a past date?

    • Afile_type
    • Brealtime_start and realtime_end (or vintage_dates)
    • Cseries_id
    • Dfrequency
    Show answer

    Answer B. realtime_start and realtime_end (or vintage_dates)

    The real-time period says what was known and when. By default it is today: FRED returns the current version. With a past period or with vintage_dates, it returns what was known back then; output_type=4 returns only the first release.

  3. A Valdoria figure is released at 08:30 New York time. At what UTC time does it appear in the calendar?

    • AAlways at 13:30
    • BAt 08:30
    • CAt 12:30 during US daylight saving time and at 13:30 outside it
    • DAt 14:30
    Show answer

    Answer C. At 12:30 during US daylight saving time and at 13:30 outside it

    New York is at UTC−4 in summer and UTC−5 in winter. Storing the calendar in UTC is right, but a fixed rule such as “13:30 UTC” gets the time wrong for about seven months of the year.

  4. The unemployment consensus was 4.7% and 4.3% is released. What is the surprise and what sign does it have for the economy?

    • A+0.4 points: worse than expected
    • B4.3%: the surprise is the figure
    • C0: the consensus is always right
    • D−0.4 points: less unemployment than expected, a positive surprise for the economy
    Show answer

    Answer D. −0.4 points: less unemployment than expected, a positive surprise for the economy

    Surprise = released − consensus = 4.3 − 4.7 = −0.4. For unemployment, a negative sign is good news; to compare events it helps to give each one its economic sign and normalize it by the dispersion of its surprises.

  5. Why does an earnings call transcript need its own timestamp in an NLP dataset?

    • ABecause the text changes over time
    • BTo sort them alphabetically
    • CBecause the information reaches the market during the call, not at the quarter's end or when the 10-Q is filed: it is the date from which a signal can use it
    • DIt does not need one
    Show answer

    Answer C. Because the information reaches the market during the call, not at the quarter's end or when the 10-Q is filed: it is the date from which a signal can use it

    It is the same principle as with fundamentals and macro: each data point counts from the moment it becomes known. A sentiment signal built on the transcript can only trade after the call (level 6).

Key points

  • Macro data gets revised: in this data, 81% of GDP quarters change between the first and the latest version, with a mean absolute revision of 0.49 points.
  • A backtest must use the vintage known on each date. FRED returns the current version by default; ALFRED and the real-time parameters return the earlier ones.
  • What moves the price is the surprise versus consensus, not the figure. It has to be normalized by event and given its economic sign.
  • Store the calendar in UTC and convert it to the market's local time with zoneinfo: the UTC time of the same release changes with daylight saving time, and holidays push the reaction to the next session.
  • Analyst estimates and transcripts are dated data, like financial statements: they count from the moment they are published.

Resources