--- title: "1.5.2 Macro data, economic calendars and estimates" description: "Macro data with its vintages (FRED, ALFRED, ECB, INE), the economic calendar in UTC, surprises versus consensus and transcripts as a bridge to NLP." url: "https://pythonparatrading.com/en/python-trading-course/level-1/macro-data-economic-calendars-and-estimates/" lang: "en" image: "https://pythonparatrading.com/og/en/python-trading-course/level-1/macro-data-economic-calendars-and-estimates.png" --- # 1.5.2 · Macro data, economic calendars, estimates and transcripts > A macro figure is published, revised and revised again: the GDP you see today is not the one that moved the market on the day it came out. This lesson works with first releases and their revisions, the calendar of when each figure becomes known (in UTC) and the surprise versus consensus, which is what really moves prices. - Course: [Level 1: Financial data, markets and analysis](https://pythonparatrading.com/en/python-trading-course/level-1/) - Module: 1.5 · Fundamental and macro data - Duration: 60 min - Prerequisites: [0.4.4 · Files (CSV, JSON, YAML, TOML) and dates with time zones](https://pythonparatrading.com/en/python-trading-course/level-0/python-files-pathlib-time-zones/), [1.5.1 · Financial statements, ratios, XBRL and SEC EDGAR](https://pythonparatrading.com/en/python-trading-course/level-1/financial-statements-xbrl-and-sec-edgar/) ## By the end of this lesson - Tell first release, revisions and latest version (vintage) of a macro series apart and rebuild the series as it was known on a date. - Measure how much CPI, unemployment and GDP are revised and why a backtest must use the vintage known at each point in time (ALFRED). - Compute a release's surprise versus consensus and normalize it by event. - Align an economic calendar in UTC with market bars, accounting for daylight saving time and holidays. - Read the FRED, ECB and INE APIs, and place analyst estimates and transcripts as dated sources. ## One figure, several versions Macro statistics are published before they are complete. A statistics agency gives a first estimate of GDP with part of the quarter's information and corrects it when the missing surveys, tax data or trade balances come in. CPI usually has a flash estimate and a final figure; unemployment is revised the following month and in the annual survey revision. Each version is a **vintage**: the series as it was known on a date. ```mermaid flowchart LR A[Advance] --> S[Second estimate] S --> T[Third estimate] T --> R[Annual revision] ``` For a trading system this has two consequences. The first is the one from lesson 1.5.1 under another name: if a backtest uses today's series, it uses figures that did not exist when it simulates the decision. The second is more subtle: the market does not react to the figure, but to the difference between the figure and what was expected. Both are solved the same way, with a knowledge date per figure. The data in this lesson are fictional series from **Valdoria**, a made-up country with a release calendar similar to that of the US, and from **Brenia**, one with a European calendar. `macro_vintages.csv` holds every release of four series with its date: ```python import pandas as pd mv = pd.read_csv("macro_vintages.csv", parse_dates=["fecha_publicacion"]) print(mv.groupby(["serie", "vintage"]).size().unstack(fill_value=0)) mv[(mv["serie"] == "pib_trimestral") & mv["periodo"].isin(["2019T4", "2020T1"])] ``` ```output vintage 1 2 3 4 serie ipc_interanual 83 83 0 0 paro 83 82 19 0 pib_trimestral 27 27 27 14 tipo_oficial 84 0 0 0 serie periodo fecha_publicacion valor vintage 362 pib_trimestral 2019T4 2020-01-30 -0.3 1 363 pib_trimestral 2019T4 2020-02-27 2.3 2 364 pib_trimestral 2019T4 2020-03-26 1.7 3 365 pib_trimestral 2020T1 2020-04-30 -3.8 1 366 pib_trimestral 2020T1 2020-05-28 -5.0 2 367 pib_trimestral 2020T1 2020-06-25 -4.6 3 368 pib_trimestral 2020T1 2021-07-29 -4.5 4 ``` CPI has two versions per month (flash and final); unemployment, two or three; GDP, three estimates and, in half of the quarters, an annual revision; the policy rate is not revised. The fourth quarter of 2019 is the extreme example: the first estimate, on January 30, 2020, said −0.3% (annualized quarterly rate); a month later, +2.3%; the third, +1.7%. ## First release versus latest version How much is each series revised? Comparing the first version of each period with the latest: ```python def primera_y_ultima(mv, serie): s = mv[mv["serie"] == serie].sort_values("vintage") g = s.groupby("periodo")["valor"] return pd.DataFrame({"primera": g.first(), "ultima": g.last()}) filas = {} for serie in ["ipc_interanual", "paro", "pib_trimestral"]: pu = primera_y_ultima(mv, serie) rev = pu["ultima"] - pu["primera"] filas[serie] = {"periodos": len(pu), "revisados_%": 100 * (rev.abs() > 1e-9).mean(), "rev_media_abs": rev.abs().mean(), "rev_max_abs": rev.abs().max()} pd.DataFrame(filas).T.round(2) ``` ```output periodos revisados_% rev_media_abs rev_max_abs ipc_interanual 83.0 61.45 0.07 0.3 paro 83.0 54.22 0.06 0.2 pib_trimestral 27.0 81.48 0.49 2.0 ``` CPI and unemployment move little: tenths of a point, in just over half of the months (61% and 54%). GDP is revised in 81.5% of the quarters, with a mean revision of almost half a point and a maximum of 2 points, the one for the fourth quarter of 2019. The figure at the top puts the two versions of each quarter side by side. Half a point of annualized GDP is the difference between “the economy is slowing down” and “the economy is doing fine”. And in some quarters the sign changes: ```python pib = primera_y_ultima(mv, "pib_trimestral") signo = (pib["primera"] < 0) != (pib["ultima"] < 0) pib[signo] ``` ```output primera ultima periodo 2019T4 -0.3 1.7 ``` With the first estimate, the fourth quarter of 2019 was a contraction. A rule such as “reduce exposure if last quarter's GDP is negative” would have acted in February 2020 on that figure. With today's series, that signal does not exist. Neither of the two is “the right one”: to study the economy, the latest; to simulate a decision, the one that was known. ## The series as it was known: vintages and ALFRED Rebuilding the series in force on a date means filtering by release date and keeping the latest version of each period. That is the idea behind **ALFRED** (*Archival FRED*), the vintage archive of the Federal Reserve Bank of St. Louis: by default FRED returns the series as it stands today; ALFRED lets you request it as it stood on a past date[^1]. ```python def conocida_en(mv, serie, fecha): """The series as it was known at the close of `fecha`.""" s = mv[(mv["serie"] == serie) & (mv["fecha_publicacion"] <= fecha)] return (s.sort_values("vintage") .groupby("periodo")["valor"].last()) fechas = ["2020-02-15", "2020-06-30", "2025-12-31"] cortes = {f: conocida_en(mv, "pib_trimestral", f) for f in fechas} pd.DataFrame(cortes).loc["2019T3":"2020T2"] ``` ```output 2020-02-15 2020-06-30 2025-12-31 periodo 2019T3 2.8 2.8 3.0 2019T4 -0.3 1.7 1.7 2020T1 NaN -4.6 -4.5 2020T2 NaN NaN -29.4 ``` Each column is a vintage. On February 15, 2020 nobody knew anything about the first quarter of 2020 and the fourth quarter of 2019 was −0.3; by the end of June the fourth quarter of 2019 was already +1.7 and the first quarter of 2020 had been revised twice. The full table, periods in rows and vintage dates in columns, is the “revision triangle” that economists use to study the quality of first estimates. To use a macro series as a variable in a daily backtest, the join is the same as with fundamentals: a `merge_asof` by release date, with each day's version. Comparing it with the naive join (the latest version of each quarter, from the day the quarter ends): ```python sesiones = pd.read_csv("fix_indice_diario.csv", parse_dates=["fecha"])[["fecha"]] p = mv[mv["serie"] == "pib_trimestral"].copy() # end of each quarter: 2019T4 -> 2019-12-31 p["periodo_fin"] = (pd.PeriodIndex(p["periodo"].str.replace( "T", "Q"), freq="Q").end_time.normalize()) pit = pd.merge_asof( sesiones, p.sort_values("fecha_publicacion") [["fecha_publicacion", "periodo", "valor"]], left_on="fecha", right_on="fecha_publicacion", allow_exact_matches=False) ult = p.sort_values("vintage").groupby("periodo").last() ingenuo = pd.merge_asof( sesiones, ult.reset_index().sort_values("periodo_fin") [["periodo_fin", "periodo", "valor"]], left_on="fecha", right_on="periodo_fin") c = pit.merge(ingenuo, on="fecha", suffixes=("_pit", "_ing")) c = c.dropna(subset=["valor_pit", "valor_ing"]) print(f"Sessions: {len(c)}") print("Different quarter:", f"{(c['periodo_pit'] != c['periodo_ing']).mean():.1%}") print("Different value:", f"{(c['valor_pit'] != c['valor_ing']).mean():.1%}") negativo_distinto = (c["valor_pit"] < 0) != (c["valor_ing"] < 0) print(f"Different sign: {negativo_distinto.mean():.1%}") ``` ```output Sessions: 1681 Different quarter: 36.6% Different value: 89.5% Different sign: 9.8% ``` In 89.5% of the sessions the naive join uses a value different from the one that was known and in 36.6%, even a different quarter: one that had not been released yet. In one in ten (9.8%) not even the sign of growth matches. A macro model trained this way learns a relationship between the economy and the market that no investor could have exploited. ## The economic calendar: when each figure becomes known An economic calendar lists, for each release, the date and time, the consensus (the mean or median of the analyst forecasts surveyed by the calendar provider), the previous value and the released one. `calendario_economico.csv` is the 2025 calendar for Valdoria and Brenia: ```python ce = pd.read_csv("calendario_economico.csv", parse_dates=["fecha_hora"]) print(ce["fecha_hora"].dtype, len(ce), "releases") ce[["fecha_hora", "evento", "pais", "consenso", "publicado"]].head(6) ``` ```output datetime64[us, UTC] 123 releases fecha_hora evento pais consenso publicado 0 2025-01-02 09:00:00+00:00 PMI manufacturero Brenia 47.70 47.70 1 2025-01-02 15:00:00+00:00 PMI manufacturero Valdoria 51.30 51.00 2 2025-01-03 13:30:00+00:00 Tasa de paro Valdoria 4.30 4.20 3 2025-01-13 13:30:00+00:00 IPC interanual (definitivo) Valdoria 2.70 2.40 4 2025-01-16 13:30:00+00:00 Ventas minoristas, mensual Valdoria 0.70 0.60 5 2025-01-29 19:00:00+00:00 Decisión de tipos Valdoria 4.35 4.35 ``` The times are in UTC, as they should be stored. Before using it, it is worth checking that it matches the series: the released value of each Valdoria CPI flash estimate must be vintage 1 of that month, and the “previous” value, the latest known version of the prior month. ```python av = ce[(ce["pais"] == "Valdoria") & (ce["evento"] == "IPC interanual (avance)")].copy() av["fecha"] = av["fecha_hora"].dt.tz_convert( "America/New_York").dt.tz_localize(None).dt.normalize() ipc1 = mv[(mv["serie"] == "ipc_interanual") & (mv["vintage"] == 1)] cmp = av.merge(ipc1, left_on="fecha", right_on="fecha_publicacion") print(len(av), "flash estimates;", len(cmp), "with their vintage;", "equal:", (cmp["publicado"] == cmp["valor"]).all()) ``` ```output 11 flash estimates; 11 with their vintage; equal: True ``` All eleven releases match. It is a cheap check worth automating in a pipeline (lesson 1.4.8): a calendar from one provider and a series from another can disagree on the vintage, the unit or the day. ## The surprise: figure minus consensus An asset's price already discounts what the market expects. What arrives with the release is the unexpected part: $$ \text{surprise} = \text{released} - \text{consensus} $$ To compare events with different units and volatilities, the surprise is normalized by dividing it by the standard deviation of that event's surprises (standardized surprise). In a real study that deviation is estimated with the past only, with a window that does not include the release being evaluated; here, with one year of data, the full year's deviation is used to describe the distribution. ```python ce["sorpresa"] = ce["publicado"] - ce["consenso"] corto = ce["evento"].str.replace( "PIB trimestral, tasa anualizada", "PIB") ce["evento_pais"] = corto + " · " + ce["pais"] res = ce.groupby("evento_pais")["sorpresa"].agg( n="size", media="mean", desv="std", exactas=lambda s: (s.abs() < 1e-9).mean()) res.sort_values("desv", ascending=False).round(2) ``` ```output n media desv exactas evento_pais Confianza del consumidor · Valdoria 12 -0.67 1.27 0.00 PIB (segunda estimación) · Valdoria 4 -0.03 0.51 0.00 PMI manufacturero · Valdoria 12 -0.07 0.46 0.00 PIB (avance) · Valdoria 4 -0.47 0.43 0.00 PMI manufacturero · Brenia 12 -0.09 0.43 0.17 Ventas minoristas, mensual · Valdoria 12 -0.07 0.31 0.33 PIB (tercera estimación) · Valdoria 4 -0.35 0.31 0.00 IPC interanual (avance) · Valdoria 11 0.08 0.22 0.09 IPC interanual (definitivo) · Valdoria 12 -0.02 0.16 0.50 Tasa de paro · Valdoria 12 -0.07 0.15 0.50 IPC interanual (avance) · Brenia 12 -0.01 0.08 0.42 Decisión de tipos · Brenia 8 0.00 0.00 1.00 Decisión de tipos · Valdoria 8 0.00 0.00 1.00 ``` *Fig. — 2025 surprises by event (actual − consensus) divided by the standard deviation of that event's surprises; unemployment with the sign flipped (less unemployment = positive surprise).* One row per event of Valdoria (V) and Brenia (B), one mark per release between −3 and +3 standard deviations. Almost all fall between −2 and +2. The largest, highlighted, is Valdoria's June 2025 unemployment: 4.3% actual against 4.7% consensus, +2.67 deviations with the economic sign. Rate decisions are not shown: they never surprised. Rate decisions never surprise in this year: the consensus gets all sixteen right, as expected from central banks that prepare the market before deciding. Unemployment and final CPI are right half the time: they move by tenths and, for the final figure, the flash estimate is already out. The most dispersed surprises are those of consumer confidence (1.27 index points), GDP and the PMIs. Three of the four surprises in the GDP advance estimate were negative (mean of −0.47 points), but with four data points that is not a consensus bias, it is an anecdote. The deviations are in different units (index points, percentages): that is why they have to be normalized. An event without surprises barely moves the price; one with large, frequent surprises is the one that deserves an event study (level 2). ```python desv = ce.groupby("evento_pais")["sorpresa"].transform("std") ce["z"] = ce["sorpresa"] / desv # for unemployment, more is worse: flip the sign ce.loc[ce["evento"] == "Tasa de paro", "z"] *= -1 top = ce.dropna(subset=["z"]).reindex( ce["z"].abs().sort_values(ascending=False).index) top[["fecha_hora", "evento_pais", "consenso", "publicado", "z"]].head(5).round({"z": 2}) ``` ```output fecha_hora evento_pais consenso publicado z 54 2025-06-06 12:30:00+00:00 Tasa de paro · Valdoria 4.7 4.3 2.67 50 2025-05-30 14:00:00+00:00 Confianza del consumidor · Valdoria 95.3 92.3 -2.36 46 2025-05-16 12:30:00+00:00 Ventas minoristas, mensual · Valdoria 0.7 -0.0 -2.25 37 2025-04-24 12:30:00+00:00 PIB (avance) · Valdoria -0.3 -1.2 -2.07 27 2025-03-27 12:30:00+00:00 PIB (tercera estimación) · Valdoria 3.0 2.4 -1.93 ``` With the standardized surprise and the economic sign (for unemployment, less is better), releases of different indicators can be put on the same scale and ranked by how unexpected they were. ## Times in UTC, markets in local time Valdoria releases its data at 08:30 New York time and decides on rates at 14:00; Brenia releases at 10:00 or 11:00 Madrid time and decides at 14:15. In UTC, those times change with daylight saving time, and not at the same moment: the US switches on the second Sunday of March and the first Sunday of November; the EU, on the last Sunday of March and of October (lesson 0.4.4). ```python ny = ce["fecha_hora"].dt.tz_convert("America/New_York") mad = ce["fecha_hora"].dt.tz_convert("Europe/Madrid") ce["hora_local"] = ny.dt.strftime("%H:%M").where( ce["pais"] == "Valdoria", mad.dt.strftime("%H:%M")) ce["hora_utc"] = ce["fecha_hora"].dt.strftime("%H:%M") print(pd.crosstab([ce["pais"], ce["hora_local"]], ce["hora_utc"])) # 2025 clock changes according to the time zone database (tzdata) dias = pd.date_range("2025-01-01", "2025-12-31", freq="D", tz="UTC") + pd.Timedelta(hours=12) for zona in ["America/New_York", "Europe/Madrid"]: off = dias.tz_convert(zona).map(lambda t: t.utcoffset()) cambio = dias[1:][off[1:] != off[:-1]] print(zona, [str(d.date()) for d in cambio]) ``` ```output hora_utc 08:00 09:00 10:00 12:15 12:30 13:15 13:30 14:00 15:00 18:00 19:00 pais hora_local Brenia 10:00 7 5 0 0 0 0 0 0 0 0 0 11:00 0 7 5 0 0 0 0 0 0 0 0 14:15 0 0 0 4 0 4 0 0 0 0 0 Valdoria 08:30 0 0 0 0 39 0 20 0 0 0 0 10:00 0 0 0 0 0 0 0 15 9 0 0 14:00 0 0 0 0 0 0 0 0 0 6 2 America/New_York ['2025-03-09', '2025-11-02'] Europe/Madrid ['2025-03-30', '2025-10-26'] ``` *Fig. — UTC time of the 2025 releases at 08:30 New York (Valdoria) and 10:00 Madrid (Brenia); in the bands, the two daylight-saving schedules do not overlap.* Two step lines across 2025. Valdoria's at 13:30 UTC in winter and 12:30 from 9 March to 2 November; Brenia's at 9:00 UTC in winter and 8:00 from 30 March to 26 October. The vertical marks are the actual calendar releases. Two highlighted bands, 9 to 30 March and 26 October to 2 November, mark when New York and Madrid are five hours apart instead of six. 08:30 in New York is 12:30 UTC during daylight saving time and 13:30 outside it. A filter written as “do not trade between 13:25 and 13:45 UTC” is right in winter and an hour late from March to November. According to the time zone database, in 2025 New York switched on March 9 and November 2, and Madrid on March 30 and October 26. Between those dates, New York and Madrid were five hours apart instead of six: two windows a year in which a rule like “Madrid time minus six” fails. ```mermaid flowchart TB C[UTC calendar] --> L[Local time] L --> S{Market open?} S -- yes --> B[Event bar] S -- no --> N[Next session] B --> V[Reaction window] N --> V ``` For a reaction study, each release has to be joined to the first bar in which the market can react. With NOVA's 5-minute bars (09:30 to 16:00 New York session) and a forward `merge_asof`: ```python barras = pd.read_parquet("nova_barras_5min.parquet") ev = ce[ce["pais"] == "Valdoria"].sort_values("fecha_hora") m = pd.merge_asof(ev, barras[["ts"]], left_on="fecha_hora", right_on="ts", direction="forward") m["espera_min"] = (m["ts"] - m["fecha_hora"]).dt.total_seconds() / 60 print(m.groupby("hora_local")["espera_min"].median()) largas = m[m["espera_min"] > 120] largas[["fecha_hora", "ts", "espera_min"]] ``` ```output hora_local 08:30 60.0 10:00 0.0 14:00 0.0 Name: espera_min, dtype: float64 fecha_hora ts espera_min 11 2025-02-17 13:30:00+00:00 2025-02-18 14:30:00+00:00 1500.0 47 2025-07-04 12:30:00+00:00 2025-07-07 13:30:00+00:00 4380.0 61 2025-09-01 14:00:00+00:00 2025-09-02 13:30:00+00:00 1410.0 81 2025-11-27 13:30:00+00:00 2025-11-28 14:30:00+00:00 1500.0 89 2025-12-25 13:30:00+00:00 2025-12-26 14:30:00+00:00 1500.0 ``` The 08:30 releases arrive while the stock market is closed: the first bar that can react is the opening bar, an hour later, and whatever happened in between is in the opening gap (and in futures and currencies, which do trade). The 10:00 releases and the 14:00 decision fall within the session. And there are five releases on days when the NYSE is closed (in this fictional calendar, February 17, July 4, September 1, November 27 and December 25): their first bar is in the next session, one to three days later. A study that measured the reaction “in the 09:30 bar on the day of the release” would mix those five with sessions without news. ## The APIs: FRED, ECB and INE FRED, the ECB and INE (Spain's National Statistics Institute), three public sources of macro data for the US, the euro area and Spain, have open APIs. All of them need network access, so the code is not executed. **FRED** (Federal Reserve Bank of St. Louis) requires a 32-character key (`api_key`). The observations endpoint accepts the real-time period (`realtime_start`, `realtime_end`), a list of vintage dates (`vintage_dates`) and the output type (`output_type`: 1, by real-time period; 2, all observations by vintage; 3, only new and revised ones; 4, only the first release)[^1]: ```python skip import os import requests URL = "https://api.stlouisfed.org/fred/series/observations" base = {"series_id": "GDP", "api_key": os.environ["FRED_API_KEY"], "file_type": "json"} hoy = requests.get(URL, params=base, timeout=30).json() primeras = requests.get(URL, params=base | {"output_type": 4}, timeout=30).json() en_2020 = requests.get(URL, params=base | { "realtime_start": "2020-02-15", "realtime_end": "2020-02-15"}, timeout=30).json() # each observation: realtime_start, realtime_end, date, value ``` The response carries an `observations` list in which each row has `date` (the period), `value` (as text; `"."` when missing) and the real-time interval during which that figure was in force. It is exactly the structure of `macro_vintages.csv`, with two date columns instead of one. **ECB** (Data Portal): URLs follow the SDMX syntax `data-api.ecb.europa.eu/service/data//`, with no API key. `format=csvdata` or `jsondata` choose the format; `startPeriod` and `endPeriod`, the range; `updatedAfter`, only what changed since a date (useful for updating a local database), and `includeHistory=true` also returns earlier versions of the data[^2]: ```python skip import io url = ("https://data-api.ecb.europa.eu/service/data/" "EXR/D.USD.EUR.SP00.A") r = requests.get(url, params={"format": "csvdata", "startPeriod": "2025-01-01"}, timeout=30) eurusd = pd.read_csv(io.StringIO(r.text)) # SDMX columns: KEY, FREQ, CURRENCY, ..., TIME_PERIOD, OBS_VALUE ``` **INE**: the JSON API has the form `servicios.ine.es/wstempus/js/{language}/{function}/{input}`; the `DATOS_SERIE` function returns the data of a series by its code and the `nult` parameter limits the response to the latest periods[^3]. The date of each data point (`Fecha`) comes as milliseconds since 1970, and there is a time zone trap here. The following sample response has the API's structure with a made-up code and values: ```python import json ej = json.loads("""{"COD": "SERIE_EJEMPLO", "Nombre": "Example series. Annual change.", "Data": [ {"Fecha": 1761951600000, "FK_Periodo": 11, "Anyo": 2025, "Valor": 3.1}, {"Fecha": 1764543600000, "FK_Periodo": 12, "Anyo": 2025, "Valor": 2.8}]}""") d = pd.DataFrame(ej["Data"]) d["utc"] = pd.to_datetime(d["Fecha"], unit="ms", utc=True) d["madrid"] = d["utc"].dt.tz_convert("Europe/Madrid") d["mes_mal"] = d["utc"].dt.strftime("%Y-%m") d["mes_bien"] = d["madrid"].dt.strftime("%Y-%m") d[["FK_Periodo", "utc", "mes_mal", "mes_bien", "Valor"]] ``` ```output FK_Periodo utc mes_mal mes_bien Valor 0 11 2025-10-31 23:00:00+00:00 2025-10 2025-11 3.1 1 12 2025-11-30 23:00:00+00:00 2025-11 2025-12 2.8 ``` Read in UTC, both data points fall on the last day of the previous month: November's looks like October's. The timestamp is midnight in Madrid on the first day of the period, which in UTC is 23:00 (or 22:00 in summer) on the day before. Converting to `Europe/Madrid` before extracting the month, or using `Anyo` and `FK_Periodo`, solves it. The real request: ```python skip url = ("https://servicios.ine.es/wstempus/js/ES/DATOS_SERIE/" "") r = requests.get(url, params={"nult": 12}, timeout=30) datos = r.json()["Data"] ``` None of these APIs says when each figure was published in the default response: FRED needs the real-time period, the ECB `includeHistory`, and for INE you have to check its release calendar. If you store macro data for backtests, also store the day you downloaded it: your own database then becomes a vintage archive. ## Analyst estimates and transcripts **Analyst estimates** (earnings per share, revenue, recommendations) are the corporate version of the macro consensus: each analyst publishes a forecast, the provider computes the mean or median, and the earnings surprise is the reported earnings minus that consensus. They have the same date problems: each estimate gets revised, a quarter's consensus changes day by day until the release, and a dataset without the date of each revision is no good for simulation. Before paying for an estimates database (lesson 1.3.4), check whether it stores each forecast with its date or only the consensus at the time of download. **Transcripts** of earnings presentations (*earnings calls*) are text: what management says about the quarter and the outlook, and the analysts' questions. They are the input to the language models of level 6: tone, changes in *guidance*, words that anticipate problems. ```mermaid flowchart LR T[Transcript] --> M[Timestamp] M --> X[Clean text] X --> F[NLP signal] ``` The principle is the same as in this whole section: a transcript counts from the moment the call ends, not from the end of the quarter or from the 10-Q filing. Lessons 6.1.2 and 6.2.3 build signals on filings and transcripts with that timestamp. ## What this changes in a backtest - **Use vintages.** If the source only gives the current series, store your downloads with their date or use ALFRED. For a series without a vintage archive, delay each data point to its real release date, even if you keep using the final version, and say so. - **Trade the surprise, not the level.** The CPI level is in prices long before it is released; what is new is the difference from the consensus. - **UTC for storage, local time for trading.** Convert with `zoneinfo` every time; never with fixed offsets. - **Holidays and closed markets.** A release while the market is closed is reflected at the next open or in the markets that do trade (futures, currencies). Decide which one you measure. ## Exercises 1. For unemployment, compute the revision between vintage 1 and 2 and between 2 and 3 (when it exists). Which of the two revisions is larger in mean absolute terms? **Solution.** ```python paro = mv[mv["serie"] == "paro"].pivot( index="periodo", columns="vintage", values="valor") r12 = (paro[2] - paro[1]).abs() r23 = (paro[3] - paro[2]).abs() print(f"v1 -> v2: mean {r12.mean():.3f}, n = {r12.count()}") print(f"v2 -> v3: mean {r23.mean():.3f}, n = {r23.count()}") ``` ```output v1 -> v2: mean 0.062, n = 82 v2 -> v3: mean 0.100, n = 19 ``` The annual revision (vintage 3) only exists in the months that change, so its mean is computed over few months and is larger: when the annual revision touches a month, it moves it more than the monthly revision. For a model with unemployment data, vintage 3 arrives more than a year after the period: if it is used from the start, it brings in information from a year later. 2. For each Valdoria release, compute the New York local time and check that all the 08:30 ones fall outside the NYSE session and that the rate decision falls within it. **Solution.** ```python val = ce[ce["pais"] == "Valdoria"].copy() local = val["fecha_hora"].dt.tz_convert("America/New_York") minuto = local.dt.hour * 60 + local.dt.minute val["en_sesion"] = minuto.between(9 * 60 + 30, 16 * 60 - 1) val.groupby("hora_local")["en_sesion"].mean() ``` ```output hora_local 08:30 0.0 10:00 1.0 14:00 1.0 Name: en_sesion, dtype: float64 ``` All the 08:30 releases are outside the cash session and the 10:00 and 14:00 ones, inside. The check does not look at holidays: for that you need the exchange calendar (`exchange_calendars`, lesson 1.1.2), which is what the `merge_asof` with NOVA's real bars did. 3. A provider offers you a history of consensus figures with a single row per release (the final consensus). What study can you do with it and what study can you not? **Solution.** With the final consensus and the released figure you can measure the surprise and study the price reaction in the minutes or days after the release: it is an event study, and the final consensus is what the market expected right before. You cannot study how the expectation formed (revisions of the consensus in the preceding weeks) or build a signal that trades before the release with the consensus at that moment, because that version is not there: using the final consensus for a decision days earlier would be look-ahead. [^1]: FRED API documentation: `fred/series/observations` (parameters `realtime_start`, `realtime_end`, `vintage_dates`, `output_type`; 32-character `api_key`) and “Real-Time Periods”: by default the real-time period is today; ALFRED lets you retrieve what was known on a past date. [^2]: ECB Data Portal, “Data”: entry point `data-api.ecb.europa.eu/service/`, formats `csvdata` and `jsondata`, parameters `startPeriod`, `endPeriod`, `updatedAfter` and `includeHistory` (earlier versions of the data). [^3]: INE, “Referencia de la API” (API reference): structure `https://servicios.ine.es/wstempus/js/{language}/{function}/{input}`, function `DATOS_SERIE` and parameter `nult`. ## Self-assessment 1. The first estimate of a quarter's GDP was −0.3% and the latest version is +1.7%. Which figure should a backtest use to make a decision on the day of the first release? - A. −0.3%, the one known that day - B. +1.7%, because it is the correct one - C. The average of the two - D. Neither: GDP is useless for trading *Answer A: −0.3%, the one known that day.* The market reacted to −0.3%. The latest version did not exist that day: using it is look-ahead. That is why ALFRED exists: it keeps every version with its date. 2. In the FRED API, which parameter lets you request the data as it stood on a past date? - A. `file_type` - B. `realtime_start` and `realtime_end` (or `vintage_dates`) - C. `series_id` - D. `frequency` *Answer B: `realtime_start` and `realtime_end` (or `vintage_dates`).* The real-time period says what was known and when. By default it is today: FRED returns the current version. With a past period or with `vintage_dates`, it returns what was known back then; `output_type=4` returns only the first release. 3. A Valdoria figure is released at 08:30 New York time. At what UTC time does it appear in the calendar? - A. Always at 13:30 - B. At 08:30 - C. At 12:30 during US daylight saving time and at 13:30 outside it - D. At 14:30 *Answer C: At 12:30 during US daylight saving time and at 13:30 outside it.* New York is at UTC−4 in summer and UTC−5 in winter. Storing the calendar in UTC is right, but a fixed rule such as “13:30 UTC” gets the time wrong for about seven months of the year. 4. The unemployment consensus was 4.7% and 4.3% is released. What is the surprise and what sign does it have for the economy? - A. +0.4 points: worse than expected - B. 4.3%: the surprise is the figure - C. 0: the consensus is always right - D. −0.4 points: less unemployment than expected, a positive surprise for the economy *Answer D: −0.4 points: less unemployment than expected, a positive surprise for the economy.* Surprise = released − consensus = 4.3 − 4.7 = −0.4. For unemployment, a negative sign is good news; to compare events it helps to give each one its economic sign and normalize it by the dispersion of its surprises. 5. Why does an earnings call transcript need its own timestamp in an NLP dataset? - A. Because the text changes over time - B. To sort them alphabetically - C. Because the information reaches the market during the call, not at the quarter's end or when the 10-Q is filed: it is the date from which a signal can use it - D. It does not need one *Answer C: Because the information reaches the market during the call, not at the quarter's end or when the 10-Q is filed: it is the date from which a signal can use it.* It is the same principle as with fundamentals and macro: each data point counts from the moment it becomes known. A sentiment signal built on the transcript can only trade after the call (level 6). ## Key points - Macro data gets revised: in this data, 81% of GDP quarters change between the first and the latest version, with a mean absolute revision of 0.49 points. - A backtest must use the vintage known on each date. FRED returns the current version by default; ALFRED and the real-time parameters return the earlier ones. - What moves the price is the surprise versus consensus, not the figure. It has to be normalized by event and given its economic sign. - Store the calendar in UTC and convert it to the market's local time with zoneinfo: the UTC time of the same release changes with daylight saving time, and holidays push the reaction to the next session. - Analyst estimates and transcripts are dated data, like financial statements: they count from the moment they are published. ## Resources ### Research - [Monetary Policy in Python](https://pythonparatrading.com/en/monetary-policy-in-python/) · Research-0010 · 14 min ### Documentation - [FRED API: fred/series/observations (realtime_start, realtime_end, vintage_dates, output_type)](https://fred.stlouisfed.org/docs/api/fred/series_observations.html) · fred.stlouisfed.org - [FRED API: real-time periods (ALFRED)](https://fred.stlouisfed.org/docs/api/fred/realtime_period.html) · fred.stlouisfed.org - [ALFRED: Archival FRED](https://alfred.stlouisfed.org/) · alfred.stlouisfed.org - [ECB Data Portal: data API (includeHistory, updatedAfter, csvdata)](https://data.ecb.europa.eu/help/api/data) · data.ecb.europa.eu - [INE (Spain's statistics institute): JSON API reference (wstempus, DATOS_SERIE, nult)](https://www.ine.es/dyngs/DAB/index.htm?cid=1100) · ine.es ### Data - [Level 1 data: Valdoria macro series with their vintages (CSV)](https://pythonparatrading.com/curso/datos/nivel-1/macro_vintages.csv) · Download · macro_vintages.csv Previous lesson: [1.5.1 · Financial statements, ratios, XBRL and SEC EDGAR](https://pythonparatrading.com/en/python-trading-course/level-1/financial-statements-xbrl-and-sec-edgar/) Next lesson: [1.6.1 · Returns, volatility, rolling correlations and drawdown](https://pythonparatrading.com/en/python-trading-course/level-1/returns-volatility-correlation-and-drawdown/)