Types of alternative data: what each one measures.

The map of sources

Lesson 6.7.1 classified alternative data by origin. This one goes through the sources one by one with data. The course includes a synthetic version of almost every type, all of them about the 120 fictitious companies of level 5 and from 2018 to September 2025, with a property real data does not have: we know what each source measures. Behind all of them there is a quarterly demand shock z per company that moves sales relative to consensus; each source sees part of it, with its own noise and delay, and the ground truth is in alt6_verdad_trimestral.csv.

Before looking for a signal, each source goes through the same questions: what it measures, whom it covers, when it becomes known and what contaminates it. The path of a data point to the model has several dates, and the one that counts in a backtest is the last one:

flowchart TB
  E[Event] --> R[Collection]
  R --> T[Delivery]
  T --> M[Mapping to company]
  M --> S[Signal]

Coverage and dates

The first table is the inventory: rows, frequency, period and how many companies each source covers. Mapping stores, names and applicants to companies is a problem in itself (lesson 6.7.5); here the course's ground-truth tables are used to skip it, except for employment, where it is solved with a rule:

import numpy as np
import pandas as pd

emp = pd.read_csv("acciones6_empresas.csv")
tk = emp.set_index("id_empresa")["ticker"]
verdad = pd.read_csv("alt6_verdad_trimestral.csv",
                     parse_dates=["fin_trimestre"])
verdad["q"] = verdad["fin_trimestre"].dt.to_period("Q")
expo = pd.read_csv("alt6_exposiciones.csv")

tar = pd.read_parquet("alt6_tarjetas_pit.parquet")
com = pd.read_csv("alt6_comercios_verdad.csv")
web = pd.read_parquet("alt6_trafico_web.parquet")
empleo = pd.read_parquet("alt6_empleo.parquet")
res = pd.read_parquet("alt6_resenas.parquet").merge(
    pd.read_csv("alt6_resenas_verdad.csv"), on="id_resena")
ais = pd.read_parquet("alt6_ais_llegadas.parquet")
clima = pd.read_parquet("alt6_clima.parquet")
pat = pd.read_parquet("alt6_patentes.parquet").merge(
    pd.read_csv("alt6_patentes_verdad.csv"), on="id_patente")

primera = emp["nombre_corto"].str.split().str[0].str.lower()
empleo["ticker"] = (empleo["empresa_publicada"].str.split()
                    .str[0].str.lower().map(
                        dict(zip(primera, emp["ticker"]))))
cubre = {
    "cards": com.query("relacion != 'sin_relacion'")
                ["id_empresa"].nunique(),
    "web": web["dominio"].isin(emp["dominio"]).groupby(
        web["dominio"]).any().sum(),
    "jobs": empleo["ticker"].nunique(),
    "reviews": res["id_empresa"].nunique(),
    "ships": (expo["tipo"] == "puerto").sum(),
    "weather": (expo["tipo"] == "clima").sum(),
    "patents": pat["id_empresa"].nunique(),
}
fechas = {"cards": tar["fecha"], "web": web["semana"],
          "jobs": empleo["mes"], "reviews": res["fecha"],
          "ships": ais["fecha"], "weather": clima["fecha"],
          "patents": pat["fecha_publicacion"]}
inv = pd.DataFrame({
    "filas": {k: len(v) for k, v in fechas.items()},
    "desde": {k: v.min().date() for k, v in fechas.items()},
    "hasta": {k: v.max().date() for k, v in fechas.items()},
    "empresas": cubre})
inv

          filas       desde       hasta  empresas
cards    359410  2018-01-01  2025-09-30        25
web       12120  2018-01-07  2025-09-28        30
jobs      27900  2018-01-31  2025-09-30        60
reviews   36213  2018-01-01  2025-09-30        10
ships     42450  2018-01-01  2025-09-30        10
weather   16980  2018-01-01  2025-09-30        12
patents    5624  2016-06-27  2025-12-31        40

No source covers the whole panel: from 10 companies (reviews, ships) to 60 (jobs). And coverage is not random: cards only see businesses that sell to consumers, weather only matters to utilities and AIS to exporters. A model built with one source trades, at most, its universe (lesson 5.2.2).

Dates are the second question. The card table includes fecha_entrega (the spending date plus 3 days); the web traffic table, the delivery on the Tuesday after the week; patents, the filing date and the publication date. The rest have no delivery date, and the course assumes the data becomes known when its period ends (the day, the month); with a real vendor, that date is a due diligence question (lesson 6.7.4):

retraso = pd.Series({
    "cards": (tar["fecha_entrega"] - tar["fecha"]).dt.days,
    "web (from Sunday)":
        (web["fecha_entrega"] - web["semana"]).dt.days,
    "patents": (pat["fecha_publicacion"]
                - pat["fecha_solicitud"]).dt.days,
})
pd.DataFrame({k: v.describe()[["min", "50%", "max"]]
              for k, v in retraso.items()}).T

                     min    50%    max
cards                3.0    3.0    3.0
web (from Sunday)    2.0    2.0    2.0
patents            538.0  557.0  577.0

Three days for cards, two for web traffic (plus the six of the week it aggregates) and between 538 and 577 days for patents: about 18 months, the usual publication period for a patent application11Note 1 · WIPO, “Protecting your Inventions Abroad: Frequently Asked Questions About the Patent Cooperation Treaty (PCT)” (https://www.wipo.int/pct/en/faqs/faqs.html, accessed on 2026-11-05): “WIPO publishes the international application shortly after the expiration of 18 months from the priority date”.. The figure at the top summarizes the coverage and delay of each source.

Card transactions: the panel changes

A card data vendor sees the spending of a panel of cards (those of the banks or apps that share data with it), not that of the whole population. The panel grows and changes composition, and that shows up in gross spending as if it were demand. In the course data, a new bank joins on March 1, 2021:

panel = pd.read_csv("alt6_tarjetas_panel.csv",
                    parse_dates=["fecha"]).set_index("fecha")
diario = tar.groupby("fecha")["gasto_usd"].sum().to_frame()
diario = diario.join(panel)
diario["por_tarjeta"] = (diario["gasto_usd"]
                         / diario["tarjetas_activas"])
antes = diario.loc["2021-02-01":"2021-02-28"].mean()
despues = diario.loc["2021-03-01":"2021-03-28"].mean()
(despues / antes - 1)[["gasto_usd", "tarjetas_activas",
                       "por_tarjeta", "fraccion_jovenes"]].round(3)

gasto_usd           0.382
tarjetas_activas    0.362
por_tarjeta         0.015
fraccion_jovenes    0.147
dtype: float64

Gross spending rises 38% from February to March; active cards, 36%; spending per card, 1.5%. The rule is to normalize by panel size before anything else. A subtler bias remains: the share of young cardholders rises with the new bank, and companies with a young customer base gain weight in the panel without selling more. That composition bias, the difference between the data as delivered and the history the vendor reconstructs when selling (backfill), and how they are detected, are the subject of lesson 6.7.4.

Once normalized, the spending per card at each company's stores tracks its sales. Its year-on-year growth by quarter, against the demand shock and the sales surprise:

t = tar.merge(com.query("relacion != 'sin_relacion'"),
              on="descriptor")
t["q"] = t["fecha"].dt.to_period("Q")
tarjetas_q = (panel.groupby(panel.index.to_period("Q"))
              ["tarjetas_activas"].mean())
g = t.groupby(["id_empresa", "q"])["gasto_usd"].sum().reset_index()
g["ticker"] = g["id_empresa"].map(tk)
g["x"] = np.log(g["gasto_usd"] / g["q"].map(tarjetas_q))
g["x"] = g["x"] - g.groupby("ticker")["x"].shift(4)

def frente_a_verdad(df, col="x"):
    m = df.merge(verdad, on=["ticker", "q"]).dropna(subset=[col])
    return pd.Series({
        "n": len(m),
        "corr_z": m[col].corr(m["z_demanda"]),
        "corr_sorpresa": m[col].corr(m["sorpresa_ventas"])}).round(3)

medidas = {"cards": frente_a_verdad(g)}
medidas["cards"]

n                675.000
corr_z             0.289
corr_sorpresa      0.211
dtype: float64

A correlation of 0.29 with the demand shock and of 0.21 with the surprise over 675 company-quarters. Year-on-year growth is a crude measure (it mixes this quarter's shock with that of the same quarter of the previous year); lesson 6.7.5 builds the right measure, spending growth relative to the growth implied by consensus, and backtests it.

Web and app traffic: the bots

Web traffic arrives by week and by domain: visits (in millions) and app downloads (in thousands). Visits have a well-known problem in real data: automated traffic. In the data, 1.3% of weeks carry a bot spike that multiplies visits by 2 to 4. A simple rule detects them: divide each week by the median of the previous 8 weeks (past only, so that the rule can be applied on the delivery date):

bots = pd.read_csv("alt6_trafico_web_bots_verdad.csv",
                   parse_dates=["semana"]).assign(verdad=True)
web = web.sort_values(["dominio", "semana"])
web["mediana"] = web.groupby("dominio")["visitas_millones"].transform(
    lambda s: s.shift(1).rolling(8, min_periods=4).median())
web["ratio"] = web["visitas_millones"] / web["mediana"]
web = web.merge(bots, on=["dominio", "semana"], how="left")
web["verdad"] = web["verdad"].fillna(False).astype(bool)
for umbral in (1.5, 1.7, 2.0):
    marca = web["ratio"] > umbral
    acierto = (marca & web["verdad"]).sum()
    print(f"threshold {umbral}: flagged {marca.sum()}, "
          f"precision {acierto / marca.sum():.1%}, "
          f"recall {acierto / web['verdad'].sum():.1%}")

threshold 1.5: flagged 159, precision 100.0%, recall 98.8%
threshold 1.7: flagged 159, precision 100.0%, recall 98.8%
threshold 2.0: flagged 157, precision 100.0%, recall 97.5%

With a threshold of 1.7, the rule flags 159 weeks, all of them bots, and finds 159 of the 161. In real data bots do not always arrive in such clean spikes (there is constant automated traffic, and legitimate spikes from a campaign or a news item), so the rule is checked against other signals: app traffic, country of origin, visit duration. Replacing the flagged weeks with the median changes what the source measures:

web["limpias"] = web["visitas_millones"].mask(
    web["ratio"] > 1.7, web["mediana"])   # only the flagged ones
web["ticker"] = web["dominio"].map(
    emp.set_index("dominio")["ticker"])
web["q"] = web["semana"].dt.to_period("Q")
for col in ["visitas_millones", "limpias", "descargas_app_miles"]:
    s = web.groupby(["ticker", "q"])[col].sum().reset_index()
    s["x"] = np.log(s[col]) - np.log(
        s.groupby("ticker")[col].shift(4))
    medidas[f"web: {col}"] = frente_a_verdad(s)
pd.DataFrame({k: v for k, v in medidas.items()
              if k.startswith("web")}).T

                              n  corr_z  corr_sorpresa
web: visitas_millones     810.0   0.170          0.176
web: limpias              810.0   0.319          0.258
web: descargas_app_miles  810.0   0.365          0.291

Uncleaned, visits correlate 0.17 with the demand shock; cleaned, 0.32. App downloads, which in this data have no bots, are the best measure of the three (0.37 with the shock, 0.29 with the surprise).

Job postings: names and horizon

Job postings are collected from job boards and corporate websites, almost always by scraping (lesson 6.7.3). Each board writes the company name its own way:

variantes = empleo.groupby("ticker")["empresa_publicada"].unique()
print(variantes.iloc[0].tolist())
print("variants per company:",
      empleo.groupby("ticker")["empresa_publicada"].nunique()
      .describe()[["min", "max"]].to_dict())
print("unmapped:", empleo["ticker"].isna().sum())

['BEBVIA PETROLEUM', 'Bebvia', 'Bebvia Petroleum Holdings Inc.', 'Bebvia Petroleum', 'Bebvia Petroleum Inc']
variants per company: {'min': 5.0, 'max': 5.0}
unmapped: 0

Five variants per company, all with the first word of the short name: in the course data, that rule is enough (the made-up names start with a unique word). In real data it is not enough, because there are common names, subsidiaries and brands that share nothing with the parent; lesson 6.7.5 covers entity resolution.

What postings measure is hiring intention, and a company hires before it sells more. In the data, new postings anticipate demand by two quarters:

empleo["q"] = empleo["mes"].dt.to_period("Q")
e = (empleo.groupby(["ticker", "q"])["ofertas_nuevas"].sum()
     .reset_index().sort_values(["ticker", "q"]))
e["x0"] = np.log(e["ofertas_nuevas"]) - np.log(
    e.groupby("ticker")["ofertas_nuevas"].shift(4))
e["x"] = e.groupby("ticker")["x0"].shift(2)   # that of q-2
m = e.merge(verdad, on=["ticker", "q"]).dropna(subset=["x", "x0"])
print("same quarter, with the surprise:",
      round(m["x0"].corr(m["sorpresa_ventas"]), 3))
print("two quarters earlier, with the hiring intention:",
      round(m["x"].corr(m["h_empleo_q_menos_2"]), 3))
medidas["jobs (q-2)"] = frente_a_verdad(e)
medidas["jobs (q-2)"]

same quarter, with the surprise: -0.025
two quarters earlier, with the hiring intention: 0.691
n                1500.000
corr_z             -0.048
corr_sorpresa       0.131
dtype: float64

Growth in postings in a quarter says nothing about that quarter's surprise (−0.025), but it measures hiring intention well (0.69) and, two quarters later, it correlates 0.13 with the surprise. It is a small correlation but with a lot of coverage (60 companies): the type of source that contributes in combination, not on its own.

Reviews: fakes and defects

Product reviews have the J shape of real data: many 5-star ones, some 1-star ones and few in between. Two effects are planted in the data: bursts of fake 5-star reviews without a verified purchase, and 49 quality-defect episodes in which the stars drop and the texts talk about the fault:

print("stars:", res["estrellas"].value_counts(normalize=True)
      .sort_index().round(3).to_dict())
print(f"fake: {res['falsa'].mean():.1%}; "
      f"with defect: {res['producto_con_defecto'].mean():.1%}")
print("mean with and without defect:", res.groupby(
    "producto_con_defecto")["estrellas"].mean().round(2).to_dict())
marca = (res["estrellas"] == 5) & ~res["compra_verificada"]
acierto = (marca & res["falsa"]).sum()
print(f"rule 5 stars without purchase: precision "
      f"{acierto / marca.sum():.1%}, recall "
      f"{acierto / res['falsa'].sum():.1%}")

stars: {1: 0.081, 2: 0.042, 3: 0.072, 4: 0.167, 5: 0.638}
fake: 22.1%; with defect: 5.4%
mean with and without defect: {False: 4.29, True: 3.29}
rule 5 stars without purchase: precision 77.8%, recall 100.0%

22% of the reviews are fake, and the “5 stars without a verified purchase” rule finds all of them, but with a precision of 78%: there are real customers who give 5 stars without a verified purchase. Reviews of products with a defect score one star lower on average (3.29 versus 4.29). For the company, what matters is the quarterly average (centered by company, because each one has its own level):

res["ticker"] = res["id_empresa"].map(tk)
res["q"] = res["fecha"].dt.to_period("Q")
for nombre, sub in [("all", res), ("without fakes", res[~res["falsa"]]),
                    ("without flagged", res[~marca])]:
    s = sub.groupby(["ticker", "q"])["estrellas"].mean()
    s = (s - s.groupby(level="ticker").transform("mean"))
    medidas[f"reviews: {nombre}"] = frente_a_verdad(
        s.rename("x").reset_index())
pd.DataFrame({k: v for k, v in medidas.items()
              if k.startswith("reviews")}).T

                              n  corr_z  corr_sorpresa
reviews: all              310.0   0.244          0.164
reviews: without fakes    310.0   0.352          0.207
reviews: without flagged  310.0   0.336          0.186

Fake reviews dilute the signal: with all of them, 0.24 with the demand shock; without them, 0.35. The imperfect rule recovers almost everything (0.34) even though it throws away genuine reviews: precision matters less than not letting the noise through.

Ships and weather: the physical world

AIS data comes from the transponders ships carry: the International Maritime Organization requires the system on ships of 300 gross tonnage or more on international voyages, on cargo ships of 500 or more and on all passenger ships22Note 2 · IMO, “AIS transponders” (https://www.imo.org/en/OurWork/Safety/Pages/AIS.aspx, accessed on 2026-10-23): regulation V/19 of the SOLAS convention, ships of 300 GT or more on international voyages, cargo ships of 500 GT or more not engaged on international voyages and all passenger ships.. Aggregated by port and ship type, it gives daily arrivals and ships at anchor waiting for a berth. In the course, 10 exporters depend on the activity of five fictitious ports:

ais["q"] = ais["fecha"].dt.to_period("Q")
puertos = expo.query("tipo == 'puerto'").rename(
    columns={"zona": "puerto"})
for tipo in ["contenedores", "granel", "petrolero"]:
    s = (ais[ais["tipo_buque"] == tipo]
         .groupby(["puerto", "q"])["llegadas"].sum().reset_index())
    s["x"] = np.log(s["llegadas"])
    s["x"] = s["x"] - s.groupby("puerto")["x"].transform("mean")
    medidas[f"ships: {tipo}"] = frente_a_verdad(
        puertos.merge(s, on="puerto"))
pd.DataFrame({k: v for k, v in medidas.items()
              if k.startswith("ships")}).T

                         n  corr_z  corr_sorpresa
ships: contenedores  310.0   0.674          0.465
ships: granel        310.0   0.400          0.259
ships: petrolero     310.0   0.452          0.305

The quarter's container ship arrivals correlate 0.67 with the demand shock of the exporters at their port: it is the most direct source in the course after weather. Bulk carriers and tankers also move with port activity (0.40 and 0.45), although the course's exporters do not depend on them. The trap is one of mapping: a port is not a company. The more companies share a port, the more what the source says about each one is diluted.

Weather is the oldest and cheapest alternative data: it is published by weather services. For energy it is summarized in degree days: heating degree days (HDD) add up how far the day's mean temperature falls below a base, and cooling degree days (CDD), how far it rises above it33Note 3 · NOAA, Climate Prediction Center, “Weekly & Monthly Degree Day Summaries Explanation” (https://www.cpc.ncep.noaa.gov/products/analysis_monitoring/cdus/degree_days/ddayexp.shtml, accessed on 2026-10-23): “Heating degree days are summations of negative differences between the mean daily temperature and the 65°F base” (18.3 °C), and cooling degree days, of the positive differences above the same base. The course deliberately uses different bases (18 and 22 °C).. The course uses a base of 18 °C for HDD and 22 °C for CDD; the cell checks it and builds the quarter's index relative to its climatology:

print("HDD = max(0, 18 - t):", np.allclose(
    clima["hdd"], np.maximum(0, 18 - clima["temp_media"])))
print("CDD = max(0, t - 22):", np.allclose(
    clima["cdd"], np.maximum(0, clima["temp_media"] - 22)))
clima["q"] = clima["fecha"].dt.to_period("Q")
c = clima.groupby(["region", "q"])[["hdd", "cdd"]].sum()
c = c.reset_index()
c["trim"] = c["q"].dt.quarter
for col in ["hdd", "cdd"]:   # climatology with the full sample
    c[col] -= c.groupby(["region", "trim"])[col].transform("mean")
c["x"] = c["hdd"] + 1.5 * c["cdd"]
regiones = expo.query("tipo == 'clima'").rename(
    columns={"zona": "region"})
medidas["weather"] = frente_a_verdad(regiones.merge(c, on="region"))
medidas["weather"]

HDD = max(0, 18 - t): True
CDD = max(0, t - 22): True
n                372.000
corr_z             0.855
corr_sorpresa      0.730
dtype: float64

0.86 with the demand shock and 0.73 with the utilities' sales surprise. It is the strongest source in the course because the generator makes the utilities' demand depend on the weather in their region; in reality the relationship is also very direct (colder weather, more heating), but the market knows it well and the weather forecast anticipates it. There is also a shortcut in the cell: the climatology (the average for each quarter) is computed with the full sample, including years that had not yet happened. For a backtest, the climatology is computed with the past only (exercise 2).

Patents: a control and a censoring

Patents are published about 18 months after the application. Until then, nobody outside the company and the office knows about them. Counting them by filing date is look-ahead bias; and in addition, recent applications are not yet in the data:

pat["q_sol"] = pat["fecha_solicitud"].dt.to_period("Q")
por_sol = pat.groupby("q_sol").size()
print({str(k): v for k, v in por_sol.loc["2023Q1":].items()})
print("last publication:", pat["fecha_publicacion"].max().date())
print("distinct applicants:", pat["solicitante"].nunique(),
      "for", pat["id_empresa"].nunique(), "companies")
pat["ticker"] = pat["id_empresa"].map(tk)
pat["q"] = pat["fecha_publicacion"].dt.to_period("Q")
s = pat.groupby(["ticker", "q"]).size().rename("n")
rejilla = pd.MultiIndex.from_product(
    [s.index.levels[0], pd.period_range("2015Q1", "2025Q3", freq="Q")],
    names=["ticker", "q"])
s = s.reindex(rejilla, fill_value=0).reset_index()
s["x"] = np.log1p(s["n"]) - np.log1p(
    s.groupby("ticker")["n"].shift(4))
medidas["patents (control)"] = frente_a_verdad(s)
medidas["patents (control)"]

{'2023Q1': 138, '2023Q2': 159, '2023Q3': 163, '2023Q4': 157, '2024Q1': 139, '2024Q2': 118, '2024Q3': 2}
last publication: 2025-12-31
distinct applicants: 160 for 40 companies
n                1280.000
corr_z              0.023
corr_sorpresa       0.003
dtype: float64

Applications per quarter fall in the second quarter of 2024, are 2 in the third and none after that: it is not that companies stop patenting, it is that what was filed since mid-2024 has not yet been published (the last publication in the data is from December 31, 2025). A series “by filing date” always shows a drop at the end, and a model that uses it sees a signal that does not exist (and that it could not have known in any case).

Patents are the course's control: the generator gives them no effect on demand. Their year-on-year growth correlates 0.02 with the shock and 0.00 with the surprise. That is what should come out of a source with no signal; if a method finds “alpha” in them, the method is wrong (lesson 2.5.1). There are 160 applicant names for 40 companies: variants and subsidiaries, the same mapping problem as in employment.

On-chain: a different calendar

On-chain data is public by construction: anyone with a node can read the blockchain. The course covered it in lesson 3.7.3 (onchain3_metricas.csv: net flow to exchanges, active addresses, transactions and fees of two fictitious crypto assets). Its distinctive feature as alternative data is the calendar:

oc = pd.read_csv("onchain3_metricas.csv", parse_dates=["fecha"])
print(oc["activo"].value_counts().to_dict())
finde = oc["fecha"].dt.dayofweek >= 5
print(f"weekend days: {finde.mean():.1%}")

{'XBQ': 2192, 'XEQ': 2192}
weekend days: 28.6%

28.6% of the days are weekend days: the blockchain does not close. To combine on-chain data with a stock or an ETF that trades in sessions, you have to decide what to do with Saturday and Sunday (add them to Monday, or take the last value before the open) and set the day's cutoff time (in the course, 23:00 UTC). The planted signal and its controls are in lesson 3.7.3.

The profile of each source

With everything measured, the profile:

claves = ["cards", "web: descargas_app_miles",
          "web: limpias", "jobs (q-2)", "reviews: without flagged",
          "ships: contenedores", "weather", "patents (control)"]
ficha = pd.DataFrame({k: medidas[k] for k in claves}).T
ficha["n"] = ficha["n"].astype(int)
ficha.sort_values("corr_sorpresa", ascending=False)

                             n  corr_z  corr_sorpresa
weather                    372   0.855          0.730
ships: contenedores        310   0.674          0.465
web: descargas_app_miles   810   0.365          0.291
web: limpias               810   0.319          0.258
cards                      675   0.289          0.211
reviews: without flagged   310   0.336          0.186
jobs (q-2)                1500  -0.048          0.131
patents (control)         1280   0.023          0.003

Two caveats for reading the profile. The correlations come from synthetic data with planted effects: they measure how well each simple method recovers what the generator put in, not what real sources are worth. And a correlation with the surprise is not a strategy: you still need to know how much of that information is already in the price when the data arrives (in the course, more and more since 2021; lesson 6.7.5) and how much it costs (lesson 6.7.1).

Common mistakes

• Using a panel's gross spending. Without normalizing by active cards, a bank joining looks like 38% demand growth.
• Dating by the event and not by the delivery. Spending data for day t becomes known at t + 3; a patent filed today becomes known in 18 months.
• Not cleaning automated traffic. Bots cut almost in half what visits say about demand.
• Counting fake reviews. They inflate the average and disconnect it from real satisfaction.
• Building series by filing or event date when the data is published late: look-ahead bias, and an artificial drop at the end of the sample.
• Computing normals or climatologies with the full sample. It is a discreet way of using the future.
• Taking a source's correlation for a tradable signal. The price is missing: what the market already knows.

Exercises

1. Repeat the bot detection with a centered median (the 4 previous weeks and the 4 following ones) instead of the median of the previous 8. Does detection improve? Could it be used in production?

centrada = web.groupby("dominio")["visitas_millones"].transform(
    lambda s: s.rolling(9, center=True, min_periods=5).median())
marca_c = web["visitas_millones"] / centrada > 1.7
acierto = (marca_c & web["verdad"]).sum()
print(f"flagged {marca_c.sum()}, precision "
      f"{acierto / marca_c.sum():.1%}, recall "
      f"{acierto / web['verdad'].sum():.1%}")

flagged 161, precision 100.0%, recall 100.0%

2. Compute the weather index with a climatology using only the past: for each quarter, the average of the same quarter in previous years (from 2019, to have at least one year). Compare the correlation with the surprise.

c2 = clima.groupby(["region", "q"])[["hdd", "cdd"]].sum()
c2 = c2.reset_index().sort_values(["region", "q"])
c2["trim"] = c2["q"].dt.quarter
for col in ["hdd", "cdd"]:
    previa = c2.groupby(["region", "trim"])[col].transform(
        lambda s: s.shift(1).expanding().mean())
    c2[col] = c2[col] - previa
c2["x"] = c2["hdd"] + 1.5 * c2["cdd"]
print(frente_a_verdad(regiones.merge(c2, on="region")))

n                324.000
corr_z             0.826
corr_sorpresa      0.697
dtype: float64

3. Pick a type of alternative data the course does not include (email receipts, mobile geolocation or real satellite imagery) and write its profile: what it measures, likely coverage, delay, biases and legal risks.