---
title: "6.7.2 Types of alternative data: what each one measures"
description: "Cards, web traffic, jobs, reviews, ships, weather, patents and on-chain: what each source measures, coverage, delay and biases, measured with data."
url: "https://pythonparatrading.com/en/python-trading-course/level-6/types-of-alternative-data/"
lang: "en"
image: "https://pythonparatrading.com/og/en/python-trading-course/level-6/types-of-alternative-data.png"
---

# 6.7.2 · Types of alternative data: what each one measures

> Each type of alternative data sees a different part of a company's activity, with its own coverage, delay and traps. This lesson goes through the course's alternative datasets (cards, web and app traffic, job postings, reviews, ship arrivals, weather, patents and on-chain) and measures in each one what it captures, how many companies it covers, when it becomes known and what contaminates it.

- Course: [Level 6: Generative AI, LLMs, agents, big data and alternative data](https://pythonparatrading.com/en/python-trading-course/level-6/)
- Module: 6.7 · Alternative data
- Duration: 65 min
- Prerequisites: [1.4.4 · Universes without survivorship bias: delistings, mergers and symbol changes](https://pythonparatrading.com/en/python-trading-course/level-1/survivorship-bias-and-point-in-time-data/), [5.2.2 · Point-in-time fundamentals, macro and alternative data](https://pythonparatrading.com/en/python-trading-course/level-5/point-in-time-fundamentals-macro-alternative-data/), [6.7.1 · Overview and size of the alternative data market](https://pythonparatrading.com/en/python-trading-course/level-6/alternative-data-market-overview/)

## By the end of this lesson

- Describe the main types of alternative data (transactions, web and app traffic, employment, reviews, AIS, weather, patents, satellite, geolocation, on-chain) by what they measure and how they are collected.
- Measure the coverage, frequency and delivery delay of each source, and distinguish the event date, the delivery date and the publication date.
- Detect the typical biases in the data: panel changes, bot spikes, name variants, fake reviews and censoring due to late publication.
- Compare each source with its known ground truth and with the sales surprise, without confusing what it measures with what it predicts.

## The map of sources

Lesson 6.7.1 classified alternative data by origin. This one goes through the sources one by one with data. The course
includes a synthetic version of almost every type, all of them about the 120 fictitious companies of level 5 and from
2018 to September 2025, with a property real data does not have: we know what each source measures. Behind all of them
there is a quarterly **demand shock** `z` per company that moves sales relative to consensus; each source sees part of
it, with its own noise and delay, and the ground truth is in `alt6_verdad_trimestral.csv`.

| Type | What it measures | In the course | Real (collection examples) |
|---|---|---|---|
| Card transactions | Spending at the company's stores | `alt6_tarjetas_pit.parquet` | Panels from banks or payment aggregators |
| Receipts | Basket, products, price paid | — | Email receipts with the user's permission |
| Web and app traffic | Visits, downloads, usage | `alt6_trafico_web.parquet` | Browsing panels, app stores |
| Job postings | Hiring intention, by area and country | `alt6_empleo.parquet` | Scraping of job boards and careers pages (lesson 6.7.3) |
| Reviews and published prices | Satisfaction, defects, prices | `alt6_resenas.parquet` | Scraping of stores and review sites |
| Shipping (AIS) | Ship arrivals, waiting times in port | `alt6_ais_llegadas.parquet` | Ships' AIS transponders |
| Weather | Temperatures, degree days, rainfall | `alt6_clima.parquet` | National weather services |
| Patents | Published applications by class | `alt6_patentes.parquet` | Patent offices (EPO, USPTO, WIPO) |
| Satellite and geolocation | Cars in parking lots, foot traffic | `satelite6_imagenes.parquet` (lesson 6.7.6) | Imagery vendors; SDKs in mobile apps |
| On-chain | Flows, active addresses, fees | `onchain3_metricas.csv` (lesson 3.7.3) | Blockchain nodes and explorers |

Before looking for a signal, each source goes through the same questions: what it measures, whom it covers, when it
becomes known and what contaminates it. The path of a data point to the model has several dates, and the one that
counts in a backtest is the last one:

```mermaid
flowchart TB
  E[Event] --> R[Collection]
  R --> T[Delivery]
  T --> M[Mapping to company]
  M --> S[Signal]
```

## Coverage and dates

The first table is the inventory: rows, frequency, period and how many companies each source covers. Mapping stores,
names and applicants to companies is a problem in itself (lesson 6.7.5); here the course's ground-truth tables are used
to skip it, except for employment, where it is solved with a rule:

```python
import numpy as np
import pandas as pd

emp = pd.read_csv("acciones6_empresas.csv")
tk = emp.set_index("id_empresa")["ticker"]
verdad = pd.read_csv("alt6_verdad_trimestral.csv",
                     parse_dates=["fin_trimestre"])
verdad["q"] = verdad["fin_trimestre"].dt.to_period("Q")
expo = pd.read_csv("alt6_exposiciones.csv")

tar = pd.read_parquet("alt6_tarjetas_pit.parquet")
com = pd.read_csv("alt6_comercios_verdad.csv")
web = pd.read_parquet("alt6_trafico_web.parquet")
empleo = pd.read_parquet("alt6_empleo.parquet")
res = pd.read_parquet("alt6_resenas.parquet").merge(
    pd.read_csv("alt6_resenas_verdad.csv"), on="id_resena")
ais = pd.read_parquet("alt6_ais_llegadas.parquet")
clima = pd.read_parquet("alt6_clima.parquet")
pat = pd.read_parquet("alt6_patentes.parquet").merge(
    pd.read_csv("alt6_patentes_verdad.csv"), on="id_patente")

primera = emp["nombre_corto"].str.split().str[0].str.lower()
empleo["ticker"] = (empleo["empresa_publicada"].str.split()
                    .str[0].str.lower().map(
                        dict(zip(primera, emp["ticker"]))))
cubre = {
    "cards": com.query("relacion != 'sin_relacion'")
                ["id_empresa"].nunique(),
    "web": web["dominio"].isin(emp["dominio"]).groupby(
        web["dominio"]).any().sum(),
    "jobs": empleo["ticker"].nunique(),
    "reviews": res["id_empresa"].nunique(),
    "ships": (expo["tipo"] == "puerto").sum(),
    "weather": (expo["tipo"] == "clima").sum(),
    "patents": pat["id_empresa"].nunique(),
}
fechas = {"cards": tar["fecha"], "web": web["semana"],
          "jobs": empleo["mes"], "reviews": res["fecha"],
          "ships": ais["fecha"], "weather": clima["fecha"],
          "patents": pat["fecha_publicacion"]}
inv = pd.DataFrame({
    "filas": {k: len(v) for k, v in fechas.items()},
    "desde": {k: v.min().date() for k, v in fechas.items()},
    "hasta": {k: v.max().date() for k, v in fechas.items()},
    "empresas": cubre})
inv
```
```output
          filas       desde       hasta  empresas
cards    359410  2018-01-01  2025-09-30        25
web       12120  2018-01-07  2025-09-28        30
jobs      27900  2018-01-31  2025-09-30        60
reviews   36213  2018-01-01  2025-09-30        10
ships     42450  2018-01-01  2025-09-30        10
weather   16980  2018-01-01  2025-09-30        12
patents    5624  2016-06-27  2025-12-31        40
```

No source covers the whole panel: from 10 companies (reviews, ships) to 60 (jobs). And coverage is not random: cards
only see businesses that sell to consumers, weather only matters to utilities and AIS to exporters. A model built with
one source trades, at most, its universe (lesson 5.2.2).

Dates are the second question. The card table includes `fecha_entrega` (the spending date plus 3 days); the web
traffic table, the delivery on the Tuesday after the week; patents, the filing date and the publication date. The rest
have no delivery date, and the course assumes the data becomes known when its period ends (the day, the month); with a
real vendor, that date is a *due diligence* question (lesson 6.7.4):

```python
retraso = pd.Series({
    "cards": (tar["fecha_entrega"] - tar["fecha"]).dt.days,
    "web (from Sunday)":
        (web["fecha_entrega"] - web["semana"]).dt.days,
    "patents": (pat["fecha_publicacion"]
                - pat["fecha_solicitud"]).dt.days,
})
pd.DataFrame({k: v.describe()[["min", "50%", "max"]]
              for k, v in retraso.items()}).T
```
```output
                     min    50%    max
cards                3.0    3.0    3.0
web (from Sunday)    2.0    2.0    2.0
patents            538.0  557.0  577.0
```

Three days for cards, two for web traffic (plus the six of the week it aggregates) and between 538 and 577 days for
patents: about 18 months, the usual publication period for a patent application[^1]. The figure at the top summarizes
the coverage and delay of each source.

## Card transactions: the panel changes

A card data vendor sees the spending of a **panel** of cards (those of the banks or apps that share data with it), not
that of the whole population. The panel grows and changes composition, and that shows up in gross spending as if it
were demand. In the course data, a new bank joins on March 1, 2021:

```python
panel = pd.read_csv("alt6_tarjetas_panel.csv",
                    parse_dates=["fecha"]).set_index("fecha")
diario = tar.groupby("fecha")["gasto_usd"].sum().to_frame()
diario = diario.join(panel)
diario["por_tarjeta"] = (diario["gasto_usd"]
                         / diario["tarjetas_activas"])
antes = diario.loc["2021-02-01":"2021-02-28"].mean()
despues = diario.loc["2021-03-01":"2021-03-28"].mean()
(despues / antes - 1)[["gasto_usd", "tarjetas_activas",
                       "por_tarjeta", "fraccion_jovenes"]].round(3)
```
```output
gasto_usd           0.382
tarjetas_activas    0.362
por_tarjeta         0.015
fraccion_jovenes    0.147
dtype: float64
```

*Fig. — Weekly card spend in the panel, raw and per active card (February 2021 = 100): a bank joining on 1 March 2021 lifts the raw figure by 38% with no change in demand.* Two weekly lines from November 2020 to June 2021. Until February 2021 they move together near 100, with a December peak. From the week of 7 March, raw spend jumps to about 139 and rises to about 146; spend per card stays between 100 and 107. A highlighted vertical line marks 1 March 2021.

Gross spending rises 38% from February to March; active cards, 36%; spending per card, 1.5%. The rule is to normalize
by panel size before anything else. A subtler bias remains: the share of young cardholders rises with the new bank, and
companies with a young customer base gain weight in the panel without selling more. That composition bias, the
difference between the data as delivered and the history the vendor reconstructs when selling (*backfill*), and how
they are detected, are the subject of lesson 6.7.4.

Once normalized, the spending per card at each company's stores tracks its sales. Its year-on-year growth by quarter,
against the demand shock and the sales surprise:

```python
t = tar.merge(com.query("relacion != 'sin_relacion'"),
              on="descriptor")
t["q"] = t["fecha"].dt.to_period("Q")
tarjetas_q = (panel.groupby(panel.index.to_period("Q"))
              ["tarjetas_activas"].mean())
g = t.groupby(["id_empresa", "q"])["gasto_usd"].sum().reset_index()
g["ticker"] = g["id_empresa"].map(tk)
g["x"] = np.log(g["gasto_usd"] / g["q"].map(tarjetas_q))
g["x"] = g["x"] - g.groupby("ticker")["x"].shift(4)

def frente_a_verdad(df, col="x"):
    m = df.merge(verdad, on=["ticker", "q"]).dropna(subset=[col])
    return pd.Series({
        "n": len(m),
        "corr_z": m[col].corr(m["z_demanda"]),
        "corr_sorpresa": m[col].corr(m["sorpresa_ventas"])}).round(3)

medidas = {"cards": frente_a_verdad(g)}
medidas["cards"]
```
```output
n                675.000
corr_z             0.289
corr_sorpresa      0.211
dtype: float64
```

A correlation of 0.29 with the demand shock and of 0.21 with the surprise over 675 company-quarters. Year-on-year
growth is a crude measure (it mixes this quarter's shock with that of the same quarter of the previous year); lesson
6.7.5 builds the right measure, spending growth relative to the growth implied by consensus, and backtests it.

## Web and app traffic: the bots

Web traffic arrives by week and by domain: visits (in millions) and app downloads (in thousands). Visits have a
well-known problem in real data: automated traffic. In the data, 1.3% of weeks carry a bot spike that multiplies visits
by 2 to 4. A simple rule detects them: divide each week by the median of the previous 8 weeks (past only, so that the
rule can be applied on the delivery date):

```python
bots = pd.read_csv("alt6_trafico_web_bots_verdad.csv",
                   parse_dates=["semana"]).assign(verdad=True)
web = web.sort_values(["dominio", "semana"])
web["mediana"] = web.groupby("dominio")["visitas_millones"].transform(
    lambda s: s.shift(1).rolling(8, min_periods=4).median())
web["ratio"] = web["visitas_millones"] / web["mediana"]
web = web.merge(bots, on=["dominio", "semana"], how="left")
web["verdad"] = web["verdad"].fillna(False).astype(bool)
for umbral in (1.5, 1.7, 2.0):
    marca = web["ratio"] > umbral
    acierto = (marca & web["verdad"]).sum()
    print(f"threshold {umbral}: flagged {marca.sum()}, "
          f"precision {acierto / marca.sum():.1%}, "
          f"recall {acierto / web['verdad'].sum():.1%}")
```
```output
threshold 1.5: flagged 159, precision 100.0%, recall 98.8%
threshold 1.7: flagged 159, precision 100.0%, recall 98.8%
threshold 2.0: flagged 157, precision 100.0%, recall 97.5%
```

*Fig. — Weekly visits to one course domain (bosdor.example) with its 10 bot-traffic spikes, which multiply visits by 2 to 4.* Weekly line from 2018 to 2025 between about 30 and 50 million visits, with seasonality, and ten highlighted isolated spikes of up to 139 million.

With a threshold of 1.7, the rule flags 159 weeks, all of them bots, and finds 159 of the 161. In real data bots do not
always arrive in such clean spikes (there is constant automated traffic, and legitimate spikes from a campaign or a
news item), so the rule is checked against other signals: app traffic, country of origin, visit duration. Replacing
the flagged weeks with the median changes what the source measures:

```python
web["limpias"] = web["visitas_millones"].mask(
    web["ratio"] > 1.7, web["mediana"])   # only the flagged ones
web["ticker"] = web["dominio"].map(
    emp.set_index("dominio")["ticker"])
web["q"] = web["semana"].dt.to_period("Q")
for col in ["visitas_millones", "limpias", "descargas_app_miles"]:
    s = web.groupby(["ticker", "q"])[col].sum().reset_index()
    s["x"] = np.log(s[col]) - np.log(
        s.groupby("ticker")[col].shift(4))
    medidas[f"web: {col}"] = frente_a_verdad(s)
pd.DataFrame({k: v for k, v in medidas.items()
              if k.startswith("web")}).T
```
```output
                              n  corr_z  corr_sorpresa
web: visitas_millones     810.0   0.170          0.176
web: limpias              810.0   0.319          0.258
web: descargas_app_miles  810.0   0.365          0.291
```

Uncleaned, visits correlate 0.17 with the demand shock; cleaned, 0.32. App downloads, which in this data have no bots,
are the best measure of the three (0.37 with the shock, 0.29 with the surprise).

## Job postings: names and horizon

Job postings are collected from job boards and corporate websites, almost always by scraping (lesson 6.7.3). Each board
writes the company name its own way:

```python
variantes = empleo.groupby("ticker")["empresa_publicada"].unique()
print(variantes.iloc[0].tolist())
print("variants per company:",
      empleo.groupby("ticker")["empresa_publicada"].nunique()
      .describe()[["min", "max"]].to_dict())
print("unmapped:", empleo["ticker"].isna().sum())
```
```output
['BEBVIA PETROLEUM', 'Bebvia', 'Bebvia Petroleum Holdings Inc.', 'Bebvia Petroleum', 'Bebvia Petroleum Inc']
variants per company: {'min': 5.0, 'max': 5.0}
unmapped: 0
```

Five variants per company, all with the first word of the short name: in the course data, that rule is enough (the
made-up names start with a unique word). In real data it is not enough, because there are common names, subsidiaries
and brands that share nothing with the parent; lesson 6.7.5 covers entity resolution.

What postings measure is **hiring intention**, and a company hires before it sells more. In the data, new postings
anticipate demand by two quarters:

```python
empleo["q"] = empleo["mes"].dt.to_period("Q")
e = (empleo.groupby(["ticker", "q"])["ofertas_nuevas"].sum()
     .reset_index().sort_values(["ticker", "q"]))
e["x0"] = np.log(e["ofertas_nuevas"]) - np.log(
    e.groupby("ticker")["ofertas_nuevas"].shift(4))
e["x"] = e.groupby("ticker")["x0"].shift(2)   # that of q-2
m = e.merge(verdad, on=["ticker", "q"]).dropna(subset=["x", "x0"])
print("same quarter, with the surprise:",
      round(m["x0"].corr(m["sorpresa_ventas"]), 3))
print("two quarters earlier, with the hiring intention:",
      round(m["x"].corr(m["h_empleo_q_menos_2"]), 3))
medidas["jobs (q-2)"] = frente_a_verdad(e)
medidas["jobs (q-2)"]
```
```output
same quarter, with the surprise: -0.025
two quarters earlier, with the hiring intention: 0.691
n                1500.000
corr_z             -0.048
corr_sorpresa       0.131
dtype: float64
```

Growth in postings in a quarter says nothing about that quarter's surprise (−0.025), but it measures hiring intention
well (0.69) and, two quarters later, it correlates 0.13 with the surprise. It is a small correlation but with a lot of
coverage (60 companies): the type of source that contributes in combination, not on its own.

## Reviews: fakes and defects

Product reviews have the J shape of real data: many 5-star ones, some 1-star ones and few in between. Two effects are
planted in the data: bursts of fake 5-star reviews without a verified purchase, and 49 quality-defect episodes in which
the stars drop and the texts talk about the fault:

```python
print("stars:", res["estrellas"].value_counts(normalize=True)
      .sort_index().round(3).to_dict())
print(f"fake: {res['falsa'].mean():.1%}; "
      f"with defect: {res['producto_con_defecto'].mean():.1%}")
print("mean with and without defect:", res.groupby(
    "producto_con_defecto")["estrellas"].mean().round(2).to_dict())
marca = (res["estrellas"] == 5) & ~res["compra_verificada"]
acierto = (marca & res["falsa"]).sum()
print(f"rule 5 stars without purchase: precision "
      f"{acierto / marca.sum():.1%}, recall "
      f"{acierto / res['falsa'].sum():.1%}")
```
```output
stars: {1: 0.081, 2: 0.042, 3: 0.072, 4: 0.167, 5: 0.638}
fake: 22.1%; with defect: 5.4%
mean with and without defect: {False: 4.29, True: 3.29}
rule 5 stars without purchase: precision 77.8%, recall 100.0%
```

*Fig. — The course's 36,213 reviews by stars: the J shape of real reviews, with 22.1% fake 5-star reviews without a verified purchase.* Bars: 1 star 8.1%, 2 stars 4.2%, 3 stars 7.2%, 4 stars 16.7% and 5 stars 63.8%, of which 41.7 points are genuine and 22.1 fake (highlighted).

22% of the reviews are fake, and the “5 stars without a verified purchase” rule finds all of them, but with a precision
of 78%: there are real customers who give 5 stars without a verified purchase. Reviews of products with a defect score
one star lower on average (3.29 versus 4.29). For the company, what matters is the quarterly average (centered by
company, because each one has its own level):

```python
res["ticker"] = res["id_empresa"].map(tk)
res["q"] = res["fecha"].dt.to_period("Q")
for nombre, sub in [("all", res), ("without fakes", res[~res["falsa"]]),
                    ("without flagged", res[~marca])]:
    s = sub.groupby(["ticker", "q"])["estrellas"].mean()
    s = (s - s.groupby(level="ticker").transform("mean"))
    medidas[f"reviews: {nombre}"] = frente_a_verdad(
        s.rename("x").reset_index())
pd.DataFrame({k: v for k, v in medidas.items()
              if k.startswith("reviews")}).T
```
```output
                              n  corr_z  corr_sorpresa
reviews: all              310.0   0.244          0.164
reviews: without fakes    310.0   0.352          0.207
reviews: without flagged  310.0   0.336          0.186
```

Fake reviews dilute the signal: with all of them, 0.24 with the demand shock; without them, 0.35. The imperfect rule
recovers almost everything (0.34) even though it throws away genuine reviews: precision matters less than not letting
the noise through.

## Ships and weather: the physical world

**AIS** data comes from the transponders ships carry: the International Maritime Organization requires the system on
ships of 300 gross tonnage or more on international voyages, on cargo ships of 500 or more and on all passenger
ships[^2]. Aggregated by port and ship type, it gives daily arrivals and ships at anchor waiting for a berth. In the
course, 10 exporters depend on the activity of five fictitious ports:

```python
ais["q"] = ais["fecha"].dt.to_period("Q")
puertos = expo.query("tipo == 'puerto'").rename(
    columns={"zona": "puerto"})
for tipo in ["contenedores", "granel", "petrolero"]:
    s = (ais[ais["tipo_buque"] == tipo]
         .groupby(["puerto", "q"])["llegadas"].sum().reset_index())
    s["x"] = np.log(s["llegadas"])
    s["x"] = s["x"] - s.groupby("puerto")["x"].transform("mean")
    medidas[f"ships: {tipo}"] = frente_a_verdad(
        puertos.merge(s, on="puerto"))
pd.DataFrame({k: v for k, v in medidas.items()
              if k.startswith("ships")}).T
```
```output
                         n  corr_z  corr_sorpresa
ships: contenedores  310.0   0.674          0.465
ships: granel        310.0   0.400          0.259
ships: petrolero     310.0   0.452          0.305
```

The quarter's container ship arrivals correlate 0.67 with the demand shock of the exporters at their port: it is the
most direct source in the course after weather. Bulk carriers and tankers also move with port activity (0.40 and 0.45),
although the course's exporters do not depend on them. The trap is one of mapping: a port is not a company. The more
companies share a port, the more what the source says about each one is diluted.

**Weather** is the oldest and cheapest alternative data: it is published by weather services. For energy it is
summarized in **degree days**: heating degree days (HDD) add up how far the day's mean temperature falls below a base,
and cooling degree days (CDD), how far it rises above it[^3]. The course uses a base of 18 °C for HDD and 22 °C for
CDD; the cell checks it and builds the quarter's index relative to its climatology:

```python
print("HDD = max(0, 18 - t):", np.allclose(
    clima["hdd"], np.maximum(0, 18 - clima["temp_media"])))
print("CDD = max(0, t - 22):", np.allclose(
    clima["cdd"], np.maximum(0, clima["temp_media"] - 22)))
clima["q"] = clima["fecha"].dt.to_period("Q")
c = clima.groupby(["region", "q"])[["hdd", "cdd"]].sum()
c = c.reset_index()
c["trim"] = c["q"].dt.quarter
for col in ["hdd", "cdd"]:   # climatology with the full sample
    c[col] -= c.groupby(["region", "trim"])[col].transform("mean")
c["x"] = c["hdd"] + 1.5 * c["cdd"]
regiones = expo.query("tipo == 'clima'").rename(
    columns={"zona": "region"})
medidas["weather"] = frente_a_verdad(regiones.merge(c, on="region"))
medidas["weather"]
```
```output
HDD = max(0, 18 - t): True
CDD = max(0, t - 22): True
n                372.000
corr_z             0.855
corr_sorpresa      0.730
dtype: float64
```

0.86 with the demand shock and 0.73 with the utilities' sales surprise. It is the strongest source in the course
because the generator makes the utilities' demand depend on the weather in their region; in reality the relationship is
also very direct (colder weather, more heating), but the market knows it well and the weather forecast anticipates it.
There is also a shortcut in the cell: the climatology (the average for each quarter) is computed with the full sample,
including years that had not yet happened. For a backtest, the climatology is computed with the past only (exercise 2).

## Patents: a control and a censoring

Patents are published about 18 months after the application. Until then, nobody outside the company and the office
knows about them. Counting them by filing date is look-ahead bias; and in addition, recent applications are not yet in
the data:

```python
pat["q_sol"] = pat["fecha_solicitud"].dt.to_period("Q")
por_sol = pat.groupby("q_sol").size()
print({str(k): v for k, v in por_sol.loc["2023Q1":].items()})
print("last publication:", pat["fecha_publicacion"].max().date())
print("distinct applicants:", pat["solicitante"].nunique(),
      "for", pat["id_empresa"].nunique(), "companies")
pat["ticker"] = pat["id_empresa"].map(tk)
pat["q"] = pat["fecha_publicacion"].dt.to_period("Q")
s = pat.groupby(["ticker", "q"]).size().rename("n")
rejilla = pd.MultiIndex.from_product(
    [s.index.levels[0], pd.period_range("2015Q1", "2025Q3", freq="Q")],
    names=["ticker", "q"])
s = s.reindex(rejilla, fill_value=0).reset_index()
s["x"] = np.log1p(s["n"]) - np.log1p(
    s.groupby("ticker")["n"].shift(4))
medidas["patents (control)"] = frente_a_verdad(s)
medidas["patents (control)"]
```
```output
{'2023Q1': 138, '2023Q2': 159, '2023Q3': 163, '2023Q4': 157, '2024Q1': 139, '2024Q2': 118, '2024Q3': 2}
last publication: 2025-12-31
distinct applicants: 160 for 40 companies
n                1280.000
corr_z              0.023
corr_sorpresa       0.003
dtype: float64
```

Applications per quarter fall in the second quarter of 2024, are 2 in the third and none after that: it is not that
companies stop patenting, it is that what was filed since mid-2024 has not yet been published (the last publication in
the data is from December 31, 2025). A series “by filing date” always shows a drop at the end, and a model that uses it
sees a signal that does not exist (and that it could not have known in any case).

Patents are the course's **control**: the generator gives them no effect on demand. Their year-on-year growth
correlates 0.02 with the shock and 0.00 with the surprise. That is what should come out of a source with no signal; if
a method finds “alpha” in them, the method is wrong (lesson 2.5.1). There are 160 applicant names for 40 companies:
variants and subsidiaries, the same mapping problem as in employment.

## On-chain: a different calendar

On-chain data is public by construction: anyone with a node can read the blockchain. The course covered it in lesson
3.7.3 (`onchain3_metricas.csv`: net flow to exchanges, active addresses, transactions and fees of two fictitious crypto
assets). Its distinctive feature as alternative data is the calendar:

```python
oc = pd.read_csv("onchain3_metricas.csv", parse_dates=["fecha"])
print(oc["activo"].value_counts().to_dict())
finde = oc["fecha"].dt.dayofweek >= 5
print(f"weekend days: {finde.mean():.1%}")
```
```output
{'XBQ': 2192, 'XEQ': 2192}
weekend days: 28.6%
```

28.6% of the days are weekend days: the blockchain does not close. To combine on-chain data with a stock or an ETF that
trades in sessions, you have to decide what to do with Saturday and Sunday (add them to Monday, or take the last value
before the open) and set the day's cutoff time (in the course, 23:00 UTC). The planted signal and its controls are in
lesson 3.7.3.

## The profile of each source

With everything measured, the profile:

```python
claves = ["cards", "web: descargas_app_miles",
          "web: limpias", "jobs (q-2)", "reviews: without flagged",
          "ships: contenedores", "weather", "patents (control)"]
ficha = pd.DataFrame({k: medidas[k] for k in claves}).T
ficha["n"] = ficha["n"].astype(int)
ficha.sort_values("corr_sorpresa", ascending=False)
```
```output
                             n  corr_z  corr_sorpresa
weather                    372   0.855          0.730
ships: contenedores        310   0.674          0.465
web: descargas_app_miles   810   0.365          0.291
web: limpias               810   0.319          0.258
cards                      675   0.289          0.211
reviews: without flagged   310   0.336          0.186
jobs (q-2)                1500  -0.048          0.131
patents (control)         1280   0.023          0.003
```

*Fig. — Correlation of each course alternative source, measured with a simple method, with the quarter's sales surprise (for jobs, two quarters earlier; patents are the no-effect control).* Horizontal bars: weather 0.73 (highlighted), ships 0.47, downloads 0.29, clean visits 0.26, cards 0.21, unflagged reviews 0.19, visits with bots 0.18, jobs 0.13 and patents 0.00.

| Source | Coverage | Delay | Main bias | Corr. with the surprise |
|---|---|---|---|---|
| Weather | 12 utilities (6 regions) | Daily, immediate | Climatology using the future | 0.73 |
| Ships (AIS) | 10 exporters (5 ports) | Daily | A port is not a company | 0.47 |
| App downloads | 30 | Weekly, 2 days | — | 0.29 |
| Web visits (cleaned) | 30 | Weekly, 2 days | Bot spikes | 0.26 |
| Cards | 25 | Daily, 3 days | Panel changes, demographics | 0.21 |
| Reviews (without flagged) | 10 | Daily | Fake reviews | 0.19 |
| Jobs (q−2) | 60 | Monthly | Name variants | 0.13 |
| Patents | 40 | 18 months | Censoring, look-ahead bias | 0.00 |

Two caveats for reading the profile. The correlations come from synthetic data with planted effects: they measure how
well each simple method recovers what the generator put in, not what real sources are worth. And a correlation with
the surprise is not a strategy: you still need to know how much of that information is already in the price when the
data arrives (in the course, more and more since 2021; lesson 6.7.5) and how much it costs (lesson 6.7.1).

## Common mistakes

- **Using a panel's gross spending**. Without normalizing by active cards, a bank joining looks like 38% demand growth.
- **Dating by the event and not by the delivery**. Spending data for day t becomes known at t + 3; a patent filed today
  becomes known in 18 months.
- **Not cleaning automated traffic**. Bots cut almost in half what visits say about demand.
- **Counting fake reviews**. They inflate the average and disconnect it from real satisfaction.
- **Building series by filing or event date when the data is published late**: look-ahead bias, and an artificial
  drop at the end of the sample.
- **Computing normals or climatologies with the full sample**. It is a discreet way of using the future.
- **Taking a source's correlation for a tradable signal**. The price is missing: what the market already knows.

## Exercises

1. Repeat the bot detection with a **centered** median (the 4 previous weeks and the 4 following ones) instead of the
   median of the previous 8. Does detection improve? Could it be used in production?

**Solution.**

```python
centrada = web.groupby("dominio")["visitas_millones"].transform(
    lambda s: s.rolling(9, center=True, min_periods=5).median())
marca_c = web["visitas_millones"] / centrada > 1.7
acierto = (marca_c & web["verdad"]).sum()
print(f"flagged {marca_c.sum()}, precision "
      f"{acierto / marca_c.sum():.1%}, recall "
      f"{acierto / web['verdad'].sum():.1%}")
```
```output
flagged 161, precision 100.0%, recall 100.0%
```

The centered median finds the 161 bot weeks with no false positives (at the start of each series it already has data),
but it uses the following four weeks: in production, the flagged week could not be corrected until a month later. To
clean the history for a backtest it is only valid if the cleaning is applied with the same delay; otherwise, the
backtest sees cleaner data than it would have had on the delivery date.

2. Compute the weather index with a climatology **using only the past**: for each quarter, the average of the same
   quarter in previous years (from 2019, to have at least one year). Compare the correlation with the surprise.

**Solution.**

```python
c2 = clima.groupby(["region", "q"])[["hdd", "cdd"]].sum()
c2 = c2.reset_index().sort_values(["region", "q"])
c2["trim"] = c2["q"].dt.quarter
for col in ["hdd", "cdd"]:
    previa = c2.groupby(["region", "trim"])[col].transform(
        lambda s: s.shift(1).expanding().mean())
    c2[col] = c2[col] - previa
c2["x"] = c2["hdd"] + 1.5 * c2["cdd"]
print(frente_a_verdad(regiones.merge(c2, on="region")))
```
```output
n                324.000
corr_z             0.826
corr_sorpresa      0.697
dtype: float64
```

The correlation with the surprise falls only slightly, from 0.73 to 0.70, and the first year is lost (324
company-quarters instead of 372): the generator's weather has no trend, so the full-sample average and that of the
previous years are similar. With real data, with warming, the full-sample climatology mixes warmer winters from the
future with those of the past and shifts the index. The honest approach is the expanding version (or the official
30-year climate normals in force at each date).

3. Pick a type of alternative data the course does not include (email receipts, mobile geolocation or real satellite
   imagery) and write its profile: what it measures, likely coverage, delay, biases and legal risks.

**Solution.**

For **mobile geolocation**: it measures visits to stores, offices or factories from the location sent by SDKs embedded
in apps. Coverage: the users of those apps who gave permission, biased by age, operating system and country; the panel
changes when an app stops sharing data (like the bank jump in the cards). Delay: from days to a week. Biases: small
stores with few measured visits, badly drawn polygons (the store versus the shared parking lot), changes in the
operating systems' privacy policies. Legal risk: it is personal data (location identifies a person), so in the EU the
GDPR applies: legal basis, minimization and aggregation before selling (lessons 6.7.3 and 6.7.4). The *due diligence*
question: how the vendor obtained consent and at what level of aggregation it delivers.

[^1]: WIPO, “Protecting your Inventions Abroad: Frequently Asked Questions About the Patent Cooperation Treaty (PCT)”
    (<https://www.wipo.int/pct/en/faqs/faqs.html>, accessed on 2026-11-05): “WIPO publishes the
    international application shortly after the expiration of 18 months from the priority date”.
[^2]: IMO, “AIS transponders” (<https://www.imo.org/en/OurWork/Safety/Pages/AIS.aspx>, accessed on 2026-10-23):
    regulation V/19 of the SOLAS convention, ships of 300 GT or more on international voyages, cargo ships of 500 GT or
    more not engaged on international voyages and all passenger ships.
[^3]: NOAA, Climate Prediction Center, “Weekly & Monthly Degree Day Summaries Explanation”
    (<https://www.cpc.ncep.noaa.gov/products/analysis_monitoring/cdus/degree_days/ddayexp.shtml>, accessed on
    2026-10-23): “Heating degree days are summations of negative differences between the mean daily temperature and
    the 65°F base” (18.3 °C), and cooling degree days, of the positive differences above the same base. The course
    deliberately uses different bases (18 and 22 °C).

## Self-assessment

1. On March 1, 2021 the panel's total card spending rises 38% from one day to the next. What is the first thing to check?

   - A. Whether there was a sales campaign
   - B. Whether the number of cards in the panel changed: a new bank joining inflates gross spending without any change in demand
   - C. Whether the vendor changed currency
   - D. Nothing: a jump like that is a bullish signal

   *Answer B: Whether the number of cards in the panel changed: a new bank joining inflates gross spending without any change in demand.* In the course data a bank joins with 36% more cards; spending per active card barely moves (+1.5%). Normalizing by panel size is the first step with any panel data.

2. A patent source gives, for each application, the filing date and the publication date (about 18 months later). Which date is used in a backtest?

   - A. The filing date, because that is when the innovation happens
   - B. Either one: the difference is small
   - C. The average of the two
   - D. The publication date: before that, nobody outside the company could know about the application

   *Answer D: The publication date: before that, nobody outside the company could know about the application.* Counting patents by filing date uses information that was published a year and a half later: look-ahead bias. In addition, applications from the last 18 months do not yet appear in the data (censoring), so series by filing date fall artificially at the end.

3. In the course's web traffic, 1.3% of weeks have a bot spike of 2 to 4 times normal traffic. What happens if they are not cleaned?

   - A. The correlation of visit growth with the demand shock falls from 0.32 to 0.17
   - B. Nothing: they are few weeks
   - C. The signal improves, because bots follow companies that grow
   - D. It only affects app downloads

   *Answer A: The correlation of visit growth with the demand shock falls from 0.32 to 0.17.* A bot spike goes entirely into the quarterly sum and the year-on-year growth. With a rolling median of the previous 8 weeks almost all of them are detected (98.8%, with no false positives in this data) and the correlation with the demand shock almost doubles.

4. The course's job postings anticipate demand by two quarters. What correlation would you expect between the growth in postings in a quarter and the sales surprise of that same quarter?

   - A. High, because hiring is expensive
   - B. Negative, because more postings mean more costs
   - C. Close to zero: the information in the postings shows up in sales two quarters later
   - D. The same as with a two-quarter lag

   *Answer C: Close to zero: the information in the postings shows up in sales two quarters later.* In the data, the growth in postings in q−2 correlates 0.13 with the surprise in q and 0.69 with the hiring intention it measures; that of the same quarter, −0.02. Each source has its own horizon.

5. The course's fake reviews are 5-star and without a verified purchase. A rule that flags all 5-star reviews without a verified purchase…

   - A. is always right
   - B. finds all the fake ones, but 22% of what it flags are genuine reviews
   - C. finds none
   - D. only works in English

   *Answer B: finds all the fake ones, but 22% of what it flags are genuine reviews.* The rule has 100% recall and 78% precision: there are real customers who give 5 stars without a verified purchase. Removing the fake ones raises the correlation of the average stars with the demand shock from 0.24 to 0.35.

## Key points

- Each source measures something different: spending at stores, visits and downloads, hiring intention, satisfaction, port activity, energy demand driven by the weather. The relationship with sales depends on the company and the sector.
- Coverage is partial and uneven: in the course, 10 to 60 of the 120 companies per source. A model that uses a source only trades its universe.
- Three dates per data point: the event date, the delivery date and the publication date. Cards, 3 days; web traffic, 2 days after the week; patents, about 18 months. In the backtest, the date on which it could be known is what counts.
- Each source brings its own dirt: panel jumps, bot spikes, name variants, fake reviews, censoring of what has not yet been published. They are measured and corrected before looking for a signal.
- Correlation with the sales surprise in the course: weather 0.73, ships 0.47, downloads 0.29, cleaned visits 0.26, cards 0.21, reviews 0.19, jobs (two quarters earlier) 0.13; patents, the no-effect control, 0.00.

## Resources

### Documentation

- [IMO: automatic identification systems (AIS)](https://www.imo.org/en/OurWork/Safety/Pages/AIS.aspx) · imo.org
- [NOAA Climate Prediction Center: degree days (explanation)](https://www.cpc.ncep.noaa.gov/products/analysis_monitoring/cdus/degree_days/ddayexp.shtml) · cpc.ncep.noaa.gov
- [WIPO: Patent Cooperation Treaty (PCT), international publication at 18 months](https://www.wipo.int/pct/en/faqs/faqs.html) · wipo.int

### Data

- [Level 6 data: point-in-time card transactions (Parquet)](https://pythonparatrading.com/curso/datos/nivel-6/alt6_tarjetas_pit.parquet) · Download · alt6_tarjetas_pit.parquet
- [Level 6 data: web traffic and app downloads (Parquet)](https://pythonparatrading.com/curso/datos/nivel-6/alt6_trafico_web.parquet) · Download · alt6_trafico_web.parquet
- [Level 6 data: ship arrivals by port (Parquet)](https://pythonparatrading.com/curso/datos/nivel-6/alt6_ais_llegadas.parquet) · Download · alt6_ais_llegadas.parquet
- [Level 6 data: quarterly ground truth of the alternative data (CSV)](https://pythonparatrading.com/curso/datos/nivel-6/alt6_verdad_trimestral.csv) · Download · alt6_verdad_trimestral.csv

Previous lesson: [6.7.1 · Overview and size of the alternative data market](https://pythonparatrading.com/en/python-trading-course/level-6/alternative-data-market-overview/)

Next lesson: [6.7.3 · Responsible web scraping](https://pythonparatrading.com/en/python-trading-course/level-6/responsible-web-scraping/)