Event-based sampling: the symmetric CUSUM filter.

When to observe

Lessons 5.3.1 and 5.3.2 labeled observations; what remains is deciding which observations. The default option is one per bar: every minute, every day. It has two problems. The first is that most of those observations are noise: a midday minute in which the price does not move teaches nothing about the signal being sought. The second is that consecutive observations are nearly identical: features that barely change and labels that overlap, the problem that lesson 5.3.4 measured.

Lesson 1.4.5 tackled a similar problem when building the bars: tick, volume or dollar bars, which close when there is activity and not when the clock ticks. Event-based sampling (López de Prado, 2018, section 2.5.2) goes one step further: given a series, whether of time bars or activity bars, it observes only when something relevant has happened. The most widely used filter is the symmetric CUSUM, a variant of Page's (1954) cumulative sums that lesson 4.2.5 used to detect that a strategy had stopped working.

The symmetric CUSUM filter

With $y_t$ the log return of each bar, the filter keeps two sums, one of rises and one of falls, clipped at zero:

When $S^+_t > h_t$ there is an upward event; when $S^-_t < -h_t$, a downward one. After each event both sums go back to zero. Clipping at zero means a fall is not offset by an earlier rise: $S^+$ measures the cumulative rise since the last relative minimum, and the event arrives when that rise exceeds $h_t$. López de Prado's version subtracts from $y_t$ its expected mean, which for one-minute returns is zero for all practical purposes.

stateDiagram-v2
  [*] --> Accumulate
  Accumulate --> Accumulate: below h
  Accumulate --> Event: S+ > h or S− < −h
  Event --> Accumulate: reset to 0

The data are FNX's (level 5 synthetic data): one year of 1-minute bars of a fictitious future with a latent regime of trend or reversion, which hf5_verdad.parquet gives minute by minute. As in the previous lessons, the regime is used only for measuring.

import numpy as np
import pandas as pd

barras = pd.read_parquet("hf5_barras_1min.parquet")
verdad = pd.read_parquet("hf5_verdad.parquet")
n_ses = len(barras) // 390
C = barras["cierre"].to_numpy().reshape(n_ses, 390)
tend = (verdad["regimen"] == "tendencia").to_numpy()
tend = tend.reshape(n_ses, 390)        # the truth: only for measuring
lr = np.diff(np.log(C), axis=1)        # 389 per session
sig = (pd.Series(lr.ravel()).ewm(span=100).std()
       .to_numpy().reshape(n_ses, 389))
print(f"minutes in trend: {tend.mean():.1%}")

minutes in trend: 35.3%

35.3% of the year's minutes are in trend. The threshold of the level's primary model is $h_t = 4\,\hat\sigma_t$, with $\hat\sigma_t$ the EWMA deviation (span of 100 minutes) of the 1-minute returns computed up to that very minute: it is known at the close of the minute that fires the event, which is when the decision is made.

def cusum(lr, umbral):
    """Symmetric CUSUM filter, session by session.

    lr: log returns (sessions x minutes); umbral: same shape
    (or a number). Accumulates rises in s_pos and falls in s_neg since
    the last event; when one exceeds the threshold, event, and both
    go back to 0. Returns session, minute (index of the close that
    fires), side (+1 upward break) and the threshold used."""
    h = np.broadcast_to(umbral, lr.shape)
    filas = []
    for d in range(lr.shape[0]):
        s_pos = s_neg = 0.0
        for m in range(lr.shape[1]):
            if not np.isfinite(h[d, m]):
                continue
            s_pos = max(0.0, s_pos + lr[d, m])
            s_neg = min(0.0, s_neg + lr[d, m])
            if s_pos > h[d, m] or s_neg < -h[d, m]:
                lado = 1 if s_pos > h[d, m] else -1
                filas.append((d, m + 1, lado, h[d, m]))
                s_pos = s_neg = 0.0
    return pd.DataFrame(
        filas, columns=["sesion", "minuto", "lado", "umbral"])


ev = cusum(lr, 4.0 * sig)
print(len(ev), "events,", round(len(ev) / n_ses, 1), "per session")
print("minutes between events (median):",
      ev.groupby("sesion").minuto.diff().median())
ev[ev.sesion == 133].head(6).assign(
    umbral_pb=lambda x: (1e4 * x.umbral).round(1)).drop(
    columns="umbral")

6550 events, 26.5 per session
minutes between events (median): 13.0
      sesion  minuto  lado  umbral_pb
3525     133       4    -1       14.2
3526     133       6     1       16.0
3527     133      11     1       16.6
3528     133      38     1       18.1
3529     133      45     1       18.8
3530     133      49     1       19.6

The result is the README's 6,550 events, 26.5 per session, with a median of 13 minutes between one and the next. The header figure shows the session of January 15, 2025, the one used in the examples of lessons 5.3.2 and 5.3.4. In the morning uptrend, almost all events are upward: the price rises, the filter accumulates and fires again and again on the same side. In the afternoon reversion, upward and downward events alternate.

The first three events arrive in the first 11 minutes with thresholds of 14 to 17 bp. The EWMA carries over the volatility of the previous afternoon, lower than that of the open, so the threshold is narrow precisely when the price moves most. It is the same lag that lesson 5.3.2 measured in the barriers.

Fixed or scaled threshold

The simplest alternative is a fixed threshold in basis points. With the same average level (4 times the median of $\hat\sigma$, 16.9 bp):

h_fijo = 4.0 * np.nanmedian(sig)
ev_fijo = cusum(lr, h_fijo)
print(f"fixed threshold: {1e4 * h_fijo:.1f} bp; {len(ev_fijo)} events")
vol_ses = lr.std(axis=1)                  # volatility of each session
tercil = pd.qcut(vol_ses, 3, labels=["low", "medium", "high"])
por_sesion = pd.DataFrame({
    "tercil": tercil,
    "escalado": np.bincount(ev.sesion, minlength=n_ses),
    "fijo": np.bincount(ev_fijo.sesion, minlength=n_ses)})
por_sesion.groupby("tercil", observed=True).mean().round(1)

fixed threshold: 16.9 bp; 6897 events
        escalado  fijo
tercil
low         26.9  19.4
medium      26.6  27.6
high        26.0  36.9

With a fixed threshold, calm sessions (low volatility tercile) give 19.4 events and turbulent ones 36.9: almost twice as many. A model trained on those events learns mostly from turbulent days, and the same 17 bp move means different things depending on the day. With the scaled threshold, each tercile gives between 26.0 and 26.9 events: the filter requires the same move in units of volatility.

To know which events are better, they have to be labeled. With the triple barrier of lesson 5.3.2 and the side of the break (the momentum primary model):

def acierto(ev, h=30):
    """Triple barrier ±σ·√h (5.3.2); 1 if the side is right."""
    out = []
    for d, m, lado in ev[["sesion", "minuto", "lado"]].itertuples(
            index=False):
        b = sig[d, m - 1] * np.sqrt(h)
        ult = min(m + h, 389)
        tramo = np.log(C[d, m + 1:ult + 1] / C[d, m])
        golpes = np.flatnonzero(np.abs(tramo) >= b)
        j = golpes[0] if len(golpes) else len(tramo) - 1
        out.append(int(len(tramo) > 0 and lado * tramo[j] > 0))
    return np.array(out)


def resumen(ev):
    a = acierto(ev)
    t = tend[ev.sesion, ev.minuto]
    return {"por_sesion": len(ev) / n_ses, "en_tendencia": t.mean(),
            "acierto": a.mean(), "acierto_tend": a[t].mean(),
            "acierto_rev": a[~t].mean()}


pd.DataFrame({"scaled k = 4": resumen(ev),
              "fixed": resumen(ev_fijo)}).T.round(3)

              por_sesion  en_tendencia  acierto  acierto_tend  acierto_rev
scaled k = 4      26.518         0.388    0.507         0.604        0.446
fixed             27.923         0.388    0.504         0.588        0.450

Both filters give a fraction of events in trend of 38.8% and a similar hit rate: 50.7% for the scaled one and 50.4% for the fixed one, a difference of half a standard error. The scaled one is somewhat more accurate in trend (60.4% against 58.8%). The main reason to scale is not the average hit rate but the composition of the sample.

Threshold and number of events

The threshold decides how many events there are. Sweeping it:

barrido = pd.DataFrame({k: resumen(cusum(lr, k * sig))
                        for k in (2, 3, 4, 5, 6, 8)}).T
barrido.index.name = "k"
barrido.round(3)

   por_sesion  en_tendencia  acierto  acierto_tend  acierto_rev
k
2      68.575         0.364    0.502         0.565        0.465
3      40.308         0.373    0.503         0.575        0.460
4      26.518         0.388    0.507         0.604        0.446
5      18.785         0.396    0.506         0.612        0.436
6      13.976         0.413    0.507         0.615        0.431
8       8.636         0.441    0.507         0.637        0.404

• Events fall quickly with the threshold: 68.6 per session with $k = 2$, 26.5 with $k = 4$ and 8.6 with $k = 8$. For a random walk, the expected time until a cumulative sum travels $k\sigma$ grows with $k^2$; on FNX, from $k = 2$ to $k = 4$ the events are divided by 2.6 and from 4 to 8 by 3.1, somewhat less than 4 because the session is cut at the close and the trend episodes speed up the path.
• Events concentrate in trend: from 36.4% with $k = 2$ to 44.1% with $k = 8$, against 35.3% of the minutes. A large move is more likely during a trend, and the filter finds it.
• The total hit rate does not move (from 50.2% to 50.7%), because while the hit rate in trend rises from 56.5% to 63.7%, the one in reversion falls from 46.5% to 40.4%: in a mean-reversion regime, the larger the deviation, the stronger the pullback, and the momentum side is wrong more often.

The threshold is therefore a trade-off between quantity and composition: with $k = 8$ there are “purer” samples but a third of those with $k = 4$, and with them more variance in everything that is estimated. Like the width of the barriers (5.3.2), it is set by design and not by looking at the hit rate on the test sample.

By events or by time

Does the CUSUM gain anything over observing at fixed time intervals? With the momentum side computed over the previous interval and the same label:

def por_tiempo(n):
    """One sample every n minutes; side = sign of the previous n."""
    filas = []
    for d in range(n_ses):
        for m in range(n, 390, n):
            r = np.log(C[d, m] / C[d, m - n])
            if r != 0:
                filas.append((d, m, int(np.sign(r))))
    return pd.DataFrame(filas, columns=["sesion", "minuto", "lado"])


tiempo = pd.DataFrame({f"every {n} min": resumen(por_tiempo(n))
                       for n in (5, 15, 30)}).T
tiempo.round(3)

              por_sesion  en_tendencia  acierto  acierto_tend  acierto_rev
every 5 min       75.332         0.353    0.503         0.556        0.474
every 15 min      24.664         0.353    0.505         0.589        0.460
every 30 min      11.879         0.354    0.511         0.592        0.466

Every 15 minutes gives 24.7 observations per session, almost the same as the CUSUM with $k = 4$. The differences:

• Time-based sampling distributes the observations like the minutes: 35.3% in trend, no more and no less. The CUSUM tilts them toward trend (38.8%).
• The total hit rate is almost the same: 50.5% every 15 minutes and 50.7% with the CUSUM. With about 6,000 events each, the standard error of each hit rate is around 0.6 points: there is no measurable difference. The CUSUM does not make good a primary model that is not; it changes which moments are observed.

Where the difference does show is in when the observation happens:

def por_media_hora(ev):
    return np.bincount(ev.minuto // 30, minlength=13) / n_ses


hora = pd.DataFrame({
    "scaled CUSUM": por_media_hora(ev),
    "fixed CUSUM": por_media_hora(ev_fijo),
    "every 15 min": por_media_hora(por_tiempo(15))},
    index=[f"{9 + (30 + 30 * i) // 60}:{(30 + 30 * i) % 60:02d}"
           for i in range(13)])
hora.round(1)

       scaled CUSUM  fixed CUSUM  every 15 min
9:30            3.0          4.1           1.0
10:00           1.8          2.7           2.0
10:30           1.6          2.2           2.0
11:00           1.6          1.9           2.0
11:30           1.7          1.8           2.0
12:00           1.9          1.8           2.0
12:30           1.9          1.8           2.0
13:00           2.0          1.8           2.0
13:30           2.1          1.8           2.0
14:00           2.1          1.8           2.0
14:30           2.1          1.9           2.0
15:00           2.2          2.1           2.0
15:30           2.4          2.3           2.0

Time-based sampling is flat (the first half hour has a single observation because the interval starts at 9:45). The CUSUM follows activity: with a fixed threshold, 4.1 events per session in the first half hour and 1.8 at midday, the same U that the volume bars of lesson 1.4.5 drew. Scaled, the U flattens but does not disappear: 3.0 at the open because of the EWMA lag and a gentle rise toward the close. If sampling uniform in units of volatility were wanted, the threshold would have to use the intraday profile, like the barrier of lesson 5.3.2.

Bands against CUSUM: why the reset matters

An intuitive rule for sampling is the band rule: an event every time the price leaves a band around its moving average (in the style of Bollinger bands). With a 30-minute exponential average and bands of $\pm 0.5\,\sigma\sqrt{30}$:

lc = np.log(C)
ema = pd.DataFrame(lc.T).ewm(span=30).mean().to_numpy().T
z = np.full_like(lc, np.nan)
z[:, 1:] = (lc[:, 1:] - ema[:, 1:]) / (sig * np.sqrt(30))
fuera_arr, fuera_ab = z > 0.5, z < -0.5
cruce = np.zeros_like(lc, dtype=int)
cruce[:, 1:] = (np.where(fuera_arr[:, 1:] & ~fuera_arr[:, :-1], 1, 0)
                - np.where(fuera_ab[:, 1:] & ~fuera_ab[:, :-1], 1, 0))
d_b, m_b = np.nonzero(cruce)
bandas = pd.DataFrame({"sesion": d_b, "minuto": m_b,
                       "lado": cruce[d_b, m_b]})


def racimos(ev):
    gap = ev.groupby("sesion").minuto.diff().dropna()
    return {"por_sesion": len(ev) / n_ses,
            "a_3_min_o_menos": (gap <= 3).mean(),
            "mediana_min": gap.median()}


pd.DataFrame({"bands ±0.5": racimos(bandas),
              "CUSUM k = 4": racimos(ev)}).T.round(3)

             por_sesion  a_3_min_o_menos  mediana_min
bands ±0.5       22.514            0.149         11.0
CUSUM k = 4      26.518            0.031         13.0

The bands give 22.5 events per session, almost as many as the CUSUM, but 14.9% arrive 3 minutes or less after the previous one, against 3.1% for the CUSUM. When the price oscillates around the edge of the band, the rule fires every time it crosses it: clustered, nearly identical events that overlap labels and repeat features. The CUSUM, by resetting, requires a full move of $h$ since the last event, and that keeps them apart.

The side of the event carries information

A side effect of event-based sampling is that the sequence of sides already says something about the regime:

misma = (ev.sesion == ev.sesion.shift()).to_numpy()
igual = (ev.lado == ev.lado.shift()).to_numpy()[misma]
t_ev = tend[ev.sesion, ev.minuto][misma]
print(f"same side as the previous event: {igual.mean():.3f}")
print(f"  in trend: {igual[t_ev].mean():.3f}")
print(f"  in reversion: {igual[~t_ev].mean():.3f}")

same side as the previous event: 0.518
  in trend: 0.624
  in reversion: 0.451

Overall, 51.8% of the events repeat the side of the previous one in the same session. In trend, 62.4%; in reversion, 45.1%. It is the signature visible in the header figure, and it is information known at the time of each event: a candidate feature for the meta-model (exercise 2 of lesson 5.3.3, where it turns out to be redundant with the flow imbalance).

The same filter on the trade tape

Nothing in the filter requires 1-minute bars: it works with any series of returns. On NOVA's regular trades of June 2, 2025 (level 1 synthetic data, lesson 1.1.1), trade by trade and with a fixed threshold of 10 bp:

op = pd.read_parquet("nova_operaciones.parquet")
op = op[op["condicion"] == "regular"].reset_index(drop=True)
r_op = np.diff(np.log(op["precio"].to_numpy()))[None, :]
ev_op = cusum(r_op, 0.001)              # 10 bp, fixed threshold
hora_ny = op["ts"].iloc[ev_op.minuto].dt.tz_convert(
    "America/New_York")
print(len(op), "trades,", len(ev_op), "events")
franja = hora_ny.dt.floor("30min").dt.strftime("%H:%M")
franja.value_counts().sort_index().to_frame("eventos").T

50938 trades, 165 events
ts       09:30  10:00  10:30  11:00  11:30  12:00  12:30  13:00  13:30  14:00  14:30  15:00  15:30
eventos     23     18     10     10      6     13      6      8     16      9      9     13     24

From 50,938 trades come 165 events, with 23 in the first half hour, 6 in the 11:30 one and 24 in the last. It is the activity U that the volume and dollar bars drew in lesson 1.4.5, reached by another route: there the bar closed on accumulating volume; here, on accumulating movement. With trade data there is a nuance: the bounce between bid and ask adds and subtracts returns without the price really moving. The CUSUM tolerates it better than a crossing rule because the bounce cancels out inside $S^+$ and $S^-$, but with thresholds on the order of the bid-ask spread it would fire on microstructure noise.

Common mistakes

• Threshold computed with the future. A volatility of the full session, or a centered EWMA, makes the filter fire “knowing” how much the day is going to move.
• Choosing the threshold by the hit rate on the test sample. It is one more parameter of the search (2.5.4).
• Entering at the price that fires the event. The event is known at the close of the minute; in live trading, the order is executed afterward (on FNX, the open of the next minute coincides with that close except for rounding to the tick, but in an asset with a bid-ask spread or with longer bars it does not).
• Forgetting the daily reset. If the sums carry over from one session to the next, the overnight gap fires events at the open that cannot be traded at the closing price.

Exercises

1. Compare the primary model's hit rate on the events of the first half hour with that of the rest of the session. How does it relate to the EWMA lag?

a = acierto(ev)
apertura = (ev.minuto < 30).to_numpy()
t_a = tend[ev.sesion, ev.minuto]
for nombre, sel in (("opening", apertura), ("rest", ~apertura)):
    print(f"{nombre:>8}: {sel.sum()} events, hit rate"
          f" {a[sel].mean():.3f}, in trend {t_a[sel].mean():.3f}")

 opening: 743 events, hit rate 0.526, in trend 0.396
    rest: 5807 events, hit rate 0.505, in trend 0.387

2. Repeat the CUSUM on the NOVA tape with thresholds of 5 and 20 bp. How many events come out and what happens to the U shape?

for h in (0.0005, 0.002):
    e_h = cusum(r_op, h)
    hny = op["ts"].iloc[e_h.minuto].dt.tz_convert("America/New_York")
    franja = hny.dt.floor("30min").dt.strftime("%H:%M")
    n = franja.value_counts().sort_index()
    print(f"{1e4 * h:.0f} bp: {len(e_h)} events;"
          f" 9:30 {n.get('09:30', 0)}, 11:30 {n.get('11:30', 0)},"
          f" 15:30 {n.get('15:30', 0)}")

5 bp: 417 events; 9:30 59, 11:30 19, 15:30 57
20 bp: 70 events; 9:30 9, 11:30 2, 15:30 10

3. Lesson 4.2.5 used a CUSUM to switch off a strategy when its returns deteriorated. What changes between that use and the one in this lesson, and why is a mean subtracted in that one and not in this one?