Probability and distributions, correlation versus causation and simple regression.

Probabilities from a distribution

A probability distribution describes which values a variable can take and how often. In trading the variable is usually a day's return, and the question, how likely an extreme value is: a −3% day, a −10% week. The answer depends on the distribution you assume, and that is where the risk lies.

The most widely used is the normal, the Gaussian bell curve, defined by its mean $\mu$ and its standard deviation $\sigma$. About 68% of values fall within one $\sigma$ of the mean and about 95% within two. Its cumulative distribution function, $F(x) = P(X \le x)$, gives the probability of landing below a value. In scipy.stats, every distribution has that function (cdf) and the one for the opposite tail (sf, $1 - F$).

With SINT's mean and standard deviation (lesson 0.7.1), the probability of a day below $\mu - 3\sigma$:

import numpy as np
import pandas as pd
from scipy import stats

precios = pd.read_csv("sint_diario.csv", parse_dates=["fecha"],
                      index_col="fecha")
r = precios["cierre"].pct_change().dropna()
mu, sigma = r.mean(), r.std()
normal = stats.norm(loc=mu, scale=sigma)
umbral = mu - 3 * sigma
print(f"μ = {mu:.4%} · σ = {sigma:.4%} · μ − 3σ = {umbral:.2%}")
print(f"P(R < μ − 3σ) under the normal: {normal.cdf(umbral):.3%}")

μ = 0.0342% · σ = 1.1400% · μ − 3σ = -3.39%
P(R < μ − 3σ) under the normal: 0.135%

0.135%: one day in 741, about three years of sessions. If the normal were a good description, a fall of more than 3.4% in SINT would be a rare event. The data say otherwise.

What the normal misses: the tails

Counting how many days fell below each threshold and comparing it with what the normal expects is the most direct test:

n = len(r)
print(" k   cutoff   normal   observed   norm. days   obs. days")
for k in (1, 2, 3, 4):
    u = mu - k * sigma
    p_norm, p_obs = normal.cdf(u), (r < u).mean()
    print(f"{k:>2} {u:8.2%} {p_norm:8.3%} {p_obs:10.3%} "
          f"{p_norm * n:11.1f} {int((r < u).sum()):10d}")

 k   cutoff   normal   observed   norm. days   obs. days
 1   -1.11%  15.866%    14.439%       206.6        188
 2   -2.25%   2.275%     1.997%        29.6         26
 3   -3.39%   0.135%     0.538%         1.8          7
 4   -4.53%   0.003%     0.077%         0.0          1

At one and two standard deviations, the normal and the data almost match (there are even slightly fewer −2σ days than expected). From three onward, the normal falls short: 7 days of −3σ against 1.8 expected, and one −4σ day (the worst in the series) when the normal expects 0.04, that is, one every 125 years of sessions. It is the excess kurtosis of lesson 0.7.1 seen as probability, and the reason a risk model based on the normal underestimates large losses.

Student's t is an alternative with fatter tails. Besides location and scale it has one parameter, the degrees of freedom ($\nu$): with few (3, 4, 5) the tails are very fat; with many, the t approaches the normal. stats.t.fit estimates the three parameters by maximum likelihood:

nu, loc, escala = stats.t.fit(r)
t_ajustada = stats.t(nu, loc=loc, scale=escala)
print(f"Degrees of freedom: {nu:.1f}")
for k in (3, 4):
    u = mu - k * sigma
    print(f"−{k}σ  normal {normal.cdf(u):.3%}  "
          f"t {t_ajustada.cdf(u):.3%}  observed {(r < u).mean():.3%}")

Degrees of freedom: 8.8
−3σ  normal 0.135%  t 0.397%  observed 0.538%
−4σ  normal 0.003%  t 0.073%  observed 0.077%

With about nine degrees of freedom, the t gives a −3σ day a probability of 0.40%, three times the normal's and much closer to the observed one (0.54%); and a −4σ day, 0.073%, almost exactly the observed frequency. It is not perfect (with seven days in the tail, no estimate is), but it describes the risk much better than the normal.

Correlation: how they move together

Pearson correlation measures whether two variables move together along a straight line. It ranges from −1 (always in opposite directions) to +1 (always in the same one), and 0 means there is no linear relationship:

In a portfolio it is the figure that decides how much each asset diversifies (lesson 0.7.3). With the daily returns of the four fictional assets in cartera_sint.csv:

cartera = pd.read_csv("cartera_sint.csv", parse_dates=["fecha"],
                      index_col="fecha")
rc = cartera.pct_change().dropna()
rc.corr().round(2)

      RVAR  RFIJ   ORO  TECH
RVAR  1.00 -0.19  0.12  0.85
RFIJ -0.19  1.00  0.19 -0.14
ORO   0.12  0.19  1.00  0.02
TECH  0.85 -0.14  0.02  1.00

The diagonal is 1 (each asset with itself) and the matrix is symmetric. TECH and RVAR, at 0.85, move almost together: TECH is a high-beta asset relative to equities (RVAR). RFIJ, the bond asset, has a negative correlation with both, which is what you expect from fixed income in a portfolio. ORO (gold) barely relates to anything.

The same calculation done with prices instead of returns gives a different matrix, and it is a common mistake:

precios_corr = cartera.corr().loc["ORO", "RVAR"]
rent_corr = rc.corr().loc["ORO", "RVAR"]
print(f"ORO and RVAR · prices: {precios_corr:+.2f} · "
      f"returns: {rent_corr:+.2f}")

ORO and RVAR · prices: -0.40 · returns: +0.12

With prices, ORO and RVAR seem to move in opposite directions (−0.40); with returns, in the same one, if only slightly (+0.12). The second figure is the one that matters: it says whether the day one rises the other tends to rise too. The first only says that over these five years one ended up rising and the other falling, something that can happen without any relationship.

Spurious correlations

How misleading can a price correlation be? The cleanest test is to build series that, by construction, have nothing to do with each other: pairs of independent random walks, the simplest model of a price (each day, a random step from the previous close). A thousand pairs of five years of sessions, with a seed:

rng = np.random.default_rng(2026)
pasos = rng.normal(0, 0.01, size=(1000, 1250, 2))
niveles = 100 * np.exp(pasos.cumsum(axis=1))


def corr(x, y):
    return np.corrcoef(x, y)[0, 1]


c_niv = np.array([corr(p[:, 0], p[:, 1]) for p in niveles])
c_ren = np.array([corr(d[:, 0], d[:, 1]) for d in pasos])
print(f"|ρ| > 0.5 with prices:        {np.mean(abs(c_niv) > 0.5):.1%}")
print(f"|ρ| > 0.5 with returns:       {np.mean(abs(c_ren) > 0.5):.1%}")
print(f"Largest |ρ| with returns:     {abs(c_ren).max():.3f}")

|ρ| > 0.5 with prices:        37.6%
|ρ| > 0.5 with returns:       0.0%
Largest |ρ| with returns:     0.104

More than a third of the pairs of independent prices have a correlation above 0.5 in absolute value. With returns, none: the largest does not reach 0.11. Two trending series always look related, because both drift away from their starting point and correlation mistakes that for moving together. It is the financial version of the absurd correlations collected by Tyler Vigen (link in “Resources”): with enough series, there is always some pair that fits.

Even with returns, a high correlation does not say why two assets move together:

flowchart LR
  F[Common factor] --> A[RVAR]
  F --> B[TECH]
  A -.->|correlation| B

TECH and RVAR move together because both are driven by the same causes (interest rates, growth, risk appetite), not because one causes the other. For trading, the difference matters: a relationship without a mechanism can vanish as soon as the environment changes, and a pairs strategy built on it stops working without warning.

Simple regression: an asset's beta

Simple linear regression fits a straight line to a cloud of points. With TECH's returns ($y$) against RVAR's ($x$), the model is:

The slope $\beta$ (the beta) is the sensitivity: how much TECH moves on average for each 1% of RVAR. $\alpha$ is what remains when RVAR does not move, and $\varepsilon_t$, the residual, what the line does not explain. Ordinary least squares (OLS) chooses $\alpha$ and $\beta$ so that the sum of squared residuals is as small as possible. statsmodels does it and also provides the fit statistics:

import statsmodels.api as sm

X = sm.add_constant(rc["RVAR"])
modelo = sm.OLS(rc["TECH"], X).fit()
beta = modelo.params["RVAR"]
print(f"Beta: {beta:.2f}")
print(f"Daily alpha: {modelo.params['const']:.4%}")
print(f"R²: {modelo.rsquared:.2f}")
beta_formula = rc["TECH"].cov(rc["RVAR"]) / rc["RVAR"].var()
print(f"cov / var: {beta_formula:.2f}")

Beta: 1.43
Daily alpha: -0.0128%
R²: 0.72
cov / var: 1.43

sm.add_constant adds the column of ones that estimates $\alpha$; without it, the line would pass through the origin. The beta is 1.43: on a day when RVAR rises 1%, TECH rises 1.43% on average, and the same on the way down. The last line shows that, in a simple regression, the beta is the covariance divided by the variance of $x$. The daily alpha is practically zero, and with five years of data it cannot be distinguished from zero.

The R² (0.72) is the fraction of TECH's variance that the line explains; in a simple regression it is the squared correlation ($0.85^2 \approx 0.72$). The figure at the top shows the cloud of points with the line: stretched along the diagonal, but with quite a lot of scatter around it.

modelo.summary() prints a full table (standard errors, t statistics, confidence intervals). Reading it properly requires the inference from level 2; here the slope and the R² are enough.

The beta has a direct use: hedging market risk. A long position of €10,000 in TECH moves with RVAR as if it were €14,300 of RVAR. Selling that amount of RVAR leaves only the part of TECH that does not depend on the market:

cubierta = rc["TECH"] - beta * rc["RVAR"]
def vol(s):
    return s.std() * np.sqrt(252)


print(f"TECH volatility:              {vol(rc['TECH']):.1%}")
print(f"Hedged TECH volatility:       {vol(cubierta):.1%}")
print(f"Hedged-RVAR correlation:      "
      f"{cubierta.corr(rc['RVAR']):+.2f}")

TECH volatility:              28.3%
Hedged TECH volatility:       14.9%
Hedged-RVAR correlation:      -0.00

The hedge cuts the position's volatility almost in half and leaves zero correlation with RVAR: what remains is TECH's own risk, the 28% of the variance the market does not explain. It is the basis of market-neutral strategies, and it carries the same fine print as correlation: the beta is estimated from the past and changes over time.

Exercises

1. Using the normal fitted to SINT, compute the probability of a day of −5% or worse and how many years of 252 sessions you would have to wait, on average, to see one. Repeat with the fitted t. Which is more prudent for sizing a position?

for nombre, dist in (("normal", normal), ("t", t_ajustada)):
    p = dist.cdf(-0.05)
    print(f"{nombre:<7} P = {p:.5%} · one every "
          f"{1 / p / 252:,.0f} years")

normal  P = 0.00050% · one every 790 years
t       P = 0.03827% · one every 10 years

2. Compute the 60-session rolling correlation between TECH and RVAR with rolling(60).corr(). Between which values does it move? What does that say about a correlation estimated from three months of data?

movil = rc["TECH"].rolling(60).corr(rc["RVAR"]).dropna()
print(f"Min {movil.min():.2f} · max {movil.max():.2f} · "
      f"mean {movil.mean():.2f}")

Min 0.75 · max 0.92 · mean 0.85

3. Estimate the beta of ORO against RVAR and its R². Would RVAR work to hedge a gold position?

m_oro = sm.OLS(rc["ORO"], sm.add_constant(rc["RVAR"])).fit()
print(f"Beta: {m_oro.params['RVAR']:.2f} · R²: {m_oro.rsquared:.3f}")

Beta: 0.11 · R²: 0.015