--- title: "0.7.2 Probability, distributions and simple regression" description: "Normal and Student's t for the tails of returns, correlation versus causation, spurious correlations and the beta of a simple regression." url: "https://pythonparatrading.com/en/python-trading-course/level-0/probability-distributions-and-regression/" lang: "en" image: "https://pythonparatrading.com/og/en/python-trading-course/level-0/probability-distributions-and-regression.png" --- # 0.7.2 · Probability and distributions, correlation versus causation and simple regression > How often should you expect a −3σ day? Do two assets move together, or does it only look that way? How much of TECH is explained by the market? Three trading questions answered with the bare minimum of probability, correlation and regression, and with the typical mistakes of each. - Course: [Level 0: Python from scratch and a data mindset](https://pythonparatrading.com/en/python-trading-course/level-0/) - Module: 0.7 · Minimum mathematics and statistics - Duration: 55 min - Prerequisites: [0.7.1 · Simple and log returns, compound interest and descriptive statistics](https://pythonparatrading.com/en/python-trading-course/level-0/returns-and-descriptive-statistics/) ## By the end of this lesson - Use `scipy.stats` to compute the probability of an extreme day under a normal distribution and compare it with the observed frequency. - Fit a Student's t to daily returns and explain why it describes the tails better than the normal. - Compute a correlation matrix from returns, not prices, and recognize a spurious correlation. - Estimate an asset's beta against the market with `statsmodels` and interpret the slope and the R². ## Probabilities from a distribution A **probability distribution** describes which values a variable can take and how often. In trading the variable is usually a day's return, and the question, how likely an extreme value is: a −3% day, a −10% week. The answer depends on the distribution you assume, and that is where the risk lies. The most widely used is the **normal**, the Gaussian bell curve, defined by its mean $\mu$ and its standard deviation $\sigma$. About 68% of values fall within one $\sigma$ of the mean and about 95% within two. Its cumulative distribution function, $F(x) = P(X \le x)$, gives the probability of landing below a value. In `scipy.stats`, every distribution has that function (`cdf`) and the one for the opposite tail (`sf`, $1 - F$). With SINT's mean and standard deviation (lesson 0.7.1), the probability of a day below $\mu - 3\sigma$: ```python import numpy as np import pandas as pd from scipy import stats precios = pd.read_csv("sint_diario.csv", parse_dates=["fecha"], index_col="fecha") r = precios["cierre"].pct_change().dropna() mu, sigma = r.mean(), r.std() normal = stats.norm(loc=mu, scale=sigma) umbral = mu - 3 * sigma print(f"μ = {mu:.4%} · σ = {sigma:.4%} · μ − 3σ = {umbral:.2%}") print(f"P(R < μ − 3σ) under the normal: {normal.cdf(umbral):.3%}") ``` ```output μ = 0.0342% · σ = 1.1400% · μ − 3σ = -3.39% P(R < μ − 3σ) under the normal: 0.135% ``` 0.135%: one day in 741, about three years of sessions. If the normal were a good description, a fall of more than 3.4% in SINT would be a rare event. The data say otherwise. ## What the normal misses: the tails Counting how many days fell below each threshold and comparing it with what the normal expects is the most direct test: ```python n = len(r) print(" k cutoff normal observed norm. days obs. days") for k in (1, 2, 3, 4): u = mu - k * sigma p_norm, p_obs = normal.cdf(u), (r < u).mean() print(f"{k:>2} {u:8.2%} {p_norm:8.3%} {p_obs:10.3%} " f"{p_norm * n:11.1f} {int((r < u).sum()):10d}") ``` ```output k cutoff normal observed norm. days obs. days 1 -1.11% 15.866% 14.439% 206.6 188 2 -2.25% 2.275% 1.997% 29.6 26 3 -3.39% 0.135% 0.538% 1.8 7 4 -4.53% 0.003% 0.077% 0.0 1 ``` At one and two standard deviations, the normal and the data almost match (there are even slightly fewer −2σ days than expected). From three onward, the normal falls short: 7 days of −3σ against 1.8 expected, and one −4σ day (the worst in the series) when the normal expects 0.04, that is, one every 125 years of sessions. It is the excess kurtosis of lesson 0.7.1 seen as probability, and the reason a risk model based on the normal underestimates large losses. **Student's t** is an alternative with fatter tails. Besides location and scale it has one parameter, the **degrees of freedom** ($\nu$): with few (3, 4, 5) the tails are very fat; with many, the t approaches the normal. `stats.t.fit` estimates the three parameters by maximum likelihood: ```python nu, loc, escala = stats.t.fit(r) t_ajustada = stats.t(nu, loc=loc, scale=escala) print(f"Degrees of freedom: {nu:.1f}") for k in (3, 4): u = mu - k * sigma print(f"−{k}σ normal {normal.cdf(u):.3%} " f"t {t_ajustada.cdf(u):.3%} observed {(r < u).mean():.3%}") ``` ```output Degrees of freedom: 8.8 −3σ normal 0.135% t 0.397% observed 0.538% −4σ normal 0.003% t 0.073% observed 0.077% ``` With about nine degrees of freedom, the t gives a −3σ day a probability of 0.40%, three times the normal's and much closer to the observed one (0.54%); and a −4σ day, 0.073%, almost exactly the observed frequency. It is not perfect (with seven days in the tail, no estimate is), but it describes the risk much better than the normal. ## Correlation: how they move together Pearson **correlation** measures whether two variables move together along a straight line. It ranges from −1 (always in opposite directions) to +1 (always in the same one), and 0 means there is no linear relationship: $$ \rho_{XY} = \frac{\operatorname{cov}(X, Y)}{\sigma_X\,\sigma_Y} $$ In a portfolio it is the figure that decides how much each asset diversifies (lesson 0.7.3). With the daily returns of the four fictional assets in `cartera_sint.csv`: ```python cartera = pd.read_csv("cartera_sint.csv", parse_dates=["fecha"], index_col="fecha") rc = cartera.pct_change().dropna() rc.corr().round(2) ``` ```output RVAR RFIJ ORO TECH RVAR 1.00 -0.19 0.12 0.85 RFIJ -0.19 1.00 0.19 -0.14 ORO 0.12 0.19 1.00 0.02 TECH 0.85 -0.14 0.02 1.00 ``` The diagonal is 1 (each asset with itself) and the matrix is symmetric. TECH and RVAR, at 0.85, move almost together: TECH is a high-beta asset relative to equities (RVAR). RFIJ, the bond asset, has a negative correlation with both, which is what you expect from fixed income in a portfolio. ORO (gold) barely relates to anything. The same calculation done with **prices** instead of returns gives a different matrix, and it is a common mistake: ```python precios_corr = cartera.corr().loc["ORO", "RVAR"] rent_corr = rc.corr().loc["ORO", "RVAR"] print(f"ORO and RVAR · prices: {precios_corr:+.2f} · " f"returns: {rent_corr:+.2f}") ``` ```output ORO and RVAR · prices: -0.40 · returns: +0.12 ``` With prices, ORO and RVAR seem to move in opposite directions (−0.40); with returns, in the same one, if only slightly (+0.12). The second figure is the one that matters: it says whether the day one rises the other tends to rise too. The first only says that over these five years one ended up rising and the other falling, something that can happen without any relationship. ## Spurious correlations How misleading can a price correlation be? The cleanest test is to build series that, by construction, have nothing to do with each other: pairs of independent **random walks**, the simplest model of a price (each day, a random step from the previous close). A thousand pairs of five years of sessions, with a seed: ```python rng = np.random.default_rng(2026) pasos = rng.normal(0, 0.01, size=(1000, 1250, 2)) niveles = 100 * np.exp(pasos.cumsum(axis=1)) def corr(x, y): return np.corrcoef(x, y)[0, 1] c_niv = np.array([corr(p[:, 0], p[:, 1]) for p in niveles]) c_ren = np.array([corr(d[:, 0], d[:, 1]) for d in pasos]) print(f"|ρ| > 0.5 with prices: {np.mean(abs(c_niv) > 0.5):.1%}") print(f"|ρ| > 0.5 with returns: {np.mean(abs(c_ren) > 0.5):.1%}") print(f"Largest |ρ| with returns: {abs(c_ren).max():.3f}") ``` ```output |ρ| > 0.5 with prices: 37.6% |ρ| > 0.5 with returns: 0.0% Largest |ρ| with returns: 0.104 ``` More than a third of the pairs of independent prices have a correlation above 0.5 in absolute value. With returns, none: the largest does not reach 0.11. Two trending series always look related, because both drift away from their starting point and correlation mistakes that for moving together. It is the financial version of the absurd correlations collected by Tyler Vigen (link in “Resources”): with enough series, there is always some pair that fits. Even with returns, a high correlation does not say **why** two assets move together: ```mermaid flowchart LR F[Common factor] --> A[RVAR] F --> B[TECH] A -.->|correlation| B ``` TECH and RVAR move together because both are driven by the same causes (interest rates, growth, risk appetite), not because one causes the other. For trading, the difference matters: a relationship without a mechanism can vanish as soon as the environment changes, and a pairs strategy built on it stops working without warning. ## Simple regression: an asset's beta **Simple linear regression** fits a straight line to a cloud of points. With TECH's returns ($y$) against RVAR's ($x$), the model is: $$ y_t = \alpha + \beta\,x_t + \varepsilon_t $$ The slope $\beta$ (the **beta**) is the sensitivity: how much TECH moves on average for each 1% of RVAR. $\alpha$ is what remains when RVAR does not move, and $\varepsilon_t$, the residual, what the line does not explain. Ordinary least squares (OLS) chooses $\alpha$ and $\beta$ so that the sum of squared residuals is as small as possible. `statsmodels` does it and also provides the fit statistics: ```python import statsmodels.api as sm X = sm.add_constant(rc["RVAR"]) modelo = sm.OLS(rc["TECH"], X).fit() beta = modelo.params["RVAR"] print(f"Beta: {beta:.2f}") print(f"Daily alpha: {modelo.params['const']:.4%}") print(f"R²: {modelo.rsquared:.2f}") beta_formula = rc["TECH"].cov(rc["RVAR"]) / rc["RVAR"].var() print(f"cov / var: {beta_formula:.2f}") ``` ```output Beta: 1.43 Daily alpha: -0.0128% R²: 0.72 cov / var: 1.43 ``` `sm.add_constant` adds the column of ones that estimates $\alpha$; without it, the line would pass through the origin. The beta is 1.43: on a day when RVAR rises 1%, TECH rises 1.43% on average, and the same on the way down. The last line shows that, in a simple regression, the beta is the covariance divided by the variance of $x$. The daily alpha is practically zero, and with five years of data it cannot be distinguished from zero. The **R²** (0.72) is the fraction of TECH's variance that the line explains; in a simple regression it is the squared correlation ($0.85^2 \approx 0.72$). The figure at the top shows the cloud of points with the line: stretched along the diagonal, but with quite a lot of scatter around it. `modelo.summary()` prints a full table (standard errors, t statistics, confidence intervals). Reading it properly requires the inference from level 2; here the slope and the R² are enough. The beta has a direct use: **hedging market risk**. A long position of €10,000 in TECH moves with RVAR as if it were €14,300 of RVAR. Selling that amount of RVAR leaves only the part of TECH that does not depend on the market: ```python cubierta = rc["TECH"] - beta * rc["RVAR"] def vol(s): return s.std() * np.sqrt(252) print(f"TECH volatility: {vol(rc['TECH']):.1%}") print(f"Hedged TECH volatility: {vol(cubierta):.1%}") print(f"Hedged-RVAR correlation: " f"{cubierta.corr(rc['RVAR']):+.2f}") ``` ```output TECH volatility: 28.3% Hedged TECH volatility: 14.9% Hedged-RVAR correlation: -0.00 ``` The hedge cuts the position's volatility almost in half and leaves zero correlation with RVAR: what remains is TECH's own risk, the 28% of the variance the market does not explain. It is the basis of market-neutral strategies, and it carries the same fine print as correlation: the beta is estimated from the past and changes over time. ## Exercises 1. Using the normal fitted to SINT, compute the probability of a day of −5% or worse and how many years of 252 sessions you would have to wait, on average, to see one. Repeat with the fitted t. Which is more prudent for sizing a position? **Solution.** ```python for nombre, dist in (("normal", normal), ("t", t_ajustada)): p = dist.cdf(-0.05) print(f"{nombre:<7} P = {p:.5%} · one every " f"{1 / p / 252:,.0f} years") ``` ```output normal P = 0.00050% · one every 790 years t P = 0.03827% · one every 10 years ``` The normal makes a −5% day in SINT almost impossible: one every 790 years. The t puts it at one every ten years, a time scale a trader can actually live through. In the five years of data there was none (the worst was −4.81%), but it came close. For sizing a position, the prudent distribution is the one that gives more probability to large losses: the t. 2. Compute the 60-session rolling correlation between TECH and RVAR with `rolling(60).corr()`. Between which values does it move? What does that say about a correlation estimated from three months of data? **Solution.** ```python movil = rc["TECH"].rolling(60).corr(rc["RVAR"]).dropna() print(f"Min {movil.min():.2f} · max {movil.max():.2f} · " f"mean {movil.mean():.2f}") ``` ```output Min 0.75 · max 0.92 · mean 0.85 ``` The data were generated with a constant correlation, and even so the three-month estimate ranges from 0.75 to 0.92 around its mean value, 0.85. That oscillation is pure estimation error: with 60 data points, a correlation is known with little precision. With real data, on top of that noise, the true correlation also changes. 3. Estimate the beta of ORO against RVAR and its R². Would RVAR work to hedge a gold position? **Solution.** ```python m_oro = sm.OLS(rc["ORO"], sm.add_constant(rc["RVAR"])).fit() print(f"Beta: {m_oro.params['RVAR']:.2f} · R²: {m_oro.rsquared:.3f}") ``` ```output Beta: 0.11 · R²: 0.015 ``` A small beta and an R² close to zero: RVAR explains 1.5% of gold's variance. Selling RVAR against a gold position would not reduce its risk, it would only add another one. A beta is only useful for hedging when the R² is high. ## Self-assessment 1. With normal returns, a day below the mean minus three standard deviations has a probability of 0.135%. SINT shows 0.54%. What does that indicate? - A. The tails are fatter than the normal's: extreme days are about four times as frequent as it predicts - B. The data contain an error - C. The volatility is miscalculated - D. Nothing: with 1,302 days, any difference is noise *Answer A: The tails are fatter than the normal's: extreme days are about four times as frequent as it predicts.* Seven observed days against fewer than two expected is a large difference for that sample size. It is the excess kurtosis of lesson 0.7.1 seen as probability: the normal underestimates bad days. 2. Which parameter of Student's t controls how fat the tails are? - A. The mean - B. The scale - C. The degrees of freedom: the fewer, the fatter; with many, the t resembles the normal - D. The sample size *Answer C: The degrees of freedom: the fewer, the fatter; with many, the t resembles the normal.* With few degrees of freedom (3, 4, 5) the t gives much more probability to the extremes; from about 30 upward it is almost a normal. The fit to SINT gives about 9. 3. The prices of two assets have a correlation of −0.40 over five years. Their daily returns, 0.12. Which figure describes how they move together? - A. The price one, because it uses the actual levels - B. The average of the two - C. Neither: correlation does not work for prices - D. The return one: the price correlation mixes the trends of each series and can be large without any relationship *Answer D: The return one: the price correlation mixes the trends of each series and can be large without any relationship.* Two trending series (such as two independent random walks) often have large correlations in levels. The correlation of returns measures whether each day's moves go in the same direction. 4. Regressing TECH's returns on RVAR's gives a slope of 1.43 and an R² of 0.72. Which statement is correct? - A. RVAR causes TECH's moves - B. On a day when RVAR rises 1%, TECH rises 1.43% on average, and 72% of TECH's variance moves with RVAR - C. TECH earns 43% more than RVAR every year - D. TECH rises on 72% of days *Answer B: On a day when RVAR rises 1%, TECH rises 1.43% on average, and 72% of TECH's variance moves with RVAR.* The slope (beta) is the average sensitivity; the R², the share of the variance the line explains. Neither says anything about causes or about long-term returns. ## Key points - `scipy.stats` gives each distribution its cumulative distribution function (`cdf`) and its tail (`sf`): with them you compute the probability of an extreme day. - Returns have fatter tails than the normal. A −3σ day is considerably more frequent than the normal predicts; Student's t with few degrees of freedom describes it better. - Correlations are computed from returns. With prices, two trending series look related even when they are independent. - Correlation is not causation: two assets can move together because a third factor drives them, and the relationship can change or vanish. - The beta of a simple regression measures an asset's sensitivity to the market; the R², how much of its variance it explains. It is used to hedge the market risk of a position. ## Resources ### Research - [Stochastic Processes - Random Walks](https://pythonparatrading.com/en/stochastic-processes/) · Research-0005 · 11 min ### Documentation - [SciPy: scipy.stats.t (Student's t distribution)](https://docs.scipy.org/doc/scipy/reference/generated/scipy.stats.t.html) · docs.scipy.org - [statsmodels: linear regression (OLS)](https://www.statsmodels.org/stable/regression.html) · statsmodels.org - [Tyler Vigen, Spurious Correlations](https://www.tylervigen.com/spurious-correlations) · tylervigen.com ### Data - [Level 0 data: closes of the fictional portfolio (CSV)](https://pythonparatrading.com/curso/datos/nivel-0/cartera_sint.csv) · Download · cartera_sint.csv Previous lesson: [0.7.1 · Simple and log returns, compound interest and descriptive statistics](https://pythonparatrading.com/en/python-trading-course/level-0/returns-and-descriptive-statistics/) Next lesson: [0.7.3 · Intuitive linear algebra: vectors and matrices as groundwork for portfolios](https://pythonparatrading.com/en/python-trading-course/level-0/linear-algebra-for-portfolios/)