--- title: "4.5.1 Live trading system architecture" description: "Data, signal, risk, OMS, broker and reconciliation: an event-driven live system with a logical clock and a simulated broker, actually run." url: "https://pythonparatrading.com/en/python-trading-course/level-4/live-trading-system-architecture/" lang: "en" image: "https://pythonparatrading.com/og/en/python-trading-course/level-4/live-trading-system-architecture.png" --- # 4.5.1 · Live system architecture: from data to reconciliation > A live system is not a backtest with a broker plugged in: it has state that outlives the day, messages that arrive late or never arrive, and money someone has to reconcile. This lesson builds, in cells that run, the six pieces of the chain with an event bus and a logical clock, checks the signal against the log of a real system and triggers a transport failure to see which piece discovers it. - Course: [Level 4: Portfolios, risk, execution and live trading](https://pythonparatrading.com/en/python-trading-course/level-4/) - Module: 4.5 · Live trading system - Duration: 70 min - Prerequisites: [2.4.2 · Event-driven backtest from scratch: loop, portfolio, orders and fills](https://pythonparatrading.com/en/python-trading-course/level-2/event-driven-backtest-from-scratch/), [2.4.5 · NautilusTrader: architecture, data, strategies and the same code in backtest and live](https://pythonparatrading.com/en/python-trading-course/level-2/nautilustrader-architecture-and-strategies/), [4.2.5 · Drawdown control and kill switch](https://pythonparatrading.com/en/python-trading-course/level-4/drawdown-control-and-kill-switch/) ## By the end of this lesson - Place the six pieces of a live system (data, signal, risk, OMS, broker, reconciliation) and what goes in and out of each one. - Code an event bus with a logical clock, with no `sleep` and no real clock, that makes the system reproducible event by event. - Apply pre-trade controls (maximum value, number of orders, kill switch) and calibrate them with the order history. - Reconcile the internal portfolio with the broker's every day and explain why a lost message turns into wrong orders if nobody detects it. ## From backtest to live system An event-driven backtest (lesson 2.4.2) already walks through time as the live system would: a bar arrives, the strategy decides, the simulated broker executes. What it lacks is what makes trading for real hard: - **State that survives.** The backtest starts from zero on every run. The live system starts every morning with yesterday's positions, open orders and cash, and it has to know what they are. - **Messages, not calls.** In the backtest the broker returns the fill in the same function. In live trading, the order goes out over the network, the broker answers when it answers, sometimes in a different order, sometimes twice, sometimes never (lesson 4.5.2). - **Two truths.** The system keeps its own books; the broker keeps its own, which are the ones that count. If they do not match, you have to know before the next order. - **Controls.** A programming error in a backtest gives a bad chart. In live trading it gives an order. This lesson builds the complete chain in cells that run: the same system as in the level data (10 fictitious ETFs, weights by momentum and volatility), with a simulated broker inside the process. The following lessons go deeper into each edge: orders and their states (4.5.2), paper trading and slippage (4.5.3), task scheduling and alerts (4.5.4), fault tolerance and reconciliation at startup (4.5.5) and the engines that do all this for you (4.5.6). ## Six pieces and a bus ```mermaid flowchart TB D[Data] --> S[Signal] S --> R[Risk] R --> O[OMS] O --> B[Broker] B --> C[Portfolio] B --> O C --> Q[Reconciliation] B --> Q Q --> M[Monitor] ``` | Piece | Receives | Publishes | What can go wrong | |---|---|---|---| | Data | Prices from the provider | Prices with their time | Late data, gaps, split adjustments | | Signal | Prices up to now | Target weights | Using information that does not exist yet | | Risk | Weights and proposed orders | Approved weights, blocks | Badly calibrated limits: blocking the good or letting the bad through | | OMS | Approved weights, broker reports | Orders with their own identifier | Duplicating or losing orders, badly tracked states | | Broker | Orders | Acknowledgments, fills, rejections | Rejections, partial fills, disconnections | | Portfolio | Fills | Internal positions and cash | Counting a fill twice or not counting it | | Reconciliation | Internal portfolio and the broker's | Mismatches | Arriving late: after the next order | The rule that orders everything: **each piece only knows the messages that have reached it**. The signal does not ask the broker how many shares there are; the OMS does not compute momentum; the portfolio decides nothing. That way each piece is tested on its own and can be replaced without touching the others. The header figure shows that chain on a real order from the data, from the OMS to the broker and back, in milliseconds. ## A clock that is not the computer's The heart of the system is an **event bus**: a priority queue ordered by time, to which each piece publishes messages and subscribes by type. The time is not given by `datetime.now()` but by the event itself: the bus advances its clock as it pops each message. With historical data the clock jumps from event to event without waiting; in live trading, the data adapter publishes with the market time and the bus moves at the pace of arrivals. The rest of the code does not distinguish the two cases, and that is what makes it possible to replay a production day in the lab. ```python import hashlib import heapq import itertools from collections import Counter, defaultdict from dataclasses import dataclass import exchange_calendars as xcals import numpy as np import pandas as pd @dataclass(frozen=True) class Evento: t: pd.Timestamp # logical time (UTC) tipo: str datos: dict class Bus: """Event queue by time; the clock is that of the last event.""" def __init__(self): self.cola, self.subs = [], defaultdict(list) self.n = itertools.count() # tie-break: publication order self.ahora = None self.registro = [] def suscribir(self, tipo, fn): self.subs[tipo].append(fn) def publicar(self, t, tipo, **datos): if self.ahora is not None and t < self.ahora: raise ValueError(f"{tipo} with a past time: {t}") heapq.heappush(self.cola, (t, next(self.n), Evento(t, tipo, datos))) def correr(self): while self.cola: t, _, ev = heapq.heappop(self.cola) self.ahora = t self.registro.append(ev) for fn in self.subs[ev.tipo]: fn(ev) bus = Bus() bus.suscribir("eco", lambda ev: print(ev.t.time(), ev.datos)) t0 = pd.Timestamp("2025-09-02 19:45", tz="UTC") bus.publicar(t0 + pd.Timedelta(seconds=2), "eco", n=2) bus.publicar(t0, "eco", n=1) bus.correr() ``` ```output 19:45:00 {'n': 1} 19:45:02 {'n': 2} ``` Events come out by their time, not by the order in which they were published, and an event with a time earlier than the clock is an error: nothing can happen in the past. The counter `n` breaks ties (two events at the same time come out in the order they were published), so the result depends on nothing external. ## Data and signal: what is known at each time The data are those of the level's fictitious system: 10 ETFs that track indices from the universe of lesson 4.1.1 (`universo4_precios.parquet`). The price of each ETF is its index times a scale; the scale comes from the broker statement (`vivo4_extracto_posiciones.csv`), which gives the closing price of each ETF. We take the dates before ORAU's 3-for-1 split on November 17 (what a split does to the system is exercise 3): ```python MAPA = {"ORUS": "RV_EEUU", "ORSC": "RV_EEUU_P", "OREU": "RV_EUR", "OREM": "RV_EMG", "ORTN": "RF_T710", "ORTL": "RF_T20", "ORIG": "RF_CIG", "ORAU": "MP_ORO", "ORCM": "MP_AMP", "ORRE": "INM_EEUU"} ETF = list(MAPA) u = pd.read_parquet("universo4_precios.parquet").set_index("fecha") ext = pd.read_csv("vivo4_extracto_posiciones.csv", parse_dates=["fecha"]) prim = ext[ext["fecha"] < "2025-11-17"].groupby("instrumento").first() escala = pd.Series({t: prim.loc[t, "precio_cierre"] / u.loc[prim.loc[t, "fecha"], a] for t, a in MAPA.items()}) precios = u[list(MAPA.values())].set_axis(ETF, axis=1) * escala CAL = xcals.get_calendar("XNYS") SESIONES = CAL.sessions_in_range("2025-09-02", "2025-12-31") print(len(SESIONES), "sessions;", "close on September 2:", CAL.session_close(SESIONES[0]), "| on December 31:", CAL.session_close(SESIONES[-1])) precios.loc["2025-09-02"].round(2).to_dict() ``` ```output 85 sessions; close on September 2: 2025-09-02 20:00:00+00:00 | on December 31: 2025-12-31 21:00:00+00:00 {'ORUS': 638.29, 'ORSC': 232.33, 'OREU': 61.23, 'OREM': 49.31, 'ORTN': 96.05, 'ORTL': 88.18, 'ORIG': 109.19, 'ORAU': 311.26, 'ORCM': 24.05, 'ORRE': 93.5} ``` The NYSE close is 16:00 New York time: 20:00 UTC until October 31 and 21:00 UTC from November 3, when the US changes clocks. A system that schedules the signal “at 19:45 UTC” will launch it an hour earlier than intended for half the year; one that schedules it 15 minutes before the close of each calendar session, never. Half sessions (November 28 and December 24, closing at 13:00 New York time) also come from the calendar. The data piece publishes three events per session: `precios` (prices) 15 minutes before the close (with closes **up to the previous day**), `cierre` (close) at the closing time (with today's close, used only by the simulated broker to fill orders) and `fin_dia` (end of day) half an hour later. The signal is the rule of the level's system: on each ETF with positive 20-session momentum, a weight of `min(2.5% / vol_63, 25%)`, normalized if they add up to more than 100%. ```python class Datos: """Publishes what is known at each moment, with its time.""" def __init__(self, bus, precios): self.bus, self.p = bus, precios def programar(self, sesiones): for d in sesiones: cierre = CAL.session_close(d) hist = self.p.loc[:d].iloc[-65:-1] # up to the previous day self.bus.publicar(cierre - pd.Timedelta(minutes=15), "precios", fecha=d, hist=hist) self.bus.publicar(cierre, "cierre", fecha=d, px=self.p.loc[d]) self.bus.publicar(cierre + pd.Timedelta(minutes=30), "fin_dia", fecha=d) def pesos_momentum(h): mom = h.iloc[-1] / h.iloc[-21] - 1 vol = h.pct_change().iloc[-63:].std() * np.sqrt(252) w = np.minimum(0.025 / vol, 0.25).where(mom > 0, 0.0) return w / w.sum() if w.sum() > 1 else w class Senal: def __init__(self, bus): self.bus = bus bus.suscribir("precios", self.calcular) def calcular(self, ev): h = ev.datos["hist"] self.bus.publicar(ev.t, "objetivo", fecha=ev.datos["fecha"], pesos=pesos_momentum(h), ref=h.iloc[-1]) ``` ## Risk before each order The risk piece does two things. On the weights, it trims whatever falls outside the portfolio limits (here, 25% per ETF and 100% in total). On each order the OMS proposes, it applies **pre-trade** controls: maximum value (a percentage of equity), maximum number of orders per day and a kill switch that, when activated, stops approving targets. They are three of the controls listed in article 15 of RTS 6; price collars are not needed here because the system only sends orders at the close. The design of the kill switch and its triggers is lesson 4.2.5; here all that matters is where it sits: between the signal and the OMS, so that no order gets past it. A badly calibrated limit has a cost: if it is too strict, it blocks legitimate orders and the portfolio drifts away from the target; if it is too loose, it stops nothing. The honest way to set it is to look at the history of legitimate orders. The 398 orders from the real system's OMS (`vivo4_ordenes.csv`), with their value relative to the initial capital of 1,000,000 USD: ```python ordenes = pd.read_csv("vivo4_ordenes.csv") pct = 100 * ordenes["cantidad"] * ordenes["precio_medio_envio"] / 1e6 print(f"Largest order: {pct.max():.1f}% of capital; " f"median: {pct.median():.1f}%") for lim in (10, 20, 30): print(f"Limit {lim}%: would block {(pct > lim).sum()} " "legitimate orders") error_dedo = 10 * pct.max() print(f"Fat-finger error (x10): {error_dedo:.0f}%, blocked: " f"{error_dedo > 30}") ``` ```output Largest order: 26.1% of capital; median: 2.2% Limit 10%: would block 72 legitimate orders Limit 20%: would block 5 legitimate orders Limit 30%: would block 0 legitimate orders Fat-finger error (x10): 261%, blocked: True ``` *Fig. — Value of the 398 orders sent by the live system's OMS, as a percentage of initial capital, and how many a maximum order value limit of 10, 20 or 30% would block.* Histogram in 2-point bins: 180 orders are worth less than 2% of capital and 83 between 2 and 4%; the tail reaches 26.1%. Three vertical lines mark limits: at 10% 72 legitimate orders would be blocked, at 20% five and at 30% (highlighted) none. The largest legitimate order is worth 26.1% of capital: the sale of the whole ORTN position on December 31. With a 30% limit none is blocked; with a 20% one five would be blocked, and four of them are **sales** of entire bond positions in December: the limit would prevent getting out. A fat-finger error that multiplies a quantity by ten stays far above any of the three. The orders-per-day limit is calibrated the same way: the day with the most orders in the history has 15 (at most one per ETF and one more for each limit order that does not fill), so 25 is loose without being infinite. ```python class Riesgo: """Portfolio limits and pre-trade controls on each order.""" def __init__(self, bus, peso_max=0.25, valor_max=0.30, ordenes_dia=25): self.bus, self.peso_max = bus, peso_max self.valor_max, self.ordenes_dia = valor_max, ordenes_dia self.interruptor = False # lesson 4.2.5 self.hoy = Counter() bus.suscribir("objetivo", self.revisar) def revisar(self, ev): if self.interruptor: self.bus.publicar(ev.t, "alerta", texto="kill switch") return w = ev.datos["pesos"].clip(upper=self.peso_max) w = w / w.sum() if w.sum() > 1 else w self.bus.publicar(ev.t, "objetivo_ok", **{**ev.datos, "pesos": w}) def comprobar(self, orden, patrimonio): """None if it passes; otherwise, the reason for the block.""" if orden["cantidad"] * orden["ref"] > (self.valor_max * patrimonio): return "max value" self.hoy[orden["fecha"]] += 1 if self.hoy[orden["fecha"]] > self.ordenes_dia: return "orders per day" return None ``` ## OMS, broker and portfolio The **OMS** (*order management system*) turns the approved weights into orders: it computes equity with the internal portfolio, the target number of shares of each ETF and the difference with the position; if the difference is worth less than 0.4% of equity, it does not trade (like the level's system). Each order gets its own client identifier, goes through risk and, if it passes, is sent to the broker five minutes later, 10 before the close. The OMS tracks the state of each order with the reports the broker returns; lesson 4.5.2 turns that state into a state machine with its rules. The **simulated broker** lives in the same process but only talks through the bus, with 42 ms of latency in each direction (the one-way median of the real account in the data). It rejects buys that do not fit in the cash plus the day's sales, accepts the rest and fills them in the closing auction, in one to three fills, at the close ± half a bid-ask spread of 1 bp, with the commission of the broker in the data (0.0035 USD per share, minimum 0.35 USD, maximum 1% of the trade value). It keeps its own books: it is the accounting truth, like a real broker's statement. The **portfolio** counts each fill once (by its identifier) and is what the OMS uses to decide. The **reconciler** compares, at the end of each day, the positions and cash of the portfolio with the broker's. ```python MS = pd.Timedelta(milliseconds=1) def comision(q, px): return round(min(max(0.0035 * q, 0.35), 0.01 * q * px), 2) class Cartera: def __init__(self, bus, efectivo): self.efectivo, self.pos = efectivo, pd.Series(0, index=ETF) self.vistas = set() bus.suscribir("informe", self.anotar) def anotar(self, ev): d = ev.datos if d["evento"] not in ("parcial", "ejecutada") \ or d["id_ejecucion"] in self.vistas: return self.vistas.add(d["id_ejecucion"]) s = 1 if d["lado"] == "compra" else -1 self.pos[d["inst"]] += s * d["cantidad"] self.efectivo -= s * d["cantidad"] * d["precio"] + d["comision"] def patrimonio(self, px): return self.efectivo + float((self.pos * px).sum()) class OMS: def __init__(self, bus, riesgo, cartera, umbral=0.004): self.bus, self.riesgo, self.cartera = bus, riesgo, cartera self.umbral, self.ordenes = umbral, {} bus.suscribir("objetivo_ok", self.rebalancear) bus.suscribir("informe", self.actualizar) def rebalancear(self, ev): ref, w = ev.datos["ref"], ev.datos["pesos"] pat = self.cartera.patrimonio(ref) delta = np.floor(w * pat / ref) - self.cartera.pos for inst, q in delta.sort_values().items(): # sells first if abs(q) * ref[inst] < self.umbral * pat: continue o = dict(id=f"C{len(self.ordenes) + 1:05d}", fecha=ev.datos["fecha"], inst=inst, lado="compra" if q > 0 else "venta", cantidad=int(abs(q)), ref=float(ref[inst])) self.ordenes[o["id"]] = {**o, "estado": "nueva", "recibida": 0} motivo = self.riesgo.comprobar(o, pat) if motivo: self.ordenes[o["id"]]["estado"] = "bloqueada" self.bus.publicar(ev.t, "alerta", texto=motivo) continue self.ordenes[o["id"]]["estado"] = "enviada" self.bus.publicar(ev.t + 5 * 60_000 * MS, "orden", **o) def actualizar(self, ev): o = self.ordenes[ev.datos["id"]] o["estado"] = ev.datos["evento"] o["recibida"] += ev.datos.get("cantidad", 0) o["acumulada"] = ev.datos.get("acumulada", 0) class BrokerSimulado: LAT = 42 * MS def __init__(self, bus, efectivo, media_horquilla_pb=1.0): self.bus, self.efectivo = bus, efectivo self.pos = pd.Series(0, index=ETF) self.hs, self.cola, self.seq = media_horquilla_pb / 1e4, [], 0 self.perder = set() # transport failures on purpose bus.suscribir("orden", self.recibir) bus.suscribir("cierre", self.subasta) def informar(self, t, **d): self.seq += 1 if d.get("id_ejecucion") not in self.perder: self.bus.publicar(t + self.LAT, "informe", seq=self.seq, **d) def recibir(self, ev): o, t = ev.datos, ev.t + self.LAT ventas = sum(x["cantidad"] * x["ref"] for x in self.cola if x["lado"] == "venta") compras = sum(x["cantidad"] * x["ref"] for x in self.cola if x["lado"] == "compra") coste = o["cantidad"] * o["ref"] * (o["lado"] == "compra") if coste > 0 and compras + coste > self.efectivo + ventas: self.informar(t, id=o["id"], evento="rechazada") return self.cola.append(o) self.informar(t, id=o["id"], evento="aceptada") def subasta(self, ev): for o in self.cola: s = 1 if o["lado"] == "compra" else -1 px = round(ev.datos["px"][o["inst"]] * (1 + s * self.hs), 2) n = 1 if o["cantidad"] < 300 else ( 2 if o["cantidad"] < 1500 else 3) partes = [o["cantidad"] // n] * n partes[-1] += o["cantidad"] - sum(partes) acum = 0 for k, q in enumerate(partes, 1): acum += q com = comision(q, px) self.pos[o["inst"]] += s * q self.efectivo -= s * q * px + com fin = acum == o["cantidad"] self.informar(ev.t + 300 * k * MS, id=o["id"], evento="ejecutada" if fin else "parcial", inst=o["inst"], lado=o["lado"], cantidad=q, acumulada=acum, precio=px, comision=com, id_ejecucion=f"{o['id']}.{k:02d}") self.cola = [] class Reconciliador: def __init__(self, bus, cartera, broker): self.bus, self.c, self.b = bus, cartera, broker self.diario = [] bus.suscribir("fin_dia", self.cuadrar) def cuadrar(self, ev): dif = (self.b.pos - self.c.pos) dif = dif[dif != 0].to_dict() de = round(self.b.efectivo - self.c.efectivo, 2) self.diario.append(dict(fecha=ev.datos["fecha"].date(), posiciones=dif, efectivo=de)) if dif or de: self.bus.publicar(ev.t, "alerta", texto=f"mismatch {dif}") ``` There is a design detail in the OMS: it decides with the **internal** portfolio, not with the broker's. That is normal (the internal portfolio is updated instantly with each report and costs no query), and it is exactly what makes a lost message dangerous, as will be seen shortly. ## The complete system, from September 2 to December 31 Assembling the system means creating the pieces on the same bus and letting the data schedule their events. A minimal monitor collects the alerts (lesson 4.5.4 sends them by Telegram, Slack or email): ```python def montar(perder=(), capital=1_000_000.0): bus = Bus() riesgo = Riesgo(bus) cartera = Cartera(bus, capital) broker = BrokerSimulado(bus, capital) broker.perder = set(perder) oms = OMS(bus, riesgo, cartera) rec = Reconciliador(bus, cartera, broker) Senal(bus) alertas = [] bus.suscribir("alerta", lambda ev: alertas.append( (ev.t.date(), ev.datos["texto"]))) Datos(bus, precios).programar(SESIONES) bus.correr() return dict(bus=bus, oms=oms, cartera=cartera, broker=broker, rec=rec, alertas=alertas) s = montar() print(Counter(ev.tipo for ev in s["bus"].registro)) print(Counter(o["estado"] for o in s["oms"].ordenes.values())) cierre = precios.loc[SESIONES[-1]] print(f"Final equity: {s['cartera'].patrimonio(cierre):,.2f} USD") print("Days with a mismatch:", sum(bool(r["posiciones"] or r["efectivo"]) for r in s["rec"].diario), "| alerts:", len(s["alertas"])) ``` ```output Counter({'informe': 868, 'orden': 350, 'precios': 85, 'objetivo': 85, 'objetivo_ok': 85, 'cierre': 85, 'fin_dia': 85}) Counter({'ejecutada': 344, 'rechazada': 6}) Final equity: 1,077,668.41 USD Days with a mismatch: 0 | alerts: 0 ``` Eighty-five sessions, three data events per session and the whole chain behind them: targets, approved targets, orders, broker reports. Of the 350 orders, 344 end in `ejecutada` (filled) and 6 in `rechazada` (rejected): they are buys the broker does not accept because they do not fit in the cash. The OMS computes with the previous day's close and sends the sells before the buys (which is why there are few), but the auction prices are not yesterday's and a portfolio almost 100% invested sometimes runs out of room; the next day the OMS recomputes and the difference is sent again. The internal portfolio matches the broker's on all 85 days. The final equity is not the topic of the lesson (the system is an example, and four months say nothing about a strategy, lesson 2.5.5). ```mermaid flowchart TB P[precios 19:45] --> OB[objetivo] OB --> OK[objetivo_ok] OK --> OR[orden 19:50] OR --> AC[aceptada] CI[cierre 20:00] --> EJ[fills] AC --> EJ EJ --> FD[fin_dia 20:30] ``` A September session, in the order the bus goes through it (the labels are the event types). Whether the system is correct cannot be seen in the equity; it is checked piece by piece. The signal has a known answer: the level's real system logged the target weights of each order it sent (`vivo4_senales.csv`, 347 signals). If our signal piece is the same rule, its weights have to match the logged ones: ```python obj = {ev.datos["fecha"]: ev.datos["pesos"] for ev in s["bus"].registro if ev.tipo == "objetivo"} sen = pd.read_csv("vivo4_senales.csv") sen["fecha"] = pd.to_datetime(sen["ts"].str[:10]) sen["nuestro"] = [obj[f][i] for f, i in zip(sen["fecha"], sen["instrumento"])] dif = (sen["nuestro"].round(4) - sen["peso_objetivo"]).abs() print(f"{len(sen)} signals; maximum difference {dif.max():.4f}; " f"different at 4 decimals: {(dif > 1e-9).sum()}") ``` ```output 347 signals; maximum difference 0.0001; different at 4 decimals: 1 ``` All 347 match; one differs by one unit in the fourth decimal, which is what rounding can produce when the log stores four decimals. This check is the live version of the cross-engine reconciliation of lesson 2.4.2: the same rule, with the same data up to the previous day, has to give the same numbers in the lab and in production. If it does not, there is an error in one of the two, and it is better to find it before money is involved. ## A lost message Now the failure. The broker fills all the orders, but one fill report does not arrive: the one for the first part of the ORSC buy on the first day (in live trading, a dropped connection and a broker that does not resend everything on reconnecting; the level data have two cases, lesson 4.5.2): ```python perdida = next(ev.datos for ev in s["bus"].registro if ev.tipo == "informe" and ev.datos["evento"] == "parcial" and ev.datos["inst"] == "ORSC") print(perdida["id_ejecucion"], perdida["lado"], perdida["cantidad"], perdida["precio"], perdida["comision"]) f = montar(perder={perdida["id_ejecucion"]}) o_f = f["oms"].ordenes[perdida["id"]] print("In the OMS:", o_f["estado"], "| ordered", o_f["cantidad"], "| summed", o_f["recibida"], "| cumulative", o_f["acumulada"]) for r in [r for r in f["rec"].diario if r["posiciones"] or r["efectivo"]][:3]: print(r) ``` ```output C00003.01 compra 208 232.36 0.73 In the OMS: ejecutada | ordered 416 | summed 208 | cumulative 416 {'fecha': datetime.date(2025, 9, 2), 'posiciones': {'ORSC': 208}, 'efectivo': np.float64(-48331.61)} {'fecha': datetime.date(2025, 9, 3), 'posiciones': {'ORSC': 208}, 'efectivo': np.float64(-48331.61)} {'fecha': datetime.date(2025, 9, 4), 'posiciones': {'ORSC': 208}, 'efectivo': np.float64(-48331.61)} ``` The OMS marks the order as `ejecutada` (filled), because that is what the last report says, but of the 416 shares ordered it has only summed 208; the report carries the cumulative quantity (416) and the difference is in plain sight for whoever compares them, which is what the state machine of lesson 4.5.2 does. This OMS does not compare, and the internal portfolio ends up with 208 ORSC fewer than the broker and 48,331.61 USD more cash (the 208 shares at 232.36 plus 0.73 of commission). The reconciler sees it that same afternoon, at 20:30 UTC, and every following day. ```python def ordenes_de(sim, fecha, inst): return [(o["lado"], o["cantidad"], o["estado"]) for o in sim["oms"].ordenes.values() if o["fecha"] == pd.Timestamp(fecha) and o["inst"] == inst] for dia in ("2025-09-03", "2025-09-04", "2025-09-05"): print(dia, "no failure:", ordenes_de(s, dia, "ORSC"), "| with failure:", ordenes_de(f, dia, "ORSC")) print("ORSC at the end, broker and internal portfolio:", f["broker"].pos["ORSC"], f["cartera"].pos["ORSC"], "| healthy system:", s["broker"].pos["ORSC"]) fantasma = (f["broker"].pos["ORSC"] - f["cartera"].pos["ORSC"]) \ * precios.loc[SESIONES[-1], "ORSC"] print(f"Value of the phantom position: {fantasma:,.0f} USD; days " f"with a mismatch: {sum(bool(r['posiciones']) for r in f['rec'].diario)}") ``` ```output 2025-09-03 no failure: [] | with failure: [('compra', 205, 'rechazada')] 2025-09-04 no failure: [('compra', 53, 'ejecutada')] | with failure: [('compra', 260, 'rechazada')] 2025-09-05 no failure: [('venta', 59, 'ejecutada')] | with failure: [('compra', 201, 'ejecutada')] ORSC at the end, broker and internal portfolio: 208 0 | healthy system: 0 Value of the phantom position: 57,716 USD; days with a mismatch: 85 ``` With the failure, on September 3 the OMS tries to buy 205 ORSC that the healthy system does not buy: it is replenishing what it thinks is missing. The broker rejects it, because that cash was already spent on the shares the system does not know it has; on the 4th it tries with 260 and is rejected again, and on the 5th the buy of 201 does go through, while the healthy system sells 59. On December 31 the healthy system has no ORSC; the one that lost the message has 208 at the broker and 0 in its portfolio: a **phantom position** of 57,716 USD at year-end, more than 5% of equity, that no strategy rule decided and that no stop watches, because the system does not know it exists. The only thing that warned, 85 times, was the reconciler. The practical lesson is the order of things: **reconcile before trading**. An unexplained mismatch should stop the rebalancing of that instrument (or of everything) until a person or a recovery process resolves it; lesson 4.5.5 codes that recovery: ask the broker for the day's fills, correct the portfolio and start with the good state. ## Reproducible event by event The bus uses neither the computer's clock nor threads, so the same input stream produces the same event log. It is checked with a fingerprint of the complete log: ```python def huella(sim): h = hashlib.sha256() for ev in sim["bus"].registro: datos = {k: v for k, v in ev.datos.items() if k not in ("hist", "px", "pesos", "ref")} h.update(f"{ev.t.isoformat()}|{ev.tipo}|{datos}".encode()) return h.hexdigest()[:16] print(huella(s), huella(montar()), huella(f)) ``` ```output c62d14133b310f84 c62d14133b310f84 f92fa0459e14156a ``` The two healthy runs give the same fingerprint; the one with the failure, a different one. It is the property that the site's research on state machines calls replay as “the gold standard”: if the log of a production day can be fed through the system again and gives the same result, any difference between what happened and what should have happened lies in the inputs (data, broker messages), not in the code. Storing that log, in an append-only file or in a database, is the basis of the persistent state of lesson 4.5.5. ## The same components in backtest, paper and live What changes when going live is at the edges: ```mermaid flowchart TB N[Common core] --> RL[Logical clock] N --> RR[Real clock] RL --> DH[Historical data] RR --> DS[Streaming data] DH --> BS[Simulated broker] DS --> BP[Paper account] DS --> BR[Live account] ``` | Piece | Backtest | Paper | Live | |---|---|---|---| | Clock | Logical, jumps from event to event | Real | Real | | Data | Historical | Real time | Real time | | Broker | Simulated in the process | Broker's paper account | Live account | | Signal, risk, OMS, portfolio, reconciliation | Same code | Same code | Same code | In live trading, the bus is the same but it is fed by an adapter that receives messages from the broker and from the data provider and publishes them with their arrival time. A sketch with `asyncio`, not executed (the real connections are those of lessons 4.4.2 to 4.4.4): ```python skip title="adaptador_en_vivo.py" import asyncio from datetime import datetime, timezone async def adaptador(bus, flujo_broker, flujo_datos): """Publishes each message on the bus with its arrival time.""" async def bombear(flujo, tipo): async for msg in flujo: ahora = pd.Timestamp(datetime.now(timezone.utc)) bus.publicar(ahora, tipo, **msg) bus.correr() # processes whatever is in the queue await asyncio.gather(bombear(flujo_broker, "informe"), bombear(flujo_datos, "precios")) ``` There are single-process systems, like this one, and multi-process systems that talk through a message queue. For a daily strategy, one process is enough and easier to reason about: events have a total order and the state is in one place. Several processes make sense when a piece is slow (a heavy model) or when there are many strategies sharing data and broker; in exchange, the order of events between processes is no longer guaranteed and more reconciliation is needed. Where the process runs (your own server, a VPS, a container) changes latency and availability, not the architecture; the site's research on Docker explains how to package it. ## A real order end to end The level data include the log of a system with this same architecture trading for four months (paper until October 10 and a live account afterwards). Order `R00071` is a market sell of 243 ORUS on October 28: ```python ev = pd.read_csv("vivo4_eventos_broker.csv") o = ordenes.set_index("id_orden_cliente").loc["R00071"] t0 = pd.Timestamp(o["ts_creacion"]) ms = lambda ts: round((pd.Timestamp(ts) - t0).total_seconds() * 1e3) print(o["id_senal"], o["instrumento"], o["lado"], o["cantidad"], o["tipo"], "| sent:", ms(o["ts_envio"]), "ms") r = ev[ev["id_orden_cliente"] == "R00071"] pd.DataFrame({"seq": r["seq"], "evento": r["evento"], "broker_ms": r["ts_broker"].map(ms), "recibido_ms": r["ts_recibido"].map(ms), "cantidad": r["cantidad_ultima"], "precio": r["precio_ultimo"]}).set_index("seq") ``` ```output S00180 ORUS venta 243 mercado | sent: 277 ms evento broker_ms recibido_ms cantidad precio seq 176 aceptada 331 378 0 NaN 177 parcial 355 392 122 658.09 178 parcial 668 708 12 658.06 180 ejecutada 1300 1329 109 658.18 ``` The signal went out at 19:45 UTC; the OMS created the order five minutes later (that is the system's rule) and sent it 277 ms after that. The broker acknowledged it at 331 ms and filled it in three fills in just over a second; each message takes between 30 and 50 ms more to reach the system. That is the header figure. Two observations for the following lessons: the system learned of the first fill (392 ms) only 14 ms after the acknowledgment (378 ms), and in other orders in the data the fill arrives **before** the acknowledgment (lesson 4.5.2); and the average price of the three fills against the average market price at sending is the slippage of lesson 4.5.3. ## Exercises 1. Activate the risk kill switch on October 1 (without changing anything else) and count how many sessions are left without approved targets and how many alerts are raised. What happens to the portfolio while the kill switch is active? **Solution.** ```python class RiesgoConFecha(Riesgo): def revisar(self, ev): self.interruptor = ev.datos["fecha"] >= pd.Timestamp( "2025-10-01") super().revisar(ev) def montar_con(riesgo_cls, capital=1_000_000.0): bus = Bus() riesgo = riesgo_cls(bus) cartera = Cartera(bus, capital) broker = BrokerSimulado(bus, capital) oms = OMS(bus, riesgo, cartera) Senal(bus) alertas = [] bus.suscribir("alerta", lambda e: alertas.append(e.datos["texto"])) Datos(bus, precios).programar(SESIONES) bus.correr() return oms, cartera, alertas oms_i, cart_i, al_i = montar_con(RiesgoConFecha) print("alerts:", Counter(al_i)) print("orders:", len(oms_i.ordenes), "versus", len(s["oms"].ordenes)) print(cart_i.pos.to_dict()) ``` ```output alerts: Counter({'kill switch': 64}) orders: 68 versus 350 {'ORUS': 201, 'ORSC': 397, 'OREU': 1835, 'OREM': 1765, 'ORTN': 1360, 'ORTL': 1474, 'ORIG': 1188, 'ORAU': 309, 'ORCM': 0, 'ORRE': 961} ``` From October 1 risk approves no target: a kill-switch alert is raised in each of the 64 sessions and the OMS sends no more orders (68 in total, all in September, versus 350). The portfolio stays frozen with the positions of September 30: the kill switch in this lesson stops trading, it does not liquidate. Whether it should also cancel open orders or close positions is a design decision with its own cost (lesson 4.2.5). 2. Lower the maximum value per order to 10% of equity. How many orders are blocked and when? What consequence does it have on the portfolio weights in the first days? **Solution.** ```python class RiesgoEstricto(Riesgo): def __init__(self, bus): super().__init__(bus, valor_max=0.10) oms_e, cart_e, al_e = montar_con(RiesgoEstricto) bloq = pd.Series([o["fecha"] for o in oms_e.ordenes.values() if o["estado"] == "bloqueada"]) print(len(bloq), "blocked in", bloq.nunique(), "of", len(SESIONES), "sessions") print("First days:", bloq.dt.date.value_counts().sort_index() .head(3).to_dict()) fin = precios.loc[SESIONES[-1]] pesos = lambda c: (c.pos * fin / c.patrimonio(fin)).round(3) pd.DataFrame({"limit 30%": pesos(s["cartera"]), "limit 10%": pesos(cart_e)}).T ``` ```output 356 blocked in 82 of 85 sessions First days: {datetime.date(2025, 9, 2): 6, datetime.date(2025, 9, 3): 6, datetime.date(2025, 9, 4): 5} ORUS ORSC OREU OREM ORTN ORTL ORIG ORAU ORCM ORRE limit 30% 0.000 0.000 0.0 0.0 0.0 0.0 0.0 0.109 0.000 0.000 limit 10% 0.108 0.107 0.0 0.0 0.0 0.0 0.0 0.108 0.112 0.109 ``` With 10%, the limit blocks 356 orders in 82 of the 85 sessions: every day the OMS repeats the order that did not get through. In the first days it cannot build the portfolio (6, 6 and 5 blocks), and the worst comes later. Positions are built piecemeal below 10%, but when the signal asks to get out, selling the whole position is worth more than 10% and is also blocked: on December 31 the strict system holds about 11% in ORUS, ORSC, ORCM and ORRE, which the signal wants at zero. A limit meant to stop errors prevents reducing risk. That is why it is calibrated with the history and, at many trading desks, orders that reduce the position have different limits from those that increase it. 3. Explain what would need to change in the data piece for the system's signal to work properly on November 17, the day of ORAU's 3-for-1 split, if the provider gave unadjusted prices. **Solution.** With unadjusted prices, ORAU's close on November 17 would be a third of the previous day's: the 20-session momentum would suddenly turn very negative and the 63-session volatility would spike, so the signal would sell all of ORAU for no economic reason. The data piece has to publish series adjusted for corporate actions (lesson 1.2.1), and the portfolio has to multiply the position by 3 and divide its cost price by 3 that same day; otherwise, the reconciliation with the broker will show a mismatch of 374 shares (lesson 4.5.5). What matters is that the adjustment is made in one place (the data piece and the corporate actions log), not scattered across the signal. ## Self-assessment 1. Why does the lesson's bus order events by a logical time instead of using `datetime.now()` and `time.sleep()`? - A. Because it is faster - B. Because that way the same event stream always gives the same result: a whole day can be replayed, in a backtest or after a failure, and compared event by event - C. Because brokers do not accept real times - D. Because `sleep` does not exist in Python 3.14 *Answer B: Because that way the same event stream always gives the same result: a whole day can be replayed, in a backtest or after a failure, and compared event by event.* With the clock injected, the order of events depends only on their logical times and on publication order. In the lesson, two complete runs give the same SHA-256 fingerprint of the event log. In live trading, the clock is set by the data adapter; the rest of the pieces do not change. 2. The system loses the fill of 208 ORSC on September 2. Which piece detects it and when? - A. The signal, the next day - B. The OMS, instantly - C. The reconciler, at the close of that day, when comparing the internal portfolio with the broker's - D. Nobody: the difference is too small *Answer C: The reconciler, at the close of that day, when comparing the internal portfolio with the broker's.* The OMS considers the order `ejecutada` (filled; that is what the last report says) and the internal portfolio knows nothing. The reconciler compares positions and cash with the broker's at 20:30 UTC and finds a difference of 208 ORSC and 48,331.61 USD. If nobody acts, the OMS tries to buy what it thinks is missing and the system ends up with a phantom position. 3. Which pre-trade control would have stopped an order with ten times the intended quantity (a fat-finger error)? - A. A limit on the maximum value of each order, calibrated above the largest legitimate order - B. A stop-loss - C. The end-of-day reconciliation - D. None: the broker always rejects it *Answer A: A limit on the maximum value of each order, calibrated above the largest legitimate order.* It is control b) of article 15 of Commission Delegated Regulation (EU) 2017/589. In the system's history of 398 orders, the largest is worth 26.1% of capital; a 30% limit blocks no legitimate order and stops the order multiplied by ten. 4. What changes between backtest, paper and live in the lesson's architecture? - A. Everything: they are three different programs - B. The signal and the risk - C. Only the adapters at the edges: where the clock and the data come from, and which broker the orders go to - D. Nothing, not even the broker *Answer C: Only the adapters at the edges: where the clock and the data come from, and which broker the orders go to.* Signal, risk, OMS, portfolio and reconciler are the same code. What changes is the clock (logical or real), the data source (historical or streaming) and the broker (simulated, paper account or live account). It is the idea behind NautilusTrader and LEAN (lessons 2.4.5 and 4.5.6). 5. Why does the lesson's signal use prices up to the previous day even though it is computed 15 minutes before the close? - A. For convenience - B. Because the day's close does not exist yet at that time: on the bus, the `cierre` (close) event is published at the closing time and no piece can read it earlier - C. Because that way the Sharpe is higher - D. Because the broker requires it *Answer B: Because the day's close does not exist yet at that time: on the bus, the `cierre` (close) event is published at the closing time and no piece can read it earlier.* In an event-driven system, information has a time. What a component knows is what has reached it. It is the same protection against look-ahead bias as in lesson 2.4.2, now with the real time of each message. ## Key points - The chain of a live system has six pieces: data, signal, risk, OMS, broker and reconciliation. Each one has a single responsibility and talks to the others through timestamped messages. - A bus with a logical clock makes the system reproducible: the event logs of two runs have the same fingerprint, and a production day can be replayed in the lab. - Pre-trade controls (maximum value, volume, number of messages, kill switch) are mandatory for firms engaged in algorithmic trading in the EU and sensible for anyone; they are calibrated with the history of legitimate orders. - The OMS knows what it asked for; only the broker knows what happened. The daily reconciliation between the two is the only piece that discovers a lost message. - A mismatch nobody corrects does not stay put: the OMS trades on a false position and the error turns into orders. - Backtest, paper and live share the core; the clock, the data and the broker change. ## Resources ### Research - [Finite-State Machines for Quantitative Trading Execution](https://pythonparatrading.com/en/finite-state-machines-trading-python/) · Research-0029 · 47 min - [The Working Environment - 1](https://pythonparatrading.com/en/docker-for-trading/) · Research-0013 · 4 min ### Documentation - [Commission Delegated Regulation (EU) 2017/589 (RTS 6): algorithmic trading requirements (OJ, PDF on the BOE)](https://www.boe.es/doue/2017/087/L00417-00448.pdf) · boe.es - [Python: heapq, priority queue](https://docs.python.org/3/library/heapq.html) · docs.python.org ### Data - [Level 4 data: live system signals (CSV)](https://pythonparatrading.com/curso/datos/nivel-4/vivo4_senales.csv) · Download · vivo4_senales.csv - [Level 4 data: OMS orders (CSV)](https://pythonparatrading.com/curso/datos/nivel-4/vivo4_ordenes.csv) · Download · vivo4_ordenes.csv Previous lesson: [4.4.5 · Secure credentials: environment variables, secret managers and least privilege](https://pythonparatrading.com/en/python-trading-course/level-4/secure-api-credentials-and-least-privilege/) Next lesson: [4.5.2 · Order management: states, partial fills, cancellations and reconnections](https://pythonparatrading.com/en/python-trading-course/level-4/order-management-states-fills-and-reconnections/)