--- title: "4.5.2 Order management: states, fills and reconnections" description: "An order state machine validated with 983 broker messages: partial fills, cancellations, rejections, duplicates, gaps and reconnections." url: "https://pythonparatrading.com/en/python-trading-course/level-4/order-management-states-fills-and-reconnections/" lang: "en" image: "https://pythonparatrading.com/og/en/python-trading-course/level-4/order-management-states-fills-and-reconnections.png" --- # 4.5.2 · Order management: states, partial fills, cancellations and reconnections > The broker does not answer in order, nor only once: sometimes the fill arrives before the acknowledgment, sometimes twice, sometimes never. This lesson writes the state machine of an order, runs it through the 983 broker messages in the level data, sees where it breaks if they are processed as they arrive and fixes it with a sequencer by sequence number that recovers even the lost fills. - Course: [Level 4: Portfolios, risk, execution and live trading](https://pythonparatrading.com/en/python-trading-course/level-4/) - Module: 4.5 · Live trading system - Duration: 75 min - Prerequisites: [4.5.1 · Live system architecture: from data to reconciliation](https://pythonparatrading.com/en/python-trading-course/level-4/live-trading-system-architecture/), [2.4.8 · Fill modeling: limit orders, queue, partial fills, gaps and stops](https://pythonparatrading.com/en/python-trading-course/level-2/backtest-order-fill-modeling/) ## By the end of this lesson - Model the life cycle of an order as a state machine with invariants (cumulative quantity, leaves quantity, final states) that rejects the impossible. - Process partial fills, cancellations with a remainder, rejections and the race between a cancellation and a fill. - Order the broker's messages by sequence number with a timeout on a logical clock, discard duplicates and declare gaps. - Rebuild a lost fill from the cumulative quantity and the average price of the next message. - Decide what to do with in-flight orders on reconnecting: query their state before resending. ## What the broker answers In lesson 4.5.1 the simulated broker always answered, in order and once. A real broker promises none of that. The Interactive Brokers API documentation says so plainly: it is normal to receive duplicate status messages, there is no guarantee that one arrives for each state change (a market order that fills instantly may not produce any) and that is why it recommends also watching the execution messages[^1]. The level data include four months of messages from a broker like that: `vivo4_eventos_broker.csv`, **in the order in which they arrived**, with the sequence number (`seq`) the broker assigns to each message of each account. ```python import heapq import numpy as np import pandas as pd ord_ = pd.read_csv("vivo4_ordenes.csv") ev = pd.read_csv("vivo4_eventos_broker.csv") ev["llegada"] = np.arange(len(ev)) # order of arrival for c in ("ts_broker", "ts_recibido"): ev[c] = pd.to_datetime(ev[c]) print(len(ord_), "orders;", len(ev), "messages") pd.crosstab(ev["evento"], ev["modo"], margins=True) ``` ```output 398 orders; 983 messages modo paper real All evento aceptada 134 262 396 cancelada 16 34 50 ejecutada 118 232 350 parcial 65 120 185 rechazada 1 1 2 All 334 649 983 ``` Five message types: `aceptada` (acknowledged), `parcial` (a fill that does not complete the order), `ejecutada` (the one that completes it), `cancelada` (canceled) and `rechazada` (rejected). Each one carries the quantity of that fill (`cantidad_ultima`), the cumulative quantity, the leaves quantity and the average price of everything filled: the complete state of the order, not just the change. That redundancy is what will save the system when a message is missing. A limit order from the live account that is partially filled and canceled after seven minutes, and the market order the OMS sends with the remainder: ```python print(ord_.set_index("id_orden_cliente").loc[ ["R00047", "R00048"], ["instrumento", "lado", "tipo", "cantidad", "precio_limite", "orden_origen"]]) cortos = {"id_orden_cliente": "orden", "cantidad_ultima": "q", "precio_ultimo": "precio", "cantidad_acumulada": "acum", "cantidad_pendiente": "pend"} (ev.loc[ev["id_orden_cliente"].isin(["R00047", "R00048"]), ["seq", "evento", *cortos]] .rename(columns=cortos).set_index("seq")) ``` ```output instrumento lado tipo cantidad precio_limite orden_origen id_orden_cliente R00047 ORIG venta limite 222 112.71 NaN R00048 ORIG venta mercado 91 NaN R00047 evento orden q precio acum pend seq 99 aceptada R00047 0 NaN 0 222 110 parcial R00047 131 112.71 131 91 112 cancelada R00047 0 NaN 131 0 113 aceptada R00048 0 NaN 0 91 114 ejecutada R00048 91 112.71 91 0 ``` `R00047` sells 222 ORIG at 112.71 or better. 131 are filled at the limit price; after seven minutes the OMS requests a cancellation and the broker confirms it with the cumulative quantity (131) and the leaves quantity at zero. The OMS then sends `R00048`, a market order, for the 91 still missing, with `orden_origen` pointing to the limit order. These are three rules the state machine has to check: the cumulative quantity is the sum of the fills, the leaves quantity is what was ordered minus the cumulative (zero on cancellation) and the remainder is exactly what was not filled. ## The state machine of an order The life cycle of an order has been standardized for decades in the FIX protocol (the `OrdStatus` field) and each broker uses a dialect of it; the site's research on state machines covers it in detail. The states in this lesson and their equivalents: | State in the lesson | FIX `OrdStatus` | Alpaca | Interactive Brokers | |---|---|---|---| | `enviada` (sent) | `PendingNew` (A) | `pending_new`, `accepted` | `PendingSubmit`, `ApiPending` | | `aceptada` (acknowledged) | `New` (0) | `new` | `Submitted` (`PreSubmitted` for orders simulated by IB) | | `parcial` (partial) | `PartiallyFilled` (1) | `partially_filled` | `Submitted` with `filled` > 0 | | `pendiente_cancelacion` (pending cancel) | `PendingCancel` (6) | `pending_cancel` | `PendingCancel` | | `ejecutada` (filled) | `Filled` (2) | `filled` | `Filled` | | `cancelada` (canceled) | `Canceled` (4) | `canceled` | `Cancelled`, `ApiCancelled` | | `rechazada` (rejected) | `Rejected` (8) | `rejected` | `Inactive` (with an error) | The complete lists of Alpaca and Interactive Brokers have more states (`done_for_day`, `expired`, `replaced`…) [^2]; a system that trades day orders without modifying them can treat them as variants of those in the table, but it has to recognize them: an unknown state is an error, not an `else`. ```mermaid stateDiagram-v2 [*] --> enviada enviada --> aceptada enviada --> rechazada aceptada --> parcial aceptada --> ejecutada parcial --> parcial parcial --> ejecutada aceptada --> pend_cancel parcial --> pend_cancel pend_cancel --> pend_cancel : fill pend_cancel --> cancelada pend_cancel --> ejecutada : race ejecutada --> [*] cancelada --> [*] rechazada --> [*] ``` `pend_cancel` is `pendiente_cancelacion`: the OMS has requested a cancellation and the broker has not answered yet. It is not a detail: while the request travels, the order can keep filling, and if it fills completely before the cancellation arrives, the broker answers with a fill, not with a cancellation (the “race”). A system that considered the order canceled as soon as it requested it would send the remainder as a market order and buy twice. The machine is coded as a transition table and a class that applies it and checks the invariants. Any (state, event) pair that is not in the table is an invalid transition and raises an exception: it is not ignored. ```python FINALES = {"ejecutada", "cancelada", "rechazada"} T = { ("enviada", "aceptada"): "aceptada", ("enviada", "rechazada"): "rechazada", ("aceptada", "parcial"): "parcial", ("aceptada", "ejecutada"): "ejecutada", ("parcial", "parcial"): "parcial", ("parcial", "ejecutada"): "ejecutada", ("aceptada", "pedir_cancelacion"): "pendiente_cancelacion", ("parcial", "pedir_cancelacion"): "pendiente_cancelacion", ("pendiente_cancelacion", "parcial"): "pendiente_cancelacion", ("pendiente_cancelacion", "ejecutada"): "ejecutada", ("pendiente_cancelacion", "cancelada"): "cancelada", } class Invalida(Exception): pass class Orden: def __init__(self, id_, cantidad): self.id, self.cantidad = id_, cantidad self.estado, self.acumulada, self.valor = "enviada", 0, 0.0 self.ejecuciones = {} # id_ejecucion -> (q, price) def aplicar(self, e): clave = (self.estado, e.evento) if clave not in T: raise Invalida(f"{self.id}: “{e.evento}” in " f"“{self.estado}”") if e.evento in ("parcial", "ejecutada"): if e.id_ejecucion in self.ejecuciones: raise Invalida(f"{self.id}: {e.id_ejecucion} repeated") nueva = self.acumulada + e.cantidad_ultima if nueva != e.cantidad_acumulada: raise Invalida(f"{self.id}: cumulative {self.acumulada}" f" + {e.cantidad_ultima} != " f"{e.cantidad_acumulada}") self.ejecuciones[e.id_ejecucion] = (e.cantidad_ultima, e.precio_ultimo) self.acumulada = e.cantidad_acumulada self.valor += e.cantidad_ultima * e.precio_ultimo if e.evento == "cancelada" and ( e.cantidad_acumulada != self.acumulada): raise Invalida(f"{self.id}: cumulative {self.acumulada}" f" != {e.cantidad_acumulada} on cancel") if e.evento in ("parcial", "ejecutada", "cancelada"): pend = 0 if e.evento == "cancelada" else ( self.cantidad - self.acumulada) if e.cantidad_pendiente != pend: raise Invalida(f"{self.id}: leaves " f"{e.cantidad_pendiente} != {pend}") self.estado = T[clave] ``` The order starts in `enviada` when the OMS sends it (not when the broker acknowledges it) and the OMS adds its own event, `pedir_cancelacion` (request cancel), at the time it requested it (`ts_cancelacion` in `vivo4_ordenes.csv`). The other events are the broker's. ## Processing messages as they arrive The simplest system applies each message in order of arrival. The OMS stream (submissions and cancel requests) is merged with the broker's by system time: `ts_envio` and `ts_cancelacion` for the OMS ones, `ts_recibido` for the broker's. ```python def eventos_oms(ordenes): """The events the OMS itself generates, with their time.""" filas = [] for o in ordenes.itertuples(): if pd.notna(o.ts_cancelacion): filas.append(dict(t=pd.Timestamp(o.ts_cancelacion), id_orden_cliente=o.id_orden_cliente, evento="pedir_cancelacion")) return pd.DataFrame(filas) def procesar(flujo, ordenes): """flujo: table with t, id_orden_cliente, evento and fields.""" libro = {o.id_orden_cliente: Orden(o.id_orden_cliente, o.cantidad) for o in ordenes.itertuples()} errores = [] for e in flujo.itertuples(): o = libro[e.id_orden_cliente] try: o.aplicar(e) except Invalida as x: errores.append((o.estado, e.evento, str(x))) return libro, errores def tipo_error(estado, evento, texto): if estado == "enviada" and evento in ("parcial", "ejecutada"): return "fill before acknowledgment" if estado in FINALES: return "message after a final state" return "cumulative or leaves mismatch" def mezclar(broker, ordenes, col_t): b = broker.assign(t=broker[col_t]) return (pd.concat([b, eventos_oms(ordenes)], ignore_index=True) .sort_values("t", kind="stable")) libro_n, err_n = procesar(mezclar(ev, ord_, "ts_recibido"), ord_) print(len(err_n), "errors") print(pd.Series([tipo_error(*x) for x in err_n]).value_counts() .to_string()) print([x[2] for x in err_n[:2]]) ``` ```output 53 errors fill before acknowledgment 28 cumulative or leaves mismatch 21 message after a final state 4 ['P00007: “parcial” in “enviada”', 'P00007: cumulative 0 + 241.0 != 409.0'] ``` Fifty-three invalid transitions. The first 28 are fills that arrive **before** their order's acknowledgment: the broker acknowledged and filled in milliseconds and the two messages traveled with different latencies. The 21 cumulative ones are almost all a consequence of those: the machine rejects the early fill and the order's next message no longer adds up; among them are also the one for the partial fill that arrives after the final one (`seq` 393, anomaly A09 in the data) and those for the two fills that never arrived. The last 4 are the messages repeated on reconnecting, which reach orders that are already finished. With this processing, a system that ignored the errors would have orders in absurd states; one that counted all the fills, false positions. ## A sequencer by sequence number The broker numbers the messages of each account without gaps. That makes it possible to rebuild the order: a **sequencer** waits for the `seq` following the last one delivered, holds those that arrive early and discards those it has already delivered. If the missing one does not arrive within a timeout, it declares it a **gap** and moves on: you cannot wait forever. ```mermaid flowchart TB L[seq n arrives] --> Q{n vs expected} Q -- lower --> D[Discard repeated] Q -- equal --> E[Deliver and flush] Q -- higher --> R[Hold] R --> V{Timeout expires} V -- yes --> H[Declare gap] H --> E ``` The timeout is measured with the **message clock** (`ts_recibido`), not the computer's: the sequencer receives the time of each arrival and checks expired timeouts before processing it. That way the result is the same on any machine, loaded or not; in live trading, an event loop timer would pass it the time every so often. ```python class Secuenciador: def __init__(self, plazo=pd.Timedelta(seconds=2)): self.esperado, self.plazo = 1, plazo self.retenidos = {} # seq -> (arrival, message) self.salida, self.repetidos, self.huecos = [], [], [] self.vistos = set() def _vaciar(self, t): while self.esperado in self.retenidos: _, m = self.retenidos.pop(self.esperado) self.salida.append((t, m)) self.vistos.add(self.esperado) self.esperado += 1 def vencer(self, t): """Declares the gaps whose timeout expired before t.""" while self.retenidos: limite = min(x for x, _ in self.retenidos.values()) \ + self.plazo if limite > t: return nuevo = min(self.retenidos) self.huecos.extend(range(self.esperado, nuevo)) self.esperado = nuevo self._vaciar(limite) def llegar(self, t, m): self.vencer(t) if m.seq in self.vistos or m.seq in self.retenidos: self.repetidos.append(m.seq) return if m.seq < self.esperado: # arrives after its gap self.salida.append((t, m)) self.vistos.add(m.seq) return self.retenidos[m.seq] = (t, m) self._vaciar(t) def secuenciar(mensajes): s = Secuenciador() for m in mensajes.itertuples(): s.llegar(m.ts_recibido, m) s.vencer(pd.Timestamp.max.tz_localize("UTC")) filas = [m._asdict() | {"t": t} for t, m in s.salida] return pd.DataFrame(filas), s sec, info = {}, {} for modo, g in ev.groupby("modo", sort=False): sec[modo], info[modo] = secuenciar(g) print(f"{modo}: delivered {len(sec[modo])}, " f"repeated {info[modo].repetidos}, " f"gaps {info[modo].huecos}") ``` ```output paper: delivered 334, repeated [], gaps [] real: delivered 645, repeated [61, 62, 471, 475], gaps [202, 519] ``` In paper, the 334 messages are delivered with no repeats or gaps. In the live account, the sequencer discards the four repeated messages and declares two gaps, `seq` 202 and 519: the two messages the broker did not resend on reconnecting. Message 393, which arrived half a second late (after 405), is waited for within the two-second timeout and delivered in its place. How much does waiting cost? The delay between arrival and delivery: ```python todo = pd.concat(sec.values(), ignore_index=True) retraso = (todo["t"] - todo["ts_recibido"]).dt.total_seconds() * 1e3 print(f"Delivered immediately: {(retraso == 0).mean():.1%}") print("Held:", (retraso > 0).sum(), "| median", f"{retraso[retraso > 0].median():.1f} ms | maximum", f"{retraso.max():,.0f} ms") ``` ```output Delivered immediately: 93.2% Held: 67 | median 7.1 ms | maximum 2,000 ms ``` 93.2% of the messages are delivered immediately. The 67 held messages wait a median of 7 ms (the fill that overtook its acknowledgment) and the worst ones, those behind a gap, the full timeout: two seconds. The timeout is a trade-off: shorter, and a message that is merely late is declared a gap; longer, and the system learns late about everything behind it. ## Recovering what did not arrive With the messages in order, the state machine only finds what is really missing: ```python flujo = pd.concat([pd.concat(sec.values()), eventos_oms(ord_)], ignore_index=True).sort_values("t", kind="stable") libro, errores = procesar(flujo, ord_) print(len(errores), "errors") for x in errores: print(" ", x[2]) print(pd.Series({k: o.estado for k, o in libro.items()}) .value_counts().to_dict()) ``` ```output 2 errors R00079: cumulative 0 != 112.0 on cancel R00214: cumulative 0 + 1016.0 != 1573.0 {'ejecutada': 345, 'cancelada': 49, 'rechazada': 2, 'pendiente_cancelacion': 1, 'aceptada': 1} ``` Two errors, and they are the two gaps: in `R00079` the cancellation arrives with 112 shares cumulative when the system had counted only 0; in `R00214`, the final fill with 1,573 cumulative when it had counted 0. All the other orders go through the machine without a single invalid transition and end in a final state, except those two, which stay where the error left them. The message following the gap brings what is needed to rebuild what was lost. The quantity is the difference between the message's cumulative quantity and what was counted plus the message's own fill; the price comes from the average price: $$ q_{\text{perdida}} = A - a - q, \qquad p_{\text{perdida}} = \frac{\overline{p}\,A - V - q\,p}{q_{\text{perdida}}} $$ with $A$ and $\overline{p}$ the cumulative quantity and average price of the message, $a$ and $V$ the quantity and value counted up to then, and $q$, $p$ the message's own fill (zero on a cancellation). ```python class OrdenRecuperable(Orden): def aplicar(self, e): q = e.cantidad_ultima if e.evento in ("parcial", "ejecutada") else 0 if e.cantidad_acumulada > self.acumulada + q: perdida = e.cantidad_acumulada - self.acumulada - q px = ((e.precio_medio * e.cantidad_acumulada - self.valor - q * (e.precio_ultimo if q else 0)) / perdida) self.ejecuciones[f"inferida-{e.seq}"] = (perdida, round(px, 2)) self.acumulada += perdida self.valor += perdida * px self.inferida = (perdida, round(px, 2)) super().aplicar(e) def procesar_rec(flujo, ordenes): libro = {o.id_orden_cliente: OrdenRecuperable(o.id_orden_cliente, o.cantidad) for o in ordenes.itertuples()} errores = [] for e in flujo.itertuples(): try: libro[e.id_orden_cliente].aplicar(e) except Invalida as x: errores.append(str(x)) return libro, errores libro_r, err_r = procesar_rec(flujo, ord_) print(len(err_r), "errors") for k, o in libro_r.items(): if hasattr(o, "inferida"): print(k, o.estado, "inferred:", o.inferida) verdad = pd.read_csv("vivo4_anomalias_verdad.csv") verdad.loc[verdad["tipo"] == "falta_ejecucion", ["id_anomalia", "id_orden_cliente", "descripcion"]] ``` ```output 0 errors R00079 cancelada inferred: (112.0, 99.06) R00214 ejecutada inferred: (557.0, 114.03) id_anomalia id_orden_cliente descripcion 3 A04 R00079 La ejecución parcial E710000553.01 (112 título... 11 A12 R00214 La ejecución parcial E710001498.01 (557 título... ``` Zero errors, and the two inferred fills are exactly those of the ground truth in the data: 112 shares at 99.06 in `R00079` and 557 at 114.03 in `R00214`. The inference works because the broker sends the complete state in every message; with a broker that only sent changes there would be nothing to solve for. Even so, an inferred fill is not a confirmed fill: it has no `id_ejecucion` and no commission. The system must flag it, ask the broker for the day's fills (every API has that query) and replace it with the real one; if it does not match, it is a mismatch for the reconciliation (lesson 4.5.5). ## Partial fills and cancellations An order being filled in several parts is the norm, not the exception. With the book already rebuilt: ```python n_ej = pd.Series({k: len(o.ejecuciones) for k, o in libro_r.items()}) tabla = ord_.assign(ejecuciones=ord_["id_orden_cliente"].map(n_ej)) print(pd.crosstab(tabla["tipo"], tabla["ejecuciones"])) lim = tabla[tabla["tipo"] == "limite"].copy() fin = lim["id_orden_cliente"].map(lambda k: libro_r[k].estado) acu = lim["id_orden_cliente"].map(lambda k: libro_r[k].acumulada) lim["destino"] = np.where(fin == "ejecutada", "full", np.where(acu > 0, "partial", "none")) pd.crosstab(lim["modo"], lim["destino"], margins=True) ``` ```output ejecuciones 0 1 2 3 4 tipo cierre 0 56 0 0 0 limite 18 75 7 5 6 mercado 2 165 25 22 17 destino full none partial All modo paper 22 6 10 38 real 39 12 22 73 All 61 18 32 111 ``` *Fig. — Number of fills of each of the live system's 398 orders, by order type (with the two lost fills reconstructed).* Three bar panels. The 56 at-the-close orders fill in a single part. Of the 111 limit orders, 18 do not fill at all (highlighted), 75 fill in one part and 18 in two to four. Of the 231 market orders, 165 fill in one part, 64 in two to four and 2 (the rejected ones) in none. Market-on-close orders (MOC) are always filled in a single part, in the auction. Large market orders are split into two to four fills; limit orders, almost always into one, because they are filled against orders that arrive at the price. What sets limit orders apart is their fate: in the live account, 39 of 73 were filled completely, 22 partially and 12 not at all. The 34 that were not filled completely were canceled after seven minutes and their remainder was sent as a market order. The OMS rule for that remainder is checked on all of them: ```python resto = ord_[ord_["motivo"] == "resto_de_limite"] pedido = resto["cantidad"].to_numpy() falta = resto["orden_origen"].map( lambda k: libro_r[k].cantidad - libro_r[k].acumulada).to_numpy() cancel = resto["orden_origen"].map(lambda k: libro_r[k].estado) print(len(resto), "remainders;", "all from canceled orders:", (cancel == "cancelada").all(), "| correct quantity:", (pedido == falta).all()) ``` ```output 50 remainders; all from canceled orders: True | correct quantity: True ``` The 50 remainders (16 in paper, 34 in live) come from a canceled limit order and ask for exactly what was missing. Note that the check uses the **rebuilt** cumulative quantity: `R00079` is one of those limit orders, and without the inferred fill of 112 shares the remainder would have looked 112 shares too large. An OMS that computed the remainder with what it had counted, and not with the cumulative quantity the broker confirms on cancellation, would have sold 154 ORTN instead of 42: 112 too many. Rejections are few and should be treated as what they are, a final answer: ```python r = ev[ev["evento"] == "rechazada"] r[["id_orden_cliente", "ts_broker", "motivo"]].merge( ord_[["id_orden_cliente", "instrumento", "cantidad"]]) ``` ```output id_orden_cliente ts_broker ... instrumento cantidad 0 P00005 2025-09-02 19:50:00.862198+00:00 ... ORTL 1639 1 R00007 2025-10-14 13:35:00.585315+00:00 ... ORIG 1260 [2 rows x 5 columns] ``` Two buys rejected for lack of buying power, one in each account and both in the first days, when the portfolio is being built from cash. Retrying the same order would give the same rejection: the OMS does not retry, and the next signal recomputes the difference with what is really there. A rejection for a transient cause (the market has not opened yet, a request limit) is handled differently, and that is why the reason must be stored with the order. ## Disconnections and reconnections The connection with the broker has its own log (`vivo4_conexion.csv`): each session starts with `conectado` (connected) and ends with `desconectado` (disconnected, with the detail `fin de sesión`, “end of session”), and in between there are seven drops. Each drop follows the same script: the system stops receiving the broker's heartbeat, after three seconds it logs it (`latido_perdido`), the connection closes, the system reconnects with a new session and the broker resends what the system missed. ```python con = pd.read_csv("vivo4_conexion.csv", parse_dates=["ts"]) cortes = con[con["evento"] != "conectado"] cortes = cortes[cortes["detalle"] != "fin de sesión"] filas = [] for (modo, dia), g in cortes.groupby(["modo", cortes["ts"].dt.date]): t_d = g.loc[g["evento"] == "desconectado", "ts"].iloc[0] t_r = g.loc[g["evento"] == "reconectado", "ts"].iloc[0] cuenta = ev[ev["modo"] == modo] rep = cuenta[(cuenta["ts_broker"] < t_r) & (cuenta["ts_recibido"] >= t_r)] seqs = range(rep["seq"].min(), rep["seq"].max() + 2) \ if len(rep) else [] filas.append(dict(modo=modo, dia=dia, segundos=int((t_r - t_d).total_seconds()), reenviados=len(rep), repetidos=int(rep["seq"].duplicated().sum()), huecos=[h for h in info[modo].huecos if h in seqs])) pd.DataFrame(filas) ``` ```output modo dia segundos reenviados repetidos huecos 0 paper 2025-09-23 42 0 0 [] 1 real 2025-10-21 92 16 2 [] 2 real 2025-11-03 137 4 0 [202] 3 real 2025-11-13 72 14 0 [] 4 real 2025-12-03 117 18 0 [] 5 real 2025-12-09 57 15 2 [] 6 real 2025-12-15 107 6 0 [519] ``` Drops of between 42 and 137 seconds, all at the most delicate time: right when the system sends the closing orders. The paper one fell on a day without orders and left nothing to resend. In those of the live account, on reconnecting the broker resends between 4 and 18 messages generated during the drop; in two of them it sends two fills twice (the repeated messages the sequencer discarded) and in another two it leaves one out (gaps 202 and 519). The header figure summarizes what each drop left behind: repeats, gaps, an out-of-order message and a duplicated order. The duplicated order is the most expensive one, and it is not the broker's fault. On November 13, the OMS sent `R00113` (sell 7 ORUS) at 20:50:00, when the heartbeat was no longer arriving and two seconds before the connection closed. On reconnecting, without the acknowledgment, it considered it lost and resent it as `R00114`: ```python dup = ord_.set_index("id_orden_cliente").loc[["R00113", "R00114"]] print(dup[["ts_envio", "lado", "cantidad", "motivo", "orden_origen"]]) t_rec = pd.Timestamp("2025-11-13 20:51:14", tz="UTC") m = ev[ev["id_orden_cliente"].isin(["R00113", "R00114"])] hora = lambda c: m[c].dt.strftime("%H:%M:%S.%f").str[:12] m.assign(broker=hora("ts_broker"), recibido=hora("ts_recibido"))[ ["seq", "id_orden_cliente", "evento", "broker", "recibido"]] ``` ```output ts_envio lado cantidad motivo orden_origen id_orden_cliente R00113 2025-11-13T20:50:00.246596Z venta 7 rebalanceo NaN R00114 2025-11-13T20:51:14.296596Z venta 7 reenvio_sin_respuesta R00113 seq id_orden_cliente evento broker recibido 628 308 R00114 aceptada 20:51:14.345 20:51:14.375 629 309 R00114 ejecutada 20:51:14.389 20:51:14.422 631 295 R00113 aceptada 20:50:00.290 20:51:14.586 634 298 R00113 ejecutada 20:50:00.343 20:51:14.706 ``` The broker had acknowledged and filled `R00113` at 20:50:00, but its messages did not arrive before the drop. The OMS resent at 20:51:14.296, right on reconnecting, and the messages for `R00113` arrived 290 ms **after** the resend. Result: two sales of 7 ORUS where the signal asked for one. The correct alternative is to ask before resending. With the state the broker had at the instant of the resend: ```python def estado_en_broker(id_cliente, t): """What the query of the order by its client id would return (simulated with the messages the broker had already generated).""" m = ev[(ev["id_orden_cliente"] == id_cliente) & (ev["ts_broker"] <= t)].sort_values("seq") return "unknown" if m.empty else m["evento"].iloc[-1] def al_reconectar(en_vuelo, t): for id_cliente in en_vuelo: estado = estado_en_broker(id_cliente, t) accion = "send with the same id" if ( estado == "unknown") else "do not resend" print(f"{id_cliente}: the broker says “{estado}” -> {accion}") al_reconectar(["R00113"], pd.Timestamp("2025-11-13 20:51:14.296", tz="UTC")) ``` ```output R00113: the broker says “ejecutada” -> do not resend ``` The query would have said `ejecutada` (filled) and there would have been no resend. If the broker did not know the order, the safe thing is to send it **with the same client identifier**: if the first one arrived, a broker that requires unique client identifiers rejects the second as a duplicate. It is the idempotency key, which lesson 3.7.4 showed with ccxt and which lesson 4.5.5 develops together with persistent state. The order of operations on reconnecting is then: ```mermaid flowchart TB R[Reconnect] --> RE[Receive resend] RE --> S[Sequence] S --> C[Query in-flight] C --> E[Request fills] E --> O[Resume trading] ``` Receive and sequence the resend, query the orders that still have no final state, request the day's fills to confirm the inferred ones and, only then, start sending orders again. Meanwhile, the signal can keep computing, but the OMS does not send. ## The heartbeat, with an injected clock The disconnection is detected by the **heartbeat**: the broker sends a message every so often even when there is nothing to say, and if three seconds pass without any, the connection is considered dead. Like the sequencer, the heartbeat monitor must not read the computer's clock: it receives the time as an argument, and that way it can be tested with made-up times: ```python class Latido: def __init__(self, limite=pd.Timedelta(seconds=3)): self.limite, self.ultimo, self.caido = limite, None, False def recibir(self, t): self.ultimo, self.caido = t, False def comprobar(self, t): if not self.caido and t - self.ultimo >= self.limite: self.caido = True return f"{t.time()} no heartbeat since {self.ultimo.time()}" return None t0 = pd.Timestamp("2025-12-15 20:49:50", tz="UTC") lat = Latido() for s in range(0, 6): # heartbeats every second lat.recibir(t0 + pd.Timedelta(seconds=s)) for s in range(6, 12): # the broker goes silent aviso = lat.comprobar(t0 + pd.Timedelta(seconds=s)) if aviso: print(aviso) ``` ```output 20:49:58 no heartbeat since 20:49:55 ``` The last heartbeat arrives at 20:49:55 and the monitor fires at 20:49:58: the time at which the connection log in the data records “sin latido del broker en 3 s” (no broker heartbeat for 3 s) in the drop of December 15. The same class, in production, receives the time from an event loop timer; in tests, whatever time you want, without waiting a single second. ## Exercises 1. Change the sequencer's timeout to 0.2 seconds. What happens to message 393 of the live account? Does the state machine handle it correctly? **Solution.** ```python s2 = Secuenciador(plazo=pd.Timedelta(seconds=0.2)) for m in ev[ev["modo"] == "real"].itertuples(): s2.llegar(m.ts_recibido, m) s2.vencer(pd.Timestamp.max.tz_localize("UTC")) print("gaps:", s2.huecos) orden_393 = [m.seq for _, m in s2.salida].index(393) print("delivery position of 393:", orden_393, "| before it:", [m.seq for _, m in s2.salida][orden_393 - 3:orden_393]) sal = pd.DataFrame([m._asdict() | {"t": t} for t, m in s2.salida]) ord_r = ord_[ord_["modo"] == "real"] fl = pd.concat([sal, eventos_oms(ord_r)], ignore_index=True) _, err2 = procesar_rec(fl.sort_values("t", kind="stable"), ord_r) print(err2) ``` ```output gaps: [202, 295, 296, 297, 298, 299, 300, 301, 302, 303, 304, 305, 306, 307, 393, 398, 399, 400, 401, 402, 403, 404, 519] delivery position of 393: 403 | before it: [402, 403, 404] ['R00162: “parcial” in “ejecutada”'] ``` With 0.2 seconds, the sequencer declares 23 gaps where there were only 2. On reconnecting on November 13, the broker first sends the messages of the new session (308 and 309, those of `R00114`) and then the resend (295 to 307): with such a short timeout, all thirteen are considered lost and delivered later as late arrivals (in the drop of December 3 the same happens with another eight). Most orders are unaffected, because their own messages are still in order relative to each other, but `R00162` is: message 393 arrives half a second late, when the order is already `ejecutada` (the rebuild by cumulative quantity had inferred its 857 shares as lost), and the machine rejects it. A timeout that is too short turns late messages into false gaps and forces reconciling what was not broken. 2. Check that, in the rebuilt book, no order in a final state has a leaves quantity and that the cumulative quantity of each order is the sum of its fills (including the inferred ones). **Solution.** ```python mal = [] for k, o in libro_r.items(): suma = sum(q for q, _ in o.ejecuciones.values()) if suma != o.acumulada: mal.append((k, "sum")) if o.estado == "ejecutada" and o.acumulada != o.cantidad: mal.append((k, "incomplete fill")) if o.estado not in FINALES: mal.append((k, o.estado)) print(len(libro_r), "orders;", len(mal), "violate them:", mal) ``` ```output 398 orders; 0 violate them: [] ``` All 398 orders satisfy the three invariants: all of them end in a final state, the filled ones have their complete quantity and the cumulative quantity is the sum of their fills. It is the test worth running at the end of each session, before the reconciliation with the broker. 3. A limit order is in `pendiente_cancelacion` and a fill arrives that completes it. What should the OMS do with the remainder it was planning to send as a market order? **Solution.** Nothing: there is no remainder. The fill completes the order (transition from `pendiente_cancelacion` to `ejecutada`, the race between the cancellation and the fill) and the broker will answer the cancellation with a rejection of the “too late to cancel” kind. The OMS must only send the remainder when it receives the **confirmation** of the cancellation, and for the quantity that confirmation reports as unfilled (ordered minus cumulative), not for the one it had counted itself. If it sent the remainder when requesting the cancellation, in a race it would buy twice. [^1]: Interactive Brokers, “TWS API: Placing Orders”, sections on order states and `orderStatus` (accessed on October 22, 2026; link in the resources). [^2]: Alpaca, “Orders at Alpaca”, “Order Lifecycle” section (accessed on October 22, 2026; link in the resources). ## Self-assessment 1. A fill arrives for an order whose acknowledgment has not arrived yet. What does the lesson's system do? - A. It holds it until the message with the previous sequence number (the acknowledgment) arrives and applies both in order - B. It rejects it: an order that has not been acknowledged cannot be filled - C. It applies it and forgets about the acknowledgment - D. It cancels the order as a precaution *Answer A: It holds it until the message with the previous sequence number (the acknowledgment) arrives and applies both in order.* The broker numbers its messages per account. If `seq` 8 arrives and 7 was expected, the sequencer stores 8 until 7 arrives or the timeout expires. In the data, 29 orders have some message that arrives before an earlier one (in 28, the fill before the acknowledgment); with the sequencer, none produces an invalid transition. 2. After a reconnection, the fill `E710000210.01` arrives again, with the same `seq`. What happens if it is counted? - A. Nothing: the broker already takes it into account - B. The internal position will have 41 ORAU too many and the cash 41 times its price too little - C. The order becomes canceled - D. The broker rejects it *Answer B: The internal position will have 41 ORAU too many and the cash 41 times its price too little.* A repeated message is not a new fill. It is recognized by its `seq` (already delivered) and, once more, by its `id_ejecucion` (already counted). In the data there are four repeated messages, all on reconnecting. 3. The message with `seq` 202 is missing and the next one for that order is its cancellation with `cantidad_acumulada` 112 and `precio_medio` 99.06. What can be known about the lost fill? - A. Nothing: you have to wait for the statement - B. That the order was not filled - C. That it was 112 shares at 99.06: the cumulative quantity goes from 0 to 112 without any fill received, and the average price of everything filled is 99.06 - D. That it was a duplicate *Answer C: That it was 112 shares at 99.06: the cumulative quantity goes from 0 to 112 without any fill received, and the average price of everything filled is 99.06.* Status messages carry the cumulative quantity and the average price. The difference between the cumulative and what was received is the lost quantity; the price comes from solving the average price. It matches the ground truth in the data (anomaly A04). Even so, the inferred fill is flagged and confirmed with the broker (lesson 4.5.5). 4. An order is sent, the connection drops and on reconnecting its acknowledgment has not arrived. What is the right thing to do? - A. Resend it with a new identifier - B. Consider it lost - C. Cancel all orders and shut down the system - D. Wait for the broker to finish resending messages or query the order's state by its client identifier; only if the broker does not know it, send it, with the same identifier *Answer D: Wait for the broker to finish resending messages or query the order's state by its client identifier; only if the broker does not know it, send it, with the same identifier.* No answer does not mean no order. In the data, the OMS resent `R00113` as `R00114` at the very moment of the reconnection and the broker filled both: 7 ORUS sold too many. At that time, the broker already had `R00113` filled. 5. In the live account, what fraction of limit orders were filled completely before being canceled after seven minutes? - A. All of them - B. None - C. About 10% - D. Slightly more than half: 39 of 73; 22 were partially filled and 12 not at all *Answer D: Slightly more than half: 39 of 73; 22 were partially filled and 12 not at all.* A passive limit order has no guaranteed fill. The remainder of the 34 that were not filled was sent as a market order (`resto_de_limite`), with its cost (lesson 4.5.3). ## Key points - An order is a state machine: sent, acknowledged, partial, pending cancel and three final states (filled, canceled, rejected) that cannot be left. - The invariants are as important as the transitions: the cumulative quantity is the sum of the fills and never goes down, the leaves quantity is what was ordered minus the cumulative, each `id_ejecucion` counts once. - The broker's messages are ordered by their sequence number, not by order of arrival: hold what comes early, discard what is repeated and, if a gap is not filled within a timeout, declare it. - A gap is not the end: the cumulative quantity and the average price of the next message make it possible to rebuild the lost fill, which is flagged as inferred until it is confirmed. - On reconnecting, state first and orders afterwards: resending without querying duplicated an order in the data. - The sequencer's timeout and the heartbeat's are measured with the message clock, not the computer's: the result does not depend on the machine's load. ## Resources ### Research - [Finite-State Machines for Quantitative Trading Execution](https://pythonparatrading.com/en/finite-state-machines-trading-python/) · Research-0029 · 47 min ### Documentation - [Interactive Brokers, TWS API: order states (orderStatus) and duplicate messages](https://interactivebrokers.github.io/tws-api/order_submission.html) · interactivebrokers.github.io - [Alpaca: orders and their life cycle (order states)](https://docs.alpaca.markets/docs/orders-at-alpaca) · docs.alpaca.markets - [FIX Trading Community: FIX protocol standards](https://www.fixtrading.org/standards/) · fixtrading.org ### Data - [Level 4 data: broker messages in order of arrival (CSV)](https://pythonparatrading.com/curso/datos/nivel-4/vivo4_eventos_broker.csv) · Download · vivo4_eventos_broker.csv - [Level 4 data: broker connection log (CSV)](https://pythonparatrading.com/curso/datos/nivel-4/vivo4_conexion.csv) · Download · vivo4_conexion.csv Previous lesson: [4.5.1 · Live system architecture: from data to reconciliation](https://pythonparatrading.com/en/python-trading-course/level-4/live-trading-system-architecture/) Next lesson: [4.5.3 · Paper trading, shadow trading and slippage monitoring](https://pythonparatrading.com/en/python-trading-course/level-4/paper-trading-shadow-trading-and-slippage/)