--- title: "1.3.6 Brokers as market data sources" description: "Interactive Brokers with ib_async, MetaTrader 5 and OANDA as data sources: architecture, pacing limits, market data subscriptions and response formats." url: "https://pythonparatrading.com/en/python-trading-course/level-1/brokers-as-market-data-sources/" lang: "en" image: "https://pythonparatrading.com/og/en/python-trading-course/level-1/brokers-as-market-data-sources.png" --- # 1.3.6 · Brokers as market data sources > The broker you trade with already gives you data: history, quotes and, sometimes, the order book. They are the prices you execute at, but they come with their own rules: an intermediate program, strict pacing limits and market data subscriptions. This lesson walks through them with Interactive Brokers, MetaTrader 5 and OANDA, and simulates the download pace without connecting to anything. - Course: [Level 1: Financial data, markets and analysis](https://pythonparatrading.com/en/python-trading-course/level-1/) - Module: 1.3 · Market data sources - Duration: 50 min - Prerequisites: [1.2.4 · Forex: pairs, pips, lots, rollover and CFDs](https://pythonparatrading.com/en/python-trading-course/level-1/forex-pairs-pips-lots-and-rollover/), [1.3.1 · Types of market data: EOD, bars, ticks, order book and reference](https://pythonparatrading.com/en/python-trading-course/level-1/types-of-market-data/), [1.3.3 · Retail and prosumer market data APIs](https://pythonparatrading.com/en/python-trading-course/level-1/retail-and-prosumer-market-data-apis/) ## By the end of this lesson - Explain the architecture of the Interactive Brokers API (TWS or IB Gateway, ports, `clientId`) and what `ib_async` is relative to `ib_insync`. - Apply the IBKR historical data pacing limits and estimate how long a download takes under them. - Read the candle format of OANDA v20 and of MetaTrader 5 `copy_rates_*` and turn it into a DataFrame in UTC. - Decide when to use the broker as a data source and when to use an independent provider. ## The broker is also a provider Any broker with an API provides data in addition to execution: real-time quotes, historical candles and, in some cases, order book depth. For a system that trades with that broker, this data has an advantage no other provider can match: they are **the prices of the venue where execution happens**, with its bid-ask spread, its trading hours and its commissions. A currency system that computes signals with its broker's prices and executes there does not have to worry about the data source and the execution market not matching. The drawbacks are the opposite: short history, request limits designed for trading rather than for downloading years, and no delisted securities (the broker only offers what can be traded today). This lesson walks through three brokers with APIs that are widely used from Python: Interactive Brokers, MetaTrader 5 (a platform offered by many brokers) and OANDA. ## Interactive Brokers: TWS, Gateway and `ib_async` The Interactive Brokers (IBKR) API has an architecture that surprises people the first time: the program does not connect to the IBKR servers, but to an application running on the same machine, **TWS** (Trader Workstation, the desktop platform) or **IB Gateway** (the same connection without a graphical interface, lighter for a server). That application holds the session with IBKR and exposes a local port: ```mermaid flowchart LR P[Your program] --> A[ib_async] A --> G[TWS or Gateway] G --> S[IBKR servers] ``` | Application | Live account | Paper account | |---|---|---| | TWS | 7496 | 7497 | | IB Gateway | 4001 | 4002 | These are the default ports according to the IBKR documentation[^1]; each application lets you change them. Each connection also carries a `clientId`: several programs can connect to the same TWS at once if they use different identifiers. Connecting to the paper account (7497 or 4002) is the safe way to test: the same data and the same API, without real money. The official IBKR library for Python (`ibapi`) is low level: callbacks and messages. The one almost everybody uses is **`ib_async`**, the continuation of `ib_insync`. According to its README, Ewald de Wit created it in 2017 and maintained it until he passed away in early 2024; the maintainers renamed it `ib_async` under a new GitHub organization (`ib-api-reloaded`) because they could no longer modify the original repositories. It requires Python 3.10 or later and does not need the `ibapi` package[^2]. `ib_insync` code works almost unchanged by switching the import. ```python skip title="ibkr_historico.py" from ib_async import IB, Stock, util # uv add ib_async ib = IB() ib.connect("127.0.0.1", 7497, clientId=1) # TWS, paper account contrato = Stock("AAPL", "SMART", "USD") barras = ib.reqHistoricalData( contrato, endDateTime="", durationStr="30 D", barSizeSetting="1 hour", whatToShow="MIDPOINT", useRTH=True, ) df = util.df(barras) ib.disconnect() ``` `durationStr` is how far back to go (`"30 D"`), `barSizeSetting` the bar size, `whatToShow` which price (`TRADES`, `MIDPOINT`, `BID`, `ASK`, `BID_ASK`…) and `useRTH=True` restricts to the regular session. `util.df` converts the list of bars into a DataFrame. ### Market data subscriptions The API working does not mean there is data. According to the IBKR documentation, except for delayed data, receiving market data through the API **requires a paid subscription** to the corresponding market; to request delayed data without a subscription, call `ib.reqMarketDataType(3)` before requesting quotes. Each user also has a number of **market data lines** (100 by default) shared by the platform and the API: each symbol followed in real time takes up one[^3]. ## Pacing limits: what decides how long a download takes IBKR carefully limits what it is asked for. There are two kinds of limit: - **General.** The maximum number of requests per second is the number of market data lines divided by two: with the default 100 lines, **50 requests per second**[^3]. - **Small-bar historical data.** For bars of 30 seconds or less, the documentation of the previous version of the API (marked as deprecated, but the only one that publishes the rules) considers a pacing violation: repeating an **identical request within 15 seconds**; making **6 or more** requests for the same contract, exchange and data type **within 2 seconds**; and making **more than 60 requests within any 10-minute period**. With `BID_ASK`, each request counts twice. For bars of 1 minute or more, those rules were lifted, although the server still throttles anyone who asks for too much[^4]. In addition, each request has a maximum duration depending on the bar size: the current documentation gives, for example, 2,000 seconds for 1-second bars and 86,400 seconds (one day) for 5-second bars[^5]. With those rules you can simulate a downloader that never violates them, with a virtual clock: ```python import math from collections import deque class RitmoIBKR: """Rules for bars of 30 s or less (previous documentation).""" def __init__(self): self.hechas = deque() # (t, contract, key, weight) def espera(self, t, contrato, clave, peso=1): while self.hechas and self.hechas[0][0] <= t - 600: self.hechas.popleft() e = 0.0 # more than 60 in 10 minutes (BID_ASK weighs 2) usadas = sum(h[3] for h in self.hechas) if usadas + peso > 60: acum = 0 for h in self.hechas: acum += h[3] if usadas - acum + peso <= 60: e = max(e, h[0] + 600 - t) break # 6 or more for the same contract in 2 seconds mismo = [h[0] for h in self.hechas if h[1] == contrato] if len(mismo) >= 5: e = max(e, mismo[-5] + 2 - t) # identical within 15 seconds for h in self.hechas: if h[2] == clave: e = max(e, h[0] + 15 - t) return max(e, 0.0) def registra(self, t, contrato, clave, peso=1): self.hechas.append((t, contrato, clave, peso)) def descarga(n_peticiones, peso=1, latencia=0.5): ritmo, t, horas = RitmoIBKR(), 0.0, [] for i in range(n_peticiones): clave = ("NOVA", i) # each request, a different stretch while (e := ritmo.espera(t, "NOVA", clave, peso)) > 0: t += e ritmo.registra(t, "NOVA", clave, peso) horas.append(t) t += latencia return horas horas = descarga(252) # one day per request print(f"252 requests: {horas[-1] / 60:.1f} minutes") print("first:", [round(h, 1) for h in horas[:7]]) ``` ```output 252 requests: 40.1 minutes first: [0.0, 0.5, 1.0, 1.5, 2.0, 2.5, 3.0] ``` The simulation assumes each response takes half a second. At that pace, four requests go out every two seconds, so the 6-in-2-seconds rule is never triggered (with faster responses it would be). The binding rule is 60 in 10 minutes: the downloader makes 60 requests in half a minute and waits out the rest of the window. Four complete windows and 12 more requests: 40.1 minutes (figure at the top). Now the calculation that matters: how long one year of bars of each size takes for **one** contract, with the maximum duration of each request: ```python sesion = int(6.5 * 3600) # 23,400 s of regular session casos = { "5 s, TRADES": (86_400, 1), "5 s, BID_ASK": (86_400, 2), "1 s, TRADES": (2_000, 1), "1 s, BID_ASK": (2_000, 2), } for nombre, (max_seg, peso) in casos.items(): por_dia = math.ceil(sesion / max_seg) n = 252 * por_dia h = descarga(n, peso=peso) print(f"{nombre:13} {n:5} requests {h[-1] / 3600:5.2f} h") ``` ```output 5 s, TRADES 252 requests 0.67 h 5 s, BID_ASK 252 requests 1.33 h 1 s, TRADES 3024 requests 8.34 h 1 s, BID_ASK 3024 requests 16.67 h ``` *Fig. — Download time for one year of small bars of one contract under IBKR pacing rules, by bar size and data type.* Horizontal bars: 5-second bars with TRADES, 0.67 hours; with BID_ASK, 1.33; 1-second bars with TRADES, 8.34 hours, highlighted; with BID_ASK, 16.67. One year of 5-second bars fits in 252 requests and about 40 minutes; the same year in 1-second bars needs 12 requests per day (23,400 / 2,000, rounded up), 3,024 in total and more than 8 hours, or twice that if bid and ask are requested. For a hundred contracts, multiply. That is why long histories of small bars are not downloaded from the broker but from a provider (lessons 1.3.3 and 1.3.4), and the broker is kept for recent data and real time. ### What happens if the rules are not respected A loop that fires off requests without a limiter does not know it is violating the rules until IBKR replies with a pacing violation error, and with repeated violations the connection may end up being cut. With the same class you can count how many requests in a naive loop would be violations: ```python def violaciones(n, latencia=0.5): ritmo, t, malas = RitmoIBKR(), 0.0, 0 for i in range(n): clave = ("NOVA", i) if ritmo.espera(t, "NOVA", clave) > 0: malas += 1 ritmo.registra(t, "NOVA", clave) t += latencia return malas print(violaciones(252), "of 252 requests outside the rules") ``` ```output 192 of 252 requests outside the rules ``` The first 60 fit in the 10-minute window; the next 192 violate it. The loop finishes in two minutes, but it would have received 192 errors (and, long before that, probably a disconnection). The solution is not to retry faster, but to wait for what the rules require before asking. ## MetaTrader 5: the terminal as a data server MetaTrader 5 is a trading platform offered by many forex and CFD brokers. Its Python package (`MetaTrader5` on PyPI) does not talk to any server: it connects to an **installed MetaTrader 5 terminal** on the same machine, which is the one holding the session with the broker. `initialize()` looks for the terminal or receives the path to its executable; the wheels published on PyPI are for Windows only (`win_amd64`)[^6]. ```python skip title="mt5_velas.py" from datetime import datetime, timezone import MetaTrader5 as mt5 # Windows only, with the terminal if not mt5.initialize(): raise RuntimeError(mt5.last_error()) desde = datetime(2025, 6, 2, tzinfo=timezone.utc) velas = mt5.copy_rates_from("EURUSD", mt5.TIMEFRAME_H1, desde, 500) ultimo = mt5.symbol_info_tick("EURUSD") mt5.shutdown() ``` `copy_rates_from` returns the candles before the given date as a **numpy array** with named fields: `time`, `open`, `high`, `low`, `close`, `tick_volume`, `spread` and `real_volume`. The documentation warns about two things: the terminal stores the open time in **UTC** (which is why the date must be created in UTC), and the number of candles it can return is limited by the terminal's “Max. bars in chart” setting[^7]. With an array of that shape, built from the course's synthetic EURUSD: ```python import numpy as np import pandas as pd eur = pd.read_parquet("eurusd_1h.parquet").set_index("ts") h1 = eur.loc["2025-06-02", "mid"] tipo = np.dtype([("time", " B[Broker data] Q -- long backtest --> P[Provider] Q -- equity universe --> P B --> C[Cross-check] P --> C ``` | Use | Broker? | Why | |---|---|---| | Real-time signals and execution | Yes | They are the prices where execution happens | | Costs of a forex or CFD system | Yes | Its bid-ask spread and its financing are the ones you pay | | Long history of small bars | No | Pacing limits and maximum duration per request | | Equity universe for a backtest | No | It only has what trades today: survivorship bias (lesson 1.3.8) | | Validating a provider | Yes, as a cross-check | Comparing a few days of prices from both detects differences in time stamps or adjustments | The usual combination is a provider for history and research and the broker for live trading, with a periodic check that both tell the same story on the days where they overlap. ## Exercises 1. With `descarga`, how long does one year of 5-second bars with `BID_ASK` take for 10 contracts, if they are requested one after another with the same limiter? (Hint: the 10-minute rules count all requests, not per contract.) **Solution.** ```python h = descarga(10 * 252, peso=2) print(f"{len(h)} requests: {h[-1] / 3600:.1f} hours") ``` ```output 2520 requests: 13.8 hours ``` 2,520 requests that count double are 5,040 units out of the 60 per 10-minute window: about 84 windows, almost 14 hours. With ten contracts the download is no longer “a moment before starting” but an overnight job that has to be resumable if it is interrupted (lesson 1.4.8). 2. In the OANDA response, compute the average bid-ask spread of the complete candles in pips using the close, and compare it with that of the open. Why do they not have to match? **Solution.** ```python oanda["spread_ap"] = 1e4 * (oanda["ask_o"] - oanda["bid_o"]) oanda[["spread_ap", "spread_pips"]].mean().round(2) ``` ```output spread_ap 0.53 spread_pips 0.33 dtype: float64 ``` The opening and closing spreads are two snapshots at two different moments of the candle; the spread changes with the session (wider in Asia and around the rollover, tighter during the London–New York overlap, lesson 1.2.4). For a backtest, the spread that matters is the one at the moment of the order: if the system trades at the candle close, the closing one. 3. Your system trades EURUSD with 4-hour candles at a broker with MetaTrader 5 and you want 15 years of history to test it. Where would you get the data from and what would you check? **Solution.** The terminal's history is limited by “Max. bars in chart” and by whatever the broker has stored; for 15 years, a forex provider with long history is a better fit (lesson 1.3.3), using the broker for recent years and live trading. Checks: that times are UTC in both sources (MT5 stores the open in UTC, but many brokers display server time in another time zone), that the 4-hour candles start at the same hours, the price difference on overlapping days (mid price versus bid), and that the bid-ask spread in the backtest is the broker's (`spread`), not the provider's. [^1]: IBKR: [Configure the Connecting Socket Port](https://www.interactivebrokers.com/docs/excel/dde/dde-socket-bridge/configuring-the-socket-bridge/configure-the-connecting-socket-port), accessed on October 21, 2026. [^2]: [ib_async](https://github.com/ib-api-reloaded/ib_async) README, accessed on October 21, 2026; version 2.1.0 on [PyPI](https://pypi.org/project/ib_async/). [^3]: IBKR: [Pacing Limitations](https://www.interactivebrokers.com/docs/tws-api/doc/pacing-limitations/introduction), [Market Data Lines](https://www.interactivebrokers.com/docs/general/market-data-subscriptions/market-data-lines/introduction) and [Market Data](https://www.interactivebrokers.com/docs/tws-api/doc/md/introduction), accessed on October 21, 2026. [^4]: IBKR, previous API documentation: [Historical Data Limitations](https://interactivebrokers.github.io/tws-api/historical_limitations.html) (“This documentation is now deprecated”), accessed on October 21, 2026. [^5]: IBKR: [Historical Bars (OHLC)](https://www.interactivebrokers.com/docs/tws-api/doc/md/historical-bars), table of maximum duration by bar size. [^6]: [MetaTrader 5: Python Integration](https://www.mql5.com/en/docs/python_metatrader5) and [initialize](https://www.mql5.com/en/docs/python_metatrader5/mt5initialize_py); `win_amd64` wheels of version 5.0.6231 on [PyPI](https://pypi.org/project/MetaTrader5/). [^7]: [MetaTrader 5: copy_rates_from](https://www.mql5.com/en/docs/python_metatrader5/mt5copyratesfrom_py). [^8]: OANDA v20: [Development Guide](https://developer.oanda.com/rest-live-v20/development-guide/), [Best Practices](https://developer.oanda.com/rest-live-v20/best-practices/) and [Troubleshooting](https://developer.oanda.com/rest-live-v20/troubleshooting-errors/), accessed on October 21, 2026. [^9]: OANDA v20: [Pricing](https://developer.oanda.com/rest-live-v20/pricing-ep/) (candles endpoint), [Instrument](https://developer.oanda.com/rest-live-v20/instrument-df/) (`Candlestick` schema) and [Primitives](https://developer.oanda.com/rest-live-v20/primitives-df/) (`DateTime`). ## Self-assessment 1. You connect `ib_async` to port 7497. What are you connecting to, according to the IBKR port table? - A. TWS with the live account - B. IB Gateway with the live account - C. TWS with the paper (simulated) account - D. Directly to the IBKR servers *Answer C: TWS with the paper (simulated) account.* The default ports are 7496 (TWS live), 7497 (TWS paper), 4001 (Gateway live) and 4002 (Gateway paper). The API never talks directly to the servers: it always goes through TWS or IB Gateway on your machine. 2. What is the relationship between `ib_async` and `ib_insync`? - A. `ib_async` is the continuation of `ib_insync`, renamed under a new GitHub organization after its author passed away in 2024 - B. They are unrelated rival libraries - C. `ib_async` is the official IBKR library - D. `ib_insync` is the new version *Answer A: `ib_async` is the continuation of `ib_insync`, renamed under a new GitHub organization after its author passed away in 2024.* Its README explains that Ewald de Wit created `ib_insync` in 2017 and maintained it until he passed away in early 2024; the maintainers renamed it because they could no longer modify the original repository. 3. Under the IBKR pacing rules for bars of 30 seconds or less, how many historical data requests can you make in 10 minutes? - A. Unlimited - B. 600 - C. 6 - D. 60 *Answer D: 60.* More than 60 requests within any 10-minute period is a pacing violation. In addition, you cannot repeat an identical request within 15 seconds or make 6 or more requests for the same contract within 2 seconds. With `BID_ASK`, each request counts twice. 4. OANDA returns a candle with `"complete": false`. What does it mean? - A. That prices are missing - B. That it is the current candle: its end is in the future and its values will change - C. That the market was closed - D. That there was an error *Answer B: That it is the current candle: its end is in the future and its values will change.* The documentation defines a complete candle as one whose end is not in the future. An incomplete candle is not stored as history. 5. What is the main advantage of using the broker as a data source? - A. That it is always free and unlimited - B. That it has more history than any provider - C. That they are the prices of the venue where you execute, with its bid-ask spread and its trading hours, and no other contract is needed - D. That it has no survivorship bias *Answer C: That they are the prices of the venue where you execute, with its bid-ask spread and its trading hours, and no other contract is needed.* For a system that trades with that broker, its prices are the executable ones. In exchange, the history is usually shorter, the limits stricter and there are no delisted securities (lesson 1.3.8). ## Key points - The IBKR API does not talk to the servers: it talks to TWS or IB Gateway on your machine, through a port (7496/7497 for TWS, 4001/4002 for Gateway) and with a `clientId`. - `ib_async` is the continuation of `ib_insync`; it requires Python 3.10 or later and does not need the IBKR `ibapi` package. - Data through the API requires a paid market data subscription (except delayed data), and requests are limited: 50 per second in general, plus specific rules for small-bar historical data. - The pacing limit decides how long a historical download takes: one year of 1-second bars for one contract is about 3,000 requests and more than 8 hours. - MetaTrader 5 delivers numpy arrays with UTC times from an installed terminal (wheels exist only for Windows); OANDA, JSON with prices as text, RFC 3339 times and a `complete` flag. - The broker is the best source for the prices you execute at, and a poor source of long history and of bias-free universes. ## Resources ### Documentation - [ib_async (GitHub, ib-api-reloaded)](https://github.com/ib-api-reloaded/ib_async) · github.com - [IBKR TWS API: Pacing Limitations](https://www.interactivebrokers.com/docs/tws-api/doc/pacing-limitations/introduction) · interactivebrokers.com - [IBKR TWS API (previous version): Historical Data Limitations](https://interactivebrokers.github.io/tws-api/historical_limitations.html) · interactivebrokers.github.io - [MetaTrader 5: Python integration (MQL5 Reference)](https://www.mql5.com/en/docs/python_metatrader5) · mql5.com - [OANDA v20 REST: Development Guide](https://developer.oanda.com/rest-live-v20/development-guide/) · developer.oanda.com ### Data - [Level 1 data: hourly synthetic EURUSD (Parquet)](https://pythonparatrading.com/curso/datos/nivel-1/eurusd_1h.parquet) · Download · eurusd_1h.parquet Previous lesson: [1.3.5 · Crypto and on-chain data](https://pythonparatrading.com/en/python-trading-course/level-1/crypto-and-on-chain-data/) Next lesson: [1.3.7 · Market data licensing and terms of use](https://pythonparatrading.com/en/python-trading-course/level-1/market-data-licensing-and-terms-of-use/)