Brokers as market data sources.

The broker is also a provider

Any broker with an API provides data in addition to execution: real-time quotes, historical candles and, in some cases, order book depth. For a system that trades with that broker, this data has an advantage no other provider can match: they are the prices of the venue where execution happens, with its bid-ask spread, its trading hours and its commissions. A currency system that computes signals with its broker's prices and executes there does not have to worry about the data source and the execution market not matching.

The drawbacks are the opposite: short history, request limits designed for trading rather than for downloading years, and no delisted securities (the broker only offers what can be traded today). This lesson walks through three brokers with APIs that are widely used from Python: Interactive Brokers, MetaTrader 5 (a platform offered by many brokers) and OANDA.

Interactive Brokers: TWS, Gateway and ib_async

The Interactive Brokers (IBKR) API has an architecture that surprises people the first time: the program does not connect to the IBKR servers, but to an application running on the same machine, TWS (Trader Workstation, the desktop platform) or IB Gateway (the same connection without a graphical interface, lighter for a server). That application holds the session with IBKR and exposes a local port:

flowchart LR
  P[Your program] --> A[ib_async]
  A --> G[TWS or Gateway]
  G --> S[IBKR servers]

These are the default ports according to the IBKR documentation11Note 1 · IBKR: Configure the Connecting Socket Port, accessed on October 21, 2026.; each application lets you change them. Each connection also carries a clientId: several programs can connect to the same TWS at once if they use different identifiers. Connecting to the paper account (7497 or 4002) is the safe way to test: the same data and the same API, without real money.

The official IBKR library for Python (ibapi) is low level: callbacks and messages. The one almost everybody uses is ib_async, the continuation of ib_insync. According to its README, Ewald de Wit created it in 2017 and maintained it until he passed away in early 2024; the maintainers renamed it ib_async under a new GitHub organization (ib-api-reloaded) because they could no longer modify the original repositories. It requires Python 3.10 or later and does not need the ibapi package22Note 2 · ib_async README, accessed on October 21, 2026; version 2.1.0 on PyPI.. ib_insync code works almost unchanged by switching the import.

from ib_async import IB, Stock, util     # uv add ib_async

ib = IB()
ib.connect("127.0.0.1", 7497, clientId=1)   # TWS, paper account

contrato = Stock("AAPL", "SMART", "USD")
barras = ib.reqHistoricalData(
    contrato, endDateTime="", durationStr="30 D",
    barSizeSetting="1 hour", whatToShow="MIDPOINT", useRTH=True,
)
df = util.df(barras)
ib.disconnect()

durationStr is how far back to go ("30 D"), barSizeSetting the bar size, whatToShow which price (TRADES, MIDPOINT, BID, ASK, BID_ASK…) and useRTH=True restricts to the regular session. util.df converts the list of bars into a DataFrame.

Market data subscriptions

The API working does not mean there is data. According to the IBKR documentation, except for delayed data, receiving market data through the API requires a paid subscription to the corresponding market; to request delayed data without a subscription, call ib.reqMarketDataType(3) before requesting quotes. Each user also has a number of market data lines (100 by default) shared by the platform and the API: each symbol followed in real time takes up one33Note 3 · IBKR: Pacing Limitations, Market Data Lines and Market Data, accessed on October 21, 2026..

Pacing limits: what decides how long a download takes

IBKR carefully limits what it is asked for. There are two kinds of limit:

• General. The maximum number of requests per second is the number of market data lines divided by two: with the default 100 lines, 50 requests per second³.
• Small-bar historical data. For bars of 30 seconds or less, the documentation of the previous version of the API (marked as deprecated, but the only one that publishes the rules) considers a pacing violation: repeating an identical request within 15 seconds; making 6 or more requests for the same contract, exchange and data type within 2 seconds; and making more than 60 requests within any 10-minute period. With BID_ASK, each request counts twice. For bars of 1 minute or more, those rules were lifted, although the server still throttles anyone who asks for too much44Note 4 · IBKR, previous API documentation: Historical Data Limitations (“This documentation is now deprecated”), accessed on October 21, 2026..

In addition, each request has a maximum duration depending on the bar size: the current documentation gives, for example, 2,000 seconds for 1-second bars and 86,400 seconds (one day) for 5-second bars55Note 5 · IBKR: Historical Bars (OHLC), table of maximum duration by bar size.. With those rules you can simulate a downloader that never violates them, with a virtual clock:

import math
from collections import deque

class RitmoIBKR:
    """Rules for bars of 30 s or less (previous documentation)."""

    def __init__(self):
        self.hechas = deque()          # (t, contract, key, weight)

    def espera(self, t, contrato, clave, peso=1):
        while self.hechas and self.hechas[0][0] <= t - 600:
            self.hechas.popleft()
        e = 0.0
        # more than 60 in 10 minutes (BID_ASK weighs 2)
        usadas = sum(h[3] for h in self.hechas)
        if usadas + peso > 60:
            acum = 0
            for h in self.hechas:
                acum += h[3]
                if usadas - acum + peso <= 60:
                    e = max(e, h[0] + 600 - t)
                    break
        # 6 or more for the same contract in 2 seconds
        mismo = [h[0] for h in self.hechas if h[1] == contrato]
        if len(mismo) >= 5:
            e = max(e, mismo[-5] + 2 - t)
        # identical within 15 seconds
        for h in self.hechas:
            if h[2] == clave:
                e = max(e, h[0] + 15 - t)
        return max(e, 0.0)

    def registra(self, t, contrato, clave, peso=1):
        self.hechas.append((t, contrato, clave, peso))

def descarga(n_peticiones, peso=1, latencia=0.5):
    ritmo, t, horas = RitmoIBKR(), 0.0, []
    for i in range(n_peticiones):
        clave = ("NOVA", i)        # each request, a different stretch
        while (e := ritmo.espera(t, "NOVA", clave, peso)) > 0:
            t += e
        ritmo.registra(t, "NOVA", clave, peso)
        horas.append(t)
        t += latencia
    return horas

horas = descarga(252)                  # one day per request
print(f"252 requests: {horas[-1] / 60:.1f} minutes")
print("first:", [round(h, 1) for h in horas[:7]])

252 requests: 40.1 minutes
first: [0.0, 0.5, 1.0, 1.5, 2.0, 2.5, 3.0]

The simulation assumes each response takes half a second. At that pace, four requests go out every two seconds, so the 6-in-2-seconds rule is never triggered (with faster responses it would be). The binding rule is 60 in 10 minutes: the downloader makes 60 requests in half a minute and waits out the rest of the window. Four complete windows and 12 more requests: 40.1 minutes (figure at the top).

Now the calculation that matters: how long one year of bars of each size takes for one contract, with the maximum duration of each request:

sesion = int(6.5 * 3600)              # 23,400 s of regular session
casos = {
    "5 s, TRADES": (86_400, 1),
    "5 s, BID_ASK": (86_400, 2),
    "1 s, TRADES": (2_000, 1),
    "1 s, BID_ASK": (2_000, 2),
}
for nombre, (max_seg, peso) in casos.items():
    por_dia = math.ceil(sesion / max_seg)
    n = 252 * por_dia
    h = descarga(n, peso=peso)
    print(f"{nombre:13} {n:5} requests  {h[-1] / 3600:5.2f} h")

5 s, TRADES     252 requests   0.67 h
5 s, BID_ASK    252 requests   1.33 h
1 s, TRADES    3024 requests   8.34 h
1 s, BID_ASK   3024 requests  16.67 h

One year of 5-second bars fits in 252 requests and about 40 minutes; the same year in 1-second bars needs 12 requests per day (23,400 / 2,000, rounded up), 3,024 in total and more than 8 hours, or twice that if bid and ask are requested. For a hundred contracts, multiply. That is why long histories of small bars are not downloaded from the broker but from a provider (lessons 1.3.3 and 1.3.4), and the broker is kept for recent data and real time.

What happens if the rules are not respected

A loop that fires off requests without a limiter does not know it is violating the rules until IBKR replies with a pacing violation error, and with repeated violations the connection may end up being cut. With the same class you can count how many requests in a naive loop would be violations:

def violaciones(n, latencia=0.5):
    ritmo, t, malas = RitmoIBKR(), 0.0, 0
    for i in range(n):
        clave = ("NOVA", i)
        if ritmo.espera(t, "NOVA", clave) > 0:
            malas += 1
        ritmo.registra(t, "NOVA", clave)
        t += latencia
    return malas

print(violaciones(252), "of 252 requests outside the rules")

192 of 252 requests outside the rules

The first 60 fit in the 10-minute window; the next 192 violate it. The loop finishes in two minutes, but it would have received 192 errors (and, long before that, probably a disconnection). The solution is not to retry faster, but to wait for what the rules require before asking.

MetaTrader 5: the terminal as a data server

MetaTrader 5 is a trading platform offered by many forex and CFD brokers. Its Python package (MetaTrader5 on PyPI) does not talk to any server: it connects to an installed MetaTrader 5 terminal on the same machine, which is the one holding the session with the broker. initialize() looks for the terminal or receives the path to its executable; the wheels published on PyPI are for Windows only (win_amd64)66Note 6 · MetaTrader 5: Python Integration and initialize; win_amd64 wheels of version 5.0.6231 on PyPI..

from datetime import datetime, timezone

import MetaTrader5 as mt5            # Windows only, with the terminal

if not mt5.initialize():
    raise RuntimeError(mt5.last_error())
desde = datetime(2025, 6, 2, tzinfo=timezone.utc)
velas = mt5.copy_rates_from("EURUSD", mt5.TIMEFRAME_H1, desde, 500)
ultimo = mt5.symbol_info_tick("EURUSD")
mt5.shutdown()

copy_rates_from returns the candles before the given date as a numpy array with named fields: time, open, high, low, close, tick_volume, spread and real_volume. The documentation warns about two things: the terminal stores the open time in UTC (which is why the date must be created in UTC), and the number of candles it can return is limited by the terminal's “Max. bars in chart” setting77Note 7 · MetaTrader 5: copy_rates_from.. With an array of that shape, built from the course's synthetic EURUSD:

import numpy as np
import pandas as pd

eur = pd.read_parquet("eurusd_1h.parquet").set_index("ts")
h1 = eur.loc["2025-06-02", "mid"]
tipo = np.dtype([("time", "<i8"), ("open", "<f8"), ("high", "<f8"),
                 ("low", "<f8"), ("close", "<f8"),
                 ("tick_volume", "<u8"), ("spread", "<i4"),
                 ("real_volume", "<u8")])
rates = np.zeros(len(h1) - 1, dtype=tipo)
rates["time"] = h1.index[:-1].astype("int64") // 10**6   # seconds
rates["open"] = h1.values[:-1]
rates["close"] = h1.values[1:]
rates["high"] = np.maximum(rates["open"], rates["close"])
rates["low"] = np.minimum(rates["open"], rates["close"])

velas = pd.DataFrame(rates)
velas["time"] = pd.to_datetime(velas["time"], unit="s", utc=True)
velas.set_index("time")[["open", "close"]].head(3)

                               open     close
time
2025-06-02 00:00:00+00:00  1.172300  1.172725
2025-06-02 01:00:00+00:00  1.172725  1.173615
2025-06-02 02:00:00+00:00  1.173615  1.173730

The time field comes in seconds since 1970 (not in milliseconds as in ccxt, lesson 1.3.5), and pd.DataFrame(rates) keeps the field names. Here the high and the low come from the open and the close because the course EURUSD is an hourly quote, not candles; in the terminal, each candle carries its own. Two details specific to this source: tick_volume is the number of price changes, not a traded volume (spot forex has no consolidated tape, lesson 1.2.4), and spread is the bid-ask spread in the broker's points, which varies from one broker to another.

OANDA: a REST API with bid, ask and mid candles

OANDA has a REST API, v20, with two environments: practice (api-fxpractice.oanda.com) and live (api-fxtrade.oanda.com), and separate streaming servers (stream-fxpractice.oanda.com, stream-fxtrade.oanda.com). Authentication is a token in the Authorization: Bearer header. Its limits, according to the development guide: 120 requests per second on the REST API (above that, 429), 20 active streams and no more than 2 new connections per second; the best practices page recommends not exceeding 100 per second on an already open connection88Note 8 · OANDA v20: Development Guide, Best Practices and Troubleshooting, accessed on October 21, 2026..

Candles are requested at GET /v3/accounts/{accountID}/instruments/{instrument}/candles, with granularity (default S5, five seconds), count (default 500, maximum 5,000) and price, which combines M (mid), B (bid) and A (ask)99Note 9 · OANDA v20: Pricing (candles endpoint), Instrument (Candlestick schema) and Primitives (DateTime)..

import os
from urllib.parse import urlencode
from urllib.request import Request, urlopen

cuenta = os.environ["OANDA_ACCOUNT"]
url = (f"https://api-fxpractice.oanda.com/v3/accounts/{cuenta}"
       "/instruments/EUR_USD/candles?"
       + urlencode({"granularity": "H4", "count": 500, "price": "MBA"}))
req = Request(url, headers={
    "Authorization": f"Bearer {os.environ['OANDA_TOKEN']}"})
with urlopen(req, timeout=10) as r:
    crudo = r.read()

The response carries instrument, granularity and a candles list. Each candle has time (the start of the candle, in RFC 3339 with nanoseconds: YYYY-MM-DDTHH:MM:SS.nnnnnnnnnZ), volume (the number of prices created in the candle), complete and a mid, bid or ask block with o, h, l and c. Prices arrive as text, with the instrument's precision. A response with that schema, built from the synthetic EURUSD in 4-hour candles:

import json

def bloque(s):
    return {"o": f"{s.iloc[0]:.5f}", "h": f"{s.max():.5f}",
            "l": f"{s.min():.5f}", "c": f"{s.iloc[-1]:.5f}"}

dia = eur.loc["2025-06-02 00:00":"2025-06-02 15:00"]
velas_json = []
for inicio, g in dia.groupby(pd.Grouper(freq="4h")):
    velas_json.append({
        "time": inicio.strftime("%Y-%m-%dT%H:%M:%S.000000000Z"),
        "volume": len(g), "complete": len(g) == 4,
        "bid": bloque(g["bid"]), "ask": bloque(g["ask"]),
        "mid": bloque(g["mid"]),
    })
velas_json[-1]["complete"] = False          # the current candle
respuesta = json.dumps({"instrument": "EUR_USD",
                        "granularity": "H4", "candles": velas_json})
print(respuesta[:118], "...")

{"instrument": "EUR_USD", "granularity": "H4", "candles": [{"time": "2025-06-02T00:00:00.000000000Z", "volume": 4, "co ...

And reading it: text to number, complete candles only, and closing bid-ask spread in pips.

datos = json.loads(respuesta)
filas = []
for v in datos["candles"]:
    if not v["complete"]:
        continue                            # never into the history
    fila = {"time": v["time"], "volume": v["volume"]}
    for lado in ("bid", "ask"):
        for k, x in v[lado].items():
            fila[f"{lado}_{k}"] = float(x)
    filas.append(fila)

oanda = pd.DataFrame(filas)
oanda["time"] = pd.to_datetime(oanda["time"], utc=True)
oanda["spread_pips"] = 1e4 * (oanda["ask_c"] - oanda["bid_c"])
oanda.set_index("time")[["bid_c", "ask_c", "spread_pips"]].round(5)

                             bid_c    ask_c  spread_pips
time
2025-06-02 00:00:00+00:00  1.17370  1.17376          0.6
2025-06-02 04:00:00+00:00  1.17570  1.17572          0.2
2025-06-02 08:00:00+00:00  1.17466  1.17468          0.2

Three complete candles; the fourth, flagged as current, is discarded. A forex broker's candles carry bid and ask separately, which is exactly what a backtest with realistic costs needs: buy at the ask, sell at the bid (lesson 1.1.5). The “mid price” candles that many free sources provide do not allow that.

When to use the broker as a source

flowchart TB
  Q{What for?} -- live signal --> B[Broker data]
  Q -- long backtest --> P[Provider]
  Q -- equity universe --> P
  B --> C[Cross-check]
  P --> C

The usual combination is a provider for history and research and the broker for live trading, with a periodic check that both tell the same story on the days where they overlap.

Exercises

1. With descarga, how long does one year of 5-second bars with BID_ASK take for 10 contracts, if they are requested one after another with the same limiter? (Hint: the 10-minute rules count all requests, not per contract.)

h = descarga(10 * 252, peso=2)
print(f"{len(h)} requests: {h[-1] / 3600:.1f} hours")

2520 requests: 13.8 hours

2. In the OANDA response, compute the average bid-ask spread of the complete candles in pips using the close, and compare it with that of the open. Why do they not have to match?

oanda["spread_ap"] = 1e4 * (oanda["ask_o"] - oanda["bid_o"])
oanda[["spread_ap", "spread_pips"]].mean().round(2)

spread_ap      0.53
spread_pips    0.33
dtype: float64

3. Your system trades EURUSD with 4-hour candles at a broker with MetaTrader 5 and you want 15 years of history to test it. Where would you get the data from and what would you check?