--- title: "7.3.4 Model serving: ONNX, CPU or GPU and LLM costs" description: "A scikit-learn model exported to ONNX with verified parity, served over HTTP with a contract, and LLM costs with real prices and a cap." url: "https://pythonparatrading.com/en/python-trading-course/level-7/model-serving-onnx-llm-costs/" lang: "en" image: "https://pythonparatrading.com/og/en/python-trading-course/level-7/model-serving-onnx-llm-costs.png" --- # 7.3.4 · Model serving: ONNX, CPU or GPU inference and the cost of LLMs in production > A trained model is a Python object inside a process; a model in production is a file that another process loads, feeds and monitors. This lesson exports a scikit-learn classifier to ONNX, checks that it gives the same results, serves it over HTTP with an input contract, measures what each layer costs and ends with the other inference bill: that of language models, computed with the prices published by Anthropic and OpenAI and kept in check by a budget guard. - Course: [Level 7: Production, MLOps, regulation, career and capstone](https://pythonparatrading.com/en/python-trading-course/level-7/) - Module: 7.3 · MLOps for trading models - Duration: 65 min - Prerequisites: [5.6.4 · Callbacks, a Sharpe loss and the model in production](https://pythonparatrading.com/en/python-trading-course/level-5/keras-callbacks-custom-loss-export/), [6.2.1 · How an LLM works: from the next token to the API](https://pythonparatrading.com/en/python-trading-course/level-6/how-llms-work-apis-and-local-models/), [7.3.2 · Scheduled retraining, automatic validation and champion/challenger](https://pythonparatrading.com/en/python-trading-course/level-7/model-retraining-validation-champion-challenger/) ## By the end of this lesson - Export a scikit-learn pipeline to ONNX, inspect the graph and check prediction parity with ONNX Runtime. - Choose between in-process inference, an HTTP service or a model server, and between CPU and GPU, with measurements rather than intuitions. - Put an input contract on a prediction service and know what it lets through. - Compute the monthly cost of an LLM workload with explicit tokens and dated prices, and reduce it with prompt caching and batch APIs. - Cap spending with a budget guard that authorizes each call by its maximum cost and does not repeat identical requests. ## From object to file Lessons 7.3.2 and 7.3.3 left a model registered, with aliases and monitored. What remains is to decide **how it runs**: in what format it is saved, which process loads it, on what hardware and at what price. For an in-house scikit-learn model the answer is usually simple and cheap; for a language model from a provider, the bill arrives per token and can grow without anyone noticing. Three ways to serve a model, from the simplest to the most complex: ```mermaid flowchart TB A[Trading process] -->|in process| M[ONNX Runtime] A -->|local HTTP| S[In-house service] S --> M2[ONNX Runtime] A -->|HTTP or gRPC| V[Model server] V --> G[CPU or GPU] ``` - **In the process**: the trading system loads the model file and calls a function. No network, no serialization, minimum latency. Each process has its own copy. - **An in-house service**: a small process that loads the model and answers requests. Several processes share one model, and the model is deployed without touching the trading system. It costs a round trip over the local network. - **A model server** (BentoML, Ray Serve, NVIDIA Triton; vLLM for language models): it groups requests into batches, manages several versions and GPUs, and scales with load. It pays off with many models or a lot of traffic; for a daily signal it is more infrastructure than needed. And a format. Lesson 5.6.4 exported a Keras network with `torch.export` and left ONNX untested because it was not in the level 5 environment. The `ops` environment does have it: skl2onnx 1.20.0 and onnxruntime 1.31.0. ## The model and its export to ONNX The model is a next-day direction classifier with eight features, on synthetic data: a pipeline with a `StandardScaler` and a `GradientBoostingClassifier` of 200 trees of depth 3. ```python import numpy as np from sklearn.ensemble import GradientBoostingClassifier from sklearn.pipeline import make_pipeline from sklearn.preprocessing import StandardScaler rng = np.random.default_rng(734) n, K = 20_000, 8 X = rng.standard_normal((n, K)) # 8 features for one day logit = (0.6 * X[:, 0] - 0.4 * X[:, 1] + 0.3 * X[:, 2] * X[:, 3] + 0.2 * np.tanh(2 * X[:, 4])) y = (logit + rng.logistic(size=n) > 0).astype(int) # up tomorrow? Xtr, Xte, ytr, yte = X[:15_000], X[15_000:], y[:15_000], y[15_000:] gbc = GradientBoostingClassifier(n_estimators=200, max_depth=3, random_state=0) modelo = make_pipeline(StandardScaler(), gbc).fit(Xtr, ytr) print("out-of-sample accuracy:", round((modelo.predict(Xte) == yte).mean(), 3)) ``` ```output out-of-sample accuracy: 0.64 ``` A 64% out-of-sample accuracy is what the data allows (the relationship has logistic noise); here the model is the package, not the content. ONNX (Open Neural Network Exchange) describes a model as a **graph of standard operators** with versions (the *opsets*), and any runtime that implements those operators can run it: ONNX Runtime in Python, C++, C#, Java or the browser, without scikit-learn or the rest of the training environment. ```python from collections import Counter import onnx from skl2onnx import to_onnx onx = to_onnx(modelo, Xtr[:1].astype(np.float32), options={id(gbc): {"zipmap": False}}) onnx.checker.check_model(onx) with open("senal.onnx", "wb") as f: f.write(onx.SerializeToString()) g = onx.graph dims = g.input[0].type.tensor_type.shape.dim print("input:", g.input[0].name, [d.dim_value or "N" for d in dims]) print("outputs:", [o.name for o in g.output]) print("nodes:", dict(Counter(nd.op_type for nd in g.node))) print("opsets:", {o.domain or "ai.onnx": o.version for o in onx.opset_import}) print(f"size: {len(onx.SerializeToString()) / 1024:.0f} KB") ``` ```output input: X ['N', 8] outputs: ['label', 'probabilities'] nodes: {'Scaler': 1, 'TreeEnsembleClassifier': 1, 'Identity': 1} opsets: {'ai.onnx.ml': 1, 'ai.onnx': 21} size: 106 KB ``` The whole pipeline fits in three nodes: `Scaler` (the standardization, with its means and scales inside), `TreeEnsembleClassifier` (the 200 trees with their thresholds) and an `Identity` that names the output. The input is called `X` and has shape `[N, 8]`: any number of rows, eight columns. The file weighs 106 KB. `zipmap: False` makes the probabilities come out as a matrix rather than as a list of dictionaries, which is slower and more awkward. Exporting a model is translating it, and a translation gets checked. The parity test compares the predictions of the original model and of the exported one on a lot of data, and it is part of the project's continuous integration, alongside the export itself: ```python import onnxruntime as ort opciones = ort.SessionOptions() opciones.intra_op_num_threads = 1 # stable result and timing sesion = ort.InferenceSession("senal.onnx", opciones, providers=["CPUExecutionProvider"]) ENTRADA = sesion.get_inputs()[0].name etiquetas, probs = sesion.run(None, {ENTRADA: Xte.astype(np.float32)}) p_sk = modelo.predict_proba(Xte)[:, 1] print("max |difference| in probability:", f"{np.abs(probs[:, 1] - p_sk).max():.1e}") distintas = int((etiquetas != modelo.predict(Xte)).sum()) print("different labels:", distintas, "of", len(Xte)) print("available providers:", ort.get_available_providers()) ``` ```output max |difference| in probability: 2.1e-07 different labels: 0 of 5000 available providers: ['AzureExecutionProvider', 'CPUExecutionProvider'] ``` All 5,000 labels match and the probabilities differ by at most 2.1·10⁻⁷: the input was declared as float32 and the graph works at that precision, while scikit-learn uses float64. With tree models it is worth counting **labels**, not just probability differences: a value that falls exactly on a tree threshold can move to the other side when going to 32 bits. If that matters, skl2onnx supports converting to double precision. There are two available providers: `CPUExecutionProvider`, the one in use, and `AzureExecutionProvider`, which is not used (it ships with the PyPI wheel). Export does not always work. With the exact versions of this environment, scikit-learn's other tree classifier fails to convert: ```python from sklearn.ensemble import HistGradientBoostingClassifier import skl2onnx import sklearn hgb = HistGradientBoostingClassifier(max_iter=20, random_state=0) hgb.fit(Xtr, ytr) try: to_onnx(hgb, Xtr[:1].astype(np.float32), options={id(hgb): {"zipmap": False}}) except ValueError as err: print("scikit-learn", sklearn.__version__, "· skl2onnx", skl2onnx.__version__, "· onnx", onnx.__version__) print(type(err).__name__ + ":", str(err)[:46]) causa = err.__cause__ or err.__context__ print("cause:", type(causa).__name__ + ":", str(causa)) ``` ```output scikit-learn 1.9.1 · skl2onnx 1.20.0 · onnx 1.23.2 ValueError: Unable to create node 'TreeEnsembleClassifier' cause: TypeError: Field onnx.AttributeProto.ints: Expected an int, got a boolean. ``` The converter generates an attribute with boolean values where the ONNX specification expects integers, and the `onnx` library rejects it. It is a compatibility failure between versions, of the kind that appears when scikit-learn changes the internal structure of a model and the converter has not caught up yet. The practical lesson is not this particular failure, which will get fixed, but the procedure: pinned versions in the project's lockfile (lessons 0.2.2 and 7.1.1), and a test that exports, loads and compares every model going to production before each dependency update. ## Latency: row by row, in batches and over the network How much is gained with ONNX Runtime? Timings depend on the machine and cannot appear in the output of a cell that is checked on every run; the cell prints conclusions with a margin, and the numbers go in the text: ```python import timeit fila64 = Xte[:1] fila32 = fila64.astype(np.float32) lote32 = Xte[:1000].astype(np.float32) def mejor(f, n): """Best of 5 repetitions, in microseconds per call.""" return min(timeit.repeat(f, number=n, repeat=5)) / n * 1e6 t_sk = mejor(lambda: modelo.predict_proba(fila64), 50) t_ort = mejor(lambda: sesion.run(None, {ENTRADA: fila32}), 500) t_lote = mejor(lambda: sesion.run(None, {ENTRADA: lote32}), 20) print("one row: ONNX Runtime at least 10 times faster:", t_sk / t_ort > 10) print("batch of 1000: per row, cheaper than a single row:", t_lote / 1000 < t_ort) ``` ```output one row: ONNX Runtime at least 10 times faster: True batch of 1000: per row, cheaper than a single row: True ``` On the course machine, scikit-learn's `predict_proba` takes about 410 µs for one row, and ONNX Runtime about 9 µs. Almost all of scikit-learn's time is fixed work per call (validating the input, walking through the pipeline in Python), not computation. A batch of 1,000 rows takes about 2.7 ms in ONNX Runtime: about 2.7 µs per row, three times less than a single row. The computation of a tree per row is so small that the fixed cost of each call dominates, and grouping rows spreads it out. Now the service. A minimal HTTP server with the standard library, the ONNX model loaded once and an **input contract** with Pydantic (lesson 6.2.2): exact fields, strict types, eight features. It listens on `127.0.0.1`, with a port chosen by the system; never on all interfaces without authentication in front. ```python import json import threading import urllib.error import urllib.request from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer from pydantic import BaseModel, ConfigDict, Field, ValidationError class Peticion(BaseModel): model_config = ConfigDict(extra="forbid", strict=True) modelo: str = Field(pattern=r"^senal-n7$") features: list[float] = Field(min_length=K, max_length=K) class Servicio(BaseHTTPRequestHandler): def log_message(self, *args): # no logging to stderr pass def responder(self, codigo, cuerpo): datos = json.dumps(cuerpo).encode() self.send_response(codigo) self.send_header("Content-Type", "application/json") self.send_header("Content-Length", str(len(datos))) self.end_headers() self.wfile.write(datos) def do_POST(self): crudo = self.rfile.read(int(self.headers["Content-Length"])) try: p = Peticion.model_validate_json(crudo) except ValidationError as e: return self.responder(422, {"error": e.errors()[0]["msg"]}) x = np.array([p.features], dtype=np.float32) _, pr = sesion.run(None, {ENTRADA: x}) self.responder(200, {"p_sube": round(float(pr[0, 1]), 6)}) srv = ThreadingHTTPServer(("127.0.0.1", 0), Servicio) threading.Thread(target=srv.serve_forever, daemon=True).start() URL = f"http://127.0.0.1:{srv.server_port}/predecir" def pedir(cuerpo): req = urllib.request.Request(URL, json.dumps(cuerpo).encode(), {"Content-Type": "application/json"}) try: with urllib.request.urlopen(req) as r: return r.status, json.load(r) except urllib.error.HTTPError as e: return e.code, json.load(e) f0 = [float(v) for v in Xte[0]] print(pedir({"modelo": "senal-n7", "features": f0}), "| in process:", round(float(p_sk[0]), 6)) print(pedir({"modelo": "senal-n7", "features": f0[:7]})) print(pedir({"modelo": "senal-n7", "features": f0, "umbral": 0.4})) ``` ```output (200, {'p_sube': 0.51919}) | in process: 0.51919 (422, {'error': 'List should have at least 8 items after validation, not 7'}) (422, {'error': 'Extra inputs are not permitted'}) ``` The service returns the same probability as the in-process model, and rejects with a 422 a request with seven features and another with a field that is not in the contract (`umbral`, a threshold). And the price? ```python cuerpo = {"modelo": "senal-n7", "features": f0} t_http = mejor(lambda: pedir(cuerpo), 50) print("local HTTP at least 10 times slower than in process:", t_http / t_ort > 10) srv.shutdown() ``` ```output local HTTP at least 10 times slower than in process: True ``` Over local HTTP, about 540 µs per prediction: more than scikit-learn in the process. The model itself now costs only 9 µs; the rest is serializing the JSON, opening the connection, going through the server loop and coming back. That is the conclusion of the section: architecture weighs more than format. For a daily signal, 0.5 ms does not matter at all and the service brings order (one model, one deployment, one contract). For a system that reacts to every quote (lesson 4.5.1), the model goes inside the process. | Option | Latency per call (course machine) | When | |---|---|---| | scikit-learn in the process | ≈ 410 µs | research, daily signals | | ONNX Runtime in the process | ≈ 9 µs (≈ 2.7 µs per row in batches of 1,000) | real time, many models | | Local HTTP service | ≈ 540 µs | several processes, independent deployment | | Model server | depends on load and batch | many models, GPU, variable traffic | ## CPU or GPU A GPU executes thousands of identical operations in parallel. It pays off when there is a lot of computation **per call**: large networks, large batches, language models. A tree ensemble on one row is the opposite: little computation, many branches. For almost all tabular trading models, the CPU is more than enough, and it avoids copying data to GPU memory and back (lesson 6.6.5 covered the GPU for data processing, with RAPIDS). With ONNX, changing hardware means changing the execution provider. The code is this one, not executed because the course machine has no GPU (it requires the `onnxruntime-gpu` package instead of `onnxruntime`, which since version 1.27 is published on PyPI built with CUDA 13.0[^1]): ```python skip # pip install "onnxruntime-gpu[cuda,cudnn]" (another environment, with a GPU) import onnxruntime as ort sesion = ort.InferenceSession( "senal.onnx", providers=["CUDAExecutionProvider", "CPUExecutionProvider"]) print(sesion.get_providers()) # the first available one is used ``` The two techniques that most reduce inference cost on CPU are batching (the latency cell) and **quantizing**: storing the weights as 8-bit integers, or fewer bits, instead of float32. ONNX Runtime quantizes ONNX models (its documentation recommends dynamic quantization for transformers and recurrent networks, and static quantization for convolutions), and it is what makes it possible to run language models with billions of parameters on a single consumer GPU (lesson 6.2.4). For a tree model it adds nothing: there are no weight matrices to compress. ## The cost of LLMs in production With language models the calculation changes in nature. The cost is no longer a machine but a price per token (lesson 6.2.1), with four prices per model: input tokens, output tokens, input read from the prompt cache and input written to it. The ones in the table come from the official pages of Anthropic and OpenAI, consulted on October 23, 2026, in USD per million tokens, for short contexts[^2]: ```python # USD per million tokens, short context. Official pricing pages of # Anthropic and OpenAI, consulted on October 23, 2026. PRECIOS = { # input, output, cache read, cache write "Claude Opus 5.5": (4.00, 20.00, 0.20, 5.00), "Claude Sonnet 5.5": (2.00, 10.00, 0.10, 2.50), "Claude Haiku 5.5": (0.10, 0.50, 0.01, 0.125), "gpt-6-astra": (10.00, 50.00, 1.00, 12.50), "gpt-6.1-sol": (2.00, 10.00, 0.10, 2.50), "gpt-6-luna": (0.10, 0.50, 0.01, 0.125), } LOTE = 0.5 # batch API: half price at both providers def coste(modelo, entrada, salida, cache_r=0, cache_w=0, lote=False): """USD for one request; `entrada` = input tokens without cache.""" pe, ps, pr, pw = PRECIOS[modelo] usd = (entrada * pe + salida * ps + cache_r * pr + cache_w * pw) / 1e6 return usd * (LOTE if lote else 1.0) print(f"{coste('Claude Sonnet 5.5', 1_000_000, 0):.2f}", f"{coste('gpt-6-astra', 0, 1_000_000, lote=True):.2f}") ``` ```output 2.00 25.00 ``` Two discounts change the result a lot: - **Prompt caching.** An initial part of the prompt that repeats across requests (instructions, examples, a document, an agent's history) is written once to the cache, somewhat more expensive than normal input (1.25 times), and is then read much more cheaply: 0.05 times the input price on Claude Opus 5.5, Claude Sonnet 5.5 and gpt-6.1-sol, 0.1 times on the other models in the table. The cache lasts from minutes to hours, depending on the option, and is refreshed when read. - **Batch API.** Requests that can wait (hours, not seconds) are sent together and paid at half price at both providers. Three typical workloads of a research system, with the tokens written out by hand: - **Headlines**: 2,000 a day, classified with a prompt of 1,500 tokens of instructions and examples, always the same, plus the headline (60 tokens); it answers with a JSON label (30). - **Reports**: 40 a day of 25,000 tokens, summarized in 1,000. Each report is different, so the cache does not help; they can wait until night, so the batch API does. - **Agent**: 200 tasks a day of 8 steps (lesson 6.4.1). At each step it resends everything before: 3,000 initial tokens and 1,500 more per step, and it answers with 300. ```python import pandas as pd SESIONES = 21 def titulares(m, cache): """2,000 headlines a day: instructions and examples (1,500 tokens, the same in all) + headline (60) and a JSON label (30).""" if cache: # 10 writes a day; the rest, reads return (10 * coste(m, 60, 30, cache_w=1500) + 1990 * coste(m, 60, 30, cache_r=1500)) return 2000 * coste(m, 1560, 30) def informes(m, lote): """40 reports of 25,000 tokens; summary of 1,000.""" return 40 * coste(m, 25_000, 1_000, lote=lote) def agente(m, cache): """200 tasks of 8 steps. Each step resends the whole history: 3,000 fixed tokens + 1,500 per previous step; answers 300.""" total = 0.0 for i in range(8): contexto = 3000 + 1500 * i if cache and i: # reads what came before and writes the new part total += coste(m, 0, 300, cache_r=contexto - 1500, cache_w=1500) elif cache: total += coste(m, 0, 300, cache_w=contexto) else: total += coste(m, contexto, 300) return 200 * total filas = {} for m in PRECIOS: base = titulares(m, False) + informes(m, False) + agente(m, False) opt = titulares(m, True) + informes(m, True) + agente(m, True) filas[m] = {"headlines": titulares(m, False) * SESIONES, "reports": informes(m, False) * SESIONES, "agent": agente(m, False) * SESIONES, "total": base * SESIONES, "with cache and batch": opt * SESIONES} mes = pd.DataFrame(filas).T.round(0).astype(int) mes ``` ```output headlines reports agent total with cache and batch Claude Opus 5.5 287 101 1310 1698 629 Claude Sonnet 5.5 144 50 655 849 314 Claude Haiku 5.5 7 3 33 42 17 gpt-6-astra 718 252 3276 4246 1714 gpt-6.1-sol 144 50 655 849 314 gpt-6-luna 7 3 33 42 17 ``` The figure at the top shows the totals on a logarithmic scale. From the most expensive model to the cheapest there is a factor of 100: 4,246 USD a month with gpt-6-astra versus 42 with Claude Haiku 5.5 or gpt-6-luna. And on every model, caching and batching leave the bill at between 37% and 40% of the standard price. The breakdown by workload, with Claude Sonnet 5.5: *Fig. — Monthly cost of each of the lesson's workloads with Claude Sonnet 5.5, at standard rates and with the matching optimisation (caching for headlines and agent, batch API for reports).* Horizontal bars in pairs. Headlines: 144 USD per month at standard rates and 25 with caching. Reports: 50 and 25 with the batch API. Agent, highlighted: 655 USD at standard rates, 77% of the total, and 265 with caching. The agent is 77% of the spend: 655 USD out of 849. It does not answer much (300 tokens per step); what is expensive is that **each step resends the whole conversation**, 66,000 input tokens per task. Caching brings it down to 265, because what has already been sent is read at 0.05 times the price. Headlines fall from 144 to 25 with caching (the fixed prompt is 94% of each request) and reports, from 50 to 25 with batching. ## An in-house server? The alternative to paying per token is renting a GPU and serving an open model with vLLM or Ollama (lesson 6.2.1). A real price: Hetzner's GEX45 server, with a 24 GB NVIDIA RTX PRO 4000 Blackwell, costs 214 EUR a month excluding VAT plus a 209 EUR setup fee[^3]. Above what volume is it cheaper than the API? ```python GEX45_EUR = 214.00 # per month, excl. VAT (Hetzner, 2026-10-23) ALTA_EUR = 209.00 # one-off EURUSD = 1.1193 # ECB, reference rate of 2026-10-23 servidor_usd = GEX45_EUR * EURUSD def tokens_equivalentes(m, frac_salida=0.02): """Tokens per month at which the API costs as much as the server, with 2% output tokens (as in the workloads above).""" pe, ps, *_ = PRECIOS[m] usd_mtok = (1 - frac_salida) * pe + frac_salida * ps return servidor_usd / usd_mtok * 1e6 print(f"server: {servidor_usd:.2f} USD a month") for m in ("Claude Haiku 5.5", "gpt-6-luna", "Claude Sonnet 5.5"): tok = tokens_equivalentes(m) por_s = tok / (30 * 24 * 3600) print(f"{m:18} {tok / 1e9:6.2f} billion tokens a month" f" = {por_s:6.0f} tokens/s nonstop") ``` ```output server: 239.53 USD a month Claude Haiku 5.5 2.22 billion tokens a month = 856 tokens/s nonstop gpt-6-luna 2.22 billion tokens a month = 856 tokens/s nonstop Claude Sonnet 5.5 0.11 billion tokens a month = 43 tokens/s nonstop ``` Against the small API models (Claude Haiku 5.5, gpt-6-luna), the server is only cheaper above roughly 2.2 billion tokens a month: 856 tokens per second, day and night, all month long. Against Claude Sonnet 5.5, above 43 tokens per second, but a model that fits in 24 GB is not comparable with Sonnet: the honest comparison is with the small models. Whether a GPU like that sustains 856 tokens per second with a useful model is something that has to be measured with the specific model and server (vLLM batches requests and its throughput depends heavily on batch size); there is no reliable figure to give here. The decision, therefore, is rarely about price at the scale of an individual or a small team: | Reason to serve your own model | Reason to use an API | |---|---| | Confidentiality: the data does not leave the machine (6.2.1) | Quality of the large models | | Version pinned forever: reproducibility | Nothing to operate, update or monitor | | Latency without network or third-party queues | Pay per use: zero if unused | | Huge, constant volumes | Load peaks without idle capacity | ## A budget guard An API bill has no ceiling by itself. A badly written loop that repeats the same call, an agent that does not finish (lesson 6.4.5) or a change that multiplies the length of the prompt shows up at the end of the month. The control is the same as the order risk engine, applied to spending: each call is **authorized beforehand** with its maximum possible cost (the estimated input plus `max_tokens` of output), recorded with the real cost returned by the API and, once the daily limit is reached, there are no more calls. And a request identical to one already made is served from a local cache, without calling: ```mermaid flowchart TB P[Request] --> C{In cache} C -->|yes| R[Response] C -->|no| B{Within limit} B -->|no| X[Reject and alert] B -->|yes| A[API] A --> G[Record cost] G --> R ``` ```python import hashlib class Presupuesto: """Authorizes each call with its maximum possible cost and cuts off when the daily limit is reached. Identical requests are served from the cache without calling the API.""" def __init__(self, modelo, limite_usd): self.modelo, self.limite = modelo, limite_usd self.gastado, self.cache = 0.0, {} self.llamadas = self.de_cache = self.rechazadas = 0 def llamar(self, prompt, max_salida, api): clave = hashlib.sha256(f"{self.modelo}|{max_salida}|{prompt}" .encode()).hexdigest() if clave in self.cache: self.de_cache += 1 return self.cache[clave] n_in = len(prompt.split()) * 4 // 3 # prudent estimate peor = coste(self.modelo, n_in, max_salida) if self.gastado + peor > self.limite: self.rechazadas += 1 raise RuntimeError("daily budget exhausted") texto, uso = api(prompt, max_salida) self.gastado += coste(self.modelo, uso["in"], uso["out"]) self.llamadas += 1 self.cache[clave] = texto return texto def api_simulada(prompt, max_salida): """Always answers the same and counts tokens like an API.""" n = len(prompt.split()) return "neutral", {"in": n * 4 // 3, "out": min(max_salida, 40)} g = Presupuesto("Claude Sonnet 5.5", limite_usd=5.00) largo = " ".join(["word"] * 6000) # an 8,000-token report for i in range(400): # a badly written loop g.llamar("Classify: Nore rises 3%", 50, api_simulada) for i in range(400): # 400 different reports try: g.llamar(f"Report {i}: {largo}", 1000, api_simulada) except RuntimeError: break print(f"calls {g.llamadas} · from cache {g.de_cache} · " f"rejected {g.rechazadas} · spent {g.gastado:.2f} USD") ``` ```output calls 305 · from cache 399 · rejected 1 · spent 4.99 USD ``` The badly written loop repeats the same classification 400 times: one real call and 399 served from the cache. The 400 different reports exhaust the 5 USD budget at call 305 (each 8,000-token report costs about 0.016 USD, and the worst case with 1,000 output tokens, 0.026): the guard rejects the next one, the loop stops and spending stays at 4.99 USD. In production, the rejection triggers an alert (lesson 7.2.3), the cost of each call is published as a metric, and limits are also set in the provider's console, which is the last line of defense if the guard fails. ## Exercises 1. Repeat the agent cost with Claude Opus 5.5 for tasks of 16 steps instead of 8, with and without caching. Why does the cost without caching more than double? **Solution.** ```python def agente_n(m, pasos, cache): total = 0.0 for i in range(pasos): contexto = 3000 + 1500 * i if cache and i: total += coste(m, 0, 300, cache_r=contexto - 1500, cache_w=1500) elif cache: total += coste(m, 0, 300, cache_w=contexto) else: total += coste(m, contexto, 300) return 200 * total * SESIONES m = "Claude Opus 5.5" for pasos in (8, 16): print(pasos, "steps:", round(agente_n(m, pasos, False)), "USD a month without cache ·", round(agente_n(m, pasos, True)), "with cache") ``` ```output 8 steps: 1310 USD a month without cache · 529 with cache 16 steps: 4234 USD a month without cache · 1109 with cache ``` With 16 steps, the agent costs 4,234 USD a month without caching, 3.2 times more than with 8 (1,310), and 1,109 with caching, 2.1 times more (529). Without caching, the input of a task is the sum of a context that grows at every step: with $n$ steps, $3000\,n + 750\,n(n-1)$ tokens. It is quadratic: 66,000 tokens with 8 steps and 228,000 with 16, 3.5 times more. The output only doubles, and that is why the total ends up at 3.2. With caching, what has already been sent is read at 0.05 times the price (on Opus 5.5) and what is paid in full is the new part of each step, which grows linearly: the cost gets close to doubling. It is the practical reason to limit the number of steps of an agent, summarize the history when it gets long and use caching: the cost of an agent is set by the length of its conversation, not by the length of its answers. 2. A client of the HTTP service sends a request with `NaN` in the first feature (`json.dumps` writes it that way by default). Does the contract accept it? Fix it. **Solution.** ```python from pydantic import FiniteFloat crudo = json.dumps({"modelo": "senal-n7", "features": [float("nan")] + f0[1:]}) print(crudo[:40]) p = Peticion.model_validate_json(crudo) x = np.array([p.features], dtype=np.float32) print("accepted; p_sube =", round(float(sesion.run(None, {ENTRADA: x})[1][0, 1]), 6)) class PeticionFinita(Peticion): features: list[FiniteFloat] = Field(min_length=K, max_length=K) try: PeticionFinita.model_validate_json(crudo) except ValidationError as e: print("rejected:", e.errors()[0]["msg"], e.errors()[0]["loc"]) ``` ```output {"modelo": "senal-n7", "features": [NaN, accepted; p_sube = 0.47686 rejected: Input should be a finite number ('features', 0) ``` It accepts it. `json.dumps` writes `NaN` (which is not standard JSON, but Python produces and reads it by default), Pydantic converts the list to `float` without complaint because a NaN is a valid `float`, and the tree model returns a probability, 0.47686, without any warning: in the graph, the NaN follows the branch that the converter assigns to missing values. It is the worst kind of failure: a good-looking answer computed from a value that does not exist. `FiniteFloat` requires finite numbers and the request is rejected with an error that says which field and which position fail. In the service, the right contract is the one that ends in a 422; and on the client side, `json.dumps(..., allow_nan=False)` raises an exception before sending a NaN. Both defenses at once: the contract does not trust the clients. 3. A colleague proposes moving headline classification to an open model on an in-house server “to save money”. With the lesson's numbers, what would you answer? **Solution.** That with the small API models headline classification costs about 7 USD a month at the standard price (and 1 or 2 with caching), and the GPU server about 240 USD a month plus the setup fee, plus the time to install it, update it, monitor it and keep it available. On cost it does not pay off until volumes of the order of 2.2 billion tokens a month. It can pay off for other reasons: if the headlines or the prompt contain information that must not leave the firm (positions, proprietary signals), if the model has to be pinned forever for reproducibility, or if the API latency is a problem. If any of those reasons is real, the in-house server is justified by it, and it is worth saying so and measuring the quality of the open model with the task's evaluation set before switching (lesson 6.2.1). [^1]: ONNX Runtime, “CUDA Execution Provider” (https://onnxruntime.ai/docs/execution-providers/CUDA-ExecutionProvider.html): “Starting with version 1.27, GPU packages published to PyPI (onnxruntime-gpu) … are built with CUDA 13.0 by default”, and installation with `pip install onnxruntime-gpu[cuda,cudnn]`. Consulted on October 23, 2026. [^2]: Anthropic, “Pricing” (https://platform.claude.com/docs/en/about-claude/pricing): Claude Opus 5.5, 4 USD per million input tokens and 20 output; Sonnet 5.5, 2 and 10; Haiku 5.5, 0.10 and 0.50 (prompts of up to 100,000 tokens); writes to the 5-minute cache at 1.25 times input; reads at 0.05 times on Opus 5.5 and Sonnet 5.5 and at 0.1 times on the rest; batch API with a 50% discount on input and output. OpenAI, “Pricing” (https://developers.openai.com/api/docs/pricing), standard price and short context: gpt-6-astra, 10 and 50, cached input 1.00 and 30-minute write 12.50; gpt-6.1-sol, 2 and 10, 0.10 and 2.50; gpt-6-luna, 0.10 and 0.50, 0.01 and 0.125; the batch table is half. Both consulted on October 23, 2026. [^3]: Hetzner, dedicated servers with GPU (https://www.hetzner.com/dedicated-rootserver/matrix-gpu/): GEX45, “€ 214.00 per month, excl. VAT” and “€ 209.00 one-time setup fee”, with a 24 GB NVIDIA RTX PRO 4000 Blackwell SFF Edition. ECB reference rate of October 23, 2026: 1 EUR = 1.1193 USD. Consulted on October 23, 2026. ## Self-assessment 1. The model exported to ONNX gives probabilities that differ from scikit-learn's by at most 2.1·10⁻⁷. Where does that difference come from? - A. From a bug in the converter - B. From ONNX Runtime using a different seed - C. From ONNX Runtime computing in 32-bit floating point and scikit-learn in 64-bit - D. From the ONNX graph removing unimportant trees *Answer C: From ONNX Runtime computing in 32-bit floating point and scikit-learn in 64-bit.* The input is declared as float32 and the graph works at that precision; scikit-learn uses float64. Differences of 10⁻⁷ are rounding; what matters is that none of the 5,000 labels changes. With tree models, a value sitting exactly on a threshold could fall to the other side: that is why the parity test counts labels, not just probability differences. 2. On the course machine, a one-row prediction takes about 410 µs with scikit-learn, about 9 µs with ONNX Runtime in the process and about 540 µs over local HTTP. Which conclusion is correct? - A. ONNX always makes a system faster - B. HTTP is slow and must always be avoided - C. For a daily signal, none of the three latencies matters - D. Exporting to ONNX removes almost all the compute cost, but putting it behind HTTP adds more than was saved: architecture weighs more than format *Answer D: Exporting to ONNX removes almost all the compute cost, but putting it behind HTTP adds more than was saved: architecture weighs more than format.* The HTTP service has advantages (a single model for several processes, independent deployment), and its cost per call is hundreds of microseconds. For a system that decides once a day it makes no difference; for one that decides on every tick, the model has to live inside the process. The right question is how many predictions there are per second and with what latency budget. 3. Claude Sonnet 5.5 and gpt-6.1-sol have the same price per million tokens. Do they cost the same for the same job? - A. Not necessarily: each provider counts tokens with its own tokenizer, and the same text can give different numbers - B. Yes, by definition - C. No, because OpenAI charges for caching separately and Anthropic does not - D. Yes, except with the batch API *Answer A: Not necessarily: each provider counts tokens with its own tokenizer, and the same text can give different numbers.* The price is per token, and the token is defined by each model's tokenizer. Anthropic's own pricing page warns that the tokenizer of Claude 4.7 and later produces approximately 30% more tokens for the same text. For a real comparison you have to count the tokens of a sample of the real texts with each provider. 4. In the lesson's scenario, the agent accounts for 655 of the 849 USD a month with Claude Sonnet 5.5. Why is it so expensive? - A. Because it writes very long answers - B. Because each step resends the whole history: the context grows step by step and is paid in full on every call - C. Because it uses the most expensive model - D. Because the API charges per tool used *Answer B: Because each step resends the whole history: the context grows step by step and is paid in full on every call.* At step i, 3,000 + 1,500·i tokens are sent: over 8 steps, 66,000 input tokens for 2,400 of output. With prompt caching, what has already been sent is read at 0.05 times the input price (on Sonnet 5.5) and the agent's cost falls to 265 USD. 5. The lesson's budget guard authorizes each call with its maximum possible cost, not the expected one. Why? - A. Because that way it spends more - B. Because APIs always charge max_tokens - C. Because the expected cost cannot be computed - D. Because the real cost is only known when the call finishes, and a response that reaches `max_tokens` can cost quite a lot more than the average: authorizing by the maximum guarantees the limit is not exceeded *Answer D: Because the real cost is only known when the call finishes, and a response that reaches `max_tokens` can cost quite a lot more than the average: authorizing by the maximum guarantees the limit is not exceeded.* APIs charge for the tokens they generate, not for `max_tokens`; but that number is not known until the end. Authorizing by the worst case makes the limit an actual limit. In the lesson, the guard cuts off at 4.99 USD out of 5.00 after 305 calls. ## Key points - ONNX describes the model as a graph of standard operators that ONNX Runtime runs without scikit-learn: the lesson's pipeline weighs 106 KB, gives the same 5,000 labels and probabilities within 2·10⁻⁷. - The converter lags behind the library: with scikit-learn 1.9.1 and skl2onnx 1.20.0, HistGradientBoostingClassifier does not export. Pinned versions and an export-and-parity test in continuous integration. - Latency depends more on architecture than on format: about 410 µs with scikit-learn, 9 µs with ONNX Runtime in the process and 540 µs over local HTTP on the course machine. A GPU pays off with large batches and large networks, not with one-row trees. - A prediction service validates its input with a strict contract; even so, the lesson's contract let NaN through until it required finite numbers. - The cost of an LLM is computed with explicit tokens and dated prices. In the lesson's workloads, the agent that resends its history is 77% of the spend; caching and the batch API bring the total down from 849 to 314 USD a month with Claude Sonnet 5.5. - A rented GPU server (about 240 USD a month) only competes on cost with the small API models above roughly 2.2 billion tokens a month; it is chosen for confidentiality or control, not price. And every API use sits behind a budget guard. ## Resources ### Research - [The Working Environment (Docker for trading)](https://pythonparatrading.com/en/docker-for-trading/) · Research-0013 · 4 min ### Documentation - [sklearn-onnx: documentation (skl2onnx)](https://onnx.ai/sklearn-onnx/) · onnx.ai - [ONNX Runtime: Execution Providers (CPU, CUDA, TensorRT…)](https://onnxruntime.ai/docs/execution-providers/) · onnxruntime.ai - [ONNX Runtime: Quantize ONNX models](https://onnxruntime.ai/docs/performance/model-optimizations/quantization.html) · onnxruntime.ai - [Anthropic: Pricing (Claude API)](https://platform.claude.com/docs/en/about-claude/pricing) · platform.claude.com - [OpenAI: API Pricing](https://developers.openai.com/api/docs/pricing) · developers.openai.com - [vLLM: documentation](https://docs.vllm.ai/) · docs.vllm.ai Previous lesson: [7.3.3 · Live model monitoring and automatic shutdown](https://pythonparatrading.com/en/python-trading-course/level-7/live-model-monitoring-automatic-shutdown/) Next lesson: [7.4.1 · MiFID II: clients, suitability and algorithmic trading](https://pythonparatrading.com/en/python-trading-course/level-7/mifid-ii-suitability-algorithmic-trading/)