Model serving: ONNX, CPU or GPU inference and the cost of LLMs in production.

From object to file

Lessons 7.3.2 and 7.3.3 left a model registered, with aliases and monitored. What remains is to decide how it runs: in what format it is saved, which process loads it, on what hardware and at what price. For an in-house scikit-learn model the answer is usually simple and cheap; for a language model from a provider, the bill arrives per token and can grow without anyone noticing.

Three ways to serve a model, from the simplest to the most complex:

flowchart TB
  A[Trading process] -->|in process| M[ONNX Runtime]
  A -->|local HTTP| S[In-house service]
  S --> M2[ONNX Runtime]
  A -->|HTTP or gRPC| V[Model server]
  V --> G[CPU or GPU]

• In the process: the trading system loads the model file and calls a function. No network, no serialization, minimum latency. Each process has its own copy.
• An in-house service: a small process that loads the model and answers requests. Several processes share one model, and the model is deployed without touching the trading system. It costs a round trip over the local network.
• A model server (BentoML, Ray Serve, NVIDIA Triton; vLLM for language models): it groups requests into batches, manages several versions and GPUs, and scales with load. It pays off with many models or a lot of traffic; for a daily signal it is more infrastructure than needed.

And a format. Lesson 5.6.4 exported a Keras network with torch.export and left ONNX untested because it was not in the level 5 environment. The ops environment does have it: skl2onnx 1.20.0 and onnxruntime 1.31.0.

The model and its export to ONNX

The model is a next-day direction classifier with eight features, on synthetic data: a pipeline with a StandardScaler and a GradientBoostingClassifier of 200 trees of depth 3.

import numpy as np
from sklearn.ensemble import GradientBoostingClassifier
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler

rng = np.random.default_rng(734)
n, K = 20_000, 8
X = rng.standard_normal((n, K))           # 8 features for one day
logit = (0.6 * X[:, 0] - 0.4 * X[:, 1] + 0.3 * X[:, 2] * X[:, 3]
         + 0.2 * np.tanh(2 * X[:, 4]))
y = (logit + rng.logistic(size=n) > 0).astype(int)   # up tomorrow?
Xtr, Xte, ytr, yte = X[:15_000], X[15_000:], y[:15_000], y[15_000:]

gbc = GradientBoostingClassifier(n_estimators=200, max_depth=3,
                                 random_state=0)
modelo = make_pipeline(StandardScaler(), gbc).fit(Xtr, ytr)
print("out-of-sample accuracy:",
      round((modelo.predict(Xte) == yte).mean(), 3))

out-of-sample accuracy: 0.64

A 64% out-of-sample accuracy is what the data allows (the relationship has logistic noise); here the model is the package, not the content. ONNX (Open Neural Network Exchange) describes a model as a graph of standard operators with versions (the opsets), and any runtime that implements those operators can run it: ONNX Runtime in Python, C++, C#, Java or the browser, without scikit-learn or the rest of the training environment.

from collections import Counter

import onnx
from skl2onnx import to_onnx

onx = to_onnx(modelo, Xtr[:1].astype(np.float32),
              options={id(gbc): {"zipmap": False}})
onnx.checker.check_model(onx)
with open("senal.onnx", "wb") as f:
    f.write(onx.SerializeToString())

g = onx.graph
dims = g.input[0].type.tensor_type.shape.dim
print("input:", g.input[0].name, [d.dim_value or "N" for d in dims])
print("outputs:", [o.name for o in g.output])
print("nodes:", dict(Counter(nd.op_type for nd in g.node)))
print("opsets:", {o.domain or "ai.onnx": o.version
                  for o in onx.opset_import})
print(f"size: {len(onx.SerializeToString()) / 1024:.0f} KB")

input: X ['N', 8]
outputs: ['label', 'probabilities']
nodes: {'Scaler': 1, 'TreeEnsembleClassifier': 1, 'Identity': 1}
opsets: {'ai.onnx.ml': 1, 'ai.onnx': 21}
size: 106 KB

The whole pipeline fits in three nodes: Scaler (the standardization, with its means and scales inside), TreeEnsembleClassifier (the 200 trees with their thresholds) and an Identity that names the output. The input is called X and has shape [N, 8]: any number of rows, eight columns. The file weighs 106 KB. zipmap: False makes the probabilities come out as a matrix rather than as a list of dictionaries, which is slower and more awkward.

Exporting a model is translating it, and a translation gets checked. The parity test compares the predictions of the original model and of the exported one on a lot of data, and it is part of the project's continuous integration, alongside the export itself:

import onnxruntime as ort

opciones = ort.SessionOptions()
opciones.intra_op_num_threads = 1        # stable result and timing
sesion = ort.InferenceSession("senal.onnx", opciones,
                              providers=["CPUExecutionProvider"])
ENTRADA = sesion.get_inputs()[0].name
etiquetas, probs = sesion.run(None, {ENTRADA: Xte.astype(np.float32)})
p_sk = modelo.predict_proba(Xte)[:, 1]
print("max |difference| in probability:",
      f"{np.abs(probs[:, 1] - p_sk).max():.1e}")
distintas = int((etiquetas != modelo.predict(Xte)).sum())
print("different labels:", distintas, "of", len(Xte))
print("available providers:", ort.get_available_providers())

max |difference| in probability: 2.1e-07
different labels: 0 of 5000
available providers: ['AzureExecutionProvider', 'CPUExecutionProvider']

All 5,000 labels match and the probabilities differ by at most 2.1·10⁻⁷: the input was declared as float32 and the graph works at that precision, while scikit-learn uses float64. With tree models it is worth counting labels, not just probability differences: a value that falls exactly on a tree threshold can move to the other side when going to 32 bits. If that matters, skl2onnx supports converting to double precision. There are two available providers: CPUExecutionProvider, the one in use, and AzureExecutionProvider, which is not used (it ships with the PyPI wheel).

Export does not always work. With the exact versions of this environment, scikit-learn's other tree classifier fails to convert:

from sklearn.ensemble import HistGradientBoostingClassifier
import skl2onnx
import sklearn

hgb = HistGradientBoostingClassifier(max_iter=20, random_state=0)
hgb.fit(Xtr, ytr)
try:
    to_onnx(hgb, Xtr[:1].astype(np.float32),
            options={id(hgb): {"zipmap": False}})
except ValueError as err:
    print("scikit-learn", sklearn.__version__, "· skl2onnx",
          skl2onnx.__version__, "· onnx", onnx.__version__)
    print(type(err).__name__ + ":", str(err)[:46])
    causa = err.__cause__ or err.__context__
    print("cause:", type(causa).__name__ + ":", str(causa))

scikit-learn 1.9.1 · skl2onnx 1.20.0 · onnx 1.23.2
ValueError: Unable to create node 'TreeEnsembleClassifier'
cause: TypeError: Field onnx.AttributeProto.ints: Expected an int, got a boolean.

The converter generates an attribute with boolean values where the ONNX specification expects integers, and the onnx library rejects it. It is a compatibility failure between versions, of the kind that appears when scikit-learn changes the internal structure of a model and the converter has not caught up yet. The practical lesson is not this particular failure, which will get fixed, but the procedure: pinned versions in the project's lockfile (lessons 0.2.2 and 7.1.1), and a test that exports, loads and compares every model going to production before each dependency update.

Latency: row by row, in batches and over the network

How much is gained with ONNX Runtime? Timings depend on the machine and cannot appear in the output of a cell that is checked on every run; the cell prints conclusions with a margin, and the numbers go in the text:

import timeit

fila64 = Xte[:1]
fila32 = fila64.astype(np.float32)
lote32 = Xte[:1000].astype(np.float32)

def mejor(f, n):
    """Best of 5 repetitions, in microseconds per call."""
    return min(timeit.repeat(f, number=n, repeat=5)) / n * 1e6

t_sk = mejor(lambda: modelo.predict_proba(fila64), 50)
t_ort = mejor(lambda: sesion.run(None, {ENTRADA: fila32}), 500)
t_lote = mejor(lambda: sesion.run(None, {ENTRADA: lote32}), 20)
print("one row: ONNX Runtime at least 10 times faster:",
      t_sk / t_ort > 10)
print("batch of 1000: per row, cheaper than a single row:",
      t_lote / 1000 < t_ort)

one row: ONNX Runtime at least 10 times faster: True
batch of 1000: per row, cheaper than a single row: True

On the course machine, scikit-learn's predict_proba takes about 410 µs for one row, and ONNX Runtime about 9 µs. Almost all of scikit-learn's time is fixed work per call (validating the input, walking through the pipeline in Python), not computation. A batch of 1,000 rows takes about 2.7 ms in ONNX Runtime: about 2.7 µs per row, three times less than a single row. The computation of a tree per row is so small that the fixed cost of each call dominates, and grouping rows spreads it out.

Now the service. A minimal HTTP server with the standard library, the ONNX model loaded once and an input contract with Pydantic (lesson 6.2.2): exact fields, strict types, eight features. It listens on 127.0.0.1, with a port chosen by the system; never on all interfaces without authentication in front.

import json
import threading
import urllib.error
import urllib.request
from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer

from pydantic import BaseModel, ConfigDict, Field, ValidationError

class Peticion(BaseModel):
    model_config = ConfigDict(extra="forbid", strict=True)
    modelo: str = Field(pattern=r"^senal-n7$")
    features: list[float] = Field(min_length=K, max_length=K)

class Servicio(BaseHTTPRequestHandler):
    def log_message(self, *args):          # no logging to stderr
        pass

    def responder(self, codigo, cuerpo):
        datos = json.dumps(cuerpo).encode()
        self.send_response(codigo)
        self.send_header("Content-Type", "application/json")
        self.send_header("Content-Length", str(len(datos)))
        self.end_headers()
        self.wfile.write(datos)

    def do_POST(self):
        crudo = self.rfile.read(int(self.headers["Content-Length"]))
        try:
            p = Peticion.model_validate_json(crudo)
        except ValidationError as e:
            return self.responder(422, {"error": e.errors()[0]["msg"]})
        x = np.array([p.features], dtype=np.float32)
        _, pr = sesion.run(None, {ENTRADA: x})
        self.responder(200, {"p_sube": round(float(pr[0, 1]), 6)})

srv = ThreadingHTTPServer(("127.0.0.1", 0), Servicio)
threading.Thread(target=srv.serve_forever, daemon=True).start()
URL = f"http://127.0.0.1:{srv.server_port}/predecir"

def pedir(cuerpo):
    req = urllib.request.Request(URL, json.dumps(cuerpo).encode(),
                                 {"Content-Type": "application/json"})
    try:
        with urllib.request.urlopen(req) as r:
            return r.status, json.load(r)
    except urllib.error.HTTPError as e:
        return e.code, json.load(e)

f0 = [float(v) for v in Xte[0]]
print(pedir({"modelo": "senal-n7", "features": f0}),
      "| in process:", round(float(p_sk[0]), 6))
print(pedir({"modelo": "senal-n7", "features": f0[:7]}))
print(pedir({"modelo": "senal-n7", "features": f0, "umbral": 0.4}))

(200, {'p_sube': 0.51919}) | in process: 0.51919
(422, {'error': 'List should have at least 8 items after validation, not 7'})
(422, {'error': 'Extra inputs are not permitted'})

The service returns the same probability as the in-process model, and rejects with a 422 a request with seven features and another with a field that is not in the contract (umbral, a threshold). And the price?

cuerpo = {"modelo": "senal-n7", "features": f0}
t_http = mejor(lambda: pedir(cuerpo), 50)
print("local HTTP at least 10 times slower than in process:",
      t_http / t_ort > 10)
srv.shutdown()

local HTTP at least 10 times slower than in process: True

Over local HTTP, about 540 µs per prediction: more than scikit-learn in the process. The model itself now costs only 9 µs; the rest is serializing the JSON, opening the connection, going through the server loop and coming back. That is the conclusion of the section: architecture weighs more than format. For a daily signal, 0.5 ms does not matter at all and the service brings order (one model, one deployment, one contract). For a system that reacts to every quote (lesson 4.5.1), the model goes inside the process.

CPU or GPU

A GPU executes thousands of identical operations in parallel. It pays off when there is a lot of computation per call: large networks, large batches, language models. A tree ensemble on one row is the opposite: little computation, many branches. For almost all tabular trading models, the CPU is more than enough, and it avoids copying data to GPU memory and back (lesson 6.6.5 covered the GPU for data processing, with RAPIDS).

With ONNX, changing hardware means changing the execution provider. The code is this one, not executed because the course machine has no GPU (it requires the onnxruntime-gpu package instead of onnxruntime, which since version 1.27 is published on PyPI built with CUDA 13.011Note 1 · ONNX Runtime, “CUDA Execution Provider” (https://onnxruntime.ai/docs/execution-providers/CUDA-ExecutionProvider.html): “Starting with version 1.27, GPU packages published to PyPI (onnxruntime-gpu) … are built with CUDA 13.0 by default”, and installation with pip install onnxruntime-gpu[cuda,cudnn]. Consulted on October 23, 2026.):

# pip install "onnxruntime-gpu[cuda,cudnn]"   (another environment, with a GPU)
import onnxruntime as ort

sesion = ort.InferenceSession(
    "senal.onnx",
    providers=["CUDAExecutionProvider", "CPUExecutionProvider"])
print(sesion.get_providers())   # the first available one is used

The two techniques that most reduce inference cost on CPU are batching (the latency cell) and quantizing: storing the weights as 8-bit integers, or fewer bits, instead of float32. ONNX Runtime quantizes ONNX models (its documentation recommends dynamic quantization for transformers and recurrent networks, and static quantization for convolutions), and it is what makes it possible to run language models with billions of parameters on a single consumer GPU (lesson 6.2.4). For a tree model it adds nothing: there are no weight matrices to compress.

The cost of LLMs in production

With language models the calculation changes in nature. The cost is no longer a machine but a price per token (lesson 6.2.1), with four prices per model: input tokens, output tokens, input read from the prompt cache and input written to it. The ones in the table come from the official pages of Anthropic and OpenAI, consulted on October 23, 2026, in USD per million tokens, for short contexts22Note 2 · Anthropic, “Pricing” (https://platform.claude.com/docs/en/about-claude/pricing): Claude Opus 5.5, 4 USD per million input tokens and 20 output; Sonnet 5.5, 2 and 10; Haiku 5.5, 0.10 and 0.50 (prompts of up to 100,000 tokens); writes to the 5-minute cache at 1.25 times input; reads at 0.05 times on Opus 5.5 and Sonnet 5.5 and at 0.1 times on the rest; batch API with a 50% discount on input and output. OpenAI, “Pricing” (https://developers.openai.com/api/docs/pricing), standard price and short context: gpt-6-astra, 10 and 50, cached input 1.00 and 30-minute write 12.50; gpt-6.1-sol, 2 and 10, 0.10 and 2.50; gpt-6-luna, 0.10 and 0.50, 0.01 and 0.125; the batch table is half. Both consulted on October 23, 2026.:

# USD per million tokens, short context. Official pricing pages of
# Anthropic and OpenAI, consulted on October 23, 2026.
PRECIOS = {   # input, output, cache read, cache write
    "Claude Opus 5.5":   (4.00, 20.00, 0.20, 5.00),
    "Claude Sonnet 5.5": (2.00, 10.00, 0.10, 2.50),
    "Claude Haiku 5.5":  (0.10, 0.50, 0.01, 0.125),
    "gpt-6-astra":       (10.00, 50.00, 1.00, 12.50),
    "gpt-6.1-sol":       (2.00, 10.00, 0.10, 2.50),
    "gpt-6-luna":        (0.10, 0.50, 0.01, 0.125),
}
LOTE = 0.5          # batch API: half price at both providers

def coste(modelo, entrada, salida, cache_r=0, cache_w=0, lote=False):
    """USD for one request; `entrada` = input tokens without cache."""
    pe, ps, pr, pw = PRECIOS[modelo]
    usd = (entrada * pe + salida * ps + cache_r * pr
           + cache_w * pw) / 1e6
    return usd * (LOTE if lote else 1.0)

print(f"{coste('Claude Sonnet 5.5', 1_000_000, 0):.2f}",
      f"{coste('gpt-6-astra', 0, 1_000_000, lote=True):.2f}")

2.00 25.00

Two discounts change the result a lot:

• Prompt caching. An initial part of the prompt that repeats across requests (instructions, examples, a document, an agent's history) is written once to the cache, somewhat more expensive than normal input (1.25 times), and is then read much more cheaply: 0.05 times the input price on Claude Opus 5.5, Claude Sonnet 5.5 and gpt-6.1-sol, 0.1 times on the other models in the table. The cache lasts from minutes to hours, depending on the option, and is refreshed when read.
• Batch API. Requests that can wait (hours, not seconds) are sent together and paid at half price at both providers.

Three typical workloads of a research system, with the tokens written out by hand:

• Headlines: 2,000 a day, classified with a prompt of 1,500 tokens of instructions and examples, always the same, plus the headline (60 tokens); it answers with a JSON label (30).
• Reports: 40 a day of 25,000 tokens, summarized in 1,000. Each report is different, so the cache does not help; they can wait until night, so the batch API does.
• Agent: 200 tasks a day of 8 steps (lesson 6.4.1). At each step it resends everything before: 3,000 initial tokens and 1,500 more per step, and it answers with 300.

import pandas as pd

SESIONES = 21

def titulares(m, cache):
    """2,000 headlines a day: instructions and examples (1,500 tokens,
    the same in all) + headline (60) and a JSON label (30)."""
    if cache:     # 10 writes a day; the rest, reads
        return (10 * coste(m, 60, 30, cache_w=1500)
                + 1990 * coste(m, 60, 30, cache_r=1500))
    return 2000 * coste(m, 1560, 30)

def informes(m, lote):
    """40 reports of 25,000 tokens; summary of 1,000."""
    return 40 * coste(m, 25_000, 1_000, lote=lote)

def agente(m, cache):
    """200 tasks of 8 steps. Each step resends the whole history:
    3,000 fixed tokens + 1,500 per previous step; answers 300."""
    total = 0.0
    for i in range(8):
        contexto = 3000 + 1500 * i
        if cache and i:    # reads what came before and writes the new part
            total += coste(m, 0, 300, cache_r=contexto - 1500,
                           cache_w=1500)
        elif cache:
            total += coste(m, 0, 300, cache_w=contexto)
        else:
            total += coste(m, contexto, 300)
    return 200 * total

filas = {}
for m in PRECIOS:
    base = titulares(m, False) + informes(m, False) + agente(m, False)
    opt = titulares(m, True) + informes(m, True) + agente(m, True)
    filas[m] = {"headlines": titulares(m, False) * SESIONES,
                "reports": informes(m, False) * SESIONES,
                "agent": agente(m, False) * SESIONES,
                "total": base * SESIONES,
                "with cache and batch": opt * SESIONES}
mes = pd.DataFrame(filas).T.round(0).astype(int)
mes

                   headlines  reports  agent  total  with cache and batch
Claude Opus 5.5          287      101   1310   1698                   629
Claude Sonnet 5.5        144       50    655    849                   314
Claude Haiku 5.5           7        3     33     42                    17
gpt-6-astra              718      252   3276   4246                  1714
gpt-6.1-sol              144       50    655    849                   314
gpt-6-luna                 7        3     33     42                    17

The figure at the top shows the totals on a logarithmic scale. From the most expensive model to the cheapest there is a factor of 100: 4,246 USD a month with gpt-6-astra versus 42 with Claude Haiku 5.5 or gpt-6-luna. And on every model, caching and batching leave the bill at between 37% and 40% of the standard price. The breakdown by workload, with Claude Sonnet 5.5:

The agent is 77% of the spend: 655 USD out of 849. It does not answer much (300 tokens per step); what is expensive is that each step resends the whole conversation, 66,000 input tokens per task. Caching brings it down to 265, because what has already been sent is read at 0.05 times the price. Headlines fall from 144 to 25 with caching (the fixed prompt is 94% of each request) and reports, from 50 to 25 with batching.

An in-house server?

The alternative to paying per token is renting a GPU and serving an open model with vLLM or Ollama (lesson 6.2.1). A real price: Hetzner's GEX45 server, with a 24 GB NVIDIA RTX PRO 4000 Blackwell, costs 214 EUR a month excluding VAT plus a 209 EUR setup fee33Note 3 · Hetzner, dedicated servers with GPU (https://www.hetzner.com/dedicated-rootserver/matrix-gpu/): GEX45, “€ 214.00 per month, excl. VAT” and “€ 209.00 one-time setup fee”, with a 24 GB NVIDIA RTX PRO 4000 Blackwell SFF Edition. ECB reference rate of October 23, 2026: 1 EUR = 1.1193 USD. Consulted on October 23, 2026.. Above what volume is it cheaper than the API?

GEX45_EUR = 214.00           # per month, excl. VAT (Hetzner, 2026-10-23)
ALTA_EUR = 209.00            # one-off
EURUSD = 1.1193              # ECB, reference rate of 2026-10-23
servidor_usd = GEX45_EUR * EURUSD

def tokens_equivalentes(m, frac_salida=0.02):
    """Tokens per month at which the API costs as much as the server,
    with 2% output tokens (as in the workloads above)."""
    pe, ps, *_ = PRECIOS[m]
    usd_mtok = (1 - frac_salida) * pe + frac_salida * ps
    return servidor_usd / usd_mtok * 1e6

print(f"server: {servidor_usd:.2f} USD a month")
for m in ("Claude Haiku 5.5", "gpt-6-luna", "Claude Sonnet 5.5"):
    tok = tokens_equivalentes(m)
    por_s = tok / (30 * 24 * 3600)
    print(f"{m:18} {tok / 1e9:6.2f} billion tokens a month"
          f" = {por_s:6.0f} tokens/s nonstop")

server: 239.53 USD a month
Claude Haiku 5.5     2.22 billion tokens a month =    856 tokens/s nonstop
gpt-6-luna           2.22 billion tokens a month =    856 tokens/s nonstop
Claude Sonnet 5.5    0.11 billion tokens a month =     43 tokens/s nonstop

Against the small API models (Claude Haiku 5.5, gpt-6-luna), the server is only cheaper above roughly 2.2 billion tokens a month: 856 tokens per second, day and night, all month long. Against Claude Sonnet 5.5, above 43 tokens per second, but a model that fits in 24 GB is not comparable with Sonnet: the honest comparison is with the small models. Whether a GPU like that sustains 856 tokens per second with a useful model is something that has to be measured with the specific model and server (vLLM batches requests and its throughput depends heavily on batch size); there is no reliable figure to give here.

The decision, therefore, is rarely about price at the scale of an individual or a small team:

A budget guard

An API bill has no ceiling by itself. A badly written loop that repeats the same call, an agent that does not finish (lesson 6.4.5) or a change that multiplies the length of the prompt shows up at the end of the month. The control is the same as the order risk engine, applied to spending: each call is authorized beforehand with its maximum possible cost (the estimated input plus max_tokens of output), recorded with the real cost returned by the API and, once the daily limit is reached, there are no more calls. And a request identical to one already made is served from a local cache, without calling:

flowchart TB
  P[Request] --> C{In cache}
  C -->|yes| R[Response]
  C -->|no| B{Within limit}
  B -->|no| X[Reject and alert]
  B -->|yes| A[API]
  A --> G[Record cost]
  G --> R

import hashlib

class Presupuesto:
    """Authorizes each call with its maximum possible cost and cuts off
    when the daily limit is reached. Identical requests are served
    from the cache without calling the API."""

    def __init__(self, modelo, limite_usd):
        self.modelo, self.limite = modelo, limite_usd
        self.gastado, self.cache = 0.0, {}
        self.llamadas = self.de_cache = self.rechazadas = 0

    def llamar(self, prompt, max_salida, api):
        clave = hashlib.sha256(f"{self.modelo}|{max_salida}|{prompt}"
                               .encode()).hexdigest()
        if clave in self.cache:
            self.de_cache += 1
            return self.cache[clave]
        n_in = len(prompt.split()) * 4 // 3     # prudent estimate
        peor = coste(self.modelo, n_in, max_salida)
        if self.gastado + peor > self.limite:
            self.rechazadas += 1
            raise RuntimeError("daily budget exhausted")
        texto, uso = api(prompt, max_salida)
        self.gastado += coste(self.modelo, uso["in"], uso["out"])
        self.llamadas += 1
        self.cache[clave] = texto
        return texto

def api_simulada(prompt, max_salida):
    """Always answers the same and counts tokens like an API."""
    n = len(prompt.split())
    return "neutral", {"in": n * 4 // 3, "out": min(max_salida, 40)}

g = Presupuesto("Claude Sonnet 5.5", limite_usd=5.00)
largo = " ".join(["word"] * 6000)        # an 8,000-token report
for i in range(400):                     # a badly written loop
    g.llamar("Classify: Nore rises 3%", 50, api_simulada)
for i in range(400):                     # 400 different reports
    try:
        g.llamar(f"Report {i}: {largo}", 1000, api_simulada)
    except RuntimeError:
        break
print(f"calls {g.llamadas} · from cache {g.de_cache} · "
      f"rejected {g.rechazadas} · spent {g.gastado:.2f} USD")

calls 305 · from cache 399 · rejected 1 · spent 4.99 USD

The badly written loop repeats the same classification 400 times: one real call and 399 served from the cache. The 400 different reports exhaust the 5 USD budget at call 305 (each 8,000-token report costs about 0.016 USD, and the worst case with 1,000 output tokens, 0.026): the guard rejects the next one, the loop stops and spending stays at 4.99 USD. In production, the rejection triggers an alert (lesson 7.2.3), the cost of each call is published as a metric, and limits are also set in the provider's console, which is the last line of defense if the guard fails.

Exercises

1. Repeat the agent cost with Claude Opus 5.5 for tasks of 16 steps instead of 8, with and without caching. Why does the cost without caching more than double?

def agente_n(m, pasos, cache):
    total = 0.0
    for i in range(pasos):
        contexto = 3000 + 1500 * i
        if cache and i:
            total += coste(m, 0, 300, cache_r=contexto - 1500,
                           cache_w=1500)
        elif cache:
            total += coste(m, 0, 300, cache_w=contexto)
        else:
            total += coste(m, contexto, 300)
    return 200 * total * SESIONES

m = "Claude Opus 5.5"
for pasos in (8, 16):
    print(pasos, "steps:", round(agente_n(m, pasos, False)),
          "USD a month without cache ·", round(agente_n(m, pasos, True)),
          "with cache")

8 steps: 1310 USD a month without cache · 529 with cache
16 steps: 4234 USD a month without cache · 1109 with cache

2. A client of the HTTP service sends a request with NaN in the first feature (json.dumps writes it that way by default). Does the contract accept it? Fix it.

from pydantic import FiniteFloat

crudo = json.dumps({"modelo": "senal-n7",
                    "features": [float("nan")] + f0[1:]})
print(crudo[:40])
p = Peticion.model_validate_json(crudo)
x = np.array([p.features], dtype=np.float32)
print("accepted; p_sube =",
      round(float(sesion.run(None, {ENTRADA: x})[1][0, 1]), 6))

class PeticionFinita(Peticion):
    features: list[FiniteFloat] = Field(min_length=K, max_length=K)

try:
    PeticionFinita.model_validate_json(crudo)
except ValidationError as e:
    print("rejected:", e.errors()[0]["msg"], e.errors()[0]["loc"])

{"modelo": "senal-n7", "features": [NaN,
accepted; p_sube = 0.47686
rejected: Input should be a finite number ('features', 0)

3. A colleague proposes moving headline classification to an open model on an in-house server “to save money”. With the lesson's numbers, what would you answer?