How an LLM works: from the next token to the API.

A model that predicts the next token

A large language model (LLM) does one thing only: given a text, it computes the probability of every possible immediate continuation and picks one. It repeats the step with the continuation appended, and that is how it writes whole paragraphs. Everything that looks like reasoning, memory or knowledge comes out of that loop. The architecture that makes it possible at scale is the transformer (Vaswani et al., 2017), which already appeared with time series (lesson 5.7.2): attention layers that let each position look at all the previous ones.

flowchart TB
  A[Text] --> B[Tokens]
  B --> C[Transformer]
  C --> D[Logits]
  D --> E[Softmax]
  E --> F[Pick token]
  F --> B

Text does not go in as letters or as words, but as tokens: pieces of a fixed vocabulary that the tokenizer learned when the model was built (lesson 6.1.1 explains BPE and why Spanish costs more tokens than English). The model returns one real number, a logit, for each piece of the vocabulary; the softmax turns them into probabilities; a decision rule (the most likely one, or a draw) picks a piece; and the chosen piece is appended to the input.

This lesson uses Qwen2.5-0.5B-Instruct, an open model from Alibaba with an Apache-2.0 license and 0.49B parameters, pinned in the course's ia environment by commit and sha256 and loaded without network access. It is small on purpose: it fits on a laptop CPU and lets you see the mechanism. It is not a commercial-quality model, and the lesson uses it to show how it works and how it fails, not to promise results.

import numpy as np
import pandas as pd
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

torch.manual_seed(0)
torch.use_deterministic_algorithms(True)
NOMBRE = "Qwen/Qwen2.5-0.5B-Instruct"
tok = AutoTokenizer.from_pretrained(NOMBRE)
llm = AutoModelForCausalLM.from_pretrained(
    NOMBRE, dtype=torch.float32).eval()

cfg, gen = llm.config, llm.generation_config
print(f"parameters: {llm.num_parameters():,}")
print(f"vocabulary: {len(tok):,} pieces")
print(f"layers: {cfg.num_hidden_layers}, "
      f"context: {cfg.max_position_embeddings:,} tokens")
print(f"defaults: do_sample={gen.do_sample}, "
      f"temperature={gen.temperature}, top_p={gen.top_p}, "
      f"top_k={gen.top_k}, "
      f"repetition_penalty={gen.repetition_penalty}")

parameters: 494,032,768
vocabulary: 151,665 pieces
layers: 24, context: 32,768 tokens
defaults: do_sample=True, temperature=0.7, top_p=0.8, top_k=20, repetition_penalty=1.1

The weights are stored in bf16 and loaded in float32, the format in which the environment was verified to be deterministic. The last line is the generation configuration the model publishes: by default it samples (do_sample=True) with temperature 0.7, top-p 0.8 and top-k 20, and penalizes repetitions with a factor of 1.1. When this lesson asks for greedy decoding (do_sample=False), the repetition penalty stays on; the other sampling parameters are ignored.

Tokens: what the model sees and what you pay for

The course's example texts and prompts are in Spanish, as is the corpus of news and filings of this level; the first cell compares a Spanish sentence with its English equivalent.

frases = [
    "La compañía prevé unas ventas de entre 2.440 y 2.569 "
    "millones de dólares.",
    "We expect revenue in the range of $2,440 million to "
    "$2,569 million.",
]
for f in frases:
    ids = tok(f)["input_ids"]
    piezas = [tok.decode([i]) for i in ids]
    print(len(f), "characters,", len(ids), "tokens")
    print("|".join(piezas))

73 characters, 25 tokens
La| compañía| prev|é| unas| ventas| de| entre| |2|.|4|4|0| y| |2|.|5|6|9| millones| de| dólares|.
67 characters, 23 tokens
We| expect| revenue| in| the| range| of| $|2|,|4|4|0| million| to| $|2|,|5|6|9| million|.

The two sentences say the same thing in 73 and 67 characters; the Spanish one uses 25 tokens and the English one 23. Two details matter in finance. Qwen writes each digit as a separate token (2|.|4|4|0): the model does not see the number 2,440 but five pieces, and the thousands separator is one more token, which means opposite things in Spanish and in English. And words are split into chunks ( prev|é), which the model has to put back together.

Counting tokens is not an academic exercise. Commercial APIs charge per input and output token, the context window is measured in tokens and compute time grows with them. The document format weighs as much as the language:

The figure measures tokens per 1,000 characters in the level's synthetic texts with the Qwen tokenizer. The HTML of a 10-K costs 277 tokens per 1,000 characters and its text without tags, 221: the <td>, <tr> and <sup> tags are 25% extra tokens that add nothing for the model. The Markdown table of the CNMV-style documents (CNMV is Spain's securities regulator) is the most expensive (330), because of the pipes and the figures with separators. Cleaning up the format before sending a document to an API is the easiest saving.

From logits to probabilities

For an input text, the last position of the transformer produces a vector of logits $z_i$, one per vocabulary piece. The probability that the next piece is $i$ is the softmax:

One forward pass is enough to see it. The prompt is a half-written news sentence, “Las acciones cayeron un 8 % tras unos resultados” (“Shares fell 8% after results”):

prompt = "Las acciones cayeron un 8 % tras unos resultados"
x = tok(prompt, return_tensors="pt")
with torch.inference_mode():
    z = llm(**x).logits[0, -1].double()   # last-position logits

p = torch.softmax(z, dim=-1).numpy()
orden = np.argsort(-p)
top = pd.DataFrame({
    "token": [repr(tok.decode([i])) for i in orden[:10]],
    "prob_%": (100 * p[orden[:10]]).round(1)})
print(f"{len(p):,} probabilities, sum {p.sum():.4f}")
top

151,936 probabilities, sum 1.0000
       token  prob_%
0     ' neg'    11.9
1  ' económ'     8.6
2      ' de'     7.4
3     ' más'     3.5
4   ' posit'     2.8
5     ' que'     1.9
6     ' muy'     1.7
7     ' baj'     1.6
8   ' optim'     1.5
9      ' fu'     1.3

The output vector has 151,936 positions, slightly more than the tokenizer's 151,665 pieces: the output layer is rounded up to a size convenient for computation, and the extra positions do not correspond to any piece (their probability is practically zero). The header figure plots the table. The most likely token is “ neg” (11.9%), the start of “negativos” (“negative”); it is followed by “ económ” (8.6%), from “económicos” (“economic”), and “ de” (7.4%, “of”). There are two lessons in these ten rows. First: the probability is very spread out; the most likely token does not reach 12% and the top ten add up to a little less than half. Second: “ posit” (2.8%) and “ optim” (1.5%) are among the top ten even though the shares fell 8%. The model does not reason about the news; it reproduces how similar sentences usually continue in its training data, and “resultados positivos” (“positive results”) is a very frequent phrase.

Temperature and top-p

The temperature $T$ divides the logits before the softmax:

With $T < 1$ the differences between logits grow and the probability concentrates on the first tokens; with $T \to 0$ all of it goes to the most likely one (greedy decoding). With $T > 1$ the distribution flattens. Nucleus sampling, or top-p (Holtzman et al., 2020), draws only from the smallest set of tokens whose probability adds up to at least $p$, and renormalizes within it. The two parameters can be combined.

def con_temperatura(z, T):
    return torch.softmax(z / T, dim=-1).numpy()

filas = []
for T in [0.2, 0.5, 0.7, 1.0, 1.5, 2.0]:
    q = np.sort(con_temperatura(z, T))[::-1]
    nucleo = int(np.searchsorted(np.cumsum(q), 0.9) + 1)
    filas.append({"T": T, "p_top_%": round(100 * q[0], 1),
                  "nucleus_top_p_0.9": nucleo})
pd.DataFrame(filas).set_index("T")

     p_top_%  nucleus_top_p_0.9
T
0.2     77.5                  2
0.5     43.9                  6
0.7     27.8                 43
1.0     11.9                370
1.5      2.8               7651
2.0      0.7              39393

With $T = 0.2$ the top token takes 77.5% and two tokens are enough to reach 90% of the probability. With $T = 1$ (the model's distribution, untouched) the 90% nucleus has 370 tokens. With $T = 2$, the top token drops to 0.7% and the nucleus exceeds 39,000 pieces: anything can come out. The figure plots the nucleus size on a logarithmic scale for the whole grid of temperatures.

What that means for a text shows up when you sample. Each configuration generates five 10-token continuations with the same seed, so the cell gives the same result on every run:

def ver(s):
    """Text for display: characters outside the Latin alphabet
    (Chinese, Cyrillic…) become “·” and line breaks
    become “¶”."""
    s = s.replace("\n", "¶")
    return "".join(c if ord(c) < 256 else "·" for c in s)

configs = [(0.2, 1.0), (1.0, 1.0), (1.0, 0.9), (1.6, 1.0)]
for T, top_p in configs:
    torch.manual_seed(42)
    with torch.inference_mode():
        o = llm.generate(**x, do_sample=True, temperature=T,
                         top_p=top_p, top_k=0, max_new_tokens=10,
                         num_return_sequences=5,
                         pad_token_id=tok.eos_token_id)
    n = x["input_ids"].shape[1]
    print(f"T={T}, top_p={top_p}")
    for fila in o:
        print("  …" + ver(tok.decode(fila[n:],
                                      skip_special_tokens=True)))

T=0.2, top_p=1.0
  … negativos en la semana pasada, mientras que
  … económicos negativos en los últimos días, mientras
  … negativos en la semana pasada, mientras que
  … negativos de la semana pasada, mientras que
  … económicos negativos en los últimos meses, pero
T=1.0, top_p=1.0
  … menos fuertes y brutalidades en marzo.
  … económicos injustamente subidas. Desde luego,
  … de carteles rentables para ERC y BBVA
  … negativos de la feria.¶······
  … sucios, asentándose en 1.
T=1.0, top_p=0.9
  … menos fuertes y oportunistas de
  … económicos aún poca posibilidad para la economía
  … de carteles - Correo¶La pizar
  … negativos de la feria.¶Además,
  … sucios, asentándose en 1.
T=1.6, top_p=1.0
  … menos fuertes sub·T pivó acompañ
  … f_PROPERTIES injusta manos pos-JPP, según
  … comparativos terribles para ERC mit pizar
  … ameritados primero, estos protagonistas buscan
  … sucieron, asent ba·ar·os desde que··

With $T = 0.2$ the five continuations are variations of “negativos en la semana pasada” (“negative last week”) and “económicos negativos” (“negative economic”), plausible and nearly repeated phrases (two are identical). With $T = 1$ absurd continuations appear (“brutalidades en marzo”, “brutalities in March”; “carteles rentables para ERC y BBVA”, “profitable cartels for ERC and BBVA”, with real names pulled out of nowhere), a line break (which the cell shows as “¶”) and characters from other alphabets (shown as “·”): Qwen's vocabulary covers many languages and the tail of the distribution holds a bit of everything. With top-p 0.9 and the same seed, the five continuations start the same as with top-p 1 (the first tokens were inside the nucleus) and diverge when the cutoff changes the distribution: “menos fuertes y brutalidades” becomes “menos fuertes y oportunistas” (“weaker and opportunistic”). With $T = 1.6$ the text falls apart into fragments such as “f_PROPERTIES” or “pos-JPP”. top_k=0 turns off the top-k 20 that comes with the default configuration, to see the effect of temperature without another filter.

The context window

The model only sees what fits in its context window, measured in tokens and shared between the input and the response. The Qwen2.5-0.5B-Instruct model card gives 32,768 tokens of context and up to 8,192 of generation; Claude Opus 5.5 supports 1 million tokens according to Anthropic's models page. An API keeps no memory between calls: in a conversation, each request resends the whole history and you pay for all of it.

from bs4 import BeautifulSoup

fil = pd.read_parquet("filings6_documentos.parquet")
ll = pd.read_parquet("llamadas6_transcripciones.parquet")

def n_tok(t):
    return len(tok(t)["input_ids"])

def texto(html):
    return BeautifulSoup(html, "html.parser").get_text(" ")

k10 = fil[fil["tipo"] == "10-K"]
llamadas = ll.groupby("id_llamada")["texto"].apply(" ".join)
emisor = fil[fil["ticker"] == k10["ticker"].iloc[0]]
filas = {
    "10-K (HTML)": k10["texto"].map(n_tok).median(),
    "10-K (text)": k10["texto"].map(texto).map(n_tok).median(),
    "full call": llamadas.map(n_tok).median(),
    "8 years of one issuer": emisor["texto"].map(n_tok).sum(),
}
t = pd.Series(filas, name="tokens").round().astype(int).to_frame()
t["fit_in_32k"] = 32_768 // t["tokens"]
t

                       tokens  fit_in_32k
10-K (HTML)               776          42
10-K (text)               538          60
full call                 344          95
8 years of one issuer   23578           1

The course's synthetic documents are short (a real 10-K has tens of thousands of words, not three thousand characters), but the calculation is the same with real documents. Here a 10-K fits dozens of times and an issuer's entire document history over eight years fits just once. A real 10-K, with its exhibits, may not fit in a 32,000-token window and has to be split into chunks (lesson 6.3.2).

Fitting does not mean being used well. Liu et al. (2024) measured that the accuracy of several models when answering with a fact from the context is higher when the fact is at the beginning or at the end and drops when it is in the middle, “even for explicitly long-context models”11Note 1 · Liu, N. F. et al. (2024): “Lost in the Middle: How Language Models Use Long Contexts”, Transactions of the Association for Computational Linguistics 12, 157-173 (arXiv:2307.03172, https://arxiv.org/abs/2307.03172).. Putting an entire annual report in the prompt and asking about a figure on page 40 is not the same as retrieving the section and giving it to the model (lesson 6.3.3).

Hallucinations: responses shaped like answers

A hallucination is a fluent, confident response that is not true. To provoke one, it is enough to ask about something the model cannot know. Busmar Hogar is a company in the course's fictitious panel: no LLM has seen it. The cell asks it the same question in three ways: plainly, with an instruction to abstain if it does not know the figure, and with the company's announcement (a synthetic CNMV-style “hecho relevante”, a material event disclosure) in the context. The question and the instructions are in Spanish, like the rest of the corpus. Decoding is greedy, and the cell also computes the mean probability the model gave to each token it wrote.

import textwrap

doc = fil.loc[fil["id_doc"] == "D000007", "texto"].iloc[0]
pregunta = ("¿Cuánto facturó Busmar Hogar en el cuarto trimestre "
            "de 2018?")
prompts = {
    "no context": [{"role": "user", "content": pregunta}],
    "with abstention": [
        {"role": "system", "content":
         "Responde solo con datos que conozcas con certeza. Si no "
         "conoces el dato, responde exactamente: No lo sé."},
        {"role": "user", "content": pregunta}],
    "with the document": [
        {"role": "system", "content":
         "Responde solo con el documento. Si el dato no está, "
         "di: No consta."},
        {"role": "user", "content":
         f"Documento:\n{doc}\n\nPregunta: {pregunta}"}],
}
tok.padding_side = "left"
textos = [tok.apply_chat_template(m, add_generation_prompt=True,
                                  tokenize=False)
          for m in prompts.values()]
b = tok(textos, return_tensors="pt", padding=True)
with torch.inference_mode():
    o = llm.generate(**b, do_sample=False, max_new_tokens=40,
                     pad_token_id=tok.eos_token_id,
                     output_scores=True,
                     return_dict_in_generate=True)
lp = llm.compute_transition_scores(o.sequences, o.scores,
                                   normalize_logits=True)
n = b["input_ids"].shape[1]
for k, nombre in enumerate(prompts):
    nuevos = o.sequences[k, n:]
    vale = nuevos != tok.eos_token_id
    p_media = lp[k][vale].exp().mean().item()
    print(f"[{nombre}] mean prob. {p_media:.2f}")
    print(textwrap.fill(ver(tok.decode(nuevos,
                                       skip_special_tokens=True)),
                        68, initial_indent="  ",
                        subsequent_indent="  "))

[no context] mean prob. 0.58
  Busmar Hogar ha notificado al mercado que su facturación en el
  cuarto trimestre de 2018 fue de $35 millones de dólares. Este
  cifra se ref
[with abstention] mean prob. 0.91
  No lo sé.
[with the document] mean prob. 0.86
  2.942,6 millones de USD

Without context, the model makes things up: it says Busmar Hogar “ha notificado al mercado” (“has notified the market”) revenue of 35 million dollars. The real figure in the document is 2,942.6 million; the model had no way of knowing it and still writes a sentence with the structure of a verified fact. With the instruction to abstain it answers “No lo sé.” (“I don't know.”), and with the document in the context it copies the correct figure, “2.942,6 millones de USD” (USD 2,942.6 million).

The mean token probability is lower in the made-up answer (0.58) than in the other two (0.91 and 0.86). It is tempting to use it as a detector, but it is not reliable: a model can make things up with great confidence, especially in figures with a typical shape, and it can hesitate on correct answers. Farquhar et al. (2024) propose measuring the uncertainty over the meaning of several sampled answers (semantic entropy), not over the words, and even so they detect only a subset of hallucinations22Note 2 · Farquhar, S., Kossen, J., Kuhn, L. and Gal, Y. (2024): “Detecting hallucinations in large language models using semantic entropy”, Nature 630, 625-630 (https://doi.org/10.1038/s41586-024-07421-0)..

Why do they hallucinate? Kalai, Nachum, Vempala and Zhang (2025) attribute it to how they are trained and evaluated: “the training and evaluation procedures reward guessing over acknowledging uncertainty”. On a multiple-choice exam, answering something scores more than leaving it blank, and models are optimized for those exams. The practical consequence is that the instruction to abstain helps, but what really fixes the problem is giving the model the document and asking it to answer only from it. That is the idea behind RAG (lessons 6.3.1 to 6.3.3).

Commercial APIs: the same protocol, another server

The OpenAI and Anthropic APIs are HTTP services: the client sends a JSON with the messages and the parameters, and the server returns another JSON with the response and the token count. Both have official Python SDKs (openai 3.37.0 and anthropic 2.1.0 in the course environment). OpenAI recommends its Responses API for new projects and maintains the Chat Completions API; Anthropic has the Messages API. This is the real code, not run because it needs a key and network access:

from openai import OpenAI

cliente = OpenAI()           # reads OPENAI_API_KEY from env
r = cliente.responses.create(
    model="gpt-6-astra",
    instructions="Eres un analista. Responde en una frase.",
    input="¿Qué es el guidance de una empresa?",
)
print(r.output_text)

import anthropic

cliente = anthropic.Anthropic()   # reads ANTHROPIC_API_KEY
r = cliente.messages.create(
    model="claude-opus-5-5",
    max_tokens=1024,
    system="Eres un analista. Responde en una frase.",
    messages=[{"role": "user",
               "content": "¿Qué es el guidance de una empresa?"}],
)
print(r.content[0].text, r.stop_reason, r.usage.output_tokens)

The prompts ask, in Spanish, “You are an analyst. Answer in one sentence.” and “What is a company's guidance?”. The model names are the ones each provider's documentation uses as of the lesson date (gpt-6-astra in OpenAI's quickstart, claude-opus-5-5 on Anthropic's models page); they change often and are best pinned in the project configuration, not scattered across the code. The key never goes in the code: both SDKs read it from an environment variable.

Since there is no network and no keys, the lesson sets up a local server that speaks the same protocol. It answers POST /v1/chat/completions with the shape of an OpenAI chat.completion object and POST /v1/messages with that of an Anthropic message, as each API reference describes them, and generates the text with the local Qwen. It is, in miniature, what Ollama and vLLM do:

flowchart TB
  A[Your code] --> B[Official SDK]
  B -->|base_url| C[Commercial API]
  B -->|base_url| D[Local server]
  D --> E[Local model]

import json
import threading
from http.server import BaseHTTPRequestHandler
from http.server import ThreadingHTTPServer

def responder(mensajes, max_tokens):
    """Greedy generation with the local Qwen. Returns the text,
    the input and output tokens and whether it stopped by
    itself."""
    x = tok.apply_chat_template(mensajes, return_tensors="pt",
                                add_generation_prompt=True,
                                return_dict=True)
    with torch.inference_mode():
        o = llm.generate(**x, do_sample=False,
                         max_new_tokens=max_tokens,
                         pad_token_id=tok.eos_token_id)
    nuevos = o[0, x["input_ids"].shape[1]:]
    fin = bool(torch.isin(nuevos, torch.tensor(
        gen.eos_token_id)).any())
    n_out = len(nuevos) - int(fin)
    texto = tok.decode(nuevos, skip_special_tokens=True)
    return texto, x["input_ids"].shape[1], n_out, fin

class APILocal(BaseHTTPRequestHandler):
    def log_message(self, *args):        # no logging to stderr
        pass

    def do_POST(self):
        n = int(self.headers["Content-Length"])
        req = json.loads(self.rfile.read(n))
        auth = self.headers.get("Authorization", "").split()[0]
        print(f"  [server] POST {self.path} · auth={auth} · "
              f"anthropic-version="
              f"{self.headers.get('anthropic-version')}")
        if self.path == "/v1/chat/completions":
            limite = (req.get("max_completion_tokens")
                      or req.get("max_tokens") or 64)
            txt, n_in, n_out, fin = responder(req["messages"],
                                              limite)
            r = {"id": "chatcmpl-local-1",
                 "object": "chat.completion",
                 "created": 1767225600, "model": req["model"],
                 "choices": [{
                     "index": 0, "logprobs": None,
                     "finish_reason": "stop" if fin else "length",
                     "message": {"role": "assistant",
                                 "content": txt, "refusal": None}}],
                 "usage": {"prompt_tokens": n_in,
                           "completion_tokens": n_out,
                           "total_tokens": n_in + n_out}}
        elif self.path == "/v1/messages":
            sis = req.get("system")
            msgs = ([{"role": "system", "content": sis}]
                    if sis else []) + req["messages"]
            txt, n_in, n_out, fin = responder(msgs,
                                              req["max_tokens"])
            r = {"id": "msg_local_1", "type": "message",
                 "role": "assistant", "model": req["model"],
                 "content": [{"type": "text", "text": txt}],
                 "stop_reason": "end_turn" if fin else "max_tokens",
                 "stop_sequence": None,
                 "usage": {"input_tokens": n_in,
                           "output_tokens": n_out}}
        else:
            self.send_error(404)
            return
        cuerpo = json.dumps(r).encode()
        self.send_response(200)
        self.send_header("Content-Type", "application/json")
        self.send_header("Content-Length", str(len(cuerpo)))
        self.end_headers()
        self.wfile.write(cuerpo)

srv = ThreadingHTTPServer(("127.0.0.1", 0), APILocal)
threading.Thread(target=srv.serve_forever, daemon=True).start()
URL = f"http://127.0.0.1:{srv.server_port}"
print("local server on 127.0.0.1, port chosen by the OS")

local server on 127.0.0.1, port chosen by the OS

The server listens only on 127.0.0.1 and on a port the operating system chooses. created is a fixed date so that the output does not depend on the clock. Now, OpenAI's official SDK with its Chat Completions method, with no change other than base_url (the Responses API has another response shape and the simulator does not mimic it):

from openai import OpenAI

oa = OpenAI(base_url=URL + "/v1", api_key="local",
            max_retries=0)
r = oa.chat.completions.create(
    model="qwen2.5-0.5b-instruct", max_completion_tokens=24,
    messages=[
        {"role": "system",
         "content": "Eres un analista. Responde en una frase."},
        {"role": "user",
         "content": "¿Qué es el guidance de una empresa?"}])
print(type(r).__name__, "·", r.choices[0].finish_reason, "·",
      r.usage.prompt_tokens, "+", r.usage.completion_tokens,
      "tokens")
print(textwrap.fill(ver(r.choices[0].message.content), 68))

  [server] POST /v1/chat/completions · auth=Bearer · anthropic-version=None
ChatCompletion · length · 34 + 24 tokens
El Guidance de una empresa se refiere a los planes y estrategias que
la empresa utiliza para alcanzar sus objet

The SDK has sent the key as Authorization: Bearer and has turned the JSON into a ChatCompletion object with typed fields. finish_reason is length: the response hit the 24-token limit and was cut off mid-sentence (it starts “a company's guidance refers to the plans and strategies the company uses to reach its…”). It is the first field to check in production, because a truncated response is an incomplete response (and a truncated JSON cannot be parsed, lesson 6.2.2). The 34 input tokens include the markers of Qwen's chat template, not just the text.

import anthropic

an = anthropic.Anthropic(base_url=URL, api_key="local",
                         max_retries=0)
r = an.messages.create(
    model="qwen2.5-0.5b-instruct", max_tokens=24,
    system="Eres un analista. Responde en una frase.",
    messages=[{"role": "user",
               "content": "¿Qué es el guidance de una empresa?"}])
print(type(r).__name__, "·", r.stop_reason, "·",
      r.usage.input_tokens, "+", r.usage.output_tokens, "tokens")
print(textwrap.fill(ver(r.content[0].text), 68))
srv.shutdown()

  [server] POST /v1/messages · auth=Bearer · anthropic-version=2026-10-21
Message · max_tokens · 34 + 24 tokens
El Guidance de una empresa se refiere a los planes y estrategias que
la empresa utiliza para alcanzar sus objet

The same response in another wrapper: the system goes separately from the messages, the content is a list of blocks (content[0].text) and the stop reason is called stop_reason (max_tokens instead of length). The server has seen the header anthropic-version: 2026-10-21, the API version that SDK 2.1.0 pins, and the key also as Authorization: Bearer; Anthropic's documentation presents x-api-key as a legacy alternative that still works. The differences between providers lie in these details, and that is why it pays to isolate the model call in a function of your own: switching provider or moving to a local model touches a single place.

Ollama and vLLM: the model on your machine

The two most widely used local servers do at scale what the simulator does in one cell:

Sources: Ollama documentation (OpenAI and Anthropic compatibility, structured outputs and FAQ) and vLLM documentation (quickstart, online serving and structured outputs), accessed on 2026-10-23. The code, not run (there is neither Ollama nor a GPU in the course environment):

ollama pull qwen2.5:0.5b-instruct
vllm serve Qwen/Qwen2.5-1.5B-Instruct

from openai import OpenAI

# Ollama: the key is required by the SDK but ignored
cliente = OpenAI(base_url="http://localhost:11434/v1/",
                 api_key="ollama")
r = cliente.chat.completions.create(
    model="qwen2.5:0.5b-instruct",
    messages=[{"role": "user", "content": "Hola"}])
# vLLM: base_url="http://localhost:8000/v1" and the name of the
# served model ("Qwen/Qwen2.5-1.5B-Instruct")

The client code is the same as in the previous cell. That compatibility is the practical reason to write the model-access layer against the OpenAI or Anthropic format: the same program is tested with a small, free local model and run with whichever one suits.

What to choose for trading research

Inside information or information subject to confidentiality (client orders, positions, committee notes) is a constraint, not one more criterion: it is either processed locally or under a contract that covers it. For research with public documents, quality usually rules, and a commercial API clearly beats a 0.5B model. Lessons 6.2.2 and 6.2.3 measure it: on which tasks the small model is enough and on which it is not.

Exercises

1. Repeat the temperature table with another prompt, “Los analistas esperan que el beneficio por acción” (“Analysts expect earnings per share”), and compute the size of the top-p 0.9 nucleus with $T = 0.7$ and with $T = 1$. Is it more or less concentrated than the one in the lesson?

x2 = tok("Los analistas esperan que el beneficio por acción",
         return_tensors="pt")
with torch.inference_mode():
    z2 = llm(**x2).logits[0, -1].double()
for T in [0.7, 1.0]:
    q = np.sort(con_temperatura(z2, T))[::-1]
    print(T, round(100 * q[0], 1),
          int(np.searchsorted(np.cumsum(q), 0.9) + 1))

0.7 50.1 7
1.0 26.5 113

2. Estimate how many input tokens it would cost to send all the documents in filings6_documentos.parquet to an API, in HTML and in plain text, and the cost with an example input price of 2 dollars per million tokens (it is not any provider's price: check their pricing page).

html = fil["texto"].map(n_tok).sum()
plano = fil["texto"].map(texto).map(n_tok).sum()
PRECIO = 2.0                       # USD per million (example)
for nombre, n in [("HTML", html), ("text", plano)]:
    print(f"{nombre}: {n:,} tokens · "
          f"{n / 1e6 * PRECIO:.2f} USD")

HTML: 1,530,557 tokens · 3.06 USD
text: 1,066,910 tokens · 2.13 USD

3. APIs return an error when the input does not fit in the context. Write a server like the one in the lesson that rejects any request with more than 50 input tokens with a 400 and an error JSON, and check which exception the OpenAI SDK raises.

import openai

class APIPequena(APILocal):
    def do_POST(self):
        n = int(self.headers["Content-Length"])
        req = json.loads(self.rfile.read(n))
        n_in = n_tok(tok.apply_chat_template(
            req["messages"], add_generation_prompt=True,
            tokenize=False))
        cuerpo = json.dumps({"error": {
            "message": f"{n_in} tokens > 50",
            "type": "invalid_request_error",
            "code": "context_length_exceeded"}}).encode()
        self.send_response(400)
        self.send_header("Content-Type", "application/json")
        self.send_header("Content-Length", str(len(cuerpo)))
        self.end_headers()
        self.wfile.write(cuerpo)

srv2 = ThreadingHTTPServer(("127.0.0.1", 0), APIPequena)
threading.Thread(target=srv2.serve_forever, daemon=True).start()
oa2 = OpenAI(base_url=f"http://127.0.0.1:{srv2.server_port}/v1",
             api_key="local", max_retries=0)
try:
    oa2.chat.completions.create(
        model="local", messages=[{"role": "user",
                                  "content": doc}])
except openai.BadRequestError as e:
    print(type(e).__name__, e.status_code, e.code)
srv2.shutdown()

BadRequestError 400 context_length_exceeded