---
title: "6.2.1 How LLMs work: APIs and local models"
description: "Next token, logits, temperature, top-p, context window and hallucinations with a local LLM, and the OpenAI and Anthropic APIs against a simulator."
url: "https://pythonparatrading.com/en/python-trading-course/level-6/how-llms-work-apis-and-local-models/"
lang: "en"
image: "https://pythonparatrading.com/og/en/python-trading-course/level-6/how-llms-work-apis-and-local-models.png"
---

# 6.2.1 · How an LLM works: from the next token to the API

> An LLM does not query a database: it computes the probability of the next token and picks one. This lesson opens that box with a 0.5B-parameter model that runs on the CPU (logits, temperature, top-p, context and a hallucination about a fictitious company) and runs the code for the OpenAI and Anthropic APIs against a local server that mimics their responses.

- Course: [Level 6: Generative AI, LLMs, agents, big data and alternative data](https://pythonparatrading.com/en/python-trading-course/level-6/)
- Module: 6.2 · LLMs for research and trading
- Duration: 70 min
- Prerequisites: [5.7.1 · PyTorch: tensors, autograd and the training loop](https://pythonparatrading.com/en/python-trading-course/level-5/pytorch-tensors-autograd-training-loop/), [5.7.2 · Time series architectures: TCN, N-BEATS, PatchTST and iTransformer](https://pythonparatrading.com/en/python-trading-course/level-5/tcn-n-beats-patchtst-itransformer/), [1.4.1 · Downloading over REST and websockets: pagination, rate limits and retries](https://pythonparatrading.com/en/python-trading-course/level-1/rest-websockets-pagination-and-retries/), [6.1.1 · Tokenization, embeddings, financial dictionaries and FinBERT](https://pythonparatrading.com/en/python-trading-course/level-6/tokenization-embeddings-loughran-mcdonald-finbert/)

## By the end of this lesson

- Explain autoregressive generation: tokens, logits, softmax and the choice of the next token.
- Measure how temperature and top-p change the next-token distribution and the continuations sampled with a fixed seed.
- Compute how many tokens a financial document takes up and what fits in a context window.
- Provoke and recognize a hallucination, and check how an instruction to abstain and the context reduce it.
- Call a model with the official OpenAI and Anthropic SDKs, and choose between a commercial API and a local model (Ollama, vLLM).

## A model that predicts the next token

A large language model (LLM) does one thing only: given a text, it computes the probability of every possible
immediate continuation and picks one. It repeats the step with the continuation appended, and that is how it writes
whole paragraphs. Everything that looks like reasoning, memory or knowledge comes out of that loop. The architecture
that makes it possible at scale is the *transformer* (Vaswani et al., 2017), which already appeared with time series
(lesson 5.7.2): attention layers that let each position look at all the previous ones.

```mermaid
flowchart TB
  A[Text] --> B[Tokens]
  B --> C[Transformer]
  C --> D[Logits]
  D --> E[Softmax]
  E --> F[Pick token]
  F --> B
```

Text does not go in as letters or as words, but as **tokens**: pieces of a fixed vocabulary that the tokenizer
learned when the model was built (lesson 6.1.1 explains BPE and why Spanish costs more tokens than English). The model
returns one real number, a **logit**, for each piece of the vocabulary; the softmax turns them into probabilities; a
decision rule (the most likely one, or a draw) picks a piece; and the chosen piece is appended to the input.

This lesson uses **Qwen2.5-0.5B-Instruct**, an open model from Alibaba with an Apache-2.0 license and 0.49B
parameters, pinned in the course's `ia` environment by commit and sha256 and loaded without network access. It is
small on purpose: it fits on a laptop CPU and lets you see the mechanism. It is not a commercial-quality model, and the
lesson uses it to show how it works and how it fails, not to promise results.

```python
import numpy as np
import pandas as pd
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

torch.manual_seed(0)
torch.use_deterministic_algorithms(True)
NOMBRE = "Qwen/Qwen2.5-0.5B-Instruct"
tok = AutoTokenizer.from_pretrained(NOMBRE)
llm = AutoModelForCausalLM.from_pretrained(
    NOMBRE, dtype=torch.float32).eval()

cfg, gen = llm.config, llm.generation_config
print(f"parameters: {llm.num_parameters():,}")
print(f"vocabulary: {len(tok):,} pieces")
print(f"layers: {cfg.num_hidden_layers}, "
      f"context: {cfg.max_position_embeddings:,} tokens")
print(f"defaults: do_sample={gen.do_sample}, "
      f"temperature={gen.temperature}, top_p={gen.top_p}, "
      f"top_k={gen.top_k}, "
      f"repetition_penalty={gen.repetition_penalty}")
```
```output
parameters: 494,032,768
vocabulary: 151,665 pieces
layers: 24, context: 32,768 tokens
defaults: do_sample=True, temperature=0.7, top_p=0.8, top_k=20, repetition_penalty=1.1
```

The weights are stored in bf16 and loaded in float32, the format in which the environment was verified to be
deterministic. The last line is the generation configuration the model publishes: by default it **samples**
(`do_sample=True`) with temperature 0.7, top-p 0.8 and top-k 20, and penalizes repetitions with a factor of 1.1. When
this lesson asks for greedy decoding (`do_sample=False`), the repetition penalty stays on; the other sampling
parameters are ignored.

## Tokens: what the model sees and what you pay for

The course's example texts and prompts are in Spanish, as is the corpus of news and filings of this level; the first
cell compares a Spanish sentence with its English equivalent.

```python
frases = [
    "La compañía prevé unas ventas de entre 2.440 y 2.569 "
    "millones de dólares.",
    "We expect revenue in the range of $2,440 million to "
    "$2,569 million.",
]
for f in frases:
    ids = tok(f)["input_ids"]
    piezas = [tok.decode([i]) for i in ids]
    print(len(f), "characters,", len(ids), "tokens")
    print("|".join(piezas))
```
```output
73 characters, 25 tokens
La| compañía| prev|é| unas| ventas| de| entre| |2|.|4|4|0| y| |2|.|5|6|9| millones| de| dólares|.
67 characters, 23 tokens
We| expect| revenue| in| the| range| of| $|2|,|4|4|0| million| to| $|2|,|5|6|9| million|.
```

The two sentences say the same thing in 73 and 67 characters; the Spanish one uses 25 tokens and the English one 23.
Two details matter in finance. Qwen **writes each digit as a separate token** (`2|.|4|4|0`): the model does not see the
number 2,440 but five pieces, and the thousands separator is one more token, which means opposite things in Spanish
and in English. And words are split into chunks (` prev|é`), which the model has to put back together.

Counting tokens is not an academic exercise. Commercial APIs charge per input and output token, the context window is
measured in tokens and compute time grows with them. The document format weighs as much as the language:

*Fig. — Qwen2.5 tokens per 1,000 characters in the level 6 synthetic texts, by format and language.* Seven horizontal bars. The 10-K in HTML needs 277 tokens per 1,000 characters (highlighted) and its tag-free text, 221. The CNMV-style Markdown table, 330. News: 280 in Spanish and 245 in English. Calls: 284 in Spanish and 211 in English.

The figure measures tokens per 1,000 characters in the level's synthetic texts with the Qwen tokenizer. The HTML of a
10-K costs 277 tokens per 1,000 characters and its text without tags, 221: the `<td>`, `<tr>` and `<sup>` tags are 25%
extra tokens that add nothing for the model. The Markdown table of the CNMV-style documents (CNMV is Spain's securities
regulator) is the most expensive (330), because of the pipes and the figures with separators. Cleaning up the format
before sending a document to an API is the easiest saving.

## From logits to probabilities

For an input text, the last position of the transformer produces a vector of logits $z_i$, one per vocabulary piece.
The probability that the next piece is $i$ is the softmax:

$$
p_i = \frac{e^{z_i}}{\sum_j e^{z_j}}
$$

One forward pass is enough to see it. The prompt is a half-written news sentence, “Las acciones cayeron un 8 % tras
unos resultados” (“Shares fell 8% after results”):

```python
prompt = "Las acciones cayeron un 8 % tras unos resultados"
x = tok(prompt, return_tensors="pt")
with torch.inference_mode():
    z = llm(**x).logits[0, -1].double()   # last-position logits

p = torch.softmax(z, dim=-1).numpy()
orden = np.argsort(-p)
top = pd.DataFrame({
    "token": [repr(tok.decode([i])) for i in orden[:10]],
    "prob_%": (100 * p[orden[:10]]).round(1)})
print(f"{len(p):,} probabilities, sum {p.sum():.4f}")
top
```
```output
151,936 probabilities, sum 1.0000
       token  prob_%
0     ' neg'    11.9
1  ' económ'     8.6
2      ' de'     7.4
3     ' más'     3.5
4   ' posit'     2.8
5     ' que'     1.9
6     ' muy'     1.7
7     ' baj'     1.6
8   ' optim'     1.5
9      ' fu'     1.3
```

The output vector has 151,936 positions, slightly more than the tokenizer's 151,665 pieces: the output layer is rounded
up to a size convenient for computation, and the extra positions do not correspond to any piece (their probability is
practically zero). The header figure plots the table. The most likely token is “ neg” (11.9%), the start of
“negativos” (“negative”); it is followed by “ económ” (8.6%), from “económicos” (“economic”), and “ de” (7.4%, “of”).
There are two lessons in these ten rows. First: the probability is very spread out; the most likely token does not
reach 12% and the top ten add up to a little less than half. Second: “ posit” (2.8%) and “ optim” (1.5%) are among the
top ten even though the shares fell 8%. The model does not reason about the news; it reproduces how similar sentences
usually continue in its training data, and “resultados positivos” (“positive results”) is a very frequent phrase.

## Temperature and top-p

The **temperature** $T$ divides the logits before the softmax:

$$
p_i(T) = \frac{e^{z_i / T}}{\sum_j e^{z_j / T}}
$$

With $T < 1$ the differences between logits grow and the probability concentrates on the first tokens; with
$T \to 0$ all of it goes to the most likely one (greedy decoding). With $T > 1$ the distribution flattens. **Nucleus
sampling**, or top-p (Holtzman et al., 2020), draws only from the smallest set of tokens whose probability adds up to
at least $p$, and renormalizes within it. The two parameters can be combined.

```python
def con_temperatura(z, T):
    return torch.softmax(z / T, dim=-1).numpy()

filas = []
for T in [0.2, 0.5, 0.7, 1.0, 1.5, 2.0]:
    q = np.sort(con_temperatura(z, T))[::-1]
    nucleo = int(np.searchsorted(np.cumsum(q), 0.9) + 1)
    filas.append({"T": T, "p_top_%": round(100 * q[0], 1),
                  "nucleus_top_p_0.9": nucleo})
pd.DataFrame(filas).set_index("T")
```
```output
     p_top_%  nucleus_top_p_0.9
T
0.2     77.5                  2
0.5     43.9                  6
0.7     27.8                 43
1.0     11.9                370
1.5      2.8               7651
2.0      0.7              39393
```

*Fig. — Size of the top-p 0.9 nucleus (tokens needed to reach 90% of the probability) by temperature, same prompt and Qwen2.5-0.5B-Instruct.* Rising line on a log scale. At temperature 0.1 one token is enough; at 0.2, two; at temperature 1 (highlighted) 370 tokens are needed; at 2, 39,393 of the 151,665 vocabulary pieces.

With $T = 0.2$ the top token takes 77.5% and two tokens are enough to reach 90% of the probability. With $T = 1$ (the
model's distribution, untouched) the 90% nucleus has 370 tokens. With $T = 2$, the top token drops to 0.7% and the
nucleus exceeds 39,000 pieces: anything can come out. The figure plots the nucleus size on a logarithmic scale for the
whole grid of temperatures.

What that means for a text shows up when you sample. Each configuration generates five 10-token continuations with the
**same seed**, so the cell gives the same result on every run:

```python
def ver(s):
    """Text for display: characters outside the Latin alphabet
    (Chinese, Cyrillic…) become “·” and line breaks
    become “¶”."""
    s = s.replace("\n", "¶")
    return "".join(c if ord(c) < 256 else "·" for c in s)

configs = [(0.2, 1.0), (1.0, 1.0), (1.0, 0.9), (1.6, 1.0)]
for T, top_p in configs:
    torch.manual_seed(42)
    with torch.inference_mode():
        o = llm.generate(**x, do_sample=True, temperature=T,
                         top_p=top_p, top_k=0, max_new_tokens=10,
                         num_return_sequences=5,
                         pad_token_id=tok.eos_token_id)
    n = x["input_ids"].shape[1]
    print(f"T={T}, top_p={top_p}")
    for fila in o:
        print("  …" + ver(tok.decode(fila[n:],
                                      skip_special_tokens=True)))
```
```output
T=0.2, top_p=1.0
  … negativos en la semana pasada, mientras que
  … económicos negativos en los últimos días, mientras
  … negativos en la semana pasada, mientras que
  … negativos de la semana pasada, mientras que
  … económicos negativos en los últimos meses, pero
T=1.0, top_p=1.0
  … menos fuertes y brutalidades en marzo.
  … económicos injustamente subidas. Desde luego,
  … de carteles rentables para ERC y BBVA
  … negativos de la feria.¶······
  … sucios, asentándose en 1.
T=1.0, top_p=0.9
  … menos fuertes y oportunistas de
  … económicos aún poca posibilidad para la economía
  … de carteles - Correo¶La pizar
  … negativos de la feria.¶Además,
  … sucios, asentándose en 1.
T=1.6, top_p=1.0
  … menos fuertes sub·T pivó acompañ
  … f_PROPERTIES injusta manos pos-JPP, según
  … comparativos terribles para ERC mit pizar
  … ameritados primero, estos protagonistas buscan
  … sucieron, asent ba·ar·os desde que··
```

With $T = 0.2$ the five continuations are variations of “negativos en la semana pasada” (“negative last week”) and
“económicos negativos” (“negative economic”), plausible and nearly repeated phrases (two are identical). With $T = 1$
absurd continuations appear (“brutalidades en marzo”, “brutalities in March”; “carteles rentables para ERC y BBVA”,
“profitable cartels for ERC and BBVA”, with real names pulled out of nowhere), a line break (which the cell shows as
“¶”) and characters from other alphabets (shown as “·”): Qwen's vocabulary covers many languages and the tail of the
distribution holds a bit of everything. With top-p 0.9 and the same seed, the five continuations start the same as
with top-p 1 (the first tokens were inside the nucleus) and diverge when the cutoff changes the distribution: “menos
fuertes y brutalidades” becomes “menos fuertes y oportunistas” (“weaker and opportunistic”). With $T = 1.6$ the text
falls apart into fragments such as “f_PROPERTIES” or “pos-JPP”. `top_k=0` turns off the top-k 20 that comes with the
default configuration, to see the effect of temperature without another filter.

<aside class="callout" data-kind="nota">

**Reproducibility.** Here every run gives the same result because the model is local, the weights are pinned and so
is the seed. With an API you control none of that. OpenAI's documentation says of the `seed` parameter that it is in
beta and that “our system will make a best effort to sample deterministically”, with no guarantee. Anthropic removed
`temperature`, `top_p` and `top_k` in its API version 2026-10-21 (sending them gives a 400 error), and its migration
guide to Claude Opus 5.5 points out that `temperature = 0` “never guaranteed identical outputs”. A backtest that calls
an API cannot be repeated exactly: you have to store the responses (lesson 6.2.5).

</aside>

## The context window

The model only sees what fits in its **context window**, measured in tokens and shared between the input and the
response. The Qwen2.5-0.5B-Instruct model card gives 32,768 tokens of context and up to 8,192 of generation; Claude
Opus 5.5 supports 1 million tokens according to Anthropic's models page. An API keeps no memory between calls: in a
conversation, each request resends the whole history and you pay for all of it.

```python
from bs4 import BeautifulSoup

fil = pd.read_parquet("filings6_documentos.parquet")
ll = pd.read_parquet("llamadas6_transcripciones.parquet")

def n_tok(t):
    return len(tok(t)["input_ids"])

def texto(html):
    return BeautifulSoup(html, "html.parser").get_text(" ")

k10 = fil[fil["tipo"] == "10-K"]
llamadas = ll.groupby("id_llamada")["texto"].apply(" ".join)
emisor = fil[fil["ticker"] == k10["ticker"].iloc[0]]
filas = {
    "10-K (HTML)": k10["texto"].map(n_tok).median(),
    "10-K (text)": k10["texto"].map(texto).map(n_tok).median(),
    "full call": llamadas.map(n_tok).median(),
    "8 years of one issuer": emisor["texto"].map(n_tok).sum(),
}
t = pd.Series(filas, name="tokens").round().astype(int).to_frame()
t["fit_in_32k"] = 32_768 // t["tokens"]
t
```
```output
                       tokens  fit_in_32k
10-K (HTML)               776          42
10-K (text)               538          60
full call                 344          95
8 years of one issuer   23578           1
```

The course's synthetic documents are short (a real 10-K has tens of thousands of words, not three thousand
characters), but the calculation is the same with real documents. Here a 10-K fits dozens of times and an issuer's
entire document history over eight years fits just once. A real 10-K, with its exhibits, may not fit in a
32,000-token window and has to be split into chunks (lesson 6.3.2).

Fitting does not mean being used well. Liu et al. (2024) measured that the accuracy of several models when answering
with a fact from the context is higher when the fact is at the beginning or at the end and drops when it is in the
middle, “even for explicitly long-context models”[^1]. Putting an entire annual report in the prompt and asking about a
figure on page 40 is not the same as retrieving the section and giving it to the model (lesson 6.3.3).

## Hallucinations: responses shaped like answers

A **hallucination** is a fluent, confident response that is not true. To provoke one, it is enough to ask about
something the model cannot know. Busmar Hogar is a company in the course's fictitious panel: no LLM has seen it. The
cell asks it the same question in three ways: plainly, with an instruction to abstain if it does not know the figure,
and with the company's announcement (a synthetic CNMV-style “hecho relevante”, a material event disclosure) in the
context. The question and the instructions are in Spanish, like the rest of the corpus. Decoding is greedy, and the
cell also computes the mean probability the model gave to each token it wrote.

```python
import textwrap

doc = fil.loc[fil["id_doc"] == "D000007", "texto"].iloc[0]
pregunta = ("¿Cuánto facturó Busmar Hogar en el cuarto trimestre "
            "de 2018?")
prompts = {
    "no context": [{"role": "user", "content": pregunta}],
    "with abstention": [
        {"role": "system", "content":
         "Responde solo con datos que conozcas con certeza. Si no "
         "conoces el dato, responde exactamente: No lo sé."},
        {"role": "user", "content": pregunta}],
    "with the document": [
        {"role": "system", "content":
         "Responde solo con el documento. Si el dato no está, "
         "di: No consta."},
        {"role": "user", "content":
         f"Documento:\n{doc}\n\nPregunta: {pregunta}"}],
}
tok.padding_side = "left"
textos = [tok.apply_chat_template(m, add_generation_prompt=True,
                                  tokenize=False)
          for m in prompts.values()]
b = tok(textos, return_tensors="pt", padding=True)
with torch.inference_mode():
    o = llm.generate(**b, do_sample=False, max_new_tokens=40,
                     pad_token_id=tok.eos_token_id,
                     output_scores=True,
                     return_dict_in_generate=True)
lp = llm.compute_transition_scores(o.sequences, o.scores,
                                   normalize_logits=True)
n = b["input_ids"].shape[1]
for k, nombre in enumerate(prompts):
    nuevos = o.sequences[k, n:]
    vale = nuevos != tok.eos_token_id
    p_media = lp[k][vale].exp().mean().item()
    print(f"[{nombre}] mean prob. {p_media:.2f}")
    print(textwrap.fill(ver(tok.decode(nuevos,
                                       skip_special_tokens=True)),
                        68, initial_indent="  ",
                        subsequent_indent="  "))
```
```output
[no context] mean prob. 0.58
  Busmar Hogar ha notificado al mercado que su facturación en el
  cuarto trimestre de 2018 fue de $35 millones de dólares. Este
  cifra se ref
[with abstention] mean prob. 0.91
  No lo sé.
[with the document] mean prob. 0.86
  2.942,6 millones de USD
```

Without context, the model makes things up: it says Busmar Hogar “ha notificado al mercado” (“has notified the
market”) revenue of 35 million dollars. The real figure in the document is 2,942.6 million; the model had no way of
knowing it and still writes a sentence with the structure of a verified fact. With the instruction to abstain it
answers “No lo sé.” (“I don't know.”), and with the document in the context it copies the correct figure, “2.942,6
millones de USD” (USD 2,942.6 million).

The mean token probability is lower in the made-up answer (0.58) than in the other two (0.91 and 0.86). It is tempting
to use it as a detector, but it is not reliable: a model can make things up with great confidence, especially in
figures with a typical shape, and it can hesitate on correct answers. Farquhar et al. (2024) propose measuring the
uncertainty over the **meaning** of several sampled answers (semantic entropy), not over the words, and even so they
detect only a subset of hallucinations[^2].

Why do they hallucinate? Kalai, Nachum, Vempala and Zhang (2025) attribute it to how they are trained and evaluated:
“the training and evaluation procedures reward guessing over acknowledging uncertainty”. On a multiple-choice exam,
answering something scores more than leaving it blank, and models are optimized for those exams. The practical
consequence is that the instruction to abstain helps, but what really fixes the problem is **giving the model the
document** and asking it to answer only from it. That is the idea behind RAG (lessons 6.3.1 to 6.3.3).

<aside class="callout" data-kind="riesgo">

In trading research, a hallucination is rarely an obvious blunder: it is a sales forecast with one wrong decimal, an
earnings date from the previous quarter or a ratio from a similar company. Every figure an LLM introduces into a
model, a report or a signal is checked automatically against its source (the document, the database). Lesson 6.2.2
shows how to force the format of the responses and 6.2.3 how to measure how many figures it gets right.

</aside>

## Commercial APIs: the same protocol, another server

The OpenAI and Anthropic APIs are HTTP services: the client sends a JSON with the messages and the parameters, and the
server returns another JSON with the response and the token count. Both have official Python SDKs (`openai` 3.37.0
and `anthropic` 2.1.0 in the course environment). OpenAI recommends its Responses API for new projects and maintains
the Chat Completions API; Anthropic has the Messages API. This is the real code, not run because it needs a key and
network access:

```python skip title="api_openai.py"
from openai import OpenAI

cliente = OpenAI()           # reads OPENAI_API_KEY from env
r = cliente.responses.create(
    model="gpt-6-astra",
    instructions="Eres un analista. Responde en una frase.",
    input="¿Qué es el guidance de una empresa?",
)
print(r.output_text)
```

```python skip title="api_anthropic.py"
import anthropic

cliente = anthropic.Anthropic()   # reads ANTHROPIC_API_KEY
r = cliente.messages.create(
    model="claude-opus-5-5",
    max_tokens=1024,
    system="Eres un analista. Responde en una frase.",
    messages=[{"role": "user",
               "content": "¿Qué es el guidance de una empresa?"}],
)
print(r.content[0].text, r.stop_reason, r.usage.output_tokens)
```

The prompts ask, in Spanish, “You are an analyst. Answer in one sentence.” and “What is a company's guidance?”. The
model names are the ones each provider's documentation uses as of the lesson date (`gpt-6-astra` in OpenAI's
quickstart, `claude-opus-5-5` on Anthropic's models page); they change often and are best pinned in the project
configuration, not scattered across the code. The key never goes in the code: both SDKs read it from an environment
variable.

Since there is no network and no keys, the lesson sets up a **local server** that speaks the same protocol. It answers
`POST /v1/chat/completions` with the shape of an OpenAI `chat.completion` object and `POST /v1/messages` with that of an
Anthropic `message`, as each API reference describes them, and generates the text with the local Qwen. It is, in
miniature, what Ollama and vLLM do:

```mermaid
flowchart TB
  A[Your code] --> B[Official SDK]
  B -->|base_url| C[Commercial API]
  B -->|base_url| D[Local server]
  D --> E[Local model]
```

```python
import json
import threading
from http.server import BaseHTTPRequestHandler
from http.server import ThreadingHTTPServer

def responder(mensajes, max_tokens):
    """Greedy generation with the local Qwen. Returns the text,
    the input and output tokens and whether it stopped by
    itself."""
    x = tok.apply_chat_template(mensajes, return_tensors="pt",
                                add_generation_prompt=True,
                                return_dict=True)
    with torch.inference_mode():
        o = llm.generate(**x, do_sample=False,
                         max_new_tokens=max_tokens,
                         pad_token_id=tok.eos_token_id)
    nuevos = o[0, x["input_ids"].shape[1]:]
    fin = bool(torch.isin(nuevos, torch.tensor(
        gen.eos_token_id)).any())
    n_out = len(nuevos) - int(fin)
    texto = tok.decode(nuevos, skip_special_tokens=True)
    return texto, x["input_ids"].shape[1], n_out, fin

class APILocal(BaseHTTPRequestHandler):
    def log_message(self, *args):        # no logging to stderr
        pass

    def do_POST(self):
        n = int(self.headers["Content-Length"])
        req = json.loads(self.rfile.read(n))
        auth = self.headers.get("Authorization", "").split()[0]
        print(f"  [server] POST {self.path} · auth={auth} · "
              f"anthropic-version="
              f"{self.headers.get('anthropic-version')}")
        if self.path == "/v1/chat/completions":
            limite = (req.get("max_completion_tokens")
                      or req.get("max_tokens") or 64)
            txt, n_in, n_out, fin = responder(req["messages"],
                                              limite)
            r = {"id": "chatcmpl-local-1",
                 "object": "chat.completion",
                 "created": 1767225600, "model": req["model"],
                 "choices": [{
                     "index": 0, "logprobs": None,
                     "finish_reason": "stop" if fin else "length",
                     "message": {"role": "assistant",
                                 "content": txt, "refusal": None}}],
                 "usage": {"prompt_tokens": n_in,
                           "completion_tokens": n_out,
                           "total_tokens": n_in + n_out}}
        elif self.path == "/v1/messages":
            sis = req.get("system")
            msgs = ([{"role": "system", "content": sis}]
                    if sis else []) + req["messages"]
            txt, n_in, n_out, fin = responder(msgs,
                                              req["max_tokens"])
            r = {"id": "msg_local_1", "type": "message",
                 "role": "assistant", "model": req["model"],
                 "content": [{"type": "text", "text": txt}],
                 "stop_reason": "end_turn" if fin else "max_tokens",
                 "stop_sequence": None,
                 "usage": {"input_tokens": n_in,
                           "output_tokens": n_out}}
        else:
            self.send_error(404)
            return
        cuerpo = json.dumps(r).encode()
        self.send_response(200)
        self.send_header("Content-Type", "application/json")
        self.send_header("Content-Length", str(len(cuerpo)))
        self.end_headers()
        self.wfile.write(cuerpo)

srv = ThreadingHTTPServer(("127.0.0.1", 0), APILocal)
threading.Thread(target=srv.serve_forever, daemon=True).start()
URL = f"http://127.0.0.1:{srv.server_port}"
print("local server on 127.0.0.1, port chosen by the OS")
```
```output
local server on 127.0.0.1, port chosen by the OS
```

The server listens only on `127.0.0.1` and on a port the operating system chooses. `created` is a fixed date so that
the output does not depend on the clock. Now, OpenAI's official SDK with its Chat Completions method, with no change
other than `base_url` (the Responses API has another response shape and the simulator does not mimic it):

```python
from openai import OpenAI

oa = OpenAI(base_url=URL + "/v1", api_key="local",
            max_retries=0)
r = oa.chat.completions.create(
    model="qwen2.5-0.5b-instruct", max_completion_tokens=24,
    messages=[
        {"role": "system",
         "content": "Eres un analista. Responde en una frase."},
        {"role": "user",
         "content": "¿Qué es el guidance de una empresa?"}])
print(type(r).__name__, "·", r.choices[0].finish_reason, "·",
      r.usage.prompt_tokens, "+", r.usage.completion_tokens,
      "tokens")
print(textwrap.fill(ver(r.choices[0].message.content), 68))
```
```output
  [server] POST /v1/chat/completions · auth=Bearer · anthropic-version=None
ChatCompletion · length · 34 + 24 tokens
El Guidance de una empresa se refiere a los planes y estrategias que
la empresa utiliza para alcanzar sus objet
```

The SDK has sent the key as `Authorization: Bearer` and has turned the JSON into a `ChatCompletion` object with typed
fields. `finish_reason` is `length`: the response hit the 24-token limit and was cut off mid-sentence (it starts “a
company's guidance refers to the plans and strategies the company uses to reach its…”). It is the first field to check
in production, because a truncated response is an incomplete response (and a truncated JSON cannot be parsed, lesson
6.2.2). The 34 input tokens include the markers of Qwen's chat template, not just the text.

```python
import anthropic

an = anthropic.Anthropic(base_url=URL, api_key="local",
                         max_retries=0)
r = an.messages.create(
    model="qwen2.5-0.5b-instruct", max_tokens=24,
    system="Eres un analista. Responde en una frase.",
    messages=[{"role": "user",
               "content": "¿Qué es el guidance de una empresa?"}])
print(type(r).__name__, "·", r.stop_reason, "·",
      r.usage.input_tokens, "+", r.usage.output_tokens, "tokens")
print(textwrap.fill(ver(r.content[0].text), 68))
srv.shutdown()
```
```output
  [server] POST /v1/messages · auth=Bearer · anthropic-version=2026-10-21
Message · max_tokens · 34 + 24 tokens
El Guidance de una empresa se refiere a los planes y estrategias que
la empresa utiliza para alcanzar sus objet
```

The same response in another wrapper: the `system` goes separately from the messages, the content is a list of blocks
(`content[0].text`) and the stop reason is called `stop_reason` (`max_tokens` instead of `length`). The server has
seen the header `anthropic-version: 2026-10-21`, the API version that SDK 2.1.0 pins, and the key also as
`Authorization: Bearer`; Anthropic's documentation presents `x-api-key` as a legacy alternative that still works. The
differences between providers lie in these details, and that is why it pays to isolate the model call in a function of
your own: switching provider or moving to a local model touches a single place.

## Ollama and vLLM: the model on your machine

The two most widely used local servers do at scale what the simulator does in one cell:

| | Ollama | vLLM |
|---|---|---|
| What for | A model on a laptop or a small server, with nothing to configure | Serving a model to many users with a GPU and high throughput |
| Default address | `127.0.0.1:11434` | `localhost:8000` |
| APIs it mimics | OpenAI (`/v1/chat/completions`, `/v1/responses`…) and a subset of Anthropic's Messages API | OpenAI (Completions, Chat Completions, Responses) and Anthropic's Messages |
| JSON outputs with a schema | `format` parameter with a JSON Schema | `response_format` with `json_schema` (xgrammar or guidance engines) |

Sources: Ollama documentation (OpenAI and Anthropic compatibility, structured outputs and FAQ) and vLLM documentation
(quickstart, *online serving* and *structured outputs*), accessed on 2026-10-23. The code, not run (there is neither
Ollama nor a GPU in the course environment):

```bash
ollama pull qwen2.5:0.5b-instruct
vllm serve Qwen/Qwen2.5-1.5B-Instruct
```

```python skip title="cliente_local.py"
from openai import OpenAI

# Ollama: the key is required by the SDK but ignored
cliente = OpenAI(base_url="http://localhost:11434/v1/",
                 api_key="ollama")
r = cliente.chat.completions.create(
    model="qwen2.5:0.5b-instruct",
    messages=[{"role": "user", "content": "Hola"}])
# vLLM: base_url="http://localhost:8000/v1" and the name of the
# served model ("Qwen/Qwen2.5-1.5B-Instruct")
```

The client code is the same as in the previous cell. That compatibility is the practical reason to write the
model-access layer against the OpenAI or Anthropic format: the same program is tested with a small, free local model
and run with whichever one suits.

## What to choose for trading research

| Criterion | Commercial API | Local model |
|---|---|---|
| Confidentiality | The data leave the building; retention and use according to contract and settings | The data stay inside |
| Quality | The best models available | Limited by the hardware: models from 0.5B to tens of billions of parameters |
| Reproducibility | Versions that get retired, no guaranteed seed | Weights pinned by hash, deterministic decoding |
| Cost | Per token, no investment | Hardware and energy; marginal cost close to zero |
| Latency and volume | Requests-per-minute limits | Those of your machine |
| Look-ahead bias | Declared training cutoff, sometimes imprecise (lesson 6.2.5) | The same problem, with the advantage of being able to choose a model with a cutoff before the period |

Inside information or information subject to confidentiality (client orders, positions, committee notes) is a
constraint, not one more criterion: it is either processed locally or under a contract that covers it. For research
with public documents, quality usually rules, and a commercial API clearly beats a 0.5B model. Lessons 6.2.2 and 6.2.3
measure it: on which tasks the small model is enough and on which it is not.

## Exercises

1. Repeat the temperature table with another prompt, “Los analistas esperan que el beneficio por acción” (“Analysts
   expect earnings per share”), and compute the size of the top-p 0.9 nucleus with $T = 0.7$ and with $T = 1$. Is it
   more or less concentrated than the one in the lesson?

**Solution.**

```python
x2 = tok("Los analistas esperan que el beneficio por acción",
         return_tensors="pt")
with torch.inference_mode():
    z2 = llm(**x2).logits[0, -1].double()
for T in [0.7, 1.0]:
    q = np.sort(con_temperatura(z2, T))[::-1]
    print(T, round(100 * q[0], 1),
          int(np.searchsorted(np.cumsum(q), 0.9) + 1))
```
```output
0.7 50.1 7
1.0 26.5 113
```

The top token (“ de”) has much more probability than in the lesson's sentence and the nucleus is much smaller: the
model has few usual ways to continue that sentence. Concentration depends on the context, not only on the
temperature: the same $T$ produces more or less varied texts depending on how open the sentence is.

2. Estimate how many input tokens it would cost to send all the documents in `filings6_documentos.parquet` to an API,
   in HTML and in plain text, and the cost with an example input price of 2 dollars per million tokens (it is not any
   provider's price: check their pricing page).

**Solution.**

```python
html = fil["texto"].map(n_tok).sum()
plano = fil["texto"].map(texto).map(n_tok).sum()
PRECIO = 2.0                       # USD per million (example)
for nombre, n in [("HTML", html), ("text", plano)]:
    print(f"{nombre}: {n:,} tokens · "
          f"{n / 1e6 * PRECIO:.2f} USD")
```
```output
HTML: 1,530,557 tokens · 3.06 USD
text: 1,066,910 tokens · 2.13 USD
```

Removing the tags saves 30% of the total tokens: from 1.53 to 1.07 million (the CNMV-style documents are already
Markdown and barely change). With 4,020 documents the amount is small; with the full EDGAR archive, with transcripts
and with several questions per document, the format decides a good part of the bill.

3. APIs return an error when the input does not fit in the context. Write a server like the one in the lesson that
   rejects any request with more than 50 input tokens with a 400 and an error JSON, and check which exception the
   OpenAI SDK raises.

**Solution.**

```python
import openai

class APIPequena(APILocal):
    def do_POST(self):
        n = int(self.headers["Content-Length"])
        req = json.loads(self.rfile.read(n))
        n_in = n_tok(tok.apply_chat_template(
            req["messages"], add_generation_prompt=True,
            tokenize=False))
        cuerpo = json.dumps({"error": {
            "message": f"{n_in} tokens > 50",
            "type": "invalid_request_error",
            "code": "context_length_exceeded"}}).encode()
        self.send_response(400)
        self.send_header("Content-Type", "application/json")
        self.send_header("Content-Length", str(len(cuerpo)))
        self.end_headers()
        self.wfile.write(cuerpo)

srv2 = ThreadingHTTPServer(("127.0.0.1", 0), APIPequena)
threading.Thread(target=srv2.serve_forever, daemon=True).start()
oa2 = OpenAI(base_url=f"http://127.0.0.1:{srv2.server_port}/v1",
             api_key="local", max_retries=0)
try:
    oa2.chat.completions.create(
        model="local", messages=[{"role": "user",
                                  "content": doc}])
except openai.BadRequestError as e:
    print(type(e).__name__, e.status_code, e.code)
srv2.shutdown()
```
```output
BadRequestError 400 context_length_exceeded
```

The SDK turns the 400 into `openai.BadRequestError` with the code from the body. A 400 is not retried (the SDKs only
retry connection errors, 408, 409, 429 and 5xx): the input has to be shortened, chunked or summarized first.

[^1]: Liu, N. F. et al. (2024): “Lost in the Middle: How Language Models Use Long Contexts”, *Transactions of the
    Association for Computational Linguistics* 12, 157-173 (arXiv:2307.03172, https://arxiv.org/abs/2307.03172).

[^2]: Farquhar, S., Kossen, J., Kuhn, L. and Gal, Y. (2024): “Detecting hallucinations in large language models using
    semantic entropy”, *Nature* 630, 625-630 (https://doi.org/10.1038/s41586-024-07421-0).

## Self-assessment

1. With the prompt “Las acciones cayeron un 8 % tras unos resultados” (“Shares fell 8% after results”), Qwen2.5-0.5B gives 11.9% to the token “ neg” and 2.8% to “ posit”. What do those figures mean?

   - A. That the model has read news about that company and knows that 11.9% of it was negative
   - B. That they are probabilities of the next piece of text according to what was learned in training: “ posit” is unlikely, but with sampling it will come out now and then
   - C. That the model is 11.9% sure the news is negative
   - D. That there is an error: a well-trained model would give 100% to “ neg”

   *Answer B: That they are probabilities of the next piece of text according to what was learned in training: “ posit” is unlikely, but with sampling it will come out now and then.* The model returns a distribution over the 151,665 pieces of its vocabulary. It is not a probability about the world or about the news, but about how a text like that usually continues. With sampling, a 2.8% token appears in one out of every 36 continuations.

2. What does raising the temperature from 0.7 to 1.5 do to the lesson's next-token distribution?

   - A. Nothing: temperature only affects the length of the response
   - B. It concentrates the probability on the most likely token
   - C. It flattens the distribution: the most likely token drops from 27.8% to 2.8% and the nucleus that adds up to 90% grows from 43 to 7,651 tokens
   - D. It removes the unlikely tokens

   *Answer C: It flattens the distribution: the most likely token drops from 27.8% to 2.8% and the nucleus that adds up to 90% grows from 43 to 7,651 tokens.* Dividing the logits by T > 1 brings them closer together and the softmax spreads the probability. With T = 1.5 the sampled continuations mix languages and nonsense words. top-p cuts the tail, but with such a flat distribution the nucleus is huge.

3. The 0.5B model answers that Busmar Hogar had revenue of “35 millones de dólares” (USD 35 million) in the fourth quarter of 2018. The company's document says 2,942.6 million. What explains it best?

   - A. Busmar Hogar is a fictitious company the model cannot know; it generates a response shaped like a correct answer because nothing in its training rewards saying it does not know
   - B. The model has old data about the company
   - C. A rounding error by the model
   - D. The temperature was too high

   *Answer A: Busmar Hogar is a fictitious company the model cannot know; it generates a response shaped like a correct answer because nothing in its training rewards saying it does not know.* Generation was greedy (no sampling). The model has no data on a made-up company and still writes a figure with its currency and its context. With the instruction to abstain it answers “No lo sé” (“I don't know”), and with the document in the context it gives 2,942.6 million.

4. An asset manager wants to use an LLM to summarize the internal notes of its investment committee, which contain inside information. Which criterion weighs most when choosing between a commercial API and a local model?

   - A. The price per million tokens
   - B. The response speed
   - C. The number of parameters of the model
   - D. Where the data are processed and kept: with a local model they do not leave the building; with an API they depend on the contract and on the provider's retention settings

   *Answer D: Where the data are processed and kept: with a local model they do not leave the building; with an API they depend on the contract and on the provider's retention settings.* Quality and cost matter, but confidentiality is a constraint, not a preference. A local model (with Ollama or vLLM, or in-process as in this lesson) keeps the data inside; with an API you have to check retention, location and use of the data in the contract.

5. With Anthropic's API version 2026-10-21, which is the one SDK 2.1.0 sends, a request with `temperature=0` to get reproducible responses…

   - A. Always returns the same response
   - B. Returns a 400 error: `temperature`, `top_p` and `top_k` were removed in that version
   - C. Is silently ignored
   - D. Turns on greedy decoding

   *Answer B: Returns a 400 error: `temperature`, `top_p` and `top_k` were removed in that version.* Anthropic's versions page says that in 2026-10-21 those parameters are removed and that sending any of them gives a 400 error on every model. Its migration guide adds that `temperature = 0` never guaranteed identical outputs. Only a local model with pinned weights and deterministic decoding gives exact reproducibility.

## Key points

- An LLM generates one token at a time: it tokenizes the text, computes one logit per vocabulary piece (151,665 in Qwen2.5), passes them through the softmax and picks a token, which is appended to the input for the next step.
- Temperature divides the logits before the softmax: below 1 it concentrates, above 1 it flattens. top-p samples only from the nucleus of tokens that adds up to p. With T = 1.5 the small model mixes languages.
- The context window limits what the model sees in one call (32,768 tokens in Qwen2.5-0.5B) and every token costs: the HTML of a 10-K takes 25% more tokens than its text.
- An LLM hallucinates with the shape of a correct answer: about a fictitious company it makes up a figure, and the mean probability of its tokens is not enough to detect it. Instructions to abstain and, above all, the document in the context fix it.
- The OpenAI and Anthropic APIs, Ollama and vLLM speak similar HTTP formats: the same official SDK points to one server or another by changing `base_url`.
- Commercial API or local model is decided by confidentiality, reproducibility (pinned version), cost, latency and quality, in that order when there is sensitive information.

## Resources

### Documentation

- [Qwen/Qwen2.5-0.5B-Instruct model card on Hugging Face](https://huggingface.co/Qwen/Qwen2.5-0.5B-Instruct) · huggingface.co
- [Ollama: OpenAI API compatibility](https://docs.ollama.com/api/openai-compatibility) · docs.ollama.com
- [vLLM: OpenAI-compatible server (online serving)](https://docs.vllm.ai/en/latest/serving/online_serving/) · docs.vllm.ai

### Books

- [Vaswani, A. et al. (2017): Attention Is All You Need (arXiv:1706.03762)](https://arxiv.org/abs/1706.03762) · arxiv.org
- [Holtzman, A. et al. (2020): The Curious Case of Neural Text Degeneration, ICLR 2020 (arXiv:1904.09751)](https://arxiv.org/abs/1904.09751) · arxiv.org
- [Kalai, A. T., Nachum, O., Vempala, S. S. and Zhang, E. (2025): Why Language Models Hallucinate (arXiv:2509.04664)](https://arxiv.org/abs/2509.04664) · arxiv.org

### Data

- [noticias6_corpus.parquet and filings6_documentos.parquet: synthetic financial texts](https://pythonparatrading.com/curso/datos/nivel-6/filings6_documentos.parquet) · Download · filings6_documentos.parquet

Previous lesson: [6.1.3 · Event studies with text signals, timestamp alignment and topic modeling](https://pythonparatrading.com/en/python-trading-course/level-6/text-event-studies-and-topic-modeling/)

Next lesson: [6.2.2 · Structured prompting: JSON you can validate](https://pythonparatrading.com/en/python-trading-course/level-6/structured-prompting-json-with-pydantic/)