--- title: "6.2.2 Structured prompting and validated JSON with Pydantic" description: "JSON Schemas from Pydantic, validation, retries, repair and constrained decoding, measured on 20 filings with a local LLM and a simulated API." url: "https://pythonparatrading.com/en/python-trading-course/level-6/structured-prompting-json-with-pydantic/" lang: "en" image: "https://pythonparatrading.com/og/en/python-trading-course/level-6/structured-prompting-json-with-pydantic.png" --- # 6.2.2 · Structured prompting: JSON you can validate > A data point extracted by an LLM is only useful if a program can read it. This lesson defines the contract with Pydantic, asks a 0.5B-parameter model for it on 20 earnings releases and measures how many responses are valid JSON, what retries and repair fix, and what the constrained decoding used by APIs with structured outputs guarantees. - Course: [Level 6: Generative AI, LLMs, agents, big data and alternative data](https://pythonparatrading.com/en/python-trading-course/level-6/) - Module: 6.2 · LLMs for research and trading - Duration: 75 min - Prerequisites: [6.2.1 · How an LLM works: from the next token to the API](https://pythonparatrading.com/en/python-trading-course/level-6/how-llms-work-apis-and-local-models/), [0.4.2 · Typing: type hints, mypy or ty, and Pydantic for configurations](https://pythonparatrading.com/en/python-trading-course/level-0/python-type-hints-and-pydantic/) ## By the end of this lesson - Define with Pydantic the contract of an extraction (types, ranges, nulls, cross-field rules) and generate its JSON Schema. - Write a structured prompt: role, task, schema, units, no-data case and delimited text. - Measure an LLM's rate of valid JSON and of correct values against a known ground truth, and classify the errors. - Implement a validation loop with retry and a deterministic repair, and measure what each one contributes. - Explain and implement constrained decoding, and use the structured outputs of the OpenAI and Anthropic SDKs. ## From free text to a contract In lesson 6.2.1 the model answered with sentences. For a program, a sentence is a problem: “the company expects between 7.23 and 7.65 billion” has to be turned into two numbers, a unit and a quarter before it is of any use, and every response arrives worded differently. What a research pipeline needs is a **contract**: a fixed format with fields, types and ranges, which the model fills in and a program checks before accepting the data point. The natural format is JSON, and the natural tool for the contract in Python is Pydantic (lesson 0.4.2): it defines the structure once, and from it come the **JSON Schema** shown to the model (or to the API) and the **validation** of what it returns. The full flow of this lesson: ```mermaid flowchart TB A[Document] --> B[Prompt and schema] B --> C[LLM] C --> D[Validate] D -->|valid| E[Data point] D -->|invalid| F[Retry] F --> C D -->|invalid| G[Repair] G --> D ``` The task is to extract the **sales guidance** from a company's earnings release: quarter, year and the two ends of the range in millions of USD. The documents are the SEC-style 8-Ks and the CNMV-style announcements (CNMV is Spain's securities regulator) in `filings6_documentos.parquet`, and the correct answer for each one is in `documentos6_extraccion_verdad.csv` (README of the level 6 data). They are synthetic data: the companies do not exist and the sentences come from templates, so the result measures the model on an easy case, not on real filings. The 8-Ks are in English and the CNMV-style announcements in Spanish; the prompts, the schema descriptions and the field names of the contract are in Spanish too, like the rest of the course's corpus. ## The contract in Pydantic ```python import json from pydantic import (BaseModel, ConfigDict, Field, ValidationError, model_validator) class Prevision(BaseModel): """Previsión de ventas del trimestre siguiente.""" model_config = ConfigDict(extra="forbid") trimestre: int | None = Field(ge=1, le=4) anio: int | None = Field(ge=2018, le=2026) bajo: float | None = Field(gt=0, description="millones de USD") alto: float | None = Field(gt=0, description="millones de USD") @model_validator(mode="after") def horquilla(self): if (self.bajo is None) != (self.alto is None): raise ValueError("bajo and alto: both or neither") if self.bajo is not None and self.bajo > self.alto: raise ValueError("bajo > alto") return self pruebas = [ '{"trimestre": 4, "anio": 2024, "bajo": 7230, "alto": 7650}', '{"trimestre": null, "anio": null, "bajo": null, "alto": null}', '{"trimestre": 4, "anio": 2024, "bajo": 7650, "alto": 7230}', '{"trimestre": "Q4", "anio": 2024, "bajo": 7230, "alto": 7650}', '{"trimestre": 4, "anio": 2024, "bajo": 7.230,0, "alto": 7650}', '{"trimestre": 4, "anio": 2024, "bajo": 1, "alto": 2, "x": 0}', ] for t in pruebas: try: Prevision.model_validate_json(t) print("valid") except ValidationError as e: err = e.errors()[0] print(f"invalid · {err['type']} · {err['msg'][:44]}") ``` ```output valid valid invalid · value_error · Value error, bajo > alto invalid · int_parsing · Input should be a valid integer, unable to p invalid · json_invalid · Invalid JSON: key must be a string at line 1 invalid · extra_forbidden · Extra inputs are not permitted ``` The fields are `trimestre` (quarter), `anio` (year), `bajo` (low) and `alto` (high); the class docstring and the field descriptions (“millones de USD”, millions of USD) stay in Spanish because they travel in the schema the model sees. The contract rejects six different ways of getting it wrong, with a message that says which one. Three design details: - **Nulls are explicit.** A document without guidance is not an error: all fields go to `null`. Without that way out, the model tends to make up a figure to fill the gap (lesson 6.2.1). - **Units go in the contract.** “Millions of USD” is in the field description and in the prompt; without it, a guidance of “7.23 billion” can arrive as 7.23, as 7,230 or as 7,230,000,000. - **`extra="forbid"`.** By default Pydantic ignores keys it does not know. With an LLM that is dangerous: a small model sometimes copies parts of the schema (`"required": [...]`) into the response, and without `forbid` that JSON would pass as valid. The rule “low ≤ high” relates two fields and cannot be written in JSON Schema: the schema shown to the model does not include it and only the validator checks it. The schema Pydantic generates, without the titles (which add nothing for the model and cost tokens), and a one-line **signature** derived from it: ```python def sin_titulos(s): if isinstance(s, dict): return {k: sin_titulos(v) for k, v in s.items() if k != "title"} if isinstance(s, list): return [sin_titulos(v) for v in s] return s def firma(esquema): """One-line summary of the schema, for the prompt.""" partes = [] for campo in esquema["required"]: prop = esquema["properties"][campo] t = next(a for a in prop["anyOf"] if a["type"] != "null") rango = (f" {t['minimum']}-{t['maximum']}" if "minimum" in t else f" > {t['exclusiveMinimum']}") partes.append(f'"{campo}": {t["type"]}{rango} o null') return "{" + ", ".join(partes) + "}" ESQUEMA = sin_titulos(Prevision.model_json_schema()) TXT_ESQUEMA = json.dumps(ESQUEMA, ensure_ascii=False, separators=(",", ":")) FIRMA = firma(ESQUEMA) print(TXT_ESQUEMA) print(FIRMA) ``` ```output {"additionalProperties":false,"description":"Previsión de ventas del trimestre siguiente.","properties":{"trimestre":{"anyOf":[{"maximum":4,"minimum":1,"type":"integer"},{"type":"null"}]},"anio":{"anyOf":[{"maximum":2026,"minimum":2018,"type":"integer"},{"type":"null"}]},"bajo":{"anyOf":[{"exclusiveMinimum":0,"type":"number"},{"type":"null"}],"description":"millones de USD"},"alto":{"anyOf":[{"exclusiveMinimum":0,"type":"number"},{"type":"null"}],"description":"millones de USD"}},"required":["trimestre","anio","bajo","alto"],"type":"object"} {"trimestre": integer 1-4 o null, "anio": integer 2018-2026 o null, "bajo": number > 0 o null, "alto": number > 0 o null} ``` ## A structured prompt A prompt for extracting data always has the same pieces, in the same order: the **role and the task** (what to extract), the **format** (the schema and nothing but JSON), the **conventions** (units, separators, how to convert “billion”), the **no-data case** and the **text**, kept apart from the instructions. The instructions go in the system message and the document in the user message: that way the document cannot pass itself off as an instruction (lesson 6.4.5 covers prompt injection). ```python import re import numpy as np import pandas as pd import torch from bs4 import BeautifulSoup from transformers import AutoModelForCausalLM, AutoTokenizer torch.manual_seed(0) torch.use_deterministic_algorithms(True) NOMBRE = "Qwen/Qwen2.5-0.5B-Instruct" tok = AutoTokenizer.from_pretrained(NOMBRE) tok.padding_side = "left" llm = AutoModelForCausalLM.from_pretrained( NOMBRE, dtype=torch.float32).eval() SISTEMA = ( "Extrae la previsión de ventas del trimestre siguiente. " "Responde solo con un objeto JSON así:\n" + FIRMA + "\nImportes en millones de USD, sin separador de miles " "(1 billion = 1000). Sin previsión: null.") fil = pd.read_parquet("filings6_documentos.parquet") items = fil["items"].fillna("").astype(str) cand = fil[fil["tipo"].isin(["8-K", "OIR", "HR"]) & ~items.str.contains("4.02")] muestra = cand.sample(20, random_state=3) def comunicado(t): """Text of the release: what follows Exhibit 99.1 (SEC) or the key figures (CNMV), without tables or formatting.""" t = re.split(r"Exhibit 99\.1|## Principales magnitudes", t)[-1] t = re.sub(r".*?
", "", t, flags=re.S) t = "\n".join(l for l in t.splitlines() if not l.startswith("|")) t = BeautifulSoup(t, "html.parser").get_text("\n") return re.sub(r"\n\s*\n+", "\n", t.replace("*", "")).strip() docs = [comunicado(t) for t in muestra["texto"]] n_tok = [len(tok(d)["input_ids"]) for d in docs] print(muestra["tipo"].value_counts().to_dict()) print("tokens: system", len(tok(SISTEMA)["input_ids"]), "· full schema", len(tok(TXT_ESQUEMA)["input_ids"]), "· signature", len(tok(FIRMA)["input_ids"]), "· document (median)", int(np.median(n_tok))) print(docs[0]) ``` ```output {'8-K': 14, 'OIR': 4, 'HR': 2} tokens: system 100 · full schema 144 · signature 51 · document (median) 76 Revenue of $6.10 billion Net income of $5,157.5 million, or $13.67 per share Stores at quarter end: 324. We currently anticipate the fourth quarter of 2024 revenue of between $7.23 billion and $7.65 billion. ``` The system message says: “Extract next quarter's sales guidance. Answer only with a JSON object like this: [signature]. Amounts in millions of USD, without a thousands separator (1 billion = 1000). No guidance: null.” The full JSON Schema takes 144 tokens; the one-line signature, 51, and the whole system message, 100. The full schema is what APIs with structured outputs receive (further down); in a handwritten prompt, a signature derived from the same Pydantic model says the same thing with a third of the tokens, and that saving is multiplied by every document (with a commercial API, literally). The releases are trimmed to their text part, without tables or tags, so that the lesson runs on the CPU; even so they keep the main trap, the sales figure for the quarter that has just closed (“Revenue of \$6.10 billion”), which is not the guidance. The ground truth for each document: ```python verdad = pd.read_csv("documentos6_extraccion_verdad.csv") g = verdad[verdad["tipo"] == "guidance"].set_index("id_doc") def esperado(id_doc): if id_doc not in g.index: return Prevision(trimestre=None, anio=None, bajo=None, alto=None) v = g.loc[id_doc] return Prevision(trimestre=int(v["periodo"][-1]), anio=int(v["periodo"][:4]), bajo=v["bajo"], alto=v["alto"]) ESPERADO = [esperado(i) for i in muestra["id_doc"]] def acierta(p, e): """Exact quarter and year; range ends with an error < 0.5%.""" if p is None: return False if e.bajo is None: return p.bajo is None and p.alto is None if p.bajo is None or (p.trimestre, p.anio) != (e.trimestre, e.anio): return False return (abs(p.bajo / e.bajo - 1) < 0.005 and abs(p.alto / e.alto - 1) < 0.005) print(sum(e.bajo is not None for e in ESPERADO), "of 20 documents contain guidance") ``` ```output 15 of 20 documents contain guidance ``` ## How many responses are valid JSON The measure has two levels: **validity** (the text is JSON and complies with the contract) and **accuracy** (in addition, the values match the ground truth). The 20 requests go in one batch with greedy decoding and a maximum of 48 tokens per response, more than enough for the expected JSON. ```python def generar(lista_mensajes, max_tokens=48, **kw): textos = [tok.apply_chat_template(m, tokenize=False, add_generation_prompt=True) for m in lista_mensajes] b = tok(textos, return_tensors="pt", padding=True) with torch.inference_mode(): o = llm.generate(**b, do_sample=False, max_new_tokens=max_tokens, pad_token_id=tok.eos_token_id, **kw) n = b["input_ids"].shape[1] return [tok.decode(r[n:], skip_special_tokens=True) for r in o] def validar(texto): try: return Prevision.model_validate_json(texto), None except ValidationError as e: return None, e.errors()[0]["type"] def resumen(nombre, salidas): res = [validar(s) for s in salidas] ok = sum(p is not None for p, _ in res) bien = sum(acierta(p, e) for (p, _), e in zip(res, ESPERADO)) print(f"{nombre}: {ok}/20 valid, {bien}/20 correct") return res mensajes = [[{"role": "system", "content": SISTEMA}, {"role": "user", "content": d}] for d in docs] libre = generar(mensajes) res_libre = resumen("free", libre) print(pd.Series([e or "valid" for _, e in res_libre]) .value_counts().to_dict()) for k in [0, 1, 5, 6, 13]: corto = libre[k].replace("`", "'").replace("\n", " ") print(f" {muestra['id_doc'].iloc[k]} " f"{res_libre[k][1] or 'valid':>16} · {corto[:58]}") ``` ```output free: 2/20 valid, 1/20 correct {'greater_than': 10, 'json_invalid': 5, 'value_error': 3, 'valid': 2} D003403 value_error · { "trimestre": 4, "anio": null, "bajo": 5157.5, "a D002322 valid · { "trimestre": 3, "anio": null, "bajo": null, "alt D002189 json_invalid · '''json { "trimestre": 4, "anio": 2022, "bajo": 14.0 D001708 greater_than · { "trimestre": 3, "anio": null, "bajo": 0, "alto": D003459 json_invalid · '''json { "trimestre": 3, "anio": null, "bajo": 530, ``` Only 2 of the 20 responses comply with the contract and only 1 is correct. The errors are not random. Half of them (10) are `greater_than`: an amount that is not positive, like the `"bajo": 0` of D001708, even though the signature says “number > 0”; 5 are not JSON (two of those the cell shows start with a Markdown block, which appears with single quotes); and 3 fail the validator's cross-field rules (low ≤ high; both amounts or neither): D003403, for example, takes net income (5,157.5) as the low end. Several leave the year as `null` even though the text states it. The valid ones are not necessarily right either: D002322 is valid with the amounts as `null`, and that document does contain guidance. *Fig. — Free extraction with Qwen2.5-0.5B-Instruct on 20 releases: first Pydantic validation error of each answer.* Four horizontal bars. 10 answers have a non-positive amount (highlighted), 5 are not valid JSON, 3 have the low end above the high end and 2 are valid. The figure counts the error types of the free extraction. What fails most is not the syntax but the content: a 0.5B model does not follow an instruction like “sin previsión: null” (“no guidance: null”) or “positive amounts” precisely. ## Retry with the error message The usual recipe with large models is to hand the model back its response and the validator's error, and ask it to fix it. With the API SDKs, the same loop takes a few lines and a maximum number of attempts. Here, one more attempt for the invalid responses (the follow-up message, in Spanish, says “Validation error: [type]. Fix it and answer only with the JSON.”): ```python malas = [k for k, (p, _) in enumerate(res_libre) if p is None] segundo = [mensajes[k] + [ {"role": "assistant", "content": libre[k]}, {"role": "user", "content": f"Error de validación: {res_libre[k][1]}. Corrige y " "responde solo con el JSON."}] for k in malas] corregidas = generar(segundo) reintento = list(libre) for k, s in zip(malas, corregidas): reintento[k] = s res_reint = resumen("with retry", reintento) for k, s in list(zip(malas, corregidas))[:4]: corto = s.replace("`", "'").replace("\n", " ") print(f" {muestra['id_doc'].iloc[k]} · {corto[:62]}") ``` ```output with retry: 2/20 valid, 1/20 correct D003403 · null D003657 · null D000169 · null D003018 · null ``` The retry fixes nothing: none of the 18 invalid responses passes validation on the second attempt, and the four the cell shows are a plain `null` (valid JSON, but not a `Prevision` object). A 0.5B model does not know how to use the error message to fix the response and just answers something short. With large models, retrying with the error does usually fix part of the failures; here the result is measured, not assumed, and each retry costs one more call with the whole history. ## Repair without the model The other route is to fix the text with deterministic code before validating it: keep the first JSON object, drop the keys that are not in the contract, convert a number written as text into a number and remove the thousands dots from a figure like `13.561.9`. It is cheap and spends no tokens, but each rule is an assumption about the error. ```python CAMPOS = list(Prevision.model_fields) def reparar(texto): m = re.search(r"\{.*?\}", texto, flags=re.S) if not m: return texto t = m.group(0) # 13.561.9 -> 13561.9: thousands dots except the last one t = re.sub(r"\d+(?:\.\d+){2,}", lambda x: x.group(0).replace(".", "", x.group(0) .count(".") - 1), t) try: d = json.loads(t) except json.JSONDecodeError: return t d = {k: v for k, v in d.items() if k in CAMPOS} for k, v in d.items(): if isinstance(v, str): num = re.sub(r"[^\d.]", "", v.replace(",", ".")) d[k] = float(num) if num else None return json.dumps(d) reparado = [reparar(s) for s in reintento] res_rep = resumen("with repair", reparado) ``` ```output with repair: 5/20 valid, 1/20 correct ``` Repair raises validity from 2 to 5 responses without spending a token, and accuracy does not move: it stays at 1. What has been repaired is the form (Markdown blocks around the JSON, separators, extra keys), not the content. That is the risk of repairing: a repaired response passes validation and enters the pipeline looking just like a correct one. In production, every repair is logged and its result is checked against the source. ## Constrained decoding: format by construction The two previous routes act **after** generation. Constrained decoding acts **during** generation: at each step, before choosing the token, it sets to $-\infty$ the logit of every token that cannot continue a valid text according to the schema. The model still picks the most likely token, but only among the allowed ones. Willard and Louf (2023) showed how to do it efficiently by turning a regular expression or a grammar into a finite-state machine over the vocabulary (it is the basis of the Outlines library); vLLM uses the xgrammar or guidance engines, and the OpenAI and Anthropic APIs offer structured outputs with a schema. For a flat object like `Prevision`, the state machine is simple: fixed literals (`{"trimestre": `, `, "anio": `…) that the model has to write verbatim, and slots for each value: ```mermaid stateDiagram-v2 [*] --> Literal Literal --> Value: literal complete Value --> Value: digit or point Value --> Literal: value closed Literal --> [*]: closing brace ``` In a slot, the allowed tokens are the digits that keep the number within its type and range, the decimal point if the field is `number`, the start of `null` and, once the number is complete, the start of the next literal. This is what the mask looks like in the first slot of the first document, after the model has written `{"trimestre": `: ```python V = len(tok) PIEZA = tok.batch_decode([[i] for i in range(V)]) POR_TEXTO = {} for i, s in enumerate(PIEZA): POR_TEXTO.setdefault(s, []).append(i) DIGITOS = [i for d in "0123456789" for i in POR_TEXTO[d]] def prefijos(resto): """Tokens that are a non-empty prefix of `resto`.""" return [i for n in range(1, len(resto) + 1) for i in POR_TEXTO.get(resto[:n], [])] texto = tok.apply_chat_template(mensajes[0], tokenize=False, add_generation_prompt=True) x = tok(texto + '{"trimestre": ', return_tensors="pt") with torch.inference_mode(): z = llm(**x).logits[0, -1] p = torch.softmax(z.double(), -1).numpy() permitidos = ({i for i in DIGITOS if PIEZA[i] in "1234"} | set(prefijos("null"))) orden = np.argsort(-p)[:8] pd.DataFrame({"token": [repr(PIEZA[i]) for i in orden], "prob_%": (100 * p[orden]).round(1), "allowed": [i in permitidos for i in orden]}) ``` ```output token prob_% allowed 0 '4' 84.4 True 1 '3' 10.3 True 2 '1' 2.9 True 3 '2' 1.0 True 4 '0' 0.4 False 5 '5' 0.4 False 6 '6' 0.2 False 7 '8' 0.1 False ``` At that step the model gives 84.4% to the token `4` (the guidance is for the fourth quarter) and spreads the rest among `3`, `1` and `2`. The tokens `0`, `5`, `6` and `8` add up to 1.1%, not much, but over 20 documents and four fields a 1% probability ends up coming out; the mask removes them. The figure plots the step. *Fig. — Constrained decoding: probability of the eight most likely tokens after “{"trimestre": ” in the first document, and which ones the mask lets through (1 to 4 or null).* Eight horizontal bars. Token 4 has 84.4% (highlighted) and is allowed; so are 3, 1 and 2 (10.3, 2.9 and 1.0%). Tokens 0, 5, 6 and 8 add up to 1.1% and the mask removes them. The full decoder reads the template from the JSON Schema itself (the fields in the order of `required`, the type and range of each one) and plugs into transformers' `generate` with `prefix_allowed_tokens_fn`, which at each step receives what has been generated so far and returns the list of allowed tokens: ```python EOS = tok.eos_token_id NUMERO = re.compile(r"null|\d+(\.\d*)?") def plantilla(esquema): """Literals and slots (type, minimum, maximum) of a flat object, in the order of `required`.""" piezas = [] for k, campo in enumerate(esquema["required"]): prop = esquema["properties"][campo] t = next(a for a in prop.get("anyOf", [prop]) if a["type"] != "null") piezas.append(("{" if k == 0 else ", ") + f'"{campo}": ') piezas.append((t["type"], t.get("minimum"), t.get("maximum"))) return piezas + ["}"] def en_hueco(v, tipo, lo, hi, siguiente): """Allowed tokens with the partial value `v` in a slot.""" ok = prefijos("null"[len(v):]) if "null".startswith(v) else [] if v.startswith("n"): return ok if tipo == "integer" and lo is not None and hi is not None: validos = [str(i) for i in range(lo, hi + 1)] ok += [d for d in DIGITOS if any(s.startswith(v + PIEZA[d]) for s in validos)] return ok + (prefijos(siguiente) if v in validos else []) entero, _, dec = v.partition(".") if not v: # no leading zeros ok += [d for d in DIGITOS if PIEZA[d] != "0"] elif len(entero) < 7 and "." not in v or ( "." in v and len(dec) < 2): ok += DIGITOS if tipo == "number" and v and "." not in v: ok += POR_TEXTO["."] cerrado = v and not v.endswith(".") return ok + (prefijos(siguiente) if cerrado else []) def permitidos_para(texto, piezas): pos = 0 for k, pieza in enumerate(piezas): resto = texto[pos:] if isinstance(pieza, str): # literal if resto.startswith(pieza): pos += len(pieza) continue return prefijos(pieza[len(resto):]) m = NUMERO.match(resto) v = m.group(0) if m else resto if pos + len(v) < len(texto): # slot already closed pos += len(v) continue return en_hueco(v, *pieza, piezas[k + 1]) return [EOS] def generar_restringido(lista_mensajes, esquema): piezas = plantilla(esquema) textos = [tok.apply_chat_template(m, tokenize=False, add_generation_prompt=True) for m in lista_mensajes] b = tok(textos, return_tensors="pt", padding=True) n = b["input_ids"].shape[1] def regla(_, ids): return permitidos_para( tok.decode(ids[n:], skip_special_tokens=True), piezas) with torch.inference_mode(): o = llm.generate(**b, do_sample=False, max_new_tokens=60, pad_token_id=EOS, prefix_allowed_tokens_fn=regla) return [tok.decode(r[n:], skip_special_tokens=True) for r in o] restringido = generar_restringido(mensajes, ESQUEMA) res_restr = resumen("constrained", restringido) for k in [0, 1, 5, 6, 13]: print(f" {muestra['id_doc'].iloc[k]} · {restringido[k]}") ``` ```output constrained: 13/20 valid, 3/20 correct D003403 · {"trimestre": 4, "anio": 2024, "bajo": 5157.5, "alto": 7650} D002322 · {"trimestre": 3, "anio": 2023, "bajo": 4.94, "alto": 5.15} D002189 · {"trimestre": 4, "anio": 2022, "bajo": 14.00, "alto": 14.73} D001708 · {"trimestre": 3, "anio": 2021, "bajo": 11.8, "alto": 43} D003459 · {"trimestre": 1, "anio": 2025, "bajo": 530, "alto": 555} ``` With the mask, 13 of the 20 responses are valid and 3 correct, compared with 2 and 1 for the free extraction. The 7 invalid ones all fail the cross-field rules (low ≤ high; both amounts or neither), which are not in the JSON Schema and which the mask therefore cannot impose (exercise 2 checks it). Accuracy improves somewhat because the model can no longer write `null` in the year or `0` in the amounts and has to pick a figure from the text, but the ones it picks show the same underlying failures: D003403 takes net income (5,157.5) as the low end; D002322 and D002189 copy the figures in billions without converting them (4.94 and 5.15 where the ground truth is 4,936 and 5,147); D003459 gets it right. The lesson's decoder implements only part of JSON Schema (types, integer ranges, no leading zeros); the engines of vLLM or of the APIs cover much more, but none of them fixes the model not knowing which number is the guidance. ## Structured outputs in the APIs Commercial APIs do this on the server. The real code with the official SDKs, not run (it needs a key and network access); both accept the Pydantic class directly: ```python skip title="salida_estructurada_openai.py" from openai import OpenAI cliente = OpenAI() r = cliente.responses.parse( model="gpt-6-astra", instructions="Extrae la previsión de ventas del trimestre " "siguiente en millones de USD.", input=texto_del_comunicado, text_format=Prevision, ) prevision = r.output_parsed # a Prevision instance ``` ```python skip title="salida_estructurada_anthropic.py" import anthropic cliente = anthropic.Anthropic() r = cliente.messages.parse( model="claude-opus-5-5", max_tokens=1024, system="Extrae la previsión de ventas del trimestre " "siguiente en millones de USD.", messages=[{"role": "user", "content": texto_del_comunicado}], output_format=Prevision, ) prevision = r.parsed_output # a Prevision instance ``` OpenAI's documentation recommends “always using Structured Outputs instead of JSON mode when possible”: JSON mode only guarantees that the response is JSON, not that it complies with the schema. Anthropic's explains that the SDK transforms the Pydantic model's schema, sends it as `output_config.format`, validates the response and returns it in `parsed_output`. Ollama accepts a JSON Schema in its `format` parameter and vLLM in `response_format` (lesson 6.2.1). To see what really travels, the lesson again sets up a local server shaped like both APIs (as in 6.2.1), but one that, when the request carries a schema, generates with the constrained decoder from the previous cell and that schema. It is what a server with structured outputs does, with a 0.5B model instead of a commercial one. ```python import threading from http.server import BaseHTTPRequestHandler from http.server import ThreadingHTTPServer class APIEstructurada(BaseHTTPRequestHandler): def log_message(self, *args): pass def do_POST(self): n = int(self.headers["Content-Length"]) req = json.loads(self.rfile.read(n)) if self.path == "/v1/chat/completions": esq = req["response_format"]["json_schema"]["schema"] msgs = req["messages"] else: # /v1/messages esq = req["output_config"]["format"]["schema"] msgs = ([{"role": "system", "content": req["system"]}] + req["messages"]) print(" [server]", self.path, "· trimestre:", json.dumps(esq["properties"]["trimestre"]["anyOf"][0], ensure_ascii=False)) txt = generar_restringido([msgs], esq)[0] if self.path == "/v1/chat/completions": r = {"id": "chatcmpl-local-2", "object": "chat.completion", "created": 1767225600, "model": req["model"], "choices": [{"index": 0, "logprobs": None, "finish_reason": "stop", "message": {"role": "assistant", "content": txt, "refusal": None}}], "usage": {"prompt_tokens": 0, "completion_tokens": 0, "total_tokens": 0}} else: r = {"id": "msg_local_2", "type": "message", "role": "assistant", "model": req["model"], "content": [{"type": "text", "text": txt}], "stop_reason": "end_turn", "stop_sequence": None, "usage": {"input_tokens": 0, "output_tokens": 0}} cuerpo = json.dumps(r).encode() self.send_response(200) self.send_header("Content-Type", "application/json") self.send_header("Content-Length", str(len(cuerpo))) self.end_headers() self.wfile.write(cuerpo) srv = ThreadingHTTPServer(("127.0.0.1", 0), APIEstructurada) threading.Thread(target=srv.serve_forever, daemon=True).start() URL = f"http://127.0.0.1:{srv.server_port}" ``` The `usage` token count is set to zero because it does not matter here. First, the OpenAI SDK with `chat.completions.parse`, which accepts the Pydantic class in `response_format`: ```python from openai import OpenAI INSTR = ("Extrae la previsión de ventas del trimestre siguiente " "en millones de USD (1 billion = 1000).") oa = OpenAI(base_url=URL + "/v1", api_key="local", max_retries=0) for k in [0, 6]: try: r = oa.chat.completions.parse( model="qwen2.5-0.5b-instruct", response_format=Prevision, messages=[{"role": "system", "content": INSTR}, {"role": "user", "content": docs[k]}]) print(" ", repr(r.choices[0].message.parsed)) except ValidationError as e: print(" ValidationError:", e.errors()[0]["msg"]) print(" expected:", repr(ESPERADO[k])) ``` ```output [server] /v1/chat/completions · trimestre: {"maximum": 4, "minimum": 1, "type": "integer"} Prevision(trimestre=1, anio=2024, bajo=7.23, alto=7.65) expected: Prevision(trimestre=4, anio=2024, bajo=7228.0, alto=7648.0) [server] /v1/chat/completions · trimestre: {"maximum": 4, "minimum": 1, "type": "integer"} Prevision(trimestre=1, anio=2021, bajo=659.0, alto=692.0) expected: Prevision(trimestre=4, anio=2021, bajo=659.0, alto=692.0) ``` The OpenAI SDK has sent the `Prevision` schema as a `response_format` of type `json_schema` with `"strict": true`, and the ranges have traveled: the server sees `minimum` 1 and `maximum` 4 on the quarter and applies them. The response arrives converted into an already validated `Prevision` object (`message.parsed`). But the values are wrong: the quarter is 1 in both documents (the guidance is for the fourth) and the first one gives the range in billions (7.23 and 7.65). The simulator's server applies the schema as a mask but does not show it to the model, and with the short instruction (“Extract next quarter's sales guidance in millions of USD”) the model does not know what each field means. What commercial servers include in the model's context is not documented in that much detail; what you do see here is that the mask is no substitute for explaining the task. ```python import anthropic an = anthropic.Anthropic(base_url=URL, api_key="local", max_retries=0) for k in [0]: try: r = an.messages.parse( model="qwen2.5-0.5b-instruct", max_tokens=200, system=INSTR, output_format=Prevision, messages=[{"role": "user", "content": docs[k]}]) print(" ", repr(r.parsed_output)) except ValidationError as e: print(" ValidationError:", e.errors()[0]["msg"]) srv.shutdown() ``` ```output [server] /v1/messages · trimestre: {"type": "integer", "description": "{maximum: 4, minimum: 1}"} ValidationError: Input should be greater than or equal to 2018 ``` Anthropic SDK 2.1.0 sends a different schema for the same Pydantic model: it has moved the ranges into the field description (`"description": "{maximum: 4, minimum: 1}"`), so the server can no longer enforce them while decoding and the model writes a year outside 2018-2026. The SDK validates the response with the Pydantic class when it receives it and raises `ValidationError`. That is exactly the role of client-side validation: whatever does not travel in the schema is only checked at the end, and that is why it is never skipped, even if the server promises a format. ## Validity is not accuracy | Method | Valid JSON | Correct values | Cost | |---|---|---|---| | Free extraction (signature in the prompt) | 2/20 | 1/20 | One call per document | | Plus one retry with the error | 2/20 | 1/20 | One more call per invalid response (18) | | Plus deterministic repair | 5/20 | 1/20 | None in tokens; risk of validating an error | | Constrained decoding | 13/20 | 3/20 | One call; the mask at every step | The header figure plots this table. The lesson's techniques act on the **form**: constrained decoding takes validity from 2 to 13 responses (and to 20 if the cross-field rule did not exist), but correct values only go from 1 to 3. With a 0.5B model, the bottleneck is understanding the document, not writing JSON. With a commercial model and structured outputs, validity stops being the main problem and accuracy is what has to be measured; lesson 6.2.3 does that with guidance, KPIs and tone, and compares the LLM with rules, with FinBERT and with a dictionary. ## Exercises 1. Change the contract so that the model also returns the `moneda` (currency) as text with possible values `"USD"` or `"EUR"` (hint: `Literal["USD", "EUR"] | None`) and look at how it appears in the JSON Schema. Could the lesson's constrained decoder handle that field as it stands? **Solution.** ```python from typing import Literal class PrevisionMoneda(Prevision): moneda: Literal["USD", "EUR"] | None = None esq = sin_titulos(PrevisionMoneda.model_json_schema()) print(json.dumps(esq["properties"]["moneda"])) print(esq["required"]) ``` ```output {"anyOf": [{"enum": ["USD", "EUR"], "type": "string"}, {"type": "null"}], "default": null} ['trimestre', 'anio', 'bajo', 'alto'] ``` The field appears as an `enum` inside `anyOf` and, since it has a default value, it does not go into `required`. The lesson's decoder only knows numeric slots and walks through `required`: it would ignore the field. Handling it would take an enumerated slot, whose allowed tokens are the prefixes of `"USD"`, `"EUR"` and `null`, the same idea as the literal. OpenAI's strict structured outputs also require every field to be in `required` (optional ones are expressed with `null`), and its SDK rewrites the schema to comply. 2. The constrained decoder guarantees that `bajo` is a positive number, but not that `bajo ≤ alto` nor that both amounts or neither are present. Count how many of the 20 constrained responses fail only because of those rules. **Solution.** ```python tipos = pd.Series([e or "valid" for _, e in res_restr]) tipos.value_counts() ``` ```output valid 13 value_error 7 Name: count, dtype: int64 ``` Cross-field rules (`model_validator`) do not exist in JSON Schema and no API imposes them while decoding. That is why validation with Pydantic is always done when the response arrives, even if the server promises a format. 3. Write a function `extraer(doc, llamar, intentos=2)` that requests the extraction with the function `llamar` (which receives the messages and returns the model's text), validates, retries with the error and, if it keeps failing, returns `None` with the reason, without repairing anything. Test it without spending a single token: with a fake function that first answers with a broken JSON and then with a valid one. Why is it worth logging the reason as well as returning `None`? **Solution.** ```python def extraer(doc, llamar, intentos=2): msgs = [{"role": "system", "content": SISTEMA}, {"role": "user", "content": doc}] for _ in range(intentos): txt = llamar(msgs) p, err = validar(txt) if p is not None: return p, None msgs = msgs + [ {"role": "assistant", "content": txt}, {"role": "user", "content": f"Error de validación: {err}. Corrige y responde " "solo con el JSON."}] return None, err respuestas = iter([ '{"trimestre": 4, "anio": 2024, "bajo": 7.230,0}', '{"trimestre": 4, "anio": 2024, "bajo": 7230, "alto": 7650}']) print(extraer(docs[0], lambda m: next(respuestas))) print(extraer(docs[0], lambda m: "No hay previsión.")) ``` ```output (Prevision(trimestre=4, anio=2024, bajo=7230.0, alto=7650.0), None) (None, 'json_invalid') ``` Separating the model call (`llamar`) from the loop lets you test the retry logic with fixed responses, in milliseconds and without network access; in production `llamar` wraps the API SDK or the local model. The reason tells a document with no recognizable format apart from a systematic failure (for example, all guidance in billions failing on units). Counting reasons by document type, language or date is how you see that a change in the source documents or in the model has broken the extraction. ## Self-assessment 1. With constrained decoding, the 20 responses of the 0.5B model comply with the JSON Schema and 13 also pass Pydantic validation. What can you conclude? - A. That the 20 extractions are correct - B. That the model has understood the task - C. Only that the format is correct: the schema's types, fields and ranges; the values can still be wrong, as the hit rate against the ground truth shows - D. That validating with Pydantic is no longer needed *Answer C: Only that the format is correct: the schema's types, fields and ranges; the values can still be wrong, as the hit rate against the ground truth shows.* The mask prevents writing anything that does not fit the schema, but it does not know which figure is the guidance: in the lesson, 3 of 20 responses are correct. The 7 that fail Pydantic fail on the cross-field rules (low ≤ high; both amounts or neither), which cannot be expressed in JSON Schema and only the validator checks. 2. On the retry, the model receives its response and Pydantic's error message. What does the 0.5B model do in most cases? - A. It fixes the field that was failing - B. It answers with something short that is not guidance (in the lesson, `null`) and none of the 18 responses passes validation - C. It refuses to answer - D. It returns the original document *Answer B: It answers with something short that is not guidance (in the lesson, `null`) and none of the 18 responses passes validation.* Such a small model does not know how to use the error message to fix the response. Retrying with the error works much better with large models; with this one, it has to be measured, not assumed, and each retry costs another call with the history. 3. Why does the lesson's Pydantic model use `extra="forbid"`? - A. So that the JSON takes fewer tokens - B. Because OpenAI does not support optional fields - C. So that Pydantic converts texts to numbers - D. So that a JSON with keys that are not in the contract (the model sometimes copies `required` or `title` from the schema) is rejected instead of silently accepted; and so that the schema carries `additionalProperties: false`, which strict structured outputs require *Answer D: So that a JSON with keys that are not in the contract (the model sometimes copies `required` or `title` from the schema) is rejected instead of silently accepted; and so that the schema carries `additionalProperties: false`, which strict structured outputs require.* By default Pydantic ignores unknown keys, and a response with `"required": [...]` inside would pass as valid. With `forbid` the error is out in the open. The OpenAI SDK sends the schema with `additionalProperties: false`, a requirement of its strict mode. 4. Anthropic SDK 2.1.0 sends the `Prevision` schema with the ranges (`minimum`, `maximum`) moved into the field description. What is the consequence? - A. None: the server still applies them - B. The server can no longer enforce the range while decoding; the SDK validates the response with the Pydantic model when it receives it, and that is where a quarter 0 or 5 is caught - C. The response stops being JSON - D. The model receives a 400 error *Answer B: The server can no longer enforce the range while decoding; the SDK validates the response with the Pydantic model when it receives it, and that is where a quarter 0 or 5 is caught.* What does not travel in the schema cannot be imposed by constrained decoding: the server only sees the description text. Client-side validation is the last barrier and cannot be skipped. 5. An analyst proposes asking the LLM for the guidance in JSON and, if it fails, “fixing” the text with regular expressions that remove thousands dots. What is the main risk? - A. That the repair turns into valid JSON one with a misread number (for example, a figure in billions read as millions), which then passes validation and enters the pipeline without warning - B. That it takes too long - C. That Pydantic does not accept repaired numbers - D. None: repair always improves the data *Answer A: That the repair turns into valid JSON one with a misread number (for example, a figure in billions read as millions), which then passes validation and enters the pipeline without warning.* Repair fixes the syntax, not the meaning. In the lesson it raises validity from 2 to 5 responses and correct values stay at 1. Every repair is logged and its result is checked against the source or against plausibility rules. ## Key points - The contract comes first: a Pydantic model with types, ranges, nulls, `extra="forbid"` and cross-field rules. From it come the JSON Schema for the prompt or the API and the validation of the response. - A structured prompt states the role, the task, the schema, the units, what to do if there is no data and where the text starts. The schema costs tokens on every call: 144 for the full JSON Schema and 51 for a one-line signature derived from it. - With a 0.5B model, 2 of 20 responses are valid and 1 correct; the typical errors are zero amounts, Markdown around the JSON, figures of different magnitudes mixed together and billions left unconverted. - Retrying with the error message does not help such a small model; deterministic repair raises validity from 2 to 5 without improving accuracy, and can make a wrong data point valid. - Constrained decoding masks the tokens that do not fit the schema: 13 of 20 valid responses (the rest fail a cross-field rule) and only 3 correct. Validity is not accuracy. - APIs with structured outputs (OpenAI `parse`, Anthropic `messages.parse`, Ollama `format`, vLLM `response_format`) apply that idea on the server; whatever the SDK does not send in the schema is only checked by Pydantic when the response arrives. ## Resources ### Documentation - [Pydantic: JSON Schema (`model_json_schema`)](https://pydantic.dev/docs/validation/latest/concepts/json_schema/) · pydantic.dev - [OpenAI: Structured Outputs](https://developers.openai.com/api/docs/guides/structured-outputs) · developers.openai.com - [Anthropic: Structured outputs](https://platform.claude.com/docs/en/build-with-claude/structured-outputs) · platform.claude.com - [vLLM: Structured Outputs](https://docs.vllm.ai/en/latest/features/structured_outputs/) · docs.vllm.ai ### Books - [Willard, B. T. and Louf, R. (2023): Efficient Guided Generation for Large Language Models (arXiv:2307.09702)](https://arxiv.org/abs/2307.09702) · arxiv.org - [Tam, Z. R. et al. (2024): Let Me Speak Freely? A Study on the Impact of Format Restrictions on Performance of Large Language Models (arXiv:2408.02442)](https://arxiv.org/abs/2408.02442) · arxiv.org ### Data - [documentos6_extraccion_verdad.csv: guidance and KPIs contained in each document (synthetic data)](https://pythonparatrading.com/curso/datos/nivel-6/documentos6_extraccion_verdad.csv) · Download · documentos6_extraccion_verdad.csv Previous lesson: [6.2.1 · How an LLM works: from the next token to the API](https://pythonparatrading.com/en/python-trading-course/level-6/how-llms-work-apis-and-local-models/) Next lesson: [6.2.3 · Guidance, KPIs and tone: what an LLM extracts well](https://pythonparatrading.com/en/python-trading-course/level-6/extracting-guidance-kpis-and-tone-with-llms/)