--- title: "6.7.3 Responsible web scraping: httpx, robots.txt and GDPR" description: "Scraping with httpx, requests and BeautifulSoup against a local server: robots.txt (RFC 9309), pacing, retries, conditional requests and GDPR." url: "https://pythonparatrading.com/en/python-trading-course/level-6/responsible-web-scraping/" lang: "en" image: "https://pythonparatrading.com/og/en/python-trading-course/level-6/responsible-web-scraping.png" --- # 6.7.3 · Responsible web scraping > A good share of alternative data (prices, job postings, reviews) is collected from websites. This lesson builds a complete crawler against a local shop with its own robots.txt: it reads it with urllib.robotparser, identifies itself without personal data, limits its pace, honors 429s, uses conditional requests, parses with BeautifulSoup and JSON-LD, and leaves personal data out. It closes with the legal framework in the EU, cited from the primary sources and with no legal advice. - Course: [Level 6: Generative AI, LLMs, agents, big data and alternative data](https://pythonparatrading.com/en/python-trading-course/level-6/) - Module: 6.7 · Alternative data - Duration: 75 min - Prerequisites: [1.3.7 · Market data licensing and terms of use](https://pythonparatrading.com/en/python-trading-course/level-1/market-data-licensing-and-terms-of-use/), [1.4.1 · Downloading over REST and websockets: pagination, rate limits and retries](https://pythonparatrading.com/en/python-trading-course/level-1/rest-websockets-pagination-and-retries/), [6.1.2 · Text sources: EDGAR, CNMV, transcripts and social media](https://pythonparatrading.com/en/python-trading-course/level-6/text-sources-edgar-cnmv-transcripts-social-media/) ## By the end of this lesson - Decide when to scrape and when not to: an API, an official download or a license before crawling, and never behind a login. - Read and apply a robots.txt with `urllib.robotparser` following RFC 9309 (longest match, wildcards, 4xx and 5xx errors, 24-hour cache) and know what it does not guarantee. - Write a client with httpx (and requests) that identifies itself without personal data, limits its pace, honors `Retry-After`, retries with growing backoff and uses `ETag` and `If-Modified-Since`. - Extract prices and job postings with BeautifulSoup and JSON-LD, measure accuracy against the server's ground truth and discard personal data at ingestion. - Place scraping within the EU framework: terms of use, GDPR (articles 4, 5, 6 and 14), the sui generis database right and the text and data mining exception. ## When to scrape and when not to Shop prices, job postings, reviews, product pages: much of the alternative data from lesson 6.7.2 is collected from public websites with a program that downloads pages and extracts fields, **web scraping**. It is technically simple and that is why it gets abused. Before writing a line, four questions: ```mermaid flowchart TB A{Official route} -- yes --> U[Use it] A -- no --> B{Behind login} B -- yes --> N[Do not crawl] B -- no --> R{robots and terms} R -- forbid --> N R -- allow --> P{Personal data} P -- yes --> M[Minimize or skip] P -- no --> S[Crawl slowly] ``` - **Is there an official route?** An API, a download file, a feed or a license from the site itself or from a vendor (lessons 1.3.7 and 6.7.4). It is always preferable: it is stable, has clear terms and does not load the server. - **Do you need to log in?** What sits behind a username and a password is not public: it is governed by the account contract. Nor do you get around blocks, CAPTCHAs or technical limits. - **What do the robots.txt and the terms of use say?** The first is a technical convention; the second, a legal text. Neither replaces the other. - **Is there personal data?** Names, emails, photos, signed opinions. If the purpose does not need them, they are not collected. This lesson does not crawl any real website. All the code runs against a **local server** that the first cell starts on your own machine: a fictitious shop with products, prices, job postings and its robots.txt. The code for real websites is the same with another base URL; where something is needed that is not installed (Playwright), it goes in a `skip` block. ## A local shop to practice on The server serves, with the products and brands from the level 6 reviews, a paginated listing, a page per product (with the price on the page and in JSON-LD, schema.org's structured data format), job postings with the contact person, a page whose price is loaded by JavaScript, and a robots.txt. The shop is Spanish and so is its content: its pages, job titles and prices (in Spanish number format, `1.299,00 €`) stay as they are, because the parsers depend on them. It behaves like a real server in what matters to the client: it answers 429 (*Too Many Requests*) with `Retry-After` on one in every 15 requests (always the same ones, so the output does not depend on the clock), gives `ETag` and `Last-Modified`, answers 304 to conditional requests and records the `User-Agent` of each request. The price of each product is decided by the server (the ground truth) and changes for 20% of the products on “day 2”: ```python import hashlib import html import json import threading from email.utils import formatdate from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer from urllib.parse import parse_qs, urlparse import numpy as np import pandas as pd rng = np.random.default_rng(673) cat = (pd.read_parquet("alt6_resenas.parquet")[["producto", "marca"]] .drop_duplicates().sort_values("producto") .reset_index(drop=True)) cat["id"] = [f"P{i:02d}" for i in range(1, len(cat) + 1)] cat["precio1"] = np.round(rng.uniform(20, 1500, len(cat)), 0) - 0.01 cambia = rng.random(len(cat)) < 0.2 cat["precio2"] = np.where( cambia, np.round(cat["precio1"] * rng.uniform(0.85, 1.1, len(cat)), 2), cat["precio1"]) cat["rebaja"] = rng.random(len(cat)) < 0.3 # struck-through price emp = pd.read_csv("acciones6_empresas.csv").head(8) ofertas = pd.DataFrame({ "id": [f"E{i:02d}" for i in range(1, 9)], "empresa": emp["nombre_corto"], "puesto": ["Analista de datos", "Ingeniera de backend", "Jefe de tienda", "Técnico de mantenimiento", "Comercial", "Controller financiero", "Científica de datos", "Responsable de compras"], "contacto": ["Laura Gómez", "Pablo Ruiz", "Marta Sanz", "Jorge Pardo", "Elena Vidal", "Iván Mora", "Sara León", "Raúl Ibáñez"]}) FECHAS = {1: 1790841600, 2: 1790928000} # Oct 1 and 2, 2026, 8:00 ROBOTS = """User-agent: * Disallow: /cuenta/ Disallow: /buscar Disallow: /*?orden= Disallow: /empleo/ Allow: /empleo/ofertas/ Crawl-delay: 2 User-agent: AgresivoBot Disallow: / """ def euros(x): s = f"{x:,.2f}".replace(",", "X").replace(".", ",") return s.replace("X", ".") + " €" def ficha(fila, dia): p = fila[f"precio{dia}"] ld = {"@context": "https://schema.org", "@type": "Product", "name": fila["producto"], "brand": fila["marca"], "sku": fila["id"], "offers": {"@type": "Offer", "price": f"{p:.2f}", "priceCurrency": "EUR"}} antes = (f'{euros(p * 1.25)}' ' ' if fila["rebaja"] else "") return (f"{fila['producto']}" '" f"

{fila['producto']}

Marca: {fila['marca']}" f'

{antes}{euros(p)}' "

") def oferta(o): ld = {"@context": "https://schema.org", "@type": "JobPosting", "title": o["puesto"], "datePosted": "2026-10-01", "hiringOrganization": {"@type": "Organization", "name": o["empresa"]}, "jobLocation": {"@type": "Place", "address": "Madrid"}} correo = (o["contacto"].lower().replace(" ", ".") .translate(str.maketrans("áéíóú", "aeiou"))) return ('

" f"{o['puesto']}

Contacto: {o['contacto']}, " f"{correo}@tienda.example

") class Tienda(BaseHTTPRequestHandler): dia, n, uas = 1, 0, [] def do_GET(self): url = urlparse(self.path) Tienda.uas.append(self.headers.get("User-Agent", "")) if url.path == "/robots.txt": return self.enviar(200, ROBOTS, "text/plain") Tienda.n += 1 if Tienda.n % 15 == 0: return self.enviar(429, "despacio", "text/plain", {"Retry-After": "5"}) if url.path == "/productos": pag = int(parse_qs(url.query).get("pagina", ["1"])[0]) trozo = cat.iloc[(pag - 1) * 20: pag * 20] items = "".join( f'
  • ' f"{html.escape(p)}
    • " for i, p in zip(trozo["id"], trozo["producto"])) sig = (f'" if pag * 20 < len(cat) else "") return self.enviar(200, f"{sig}") if url.path.startswith("/producto/"): fila = cat[cat["id"] == url.path.rsplit("/", 1)[1]] if fila.empty: return self.enviar(404, "no existe") fila = fila.iloc[0] cuerpo = ficha(fila, Tienda.dia) etag = '"' + hashlib.sha256( cuerpo.encode()).hexdigest()[:16] + '"' dia_mod = 2 if (Tienda.dia == 2 and fila["precio2"] != fila["precio1"]) else 1 lm = formatdate(FECHAS[dia_mod], usegmt=True) if (self.headers.get("If-None-Match") == etag or self.headers.get("If-Modified-Since") == lm): return self.enviar(304, "", extra={"ETag": etag}) return self.enviar(200, cuerpo, extra={ "ETag": etag, "Last-Modified": lm}) if url.path == "/empleo/ofertas/": items = "".join(f'{i}' for i in ofertas["id"]) return self.enviar(200, f"{items}") if url.path.startswith("/empleo/ofertas/"): o = ofertas[ofertas["id"] == url.path.rsplit("/", 1)[1]] return self.enviar(200, oferta(o.iloc[0])) if url.path.startswith("/precio-dinamico/"): pid = url.path.rsplit("/", 1)[1] return self.enviar(200, ( 'cargando")) if url.path.startswith("/api/precio/"): fila = cat[cat["id"] == url.path.rsplit("/", 1)[1]].iloc[0] return self.enviar(200, json.dumps( {"id": fila["id"], "precio": fila[f"precio{Tienda.dia}"]}), "application/json") self.enviar(404, "no existe") def enviar(self, codigo, texto, tipo="text/html", extra=None): cuerpo = texto.encode() self.send_response(codigo) self.send_header("Content-Type", f"{tipo}; charset=utf-8") self.send_header("Content-Length", str(len(cuerpo))) for k, v in (extra or {}).items(): self.send_header(k, v) self.end_headers() if codigo != 304: self.wfile.write(cuerpo) def log_message(self, *args): pass servidor = ThreadingHTTPServer(("127.0.0.1", 0), Tienda) threading.Thread(target=servidor.serve_forever, daemon=True).start() BASE = f"http://127.0.0.1:{servidor.server_port}" print(len(cat), "products;", int(cambia.sum()), "change price on day 2;", int(cat["rebaja"].sum()), "with a struck-through price;", len(ofertas), "job postings") ``` ```output 30 products; 7 change price on day 2; 12 with a struck-through price; 8 job postings ``` ## robots.txt: what it says and what it does not The **Robots Exclusion Protocol** is a text file at the root of the site, `/robots.txt`, with groups of rules per crawler. It had been used as a convention since 1994 and in September 2022 it was published as a standard, RFC 9309[^1]. Its main rules: | RFC 9309 rule | What it implies for the crawler | |---|---| | The group is chosen by the crawler's product name, case-insensitively; if there is no group of its own, the `*` one | It needs a name of its own and must really use it in the `User-Agent` | | Between `allow` and `disallow`, the longest match wins; on a tie, `allow` | The order of the lines does not matter | | `*` is any sequence of characters and `$` the end of the path | `Disallow: /*?orden=` blocks any URL with that parameter | | A 4xx when requesting the robots.txt (does not exist) allows crawling; a 5xx forces the crawler to assume complete disallow | A server error is not a permission | | The cached copy should not be used for more than 24 hours, unless the file is unreachable | It has to be requested again every day | | The rules “are not a form of access authorization” | Complying with the robots.txt grants no right to anything; breaking it is not fixed with a technical argument | `Crawl-delay` (seconds between requests) is not in the RFC, but many sites use it and `urllib.robotparser` reads it: honoring it is the prudent choice. The standard library module has a trap: `read()` downloads the file with `urllib` and its default `User-Agent`, not with the crawler's. The cell shows this by asking the server to say which header it received, and then downloads the robots.txt with httpx and the crawler's own name: ```python import urllib.robotparser import httpx UA = "CursoPPT-precios/0.1 (+https://example.com/rastreador)" rp_std = urllib.robotparser.RobotFileParser(f"{BASE}/robots.txt") rp_std.read() print("read() identifies itself as:", Tienda.uas[-1]) cliente = httpx.Client(base_url=BASE, timeout=10.0, headers={"User-Agent": UA}) r = cliente.get("/robots.txt") print("httpx identifies itself as:", Tienda.uas[-1], "->", r.status_code) rp = urllib.robotparser.RobotFileParser() rp.parse(r.text.splitlines()) print("Crawl-delay:", rp.crawl_delay(UA)) ``` ```output read() identifies itself as: Python-urllib/3.14 httpx identifies itself as: CursoPPT-precios/0.1 (+https://example.com/rastreador) -> 200 Crawl-delay: 2 ``` With the robots.txt loaded, `can_fetch` answers for each URL. The table tests the rules that matter: the disallowed path, the parameter with a wildcard, the long rule that beats the short one and another crawler's group: ```python pruebas = ["/productos?pagina=1", "/producto/P01", "/productos?orden=precio", "/cuenta/pedidos", "/buscar?q=cafetera", "/empleo/candidatos/", "/empleo/ofertas/E01", "/api/precio/P01"] pd.DataFrame({ "CursoPPT": [rp.can_fetch(UA, BASE + u) for u in pruebas], "AgresivoBot": [rp.can_fetch("AgresivoBot/2.0", BASE + u) for u in pruebas]}, index=pruebas) ``` ```output CursoPPT AgresivoBot /productos?pagina=1 True False /producto/P01 True False /productos?orden=precio False False /cuenta/pedidos False False /buscar?q=cafetera False False /empleo/candidatos/ False False /empleo/ofertas/E01 True False /api/precio/P01 True False ``` `/empleo/ofertas/E01` is allowed even though `Disallow: /empleo/` appears first: `Allow: /empleo/ofertas/` matches more characters. That behavior is the RFC's, but not that of every Python version: with the same file, Python 3.12.14 applies the first rule that matches and does not understand wildcards (it blocks the job postings and lets `?orden=` through); Python 3.13.15 and 3.14.8, the one in the course environment, follow the RFC (checked when writing the lesson, on 2026-10-23). The module's documentation refers to RFC 9309 without detailing the matching[^2]: if the crawler runs on another version, it has to be tested with a known robots.txt, like this table. ## A client that identifies itself and goes slowly The client from lesson 1.4.1 already retried with growing backoff. Here it gets what it needs to crawl someone else's site: ```mermaid flowchart TB U[URL] --> R{robots.txt} R -- yes --> L[Rate limit] L --> G[GET] G -- 429 --> W[Retry-After] W --> L G -- 200 --> C[Cache] ``` - **Identification**: a product name of its own and a URL that explains what the crawler does and how to get in touch, with no personal data (neither a person's name nor email). With a name of its own, the site can write a group for it in the robots.txt; with a browser's `User-Agent`, it cannot. - **Pace**: one request every `Crawl-delay` seconds or more, never in parallel against the same site. - **Errors**: on a 429, wait for what `Retry-After` says (RFC 9110, section 10.2.3[^3]); on a 5xx, retry with growing backoff and some randomness, and give up after a few attempts. The `retries` parameter of the httpx transport only retries connection errors (`ConnectError`, `ConnectTimeout`), not 429 or 503 responses[^4]: that logic belongs to the client. So that the output does not depend on the machine's clock, the client uses a **simulated clock** that moves forward when the client “sleeps”; in production, `time.monotonic()` and `time.sleep()` take its place without touching anything else: ```python class Reloj: """Simulated clock: sleep() moves time forward without waiting.""" def __init__(self): self.t = 0.0 def now(self): return self.t def sleep(self, s): self.t += s reloj = Reloj() azar = np.random.default_rng(0) registro = [] # (time, url, status code) class Rastreador: def __init__(self, cliente, rp, ua, reloj, intentos=4): self.c, self.rp, self.ua = cliente, rp, ua self.reloj, self.intentos = reloj, intentos self.pausa = max(rp.crawl_delay(ua) or 0, 1.0) self.ultima = -np.inf def get(self, ruta, cabeceras=None): if not self.rp.can_fetch(self.ua, BASE + ruta): raise PermissionError(f"robots.txt disallows {ruta}") for n in range(self.intentos): espera = self.ultima + self.pausa - self.reloj.now() if espera > 0: self.reloj.sleep(espera) self.ultima = self.reloj.now() r = self.c.get(ruta, headers=cabeceras) registro.append((self.reloj.now(), ruta, r.status_code)) if r.status_code == 429: self.reloj.sleep(float(r.headers["Retry-After"])) elif r.status_code >= 500: self.reloj.sleep(2 ** n + azar.uniform(0, 1)) else: return r r.raise_for_status() bot = Rastreador(cliente, rp, UA, reloj) try: bot.get("/cuenta/pedidos") except PermissionError as e: print(e) r = bot.get("/productos?pagina=1") print(r.status_code, len(r.text), "bytes; clock:", reloj.now(), "s") ``` ```output robots.txt disallows /cuenta/pedidos 200 1352 bytes; clock: 0.0 s ``` ## Extracting the data: BeautifulSoup and JSON-LD The listing has links to the product pages and a `rel="next"` link to the next page. Following that link, instead of inventing URLs, means the crawler does not depend on how the site numbers its pages. BeautifulSoup turns the HTML into a tree that is queried with CSS selectors; lxml is not in the `ia` environment, so the parser is the standard library's, `html.parser`: ```python from bs4 import BeautifulSoup fichas, ruta = {}, "/productos?pagina=1" while ruta: sopa = BeautifulSoup(bot.get(ruta).text, "html.parser") for a in sopa.select("li.producto a"): fichas[a["href"]] = None sig = sopa.find("a", rel="next") ruta = sig["href"] if sig else None cache = {} for ruta in fichas: r = bot.get(ruta) cache[ruta] = {"etag": r.headers.get("ETag"), "lm": r.headers.get("Last-Modified"), "html": r.text} bytes_dia1 = sum(len(v["html"].encode()) for v in cache.values()) print(len(cache), "product pages;", bytes_dia1, "bytes; clock:", reloj.now(), "s") print(pd.Series([c for _, _, c in registro]).value_counts() .to_dict()) ``` ```output 30 product pages; 11888 bytes; clock: 74.0 s {200: 33, 429: 2} ``` Thirty product pages and two listing pages, with one request every 2 seconds (the `Crawl-delay`) and two 429s along the way, which add 5 seconds of waiting each; the header figure draws that path. The price appears twice on each product page: in the visible text (`1.299,00 €`, in Spanish format) and in JSON-LD (`"price": "1299.00"`). The two ways of reading it, against the server's ground truth: ```python import re def precio_texto(h): s = re.search(r"([\d.]+,\d{2}) €", h).group(1) # the first one return float(s.replace(".", "").replace(",", ".")) def precio_css(sopa): s = sopa.select_one("span.precio").get_text() return float(s.replace(" €", "").replace(".", "") .replace(",", ".")) def precio_jsonld(sopa): ld = json.loads(sopa.find( "script", type="application/ld+json").string) return float(ld["offers"]["price"]), ld["sku"] filas = [] for ruta, v in cache.items(): sopa = BeautifulSoup(v["html"], "html.parser") p_ld, sku = precio_jsonld(sopa) filas.append({"id": sku, "regex": precio_texto(v["html"]), "css": precio_css(sopa), "jsonld": p_ld}) ext = pd.DataFrame(filas).merge(cat[["id", "precio1", "rebaja"]]) for m in ["regex", "css", "jsonld"]: ok = np.isclose(ext[m], ext["precio1"]) print(f"{m:7s} correct: {ok.mean():.0%}") ``` ```output regex correct: 60% css correct: 100% jsonld correct: 100% ``` The regular expression “the first amount in euros” fails on the products with a struck-through price: it reads the previous price. The CSS selector (`span.precio`) and the JSON-LD get all of them right. Of the two, JSON-LD is the more stable: the site publishes it for search engines, with the price as a number and the currency separately, and it does not change when the page design changes. If a redesign renames the `precio` class, the selector breaks without warning; that is why a production crawler checks every day how many empty or out-of-range fields it has extracted. ## Conditional requests and caching The next day, the crawler comes back. Most of the product pages have not changed, and the HTTP protocol has a way of not downloading them again: the **conditional request**. The client sends the `ETag` it saved (`If-None-Match`) or the date of the last modification (`If-Modified-Since`); if nothing has changed, the server answers **304 Not Modified**, without a body (RFC 9110, section 13[^3]): ```python Tienda.dia = 2 bytes_dia2, codigos, cambios = 0, [], [] for ruta, v in cache.items(): r = bot.get(ruta, {"If-None-Match": v["etag"]}) codigos.append(r.status_code) if r.status_code == 200: bytes_dia2 += len(r.content) v.update(etag=r.headers["ETag"], html=r.text, lm=r.headers["Last-Modified"]) cambios.append(ruta.rsplit("/", 1)[1]) print(pd.Series(codigos).value_counts().to_dict()) print("product page bytes: day 1", bytes_dia1, "| day 2", bytes_dia2, f"({1 - bytes_dia2 / bytes_dia1:.0%} less)") verdad = cat.loc[cat["precio2"] != cat["precio1"], "id"].tolist() print("changes detected = ground truth:", sorted(cambios) == verdad) ``` ```output {304: 23, 200: 7} product page bytes: day 1 11888 | day 2 2832 (76% less) changes detected = ground truth: True ``` *Fig. — Bytes the lesson crawler downloads for the 30 product pages: with conditional requests, on day 2 it only fetches the 7 whose price changed.* Horizontal bars: day 1, 11,888 bytes; day 2 without conditional requests, 11,886; day 2 with If-None-Match, 2,832 (highlighted, 76% less). Seven product pages come back with 200 (the ones whose price changed, all detected) and twenty-three with 304: 76% fewer bytes. The 304s also count as requests and respect the pace, but the server neither generates nor sends the page. `If-Modified-Since` works the same way with the date; it is less precise (one-second resolution) and is useful when the site gives no `ETag`: ```python v = cache["/producto/P01"] r = bot.get("/producto/P01", {"If-Modified-Since": v["lm"]}) print(v["lm"], "->", r.status_code) ``` ```output Thu, 01 Oct 2026 08:00:00 GMT -> 304 ``` For research, the cache has another advantage: saving each day's original HTML (not just the extracted fields) lets you parse it again when a parser bug is fixed, without crawling again, and reconstruct what was known on each date (*point-in-time*, lesson 1.4.4). ## Pages with JavaScript Some pages do not carry the data in the HTML: the browser requests it with JavaScript after loading. An HTTP client like httpx does not run JavaScript and sees the placeholder text (`cargando`, “loading”): ```python r = bot.get("/precio-dinamico/P01") sopa = BeautifulSoup(r.text, "html.parser") print("price in the HTML:", sopa.select_one("#precio").get_text()) api = re.search(r"fetch\('([^']+)'\)", r.text).group(1) print("the script requests:", api) print("allowed?", rp.can_fetch(UA, BASE + api)) print(bot.get(api).json()) ``` ```output price in the HTML: cargando the script requests: /api/precio/P01 allowed? True {'id': 'P01', 'precio': 1492.99} ``` The script requests the data from a JSON endpoint on the site itself; if the robots.txt and the terms allow it, requesting that JSON directly is the most efficient option (one small request instead of loading a browser). When there is no way to avoid the browser (the data is computed in the page, or the page requires interaction), an automated browser such as **Playwright** is used, which drives Chromium, Firefox or WebKit. It is not installed in the course environments (checked on 2026-10-23), so its code goes unexecuted. It is installed with `pip install playwright` followed by `playwright install`, which downloads the browsers[^5]: ```python skip from playwright.sync_api import sync_playwright with sync_playwright() as p: navegador = p.chromium.launch() contexto = navegador.new_context(user_agent=UA, locale="es-ES") pagina = contexto.new_page() pagina.goto(f"{BASE}/precio-dinamico/P01") pagina.wait_for_function( "document.getElementById('precio').textContent" " !== 'cargando'") print(pagina.locator("#precio").inner_text()) navegador.close() ``` An automated browser loads everything a person loads (scripts, images, fonts): it weighs more on the server and on your own machine. The same rules of pace, robots.txt and identification apply to it (`user_agent` in the context). The **requests** library, the most widespread, does the same as httpx in synchronous mode (httpx adds an asynchronous client and optional HTTP/2). The pattern is the same: a session with the fixed header and a timeout: ```python import requests with requests.Session() as s: s.headers["User-Agent"] = UA r = s.get(f"{BASE}/robots.txt", timeout=10) print(r.status_code, r.text.splitlines()[0], "|", Tienda.uas[-1]) ``` ```output 200 User-agent: * | CursoPPT-precios/0.1 (+https://example.com/rastreador) ``` ## Personal data: do not collect it The shop's job postings include the name and email of the contact person, as many real ones do. For a hiring study (how many postings, from which companies and categories) they are not needed. The safest way to comply is not to store them: the parser extracts from the JSON-LD only the study's fields and never touches the free text: ```python CAMPOS = ["title", "datePosted"] sopa = BeautifulSoup(bot.get("/empleo/ofertas/").text, "html.parser") guardadas, correos_vistos = [], 0 for a in sopa.find_all("a"): h = bot.get(a["href"]).text correos_vistos += len(re.findall(r"[\w.]+@[\w.]+", h)) ld = json.loads(BeautifulSoup(h, "html.parser").find( "script", type="application/ld+json").string) fila = {k: ld[k] for k in CAMPOS} fila["empresa"] = ld["hiringOrganization"]["name"] guardadas.append(fila) ofe = pd.DataFrame(guardadas) texto = ofe.to_csv(index=False) print("emails in the pages:", correos_vistos, "| in what was stored:", len(re.findall(r"@", texto))) ofe.head(3) ``` ```output emails in the pages: 8 | in what was stored: 0 title datePosted empresa 0 Analista de datos 2026-10-01 Bagvor Telecom 1 Ingeniera de backend 2026-10-01 Bakdor Pharmaceuticals 2 Jefe de tienda 2026-10-01 Bebvia Petroleum ``` Eight emails in the pages and none in the dataset. The allowlist of fields (`CAMPOS`) is the important decision: what is not on the list is not stored, even if tomorrow the site adds a new field with personal data. When finished, the client and the server are closed: ```python print("requests:", len(registro), "| 429:", sum(c == 429 for _, _, c in registro), "| simulated clock:", reloj.now(), "s") cliente.close() servidor.shutdown() ``` ```output requests: 80 | 429: 5 | simulated clock: 173.0 s ``` ## The legal framework in the EU Four frameworks intersect in a crawl: 1. **Terms of use**. They are a contract between the site and whoever uses it, and they can forbid automated access or the commercial reuse of the content. A robots.txt that allows a path does not override a prohibition in the terms. For market data, lesson 1.3.7 covered licenses; the same logic (which use is allowed, whether there is redistribution) applies to a website's content. 2. **Data protection (GDPR)**. If what is collected makes it possible to identify a person, it is personal data: “any information relating to an identified or identifiable natural person” (article 4.1). Collecting it is processing and needs a legal basis from article 6; for scraping, the debate centers on legitimate interest under 6.1.f, which requires that the data subject's rights do not override it. The data have to be “adequate, relevant and limited to what is necessary” (5.1.c, data minimization), and whoever obtains them without going through the data subject has a duty to inform (article 14), with exceptions such as disproportionate effort (14.5.b)[^6]. The AEPD (the Spanish data protection authority) sums up that scraping, “when it collects personal data, does involve processing”, and closed a case because the tool only showed aggregated information, with no identifiable persons[^7]. On July 8, 2026 the European Data Protection Board adopted guidelines on scraping for generative AI, in public consultation until October 30, 2026, which develop legitimate interest and data minimization[^8]. 3. **Sui generis database right**. Directive 96/9/EC allows the maker of a database who has made a “substantial investment” to prohibit the extraction or reuse of the whole or of a substantial part (article 7.1) and also the “repeated and systematic” extraction of insubstantial parts that conflicts with its normal exploitation (7.5), for fifteen years (article 10.1). In Spain it is in article 133 of the Intellectual Property Act (Ley de Propiedad Intelectual)[^9]. A daily crawler of a whole price catalog is precisely a repeated and systematic extraction. 4. **Text and data mining**. Directive (EU) 2019/790 created an exception for text and data mining of lawfully accessible content; in Spain, article 67 of Royal Decree-Law 24/2021. It does not apply when the rightholders “have expressly reserved the use of the works to machine-readable means or other appropriate means” (67.3)[^10]. The law does not say what form that reservation takes; a robots.txt or a clause in the terms that expresses it are the obvious candidates, and the prudent course is to respect them as such. For a fund, there is a further risk, covered in lesson 6.7.4: that the collected data is inside information or was obtained in breach of a duty of confidentiality. Fund compliance teams usually require a review before any scraped data is used. ## Common mistakes - **Crawling what has an API or a license**. More work, less stability and more risk than the official route. - **Disguising as a browser**. It prevents the site from applying its rules and is the first sign of bad faith if there is a dispute. - **Putting personal data in the `User-Agent`**. A product name and an information URL are enough. - **Using `robotparser.read()` without thinking**: it downloads with another `User-Agent` and does not store the date so it can be requested again within 24 hours. And not checking the matching of the Python version in use. - **Parallelizing against a single site** or ignoring `Retry-After`: the block arrives before the data. - **Parsing the visible text when there is structured data**: struck-through prices, units and redesigns break regular expressions. - **Storing everything “just in case”**: personal data that is not collected does not have to be protected, deleted or explained. ## Exercises 1. Change the server so that it answers 503 when `/robots.txt` is requested and check what `RobotFileParser.read()` does. Does it match RFC 9309? **Solution.** ```python class Caida(BaseHTTPRequestHandler): def do_GET(self): self.send_response(503) self.end_headers() def log_message(self, *args): pass s503 = ThreadingHTTPServer(("127.0.0.1", 0), Caida) threading.Thread(target=s503.serve_forever, daemon=True).start() rp503 = urllib.robotparser.RobotFileParser( f"http://127.0.0.1:{s503.server_port}/robots.txt") rp503.read() print("can anything be crawled?", rp503.can_fetch(UA, "http://127.0.0.1/productos")) s503.shutdown() ``` ```output can anything be crawled? False ``` `read()` marks everything as disallowed on a 5xx, as RFC 9309 requires. A crawler of your own that downloads the robots.txt with httpx has to do the same by hand: if the code is 5xx (or the connection fails), do not crawl and try again later; if it is 4xx, crawling is allowed (except 401 and 403, which the Python module treats as complete disallow, a more prudent decision than the RFC's). 2. Add to the `Rastreador` a robots.txt cache that requests it again if more than 24 hours have passed (with the simulated clock) and check it by moving the clock forward. **Solution.** ```python class RobotsCache: def __init__(self, cliente, reloj, vida=24 * 3600): self.c, self.reloj, self.vida = cliente, reloj, vida self.rp, self.cuando, self.descargas = None, None, 0 def get(self): if (self.cuando is None or self.reloj.now() - self.cuando > self.vida): r = self.c.get("/robots.txt") self.rp = urllib.robotparser.RobotFileParser() self.rp.parse(r.text.splitlines()) self.cuando = self.reloj.now() self.descargas += 1 return self.rp # the server is already closed: a simulated httpx transport falso = httpx.MockTransport( lambda req: httpx.Response(200, text=ROBOTS)) with httpx.Client(transport=falso, base_url="http://tienda.example", headers={"User-Agent": UA}) as c2: rc = RobotsCache(c2, Reloj()) rc.get() rc.reloj.sleep(3600) rc.get() rc.reloj.sleep(24 * 3600) print("allowed?", rc.get().can_fetch( UA, "http://tienda.example/producto/P01")) print("robots.txt downloads:", rc.descargas) ``` ```output allowed? True robots.txt downloads: 2 ``` Two downloads: the initial one and the one after 25 hours; the query after one hour uses the copy. `httpx.MockTransport` replaces the server: it answers each request with a function, without a network, which is also useful for testing the crawler. In a real crawler, the download date is stored next to the file, and if the download fails with a 5xx, the last good copy is kept (the RFC allows it if the file is unreachable) or the crawl is stopped. 3. A vendor offers you a dataset of prices scraped from 2,000 online shops. What questions would you ask about how they collect them? **Solution.** The ones from this lesson, turned toward the vendor: whether they respect each site's robots.txt and how they handle changes, which `User-Agent` they use and whether they identify themselves, at what pace they crawl and whether they have received complaints or blocks, whether they access anything behind a login, whether they have reviewed the sites' terms of use and what they do with those that forbid crawling, whether they collect personal data (signed reviews, private sellers) and on what legal basis, whether there are databases with a sui generis right among the sources, and whether they keep the original HTML with its date (to reconstruct the data *point-in-time*). A serious vendor has written answers and a legal review; lesson 6.7.4 turns these questions into a *due diligence* questionnaire. [^1]: RFC 9309, “Robots Exclusion Protocol” (September 2022, , accessed 2026-10-23): case-insensitive matching on the product name; “The most specific match is the match that has the most octets”; on a tie, `allow`; `*` and `$`; 4xx “unavailable” (crawling allowed) and 5xx “unreachable” (complete disallow); 24-hour cache at most; a parsing limit of at least 500 KiB; “These rules are not a form of access authorization”. [^2]: Python, “urllib.robotparser — Parser for robots.txt” (, accessed 2026-10-23): it refers to RFC 9309; `crawl_delay`, `request_rate` and `site_maps`. The behavior with a long `Allow` and wildcards was checked by running the same robots.txt on Python 3.12.14, 3.13.15 and 3.14.8. [^3]: RFC 9110, “HTTP Semantics” (, accessed 2026-10-23): `Retry-After` (section 10.2.3), conditional requests with `If-None-Match` and `If-Modified-Since` (section 13.1) and 304 Not Modified (section 15.4.5). The 429 code is defined in RFC 6585 (), which allows it to be accompanied by `Retry-After`. [^4]: HTTPX, “Transports” (, accessed 2026-10-23): requests “will be retried the given number of times in case an httpx.ConnectError or an httpx.ConnectTimeout occurs”; for other cases it refers to retry libraries such as tenacity. [^5]: Playwright for Python, “Getting started - Library” (, accessed 2026-10-23): `pip install playwright` and `playwright install`, which installs the browsers; `new_context` accepts `user_agent` and `locale` (). [^6]: Regulation (EU) 2016/679, text published in the OJEU and reproduced by the BOE, the Spanish Official Gazette (, accessed 2026-10-23): articles 4.1, 5.1.c, 6.1.f and 14.5.b. The EUR-Lex version is . [^7]: AEPD, criterion “Herramientas de data scraping para la evaluación de tendencias” (data scraping tools for trend analysis), November 25, 2025 (procedure AI/00059/2024, , accessed 2026-10-23; in Spanish, quotation translated). The page warns that the summary is not the official publication. [^8]: EDPB, “Guidelines 03/2026 on web scraping in the context of generative AI”, public consultation from July 8 to October 30, 2026 (), and press release of July 8, 2026 (), accessed 2026-10-23. It is a text under consultation and may change. [^9]: Directive 96/9/EC, articles 7.1, 7.5 and 10.1 (EUR-Lex: ; Spanish text read on WIPO Lex, ); Royal Legislative Decree 1/1996, Spanish Intellectual Property Act, article 133 (). Accessed 2026-10-23. [^10]: Royal Decree-Law 24/2021, of November 2, article 67, “Minería de textos y datos” (text and data mining), which transposes Directive (EU) 2019/790 (, accessed 2026-10-23; in Spanish, quotation translated). ## Self-assessment 1. A robots.txt contains `Disallow: /empleo/` and, further down, `Allow: /empleo/ofertas/`. According to RFC 9309, can `/empleo/ofertas/E01` be crawled? - A. No: the first rule that matches is `Disallow` - B. Yes: the rule that matches the most characters wins, and `Allow: /empleo/ofertas/` is longer - C. Only if the `User-agent` is `*` - D. It depends on the `Crawl-delay` *Answer B: Yes: the rule that matches the most characters wins, and `Allow: /empleo/ofertas/` is longer.* RFC 9309 applies the most specific match (the one with the most octets) and, on a tie, `Allow`. Python 3.13.15 and 3.14.8 do so; Python 3.12.14 applied the first rule that matched and returned False (checked in the lesson). 2. The server answers 503 when the robots.txt is requested. What must the crawler do according to RFC 9309? - A. Crawl everything, because there are no rules - B. Crawl only the home page - C. Assume everything is disallowed until the robots.txt is available again - D. Use the robots.txt of another similar website *Answer C: Assume everything is disallowed until the robots.txt is available again.* A 5xx error means “unreachable” and the crawler must assume complete disallow. A 4xx (does not exist) allows crawling. `urllib.robotparser.read()` follows that rule, and also treats 401 and 403 as complete disallow. 3. Which `User-Agent` is appropriate for a price crawler in a shared repository? - A. A browser's, so that it does not get blocked - B. The author's name and personal email - C. The library's default - D. Its own product name and a URL with information about the crawler, with no personal data *Answer D: Its own product name and a URL with information about the crawler, with no personal data.* Its own name lets the site apply its robots.txt rules to the crawler (RFC 9309 matches the `User-agent` against that name) and the URL gives it a contact. Disguising as a browser prevents the site from applying its rules, and a personal email exposes personal data. 4. When crawling a product again, the client sends `If-None-Match` with the saved `ETag` and the server answers 304. What does it mean? - A. That the page has not changed: the saved copy is reused and the server does not send the body - B. That the product no longer exists - C. That the crawler is blocked - D. That it has to retry later *Answer A: That the page has not changed: the saved copy is reused and the server does not send the body.* 304 Not Modified is the response to a conditional request whose condition holds. In the lesson, the second pass downloads 76% fewer bytes: less load for the site and the same data. 5. The job postings being crawled include the name and email of the contact person. What is done with them if the study only needs to count postings by company and category? - A. They are kept just in case - B. They are stored encrypted - C. They are not stored: they are discarded during parsing, because the data minimization principle limits data to what is necessary for the purpose - D. They are published with the rest of the dataset *Answer C: They are not stored: they are discarded during parsing, because the data minimization principle limits data to what is necessary for the purpose.* A name and an email are personal data (GDPR article 4) and article 5.1.c requires data “limited to what is necessary”. If the purpose does not need them, the right thing is not to collect them. ## Key points - Before crawling: is there an API, an official download or a license? What do the robots.txt and the terms of use say? Is there personal data? Never behind a login or by getting around technical controls. - The robots.txt (RFC 9309) is matched by crawler name, the longest rule wins, `*` and `$` are supported, a 5xx means complete disallow and it may be cached for 24 hours at most. It is not an authorization or an access control. - `urllib.robotparser.read()` downloads without your `User-Agent`: download the robots.txt with your client and pass it to `parse()`. Check the matching in your Python version: 3.12.14 does not follow the longest-match rule or the wildcards; 3.13.15 and 3.14.8 do. - A responsible client identifies itself without personal data, goes slowly (`Crawl-delay` or slower), honors `Retry-After` on 429s, retries with growing backoff, stores `ETag` and `Last-Modified` and asks with conditional requests. - JSON-LD and semantic attributes are more stable than visible text: in the local shop, a regular expression over the text fails on the 12 discounted products; the CSS selector and the JSON-LD get all 30 right. - In the EU, scraping personal data is processing (GDPR); a database may carry a sui generis right (Directive 96/9/EC); the text and data mining exception gives way if the rightholder reserves the use by machine-readable means. This is not legal advice. ## Resources ### Documentation - [RFC 9309: Robots Exclusion Protocol](https://www.rfc-editor.org/rfc/rfc9309.html) · rfc-editor.org - [Python: urllib.robotparser](https://docs.python.org/3/library/urllib.robotparser.html) · docs.python.org - [HTTPX: transports (retries and MockTransport)](https://www.python-httpx.org/advanced/transports/) · python-httpx.org - [RFC 9110: HTTP Semantics (conditional requests, Retry-After)](https://www.rfc-editor.org/rfc/rfc9110.html) · rfc-editor.org - [Regulation (EU) 2016/679 (GDPR), text in the BOE (Spanish)](https://www.boe.es/buscar/doc.php?id=DOUE-L-2016-80807) · boe.es - [AEPD: data scraping tools for trend analysis (criterion of 2025-11-25, Spanish)](https://www.aepd.es/informes-y-resoluciones/criterios-juridicos-aepd/herramientas-data-scraping-para-evaluacion-de-tendencias) · aepd.es - [EDPB: Guidelines 03/2026 on web scraping in the context of generative AI (public consultation)](https://www.edpb.europa.eu/public-consultations/guidelines-032026-on-web-scraping-in-the-context-of-generative-ai_en) · edpb.europa.eu Previous lesson: [6.7.2 · Types of alternative data: what each one measures](https://pythonparatrading.com/en/python-trading-course/level-6/types-of-alternative-data/) Next lesson: [6.7.4 · Alternative data vendors and due diligence](https://pythonparatrading.com/en/python-trading-course/level-6/alternative-data-vendors-and-due-diligence/)