Responsible web scraping.

When to scrape and when not to

Shop prices, job postings, reviews, product pages: much of the alternative data from lesson 6.7.2 is collected from public websites with a program that downloads pages and extracts fields, web scraping. It is technically simple and that is why it gets abused. Before writing a line, four questions:

flowchart TB
  A{Official route} -- yes --> U[Use it]
  A -- no --> B{Behind login}
  B -- yes --> N[Do not crawl]
  B -- no --> R{robots and terms}
  R -- forbid --> N
  R -- allow --> P{Personal data}
  P -- yes --> M[Minimize or skip]
  P -- no --> S[Crawl slowly]

• Is there an official route? An API, a download file, a feed or a license from the site itself or from a vendor (lessons 1.3.7 and 6.7.4). It is always preferable: it is stable, has clear terms and does not load the server.
• Do you need to log in? What sits behind a username and a password is not public: it is governed by the account contract. Nor do you get around blocks, CAPTCHAs or technical limits.
• What do the robots.txt and the terms of use say? The first is a technical convention; the second, a legal text. Neither replaces the other.
• Is there personal data? Names, emails, photos, signed opinions. If the purpose does not need them, they are not collected.

This lesson does not crawl any real website. All the code runs against a local server that the first cell starts on your own machine: a fictitious shop with products, prices, job postings and its robots.txt. The code for real websites is the same with another base URL; where something is needed that is not installed (Playwright), it goes in a skip block.

A local shop to practice on

The server serves, with the products and brands from the level 6 reviews, a paginated listing, a page per product (with the price on the page and in JSON-LD, schema.org's structured data format), job postings with the contact person, a page whose price is loaded by JavaScript, and a robots.txt. The shop is Spanish and so is its content: its pages, job titles and prices (in Spanish number format, 1.299,00 €) stay as they are, because the parsers depend on them. It behaves like a real server in what matters to the client: it answers 429 (Too Many Requests) with Retry-After on one in every 15 requests (always the same ones, so the output does not depend on the clock), gives ETag and Last-Modified, answers 304 to conditional requests and records the User-Agent of each request. The price of each product is decided by the server (the ground truth) and changes for 20% of the products on “day 2”:

import hashlib
import html
import json
import threading
from email.utils import formatdate
from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer
from urllib.parse import parse_qs, urlparse

import numpy as np
import pandas as pd

rng = np.random.default_rng(673)
cat = (pd.read_parquet("alt6_resenas.parquet")[["producto", "marca"]]
       .drop_duplicates().sort_values("producto")
       .reset_index(drop=True))
cat["id"] = [f"P{i:02d}" for i in range(1, len(cat) + 1)]
cat["precio1"] = np.round(rng.uniform(20, 1500, len(cat)), 0) - 0.01
cambia = rng.random(len(cat)) < 0.2
cat["precio2"] = np.where(
    cambia, np.round(cat["precio1"] * rng.uniform(0.85, 1.1,
                                                  len(cat)), 2),
    cat["precio1"])
cat["rebaja"] = rng.random(len(cat)) < 0.3   # struck-through price
emp = pd.read_csv("acciones6_empresas.csv").head(8)
ofertas = pd.DataFrame({
    "id": [f"E{i:02d}" for i in range(1, 9)],
    "empresa": emp["nombre_corto"],
    "puesto": ["Analista de datos", "Ingeniera de backend",
               "Jefe de tienda", "Técnico de mantenimiento",
               "Comercial", "Controller financiero",
               "Científica de datos", "Responsable de compras"],
    "contacto": ["Laura Gómez", "Pablo Ruiz", "Marta Sanz",
                 "Jorge Pardo", "Elena Vidal", "Iván Mora",
                 "Sara León", "Raúl Ibáñez"]})
FECHAS = {1: 1790841600, 2: 1790928000}  # Oct 1 and 2, 2026, 8:00
ROBOTS = """User-agent: *
Disallow: /cuenta/
Disallow: /buscar
Disallow: /*?orden=
Disallow: /empleo/
Allow: /empleo/ofertas/
Crawl-delay: 2

User-agent: AgresivoBot
Disallow: /
"""

def euros(x):
    s = f"{x:,.2f}".replace(",", "X").replace(".", ",")
    return s.replace("X", ".") + " €"

def ficha(fila, dia):
    p = fila[f"precio{dia}"]
    ld = {"@context": "https://schema.org", "@type": "Product",
          "name": fila["producto"], "brand": fila["marca"],
          "sku": fila["id"], "offers": {"@type": "Offer",
          "price": f"{p:.2f}", "priceCurrency": "EUR"}}
    antes = (f'<del class="precio-anterior">{euros(p * 1.25)}'
             '</del> ' if fila["rebaja"] else "")
    return (f"<html><head><title>{fila['producto']}</title>"
            '<script type="application/ld+json">'
            f"{json.dumps(ld)}</script></head><body>"
            f"<h1>{fila['producto']}</h1><p>Marca: {fila['marca']}"
            f'</p><p>{antes}<span class="precio">{euros(p)}'
            "</span></p></body></html>")

def oferta(o):
    ld = {"@context": "https://schema.org", "@type": "JobPosting",
          "title": o["puesto"], "datePosted": "2026-10-01",
          "hiringOrganization": {"@type": "Organization",
                                 "name": o["empresa"]},
          "jobLocation": {"@type": "Place", "address": "Madrid"}}
    correo = (o["contacto"].lower().replace(" ", ".")
              .translate(str.maketrans("áéíóú", "aeiou")))
    return ('<html><head><script type="application/ld+json">'
            f"{json.dumps(ld)}</script></head><body><h1>"
            f"{o['puesto']}</h1><p>Contacto: {o['contacto']}, "
            f"{correo}@tienda.example</p></body></html>")

class Tienda(BaseHTTPRequestHandler):
    dia, n, uas = 1, 0, []
    def do_GET(self):
        url = urlparse(self.path)
        Tienda.uas.append(self.headers.get("User-Agent", ""))
        if url.path == "/robots.txt":
            return self.enviar(200, ROBOTS, "text/plain")
        Tienda.n += 1
        if Tienda.n % 15 == 0:
            return self.enviar(429, "despacio", "text/plain",
                               {"Retry-After": "5"})
        if url.path == "/productos":
            pag = int(parse_qs(url.query).get("pagina", ["1"])[0])
            trozo = cat.iloc[(pag - 1) * 20: pag * 20]
            items = "".join(
                f'<li class="producto"><a href="/producto/{i}">'
                f"{html.escape(p)}</a></li>"
                for i, p in zip(trozo["id"], trozo["producto"]))
            sig = (f'<a rel="next" href="/productos?pagina={pag + 1}">'
                   "Siguiente</a>" if pag * 20 < len(cat) else "")
            return self.enviar(200, f"<ul>{items}</ul>{sig}")
        if url.path.startswith("/producto/"):
            fila = cat[cat["id"] == url.path.rsplit("/", 1)[1]]
            if fila.empty:
                return self.enviar(404, "no existe")
            fila = fila.iloc[0]
            cuerpo = ficha(fila, Tienda.dia)
            etag = '"' + hashlib.sha256(
                cuerpo.encode()).hexdigest()[:16] + '"'
            dia_mod = 2 if (Tienda.dia == 2 and
                            fila["precio2"] != fila["precio1"]) else 1
            lm = formatdate(FECHAS[dia_mod], usegmt=True)
            if (self.headers.get("If-None-Match") == etag or
                    self.headers.get("If-Modified-Since") == lm):
                return self.enviar(304, "", extra={"ETag": etag})
            return self.enviar(200, cuerpo, extra={
                "ETag": etag, "Last-Modified": lm})
        if url.path == "/empleo/ofertas/":
            items = "".join(f'<a href="/empleo/ofertas/{i}">{i}</a>'
                            for i in ofertas["id"])
            return self.enviar(200, f"<body>{items}</body>")
        if url.path.startswith("/empleo/ofertas/"):
            o = ofertas[ofertas["id"] == url.path.rsplit("/", 1)[1]]
            return self.enviar(200, oferta(o.iloc[0]))
        if url.path.startswith("/precio-dinamico/"):
            pid = url.path.rsplit("/", 1)[1]
            return self.enviar(200, (
                '<span id="precio">cargando</span><script>'
                f"fetch('/api/precio/{pid}').then(r => r.json())"
                ".then(d => document.getElementById('precio')"
                ".textContent = d.precio)</script>"))
        if url.path.startswith("/api/precio/"):
            fila = cat[cat["id"] == url.path.rsplit("/", 1)[1]].iloc[0]
            return self.enviar(200, json.dumps(
                {"id": fila["id"],
                 "precio": fila[f"precio{Tienda.dia}"]}),
                "application/json")
        self.enviar(404, "no existe")
    def enviar(self, codigo, texto, tipo="text/html", extra=None):
        cuerpo = texto.encode()
        self.send_response(codigo)
        self.send_header("Content-Type", f"{tipo}; charset=utf-8")
        self.send_header("Content-Length", str(len(cuerpo)))
        for k, v in (extra or {}).items():
            self.send_header(k, v)
        self.end_headers()
        if codigo != 304:
            self.wfile.write(cuerpo)
    def log_message(self, *args):
        pass

servidor = ThreadingHTTPServer(("127.0.0.1", 0), Tienda)
threading.Thread(target=servidor.serve_forever, daemon=True).start()
BASE = f"http://127.0.0.1:{servidor.server_port}"
print(len(cat), "products;", int(cambia.sum()),
      "change price on day 2;", int(cat["rebaja"].sum()),
      "with a struck-through price;", len(ofertas), "job postings")

30 products; 7 change price on day 2; 12 with a struck-through price; 8 job postings

robots.txt: what it says and what it does not

The Robots Exclusion Protocol is a text file at the root of the site, /robots.txt, with groups of rules per crawler. It had been used as a convention since 1994 and in September 2022 it was published as a standard, RFC 930911Note 1 · RFC 9309, “Robots Exclusion Protocol” (September 2022, https://www.rfc-editor.org/rfc/rfc9309.html, accessed 2026-10-23): case-insensitive matching on the product name; “The most specific match is the match that has the most octets”; on a tie, allow; * and $; 4xx “unavailable” (crawling allowed) and 5xx “unreachable” (complete disallow); 24-hour cache at most; a parsing limit of at least 500 KiB; “These rules are not a form of access authorization”.. Its main rules:

Crawl-delay (seconds between requests) is not in the RFC, but many sites use it and urllib.robotparser reads it: honoring it is the prudent choice. The standard library module has a trap: read() downloads the file with urllib and its default User-Agent, not with the crawler's. The cell shows this by asking the server to say which header it received, and then downloads the robots.txt with httpx and the crawler's own name:

import urllib.robotparser

import httpx

UA = "CursoPPT-precios/0.1 (+https://example.com/rastreador)"

rp_std = urllib.robotparser.RobotFileParser(f"{BASE}/robots.txt")
rp_std.read()
print("read() identifies itself as:", Tienda.uas[-1])

cliente = httpx.Client(base_url=BASE, timeout=10.0,
                       headers={"User-Agent": UA})
r = cliente.get("/robots.txt")
print("httpx identifies itself as:", Tienda.uas[-1], "->",
      r.status_code)
rp = urllib.robotparser.RobotFileParser()
rp.parse(r.text.splitlines())
print("Crawl-delay:", rp.crawl_delay(UA))

read() identifies itself as: Python-urllib/3.14
httpx identifies itself as: CursoPPT-precios/0.1 (+https://example.com/rastreador) -> 200
Crawl-delay: 2

With the robots.txt loaded, can_fetch answers for each URL. The table tests the rules that matter: the disallowed path, the parameter with a wildcard, the long rule that beats the short one and another crawler's group:

pruebas = ["/productos?pagina=1", "/producto/P01",
           "/productos?orden=precio", "/cuenta/pedidos",
           "/buscar?q=cafetera", "/empleo/candidatos/",
           "/empleo/ofertas/E01", "/api/precio/P01"]
pd.DataFrame({
    "CursoPPT": [rp.can_fetch(UA, BASE + u) for u in pruebas],
    "AgresivoBot": [rp.can_fetch("AgresivoBot/2.0", BASE + u)
                    for u in pruebas]}, index=pruebas)

                         CursoPPT  AgresivoBot
/productos?pagina=1          True        False
/producto/P01                True        False
/productos?orden=precio     False        False
/cuenta/pedidos             False        False
/buscar?q=cafetera          False        False
/empleo/candidatos/         False        False
/empleo/ofertas/E01          True        False
/api/precio/P01              True        False

/empleo/ofertas/E01 is allowed even though Disallow: /empleo/ appears first: Allow: /empleo/ofertas/ matches more characters. That behavior is the RFC's, but not that of every Python version: with the same file, Python 3.12.14 applies the first rule that matches and does not understand wildcards (it blocks the job postings and lets ?orden= through); Python 3.13.15 and 3.14.8, the one in the course environment, follow the RFC (checked when writing the lesson, on 2026-10-23). The module's documentation refers to RFC 9309 without detailing the matching22Note 2 · Python, “urllib.robotparser — Parser for robots.txt” (https://docs.python.org/3/library/urllib.robotparser.html, accessed 2026-10-23): it refers to RFC 9309; crawl_delay, request_rate and site_maps. The behavior with a long Allow and wildcards was checked by running the same robots.txt on Python 3.12.14, 3.13.15 and 3.14.8.: if the crawler runs on another version, it has to be tested with a known robots.txt, like this table.

A client that identifies itself and goes slowly

The client from lesson 1.4.1 already retried with growing backoff. Here it gets what it needs to crawl someone else's site:

flowchart TB
  U[URL] --> R{robots.txt}
  R -- yes --> L[Rate limit]
  L --> G[GET]
  G -- 429 --> W[Retry-After]
  W --> L
  G -- 200 --> C[Cache]

• Identification: a product name of its own and a URL that explains what the crawler does and how to get in touch, with no personal data (neither a person's name nor email). With a name of its own, the site can write a group for it in the robots.txt; with a browser's User-Agent, it cannot.
• Pace: one request every Crawl-delay seconds or more, never in parallel against the same site.
• Errors: on a 429, wait for what Retry-After says (RFC 9110, section 10.2.333Note 3 · RFC 9110, “HTTP Semantics” (https://www.rfc-editor.org/rfc/rfc9110.html, accessed 2026-10-23): Retry-After (section 10.2.3), conditional requests with If-None-Match and If-Modified-Since (section 13.1) and 304 Not Modified (section 15.4.5). The 429 code is defined in RFC 6585 (https://www.rfc-editor.org/rfc/rfc6585.html), which allows it to be accompanied by Retry-After.); on a 5xx, retry with growing backoff and some randomness, and give up after a few attempts. The retries parameter of the httpx transport only retries connection errors (ConnectError, ConnectTimeout), not 429 or 503 responses44Note 4 · HTTPX, “Transports” (https://www.python-httpx.org/advanced/transports/, accessed 2026-10-23): requests “will be retried the given number of times in case an httpx.ConnectError or an httpx.ConnectTimeout occurs”; for other cases it refers to retry libraries such as tenacity.: that logic belongs to the client.

So that the output does not depend on the machine's clock, the client uses a simulated clock that moves forward when the client “sleeps”; in production, time.monotonic() and time.sleep() take its place without touching anything else:

class Reloj:
    """Simulated clock: sleep() moves time forward without waiting."""
    def __init__(self):
        self.t = 0.0
    def now(self):
        return self.t
    def sleep(self, s):
        self.t += s

reloj = Reloj()
azar = np.random.default_rng(0)
registro = []   # (time, url, status code)

class Rastreador:
    def __init__(self, cliente, rp, ua, reloj, intentos=4):
        self.c, self.rp, self.ua = cliente, rp, ua
        self.reloj, self.intentos = reloj, intentos
        self.pausa = max(rp.crawl_delay(ua) or 0, 1.0)
        self.ultima = -np.inf

    def get(self, ruta, cabeceras=None):
        if not self.rp.can_fetch(self.ua, BASE + ruta):
            raise PermissionError(f"robots.txt disallows {ruta}")
        for n in range(self.intentos):
            espera = self.ultima + self.pausa - self.reloj.now()
            if espera > 0:
                self.reloj.sleep(espera)
            self.ultima = self.reloj.now()
            r = self.c.get(ruta, headers=cabeceras)
            registro.append((self.reloj.now(), ruta, r.status_code))
            if r.status_code == 429:
                self.reloj.sleep(float(r.headers["Retry-After"]))
            elif r.status_code >= 500:
                self.reloj.sleep(2 ** n + azar.uniform(0, 1))
            else:
                return r
        r.raise_for_status()

bot = Rastreador(cliente, rp, UA, reloj)
try:
    bot.get("/cuenta/pedidos")
except PermissionError as e:
    print(e)
r = bot.get("/productos?pagina=1")
print(r.status_code, len(r.text), "bytes; clock:", reloj.now(), "s")

robots.txt disallows /cuenta/pedidos
200 1352 bytes; clock: 0.0 s

Extracting the data: BeautifulSoup and JSON-LD

The listing has links to the product pages and a rel="next" link to the next page. Following that link, instead of inventing URLs, means the crawler does not depend on how the site numbers its pages. BeautifulSoup turns the HTML into a tree that is queried with CSS selectors; lxml is not in the ia environment, so the parser is the standard library's, html.parser:

from bs4 import BeautifulSoup

fichas, ruta = {}, "/productos?pagina=1"
while ruta:
    sopa = BeautifulSoup(bot.get(ruta).text, "html.parser")
    for a in sopa.select("li.producto a"):
        fichas[a["href"]] = None
    sig = sopa.find("a", rel="next")
    ruta = sig["href"] if sig else None
cache = {}
for ruta in fichas:
    r = bot.get(ruta)
    cache[ruta] = {"etag": r.headers.get("ETag"),
                   "lm": r.headers.get("Last-Modified"),
                   "html": r.text}
bytes_dia1 = sum(len(v["html"].encode()) for v in cache.values())
print(len(cache), "product pages;", bytes_dia1, "bytes; clock:",
      reloj.now(), "s")
print(pd.Series([c for _, _, c in registro]).value_counts()
      .to_dict())

30 product pages; 11888 bytes; clock: 74.0 s
{200: 33, 429: 2}

Thirty product pages and two listing pages, with one request every 2 seconds (the Crawl-delay) and two 429s along the way, which add 5 seconds of waiting each; the header figure draws that path. The price appears twice on each product page: in the visible text (<span class="precio">1.299,00 €</span>, in Spanish format) and in JSON-LD ("price": "1299.00"). The two ways of reading it, against the server's ground truth:

import re

def precio_texto(h):
    s = re.search(r"([\d.]+,\d{2}) €", h).group(1)  # the first one
    return float(s.replace(".", "").replace(",", "."))

def precio_css(sopa):
    s = sopa.select_one("span.precio").get_text()
    return float(s.replace(" €", "").replace(".", "")
                 .replace(",", "."))

def precio_jsonld(sopa):
    ld = json.loads(sopa.find(
        "script", type="application/ld+json").string)
    return float(ld["offers"]["price"]), ld["sku"]

filas = []
for ruta, v in cache.items():
    sopa = BeautifulSoup(v["html"], "html.parser")
    p_ld, sku = precio_jsonld(sopa)
    filas.append({"id": sku, "regex": precio_texto(v["html"]),
                  "css": precio_css(sopa), "jsonld": p_ld})
ext = pd.DataFrame(filas).merge(cat[["id", "precio1", "rebaja"]])
for m in ["regex", "css", "jsonld"]:
    ok = np.isclose(ext[m], ext["precio1"])
    print(f"{m:7s} correct: {ok.mean():.0%}")

regex   correct: 60%
css     correct: 100%
jsonld  correct: 100%

The regular expression “the first amount in euros” fails on the products with a struck-through price: it reads the previous price. The CSS selector (span.precio) and the JSON-LD get all of them right. Of the two, JSON-LD is the more stable: the site publishes it for search engines, with the price as a number and the currency separately, and it does not change when the page design changes. If a redesign renames the precio class, the selector breaks without warning; that is why a production crawler checks every day how many empty or out-of-range fields it has extracted.

Conditional requests and caching

The next day, the crawler comes back. Most of the product pages have not changed, and the HTTP protocol has a way of not downloading them again: the conditional request. The client sends the ETag it saved (If-None-Match) or the date of the last modification (If-Modified-Since); if nothing has changed, the server answers 304 Not Modified, without a body (RFC 9110, section 13³):

Tienda.dia = 2
bytes_dia2, codigos, cambios = 0, [], []
for ruta, v in cache.items():
    r = bot.get(ruta, {"If-None-Match": v["etag"]})
    codigos.append(r.status_code)
    if r.status_code == 200:
        bytes_dia2 += len(r.content)
        v.update(etag=r.headers["ETag"], html=r.text,
                 lm=r.headers["Last-Modified"])
        cambios.append(ruta.rsplit("/", 1)[1])
print(pd.Series(codigos).value_counts().to_dict())
print("product page bytes: day 1", bytes_dia1, "| day 2",
      bytes_dia2, f"({1 - bytes_dia2 / bytes_dia1:.0%} less)")
verdad = cat.loc[cat["precio2"] != cat["precio1"], "id"].tolist()
print("changes detected = ground truth:", sorted(cambios) == verdad)

{304: 23, 200: 7}
product page bytes: day 1 11888 | day 2 2832 (76% less)
changes detected = ground truth: True

Seven product pages come back with 200 (the ones whose price changed, all detected) and twenty-three with 304: 76% fewer bytes. The 304s also count as requests and respect the pace, but the server neither generates nor sends the page. If-Modified-Since works the same way with the date; it is less precise (one-second resolution) and is useful when the site gives no ETag:

v = cache["/producto/P01"]
r = bot.get("/producto/P01", {"If-Modified-Since": v["lm"]})
print(v["lm"], "->", r.status_code)

Thu, 01 Oct 2026 08:00:00 GMT -> 304

For research, the cache has another advantage: saving each day's original HTML (not just the extracted fields) lets you parse it again when a parser bug is fixed, without crawling again, and reconstruct what was known on each date (point-in-time, lesson 1.4.4).

Pages with JavaScript

Some pages do not carry the data in the HTML: the browser requests it with JavaScript after loading. An HTTP client like httpx does not run JavaScript and sees the placeholder text (cargando, “loading”):

r = bot.get("/precio-dinamico/P01")
sopa = BeautifulSoup(r.text, "html.parser")
print("price in the HTML:", sopa.select_one("#precio").get_text())
api = re.search(r"fetch\('([^']+)'\)", r.text).group(1)
print("the script requests:", api)
print("allowed?", rp.can_fetch(UA, BASE + api))
print(bot.get(api).json())

price in the HTML: cargando
the script requests: /api/precio/P01
allowed? True
{'id': 'P01', 'precio': 1492.99}

The script requests the data from a JSON endpoint on the site itself; if the robots.txt and the terms allow it, requesting that JSON directly is the most efficient option (one small request instead of loading a browser). When there is no way to avoid the browser (the data is computed in the page, or the page requires interaction), an automated browser such as Playwright is used, which drives Chromium, Firefox or WebKit. It is not installed in the course environments (checked on 2026-10-23), so its code goes unexecuted. It is installed with pip install playwright followed by playwright install, which downloads the browsers55Note 5 · Playwright for Python, “Getting started - Library” (https://playwright.dev/python/docs/library, accessed 2026-10-23): pip install playwright and playwright install, which installs the browsers; new_context accepts user_agent and locale (https://playwright.dev/python/docs/api/class-browser).:

from playwright.sync_api import sync_playwright

with sync_playwright() as p:
    navegador = p.chromium.launch()
    contexto = navegador.new_context(user_agent=UA, locale="es-ES")
    pagina = contexto.new_page()
    pagina.goto(f"{BASE}/precio-dinamico/P01")
    pagina.wait_for_function(
        "document.getElementById('precio').textContent"
        " !== 'cargando'")
    print(pagina.locator("#precio").inner_text())
    navegador.close()

An automated browser loads everything a person loads (scripts, images, fonts): it weighs more on the server and on your own machine. The same rules of pace, robots.txt and identification apply to it (user_agent in the context).

The requests library, the most widespread, does the same as httpx in synchronous mode (httpx adds an asynchronous client and optional HTTP/2). The pattern is the same: a session with the fixed header and a timeout:

import requests

with requests.Session() as s:
    s.headers["User-Agent"] = UA
    r = s.get(f"{BASE}/robots.txt", timeout=10)
print(r.status_code, r.text.splitlines()[0], "|", Tienda.uas[-1])

200 User-agent: * | CursoPPT-precios/0.1 (+https://example.com/rastreador)

Personal data: do not collect it

The shop's job postings include the name and email of the contact person, as many real ones do. For a hiring study (how many postings, from which companies and categories) they are not needed. The safest way to comply is not to store them: the parser extracts from the JSON-LD only the study's fields and never touches the free text:

CAMPOS = ["title", "datePosted"]
sopa = BeautifulSoup(bot.get("/empleo/ofertas/").text,
                     "html.parser")
guardadas, correos_vistos = [], 0
for a in sopa.find_all("a"):
    h = bot.get(a["href"]).text
    correos_vistos += len(re.findall(r"[\w.]+@[\w.]+", h))
    ld = json.loads(BeautifulSoup(h, "html.parser").find(
        "script", type="application/ld+json").string)
    fila = {k: ld[k] for k in CAMPOS}
    fila["empresa"] = ld["hiringOrganization"]["name"]
    guardadas.append(fila)
ofe = pd.DataFrame(guardadas)
texto = ofe.to_csv(index=False)
print("emails in the pages:", correos_vistos,
      "| in what was stored:", len(re.findall(r"@", texto)))
ofe.head(3)

emails in the pages: 8 | in what was stored: 0
                  title  datePosted                 empresa
0     Analista de datos  2026-10-01          Bagvor Telecom
1  Ingeniera de backend  2026-10-01  Bakdor Pharmaceuticals
2        Jefe de tienda  2026-10-01        Bebvia Petroleum

Eight emails in the pages and none in the dataset. The allowlist of fields (CAMPOS) is the important decision: what is not on the list is not stored, even if tomorrow the site adds a new field with personal data.

When finished, the client and the server are closed:

print("requests:", len(registro), "| 429:",
      sum(c == 429 for _, _, c in registro),
      "| simulated clock:", reloj.now(), "s")
cliente.close()
servidor.shutdown()

requests: 80 | 429: 5 | simulated clock: 173.0 s

The legal framework in the EU

Four frameworks intersect in a crawl:

1. Terms of use. They are a contract between the site and whoever uses it, and they can forbid automated access or the commercial reuse of the content. A robots.txt that allows a path does not override a prohibition in the terms. For market data, lesson 1.3.7 covered licenses; the same logic (which use is allowed, whether there is redistribution) applies to a website's content.
2. Data protection (GDPR). If what is collected makes it possible to identify a person, it is personal data: “any information relating to an identified or identifiable natural person” (article 4.1). Collecting it is processing and needs a legal basis from article 6; for scraping, the debate centers on legitimate interest under 6.1.f, which requires that the data subject's rights do not override it. The data have to be “adequate, relevant and limited to what is necessary” (5.1.c, data minimization), and whoever obtains them without going through the data subject has a duty to inform (article 14), with exceptions such as disproportionate effort (14.5.b)66Note 6 · Regulation (EU) 2016/679, text published in the OJEU and reproduced by the BOE, the Spanish Official Gazette (https://www.boe.es/buscar/doc.php?id=DOUE-L-2016-80807, accessed 2026-10-23): articles 4.1, 5.1.c, 6.1.f and 14.5.b. The EUR-Lex version is https://eur-lex.europa.eu/eli/reg/2016/679/oj.. The AEPD (the Spanish data protection authority) sums up that scraping, “when it collects personal data, does involve processing”, and closed a case because the tool only showed aggregated information, with no identifiable persons77Note 7 · AEPD, criterion “Herramientas de data scraping para la evaluación de tendencias” (data scraping tools for trend analysis), November 25, 2025 (procedure AI/00059/2024, https://www.aepd.es/informes-y-resoluciones/criterios-juridicos-aepd/herramientas-data-scraping-para-evaluacion-de-tendencias, accessed 2026-10-23; in Spanish, quotation translated). The page warns that the summary is not the official publication.. On July 8, 2026 the European Data Protection Board adopted guidelines on scraping for generative AI, in public consultation until October 30, 2026, which develop legitimate interest and data minimization88Note 8 · EDPB, “Guidelines 03/2026 on web scraping in the context of generative AI”, public consultation from July 8 to October 30, 2026 (https://www.edpb.europa.eu/public-consultations/guidelines-032026-on-web-scraping-in-the-context-of-generative-ai_en), and press release of July 8, 2026 (https://www.edpb.europa.eu/news/edpb-sheds-light-on-anonymisation-and-web-scraping-for-generative-ai-and-adopts-final-version_en), accessed 2026-10-23. It is a text under consultation and may change..
3. Sui generis database right. Directive 96/9/EC allows the maker of a database who has made a “substantial investment” to prohibit the extraction or reuse of the whole or of a substantial part (article 7.1) and also the “repeated and systematic” extraction of insubstantial parts that conflicts with its normal exploitation (7.5), for fifteen years (article 10.1). In Spain it is in article 133 of the Intellectual Property Act (Ley de Propiedad Intelectual)99Note 9 · Directive 96/9/EC, articles 7.1, 7.5 and 10.1 (EUR-Lex: https://eur-lex.europa.eu/eli/dir/1996/9/oj; Spanish text read on WIPO Lex, https://www.wipo.int/wipolex/es/text/126790); Royal Legislative Decree 1/1996, Spanish Intellectual Property Act, article 133 (https://www.boe.es/buscar/act.php?id=BOE-A-1996-8930). Accessed 2026-10-23.. A daily crawler of a whole price catalog is precisely a repeated and systematic extraction.
4. Text and data mining. Directive (EU) 2019/790 created an exception for text and data mining of lawfully accessible content; in Spain, article 67 of Royal Decree-Law 24/2021. It does not apply when the rightholders “have expressly reserved the use of the works to machine-readable means or other appropriate means” (67.3)1010Note 10 · Royal Decree-Law 24/2021, of November 2, article 67, “Minería de textos y datos” (text and data mining), which transposes Directive (EU) 2019/790 (https://www.boe.es/buscar/act.php?id=BOE-A-2021-17910, accessed 2026-10-23; in Spanish, quotation translated).. The law does not say what form that reservation takes; a robots.txt or a clause in the terms that expresses it are the obvious candidates, and the prudent course is to respect them as such.

For a fund, there is a further risk, covered in lesson 6.7.4: that the collected data is inside information or was obtained in breach of a duty of confidentiality. Fund compliance teams usually require a review before any scraped data is used.

Common mistakes

• Crawling what has an API or a license. More work, less stability and more risk than the official route.
• Disguising as a browser. It prevents the site from applying its rules and is the first sign of bad faith if there is a dispute.
• Putting personal data in the User-Agent. A product name and an information URL are enough.
• Using robotparser.read() without thinking: it downloads with another User-Agent and does not store the date so it can be requested again within 24 hours. And not checking the matching of the Python version in use.
• Parallelizing against a single site or ignoring Retry-After: the block arrives before the data.
• Parsing the visible text when there is structured data: struck-through prices, units and redesigns break regular expressions.
• Storing everything “just in case”: personal data that is not collected does not have to be protected, deleted or explained.

Exercises

1. Change the server so that it answers 503 when /robots.txt is requested and check what RobotFileParser.read() does. Does it match RFC 9309?

class Caida(BaseHTTPRequestHandler):
    def do_GET(self):
        self.send_response(503)
        self.end_headers()
    def log_message(self, *args):
        pass

s503 = ThreadingHTTPServer(("127.0.0.1", 0), Caida)
threading.Thread(target=s503.serve_forever, daemon=True).start()
rp503 = urllib.robotparser.RobotFileParser(
    f"http://127.0.0.1:{s503.server_port}/robots.txt")
rp503.read()
print("can anything be crawled?",
      rp503.can_fetch(UA, "http://127.0.0.1/productos"))
s503.shutdown()

can anything be crawled? False

2. Add to the Rastreador a robots.txt cache that requests it again if more than 24 hours have passed (with the simulated clock) and check it by moving the clock forward.

class RobotsCache:
    def __init__(self, cliente, reloj, vida=24 * 3600):
        self.c, self.reloj, self.vida = cliente, reloj, vida
        self.rp, self.cuando, self.descargas = None, None, 0
    def get(self):
        if (self.cuando is None or
                self.reloj.now() - self.cuando > self.vida):
            r = self.c.get("/robots.txt")
            self.rp = urllib.robotparser.RobotFileParser()
            self.rp.parse(r.text.splitlines())
            self.cuando = self.reloj.now()
            self.descargas += 1
        return self.rp

# the server is already closed: a simulated httpx transport
falso = httpx.MockTransport(
    lambda req: httpx.Response(200, text=ROBOTS))
with httpx.Client(transport=falso, base_url="http://tienda.example",
                  headers={"User-Agent": UA}) as c2:
    rc = RobotsCache(c2, Reloj())
    rc.get()
    rc.reloj.sleep(3600)
    rc.get()
    rc.reloj.sleep(24 * 3600)
    print("allowed?", rc.get().can_fetch(
        UA, "http://tienda.example/producto/P01"))
print("robots.txt downloads:", rc.descargas)

allowed? True
robots.txt downloads: 2

3. A vendor offers you a dataset of prices scraped from 2,000 online shops. What questions would you ask about how they collect them?