Text sources: EDGAR, CNMV, transcripts and social media.

The map of text sources

Lesson 6.1.1 turned text into scores. Before that comes a less glamorous job that decides the quality of everything else: obtaining the text, timestamping it precisely and cleaning it. The usual sources in financial research:

The CNMV (Comisión Nacional del Mercado de Valores) is the Spanish securities regulator; OIR stands for otra información relevante, other relevant information.

flowchart TB
  F[Source] --> D[Download]
  D --> P[Parsing]
  P --> L[Cleaning]
  L --> X[Timestamped table]
  X --> S[Scoring 6.1.1]

The course ships a synthetic version of each source in the level 6 data, with invented documents that follow the shape of the real ones: 4,020 SEC-type and CNMV-type regulatory documents, 13,067 XBRL facts, 840 calls and 61,697 social media posts, in addition to the news of the previous lesson. All of them say in their text that they are synthetic. Part of the corpus is in Spanish (the CNMV-type documents and many of the news items, calls and posts) and is kept as it is, because it is the data the code processes. Real downloads need a network connection and go in skip blocks; where it helps, a local server imitates the API and the client code actually runs.

EDGAR: the filing history of each company

The SEC publishes on data.sec.gov, without authentication or keys, the filing history of each entity at /submissions/CIK##########.json (the CIK with ten digits and leading zeros)11Note 1 · SEC, “EDGAR Application Programming Interfaces” (https://www.sec.gov/search-filings/edgar-application-programming-interfaces, accessed on October 23, 2026): history at data.sec.gov/submissions/CIK##########.json, “at least one year's of filing or to 1,000 (whichever is more)”, additional files for earlier ones, and “these APIs do not require any authentication or API keys to access”.. Lesson 1.5.1 used the other half of the API, the XBRL facts of companyfacts; here what matters is the document index, which says what was filed, when and where it is:

flowchart TB
  S[submissions JSON] --> R[filings.recent]
  R --> A[accessionNumber]
  R --> T[acceptanceDateTime]
  R --> P[primaryDocument]
  A --> U[Document URL]
  P --> U

filings.recent is in columnar format (one list per field): accessionNumber, filingDate, reportDate, acceptanceDateTime, form, items (those of the 8-K), primaryDocument and others. It contains at least one year or the 1,000 most recent filings; earlier ones go in additional files listed in filings.files. The document is downloaded from the EDGAR archives at /Archives/edgar/data/<CIK>/<accession number without dashes>/<document>, the directory structure described by the SEC itself22Note 2 · SEC, “Accessing EDGAR Data” (https://www.sec.gov/os/accessing-edgar-data, accessed on October 23, 2026): “Current max request rate: 10 requests/second”, declared User-Agent header, paths /Archives/edgar/data/<CIK>/<accession number>/… and daily and full indexes since 1994..

The SEC's fair access policy has two rules that affect the code²: a current maximum of 10 requests per second, and a User-Agent header that identifies who is accessing (its example: “Sample Company Name AdminContact@<sample company domain>.com”). To comply without publishing personal data, the course uses a placeholder with a domain reserved for examples (example.com, RFC 2606): in real use it carries the organization's name and a role contact mailbox (not a person's email), read from the configuration and not written in the code that is shared.

import json
import os
import time
import urllib.request

# Organization and role mailbox, from the configuration (lesson 0.2.4)
UA = os.environ.get("EDGAR_USER_AGENT",
                    "PPT Course [email protected]")

def get(url):
    req = urllib.request.Request(url, headers={"User-Agent": UA})
    with urllib.request.urlopen(req, timeout=30) as r:
        datos = r.read()
    time.sleep(0.15)  # well below 10 requests per second
    return datos

cik = 320193
sub = json.loads(get("https://data.sec.gov/submissions/"
                     f"CIK{cik:010d}.json"))
reciente = sub["filings"]["recent"]
i = reciente["form"].index("10-K")
acc = reciente["accessionNumber"][i].replace("-", "")
doc = reciente["primaryDocument"][i]
html = get(f"https://www.sec.gov/Archives/edgar/data/{cik}/{acc}/{doc}")

To run that same client without a network, the next cell starts a local HTTP server that serves, with the shape of submissions, the synthetic documents of one of the course companies. Like the real one, it rejects requests without a User-Agent (with a 403); to simulate saturation, it answers 429 (Too Many Requests) to one in three document downloads, always the same one, so that the output does not depend on the clock:

import json
import re
import threading
import time
import urllib.error
import urllib.request
from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer

import numpy as np
import pandas as pd

docs = pd.read_parquet("filings6_documentos.parquet")
xbrl = pd.read_parquet("filings6_xbrl.parquet")
CIK = xbrl.merge(docs[["id_doc", "emisor"]], on="id_doc").query(
    "emisor == 'Luklux Apparel Corp.'")["cik"].iloc[0]
mios = docs[docs["emisor"] == "Luklux Apparel Corp."].sort_values("ts")
mios = mios.assign(
    acc=[f"{CIK}-{t:%y}-{n:06d}" for n, t in
         enumerate(mios["ts"], start=1)],
    archivo=[f"{i.lower()}.htm" for i in mios["id_doc"]])

def submissions():
    r = mios.iloc[::-1]  # most recent first, like the API
    ny = r["ts"].dt.tz_convert("America/New_York")
    return {"cik": CIK, "name": "Luklux Apparel Corp.",
            "filings": {"recent": {
                "accessionNumber": r["acc"].tolist(),
                "acceptanceDateTime": ny.dt.strftime(
                    "%Y-%m-%dT%H:%M:%S").tolist(),
                "form": r["tipo"].tolist(),
                "items": r["items"].tolist(),
                "primaryDocument": r["archivo"].tolist()}}}

class EdgarLocal(BaseHTTPRequestHandler):
    peticiones = 0
    def do_GET(self):
        if not self.headers.get("User-Agent", "").strip():
            return self.responder(403, b"Declare your User-Agent")
        if self.path.startswith("/Archives/"):
            EdgarLocal.peticiones += 1
            if EdgarLocal.peticiones % 3 == 0:  # simulated saturation
                return self.responder(429, b"Too Many Requests")
        if self.path == f"/submissions/CIK{CIK}.json":
            cuerpo = json.dumps(submissions()).encode()
            return self.responder(200, cuerpo)
        fila = mios[mios["archivo"] == self.path.rsplit("/", 1)[-1]]
        if len(fila):
            return self.responder(200, fila["texto"].iloc[0].encode())
        self.responder(404, b"Not Found")
    def responder(self, codigo, cuerpo):
        self.send_response(codigo)
        self.end_headers()
        self.wfile.write(cuerpo)
    def log_message(self, *args):
        pass

servidor = ThreadingHTTPServer(("127.0.0.1", 0), EdgarLocal)
threading.Thread(target=servidor.serve_forever, daemon=True).start()
BASE = f"http://127.0.0.1:{servidor.server_port}"
print(len(mios), "Luklux documents on the local server")

62 Luklux documents on the local server

The client is the one from the real block with three changes: the base URL, a retry with growing wait when the server returns 429 and a clear error when the header is missing:

from collections import Counter

UA = "PPT Course [email protected]"

reintentos = Counter()

def get(url, ua=UA, intentos=4):
    cab = {"User-Agent": ua} if ua else {}
    for n in range(intentos):
        try:
            req = urllib.request.Request(url, headers=cab)
            with urllib.request.urlopen(req, timeout=10) as r:
                return r.read()
        except urllib.error.HTTPError as e:
            if e.code != 429 or n == intentos - 1:
                raise
            reintentos[e.code] += 1
            time.sleep(0.05 * 2 ** n)  # growing wait

try:
    get(f"{BASE}/submissions/CIK{CIK}.json", ua=None)
except urllib.error.HTTPError as e:
    print("without User-Agent:", e.code, e.read().decode())

sub = json.loads(get(f"{BASE}/submissions/CIK{CIK}.json"))
rec = pd.DataFrame(sub["filings"]["recent"])
print(rec["form"].value_counts().to_dict())
rec.head(3)

{'8-K': 31, '10-Q': 21, '10-K': 7, '10-Q/A': 3}
        accessionNumber   acceptanceDateTime  form items primaryDocument
0  9900000059-25-000062  2025-11-12T16:09:51  10-Q           d003973.htm
1  9900000059-25-000061  2025-11-03T07:34:37   8-K  2.02     d003934.htm
2  9900000059-25-000060  2025-07-30T16:57:37  10-Q           d003768.htm

With the filings table, downloading all the company's 10-Ks is a loop. When the server answers 429, the client waits and retries (three times in seven downloads); with the real one, the fixed wait between requests avoids getting there:

diez_k = rec[rec["form"] == "10-K"]
textos = {}
for acc, archivo in zip(diez_k["accessionNumber"],
                        diez_k["primaryDocument"]):
    url = (f"{BASE}/Archives/edgar/data/{int(CIK)}/"
           f"{acc.replace('-', '')}/{archivo}")
    textos[acc] = get(url).decode()
print(len(textos), "10-Ks downloaded;",
      sum(len(t) for t in textos.values()), "characters;",
      "retries after a 429:", reintentos[429])

7 10-Ks downloaded; 17485 characters; retries after a 429: 3

Acceptance time and filing date

acceptanceDateTime is the instant at which EDGAR accepted the submission; filingDate, the date on which it is deemed filed. They do not always match: under rule 13 of Regulation S-T, what is submitted after 17:30 Eastern time is deemed filed on the next business day (with exceptions up to 22:00 for some forms, such as Forms 3, 4 and 5)33Note 3 · 17 CFR 232.13(a)(2) (https://www.law.cornell.edu/cfr/text/17/232.13): what is submitted after 17:30 Eastern time “shall be deemed filed as of the next business day”; the exceptions up to 22:00, in (a)(3) and (a)(4).. The synthetic 10-Ks and 10-Qs are filed after the market close, between 16:00 and 18:30 New York time:

from pandas.tseries.holiday import USFederalHolidayCalendar

festivos = USFederalHolidayCalendar().holidays("2018", "2026")
periodicos = docs[docs["tipo"].str.match(r"10-[KQ]")].copy()
ny = periodicos["ts"].dt.tz_convert("America/New_York")
dia = ny.dt.tz_localize(None).dt.normalize()
tarde = ny.dt.hour * 60 + ny.dt.minute >= 17 * 60 + 30
siguiente = np.busday_offset(dia.to_numpy().astype("datetime64[D]"),
                             1, roll="forward",
                             holidays=festivos.to_numpy()
                             .astype("datetime64[D]"))
periodicos["filing_date"] = dia.where(~tarde, pd.to_datetime(siguiente))
print(f"accepted after 17:30: {tarde.mean():.1%}")
periodicos.loc[tarde, ["tipo", "ts", "filing_date"]].head(3)

accepted after 17:30: 28.9%
     tipo                        ts filing_date
1  10-Q/A 2019-01-02 22:58:24+00:00  2019-01-03
3  10-K/A 2019-01-08 23:06:37+00:00  2019-01-09
5  10-Q/A 2019-01-24 22:42:21+00:00  2019-01-25

29% of the periodic reports (10-K, 10-Q and their amendments) carry a filing date later than their acceptance. For an event study (lesson 6.1.3) that is a one-day delay, not look-ahead: whoever uses filingDate arrives late, but does not use future information. The dangerous error is the opposite one: dating with reportDate (the end of the period) or with the date on which a database added the document without knowing which one it is.

Full-text search

The EDGAR website has a full-text search of electronic filings since 200144Note 4 · SEC, “EDGAR Full Text Search FAQ” (https://www.sec.gov/edgar/search/efts-faq.html, accessed on October 23, 2026): search in the text of electronic filings since 2001. The page does not document the JSON service it uses.. The page uses it by calling a JSON service (efts.sec.gov/LATEST/search-index) that the SEC does not document as an API: it can be queried under the same access policy, knowing that it may change without notice. For a stable process, what is documented is the submissions API and the daily and quarterly EDGAR indexes.

import urllib.parse

q = urllib.parse.urlencode({"q": '"material weakness"',
                            "forms": "8-K",
                            "startdt": "2024-01-01",
                            "enddt": "2024-12-31"})
r = json.loads(get(f"https://efts.sec.gov/LATEST/search-index?{q}"))

CNMV: inside information and other relevant information

Spanish issuers report to the CNMV whatever may affect their share price. Until February 8, 2020 they did so as hechos relevantes (relevant facts); since then, in two separate registers: inside information (that of article 17 of the Market Abuse Regulation, MAR) and other relevant information (OIR). The CNMV keeps the register of relevant facts “hasta 08/02/2020” (up to February 8, 2020) next to the two new ones, and dissemination is now regulated by articles 226 and 227 of Law 6/2023 on Securities Markets (Ley 6/2023 de los Mercados de Valores)55Note 5 · CNMV, official registers of issuers (in Spanish): “Hechos relevantes de emisores de valores hasta 08/02/2020” (relevant facts of securities issuers up to February 8, 2020), “Información privilegiada” (inside information) and “Otra información relevante” (other relevant information) (https://www.cnmv.es/portal/otra-informacion-relevante/consulta-oir?lang=es); Law 6/2023, of March 17, on Securities Markets and Investment Services (Ley 6/2023, de 17 de marzo, de los Mercados de Valores y de los Servicios de Inversión), articles 226 and 227 (https://www.boe.es/buscar/act.php?id=BOE-A-2023-7053, in Spanish). Accessed on October 23, 2026.. For a historical series, the two series have to be joined.

The CNMV offers web search of each register, email alerts and RSS 2.0 feeds of inside information and of OIR66Note 6 · CNMV, RSS feeds (https://www.cnmv.es/portal/gpage?id=RSS, accessed on October 23, 2026): inside information, other relevant information, press releases and other content.; we have not found a documented API. Reading an RSS feed is a request and an XML parse:

import xml.etree.ElementTree as ET

url = ("https://www.cnmv.es/portal/Otra-Informacion-Relevante/"
       "RSS.asmx/GetNoticiasCNMV")
req = urllib.request.Request(url, headers={"User-Agent": UA})
with urllib.request.urlopen(req, timeout=30) as r:
    canal = ET.fromstring(r.read())
items = [{c.tag: c.text for c in item}
         for item in canal.iter("item")]

The fields of each item (title, link, date) are those of RSS 2.0; the specific ones of the CNMV feed have to be checked in the feed itself before depending on them. What matters is the date: an RSS feed collects what has been published at the time of the query, and the publication time has to be read from the element itself, not from the moment of the download.

The course's synthetic CNMV-type documents are Markdown, in Spanish, with a table of key figures, a footnote and the outlook in text. The regime change is in the data:

cnmv = docs[docs["formato"] == "markdown"]
print(cnmv.groupby("tipo")["ts"].agg(["count", "min", "max"]))
hora_madrid = cnmv["ts"].dt.tz_convert("Europe/Madrid")
print("Madrid time:", hora_madrid.dt.strftime("%H:%M").min(),
      "to", hora_madrid.dt.strftime("%H:%M").max())
ultimo = cnmv["texto"].iloc[-1]
print(ultimo.split("## Principales magnitudes")[1]
      .split("## Evolución")[0].strip())

      count                       min                       max
tipo
HR       60 2019-02-04 06:50:10+00:00 2019-11-18 06:58:48+00:00
OIR     360 2020-02-10 06:21:06+00:00 2025-11-14 06:53:55+00:00
Madrid time: 07:17 to 08:12
| (millones de USD) | Tercer trimestre 2025 |
|---|---|
| Ventas | 5.044,2 |
| Resultado neto¹ | 196,8 |
| Beneficio por acción (USD) | 8,28 |
| Patrimonio neto | 4.140,6 |

¹ Atribuido a la sociedad dominante.

All of them are published between 7:17 and 8:12 Madrid time: before the opening of the Spanish continuous market (mercado continuo, 9:00) and, all the more so, of the NYSE. The code cuts the last document between its “Principales magnitudes” (key figures) and “Evolución” (performance) sections; the table gives, in millions of USD, sales, net income, earnings per share and equity for the third quarter of 2025. The Markdown table is turned into a DataFrame by splitting on |, with the Spanish number convention (period for thousands, comma for decimals) and the footnote separated from the figure:

def tabla_markdown(texto):
    filas = [l.strip().strip("|").split("|")
             for l in texto.splitlines()
             if l.startswith("|") and not l.startswith("|---")]
    cab, *cuerpo = [[c.strip() for c in f] for f in filas]
    return pd.DataFrame(cuerpo, columns=cab)

def numero_es(x):
    x = re.sub(r"[¹²³]", "", x).replace(".", "").replace(",", ".")
    return float(x)

t = tabla_markdown(cnmv["texto"].iloc[-1])
t["valor"] = t.iloc[:, 1].map(numero_es)
t["nota"] = t.iloc[:, 0].str.extract(r"([¹²³])", expand=False)
t

            (millones de USD) Tercer trimestre 2025    valor nota
0                      Ventas               5.044,2  5044.20  NaN
1             Resultado neto¹                 196,8   196.80    ¹
2  Beneficio por acción (USD)                  8,28     8.28  NaN
3             Patrimonio neto               4.140,6  4140.60  NaN

SEC-type documents: from HTML to sections and tables

The course's SEC-type documents are HTML with <h2> sections (the Items), tables and <sup> notes. Real ones are much longer and messier (inline styles, nested tables, inline XBRL), but the technique is the same: BeautifulSoup to walk the tree and pandas.read_html for the tables. A results 8-K (Item 2.02) carries the press release as an exhibit:

from io import StringIO

from bs4 import BeautifulSoup

ocho_k = docs[(docs["tipo"] == "8-K") & (docs["items"] == "2.02")]
sopa = BeautifulSoup(ocho_k["texto"].iloc[0], "html.parser")
for h2 in sopa.find_all("h2"):
    print("·", h2.get_text())
print([li.get_text() for li in sopa.find_all("li")])
tabla = pd.read_html(StringIO(str(sopa.find("table"))))[0]
tabla

· Item 2.02 Results of Operations and Financial Condition
· Exhibit 99.1
['Revenue of $22,638 million', 'Net loss of $11.5 million, or $-0.01 per share', 'Stores at quarter end: 441.']
      (USD millions)  Q4 2018
0            Revenue  22637.9
1  Net income (loss)   (11.5)
2  Diluted EPS (USD)    -0.01

Item 2.02 is furnished and not filed for purposes of the liability of section 18 of the Exchange Act; the general deadline of an 8-K is four business days from the event77Note 7 · SEC, Form 8-K, general instructions B.1 and B.2 (https://www.sec.gov/files/form8-k.pdf): four-business-day deadline and the information of Item 2.02 “shall not be deemed to be "filed" for purposes of Section 18”.. For a quant, two practical details: the negative figure goes in parentheses ((11.5)), which read_html leaves as text, and the table figure (22,637.9 million) appears rounded in the list (22,638), so you have to decide which figure rules. Convert and check:

def numero_en(x):
    x = str(x).replace(",", "").replace("$", "").strip()
    negativo = x.startswith("(") and x.endswith(")")
    return -float(x.strip("()")) if negativo else float(x)

tabla["valor"] = tabla.iloc[:, 1].map(numero_en)
tabla

      (USD millions)  Q4 2018     valor
0            Revenue  22637.9  22637.90
1  Net income (loss)   (11.5)    -11.50
2  Diluted EPS (USD)    -0.01     -0.01

For text signals, the text by section matters more than the tables. The following function returns a dictionary that maps each section to its text in any document of the corpus; lesson 6.1.3 uses it with the risk section of the 10-Ks:

def secciones(html):
    sopa = BeautifulSoup(html, "html.parser")
    out = {}
    for h2 in sopa.find_all("h2"):
        partes = []
        for nodo in h2.find_next_siblings():
            if nodo.name == "h2":
                break
            partes.append(nodo.get_text(" ", strip=True))
        out[h2.get_text(strip=True)] = " ".join(partes)
    return out

s10k = secciones(next(iter(textos.values())))
{k: len(v.split()) for k, v in s10k.items()}

{'Item 1. Business': 14, 'Item 1A. Risk Factors': 223, 'Part II, Item 8. Financial Statements': 23, "Item 7. Management's Discussion and Analysis": 13}

XBRL facts: duplicates and the fourth quarter

Lesson 1.5.1 explained XBRL facts and their point-in-time version. Those of the corpus (filings6_xbrl.parquet) have two traps of the real ones. The first: a first-quarter 10-Q carries the quarter and the year to date, which are the same period, so the same fact appears twice:

clave = ["id_doc", "concepto", "periodo_inicio", "periodo_fin"]
print("rows:", len(xbrl),
      "| duplicates:", xbrl.duplicated(clave).sum())
print("duplicates with a different value:",
      xbrl.duplicated(clave).sum()
      - xbrl.duplicated(clave + ["valor"]).sum())
hechos = xbrl.drop_duplicates(clave)

rows: 13067 | duplicates: 900
duplicates with a different value: 0

The second: the 10-K gives the year and there is no fourth-quarter 10-Q. Fourth-quarter sales are computed as those of the year minus the nine-month year to date of the third 10-Q, with the facts of the original documents (without the amendments, which lesson 1.5.1 treats as later versions). It is checked against the quarterly sales of acciones6_ventas.csv:

ventas = hechos[hechos["concepto"] == "us-gaap:Revenues"].copy()
ventas["dias"] = (ventas["periodo_fin"]
                  - ventas["periodo_inicio"]).dt.days
anual = ventas[(ventas["forma"] == "10-K") & (ventas["dias"] > 360)]
nueve = ventas[(ventas["forma"] == "10-Q")
               & ventas["dias"].between(270, 275)]
nueve = nueve.assign(periodo_fin=nueve["periodo_fin"]
                     + pd.offsets.QuarterEnd(1))
q4 = anual.merge(nueve, on=["cik", "periodo_fin"],
                 suffixes=("_año", "_9m"))
q4["q4_musd"] = (q4["valor_año"] - q4["valor_9m"]) / 1e6

emisores = docs.drop_duplicates("emisor").set_index("emisor")
cik_tk = (xbrl.merge(docs[["id_doc", "emisor"]], on="id_doc")
          .drop_duplicates("cik").set_index("cik")["emisor"]
          .map(emisores["ticker"]))
q4["ticker"] = q4["cik"].map(cik_tk)
v = pd.read_csv("acciones6_ventas.csv", parse_dates=["fin_trimestre"])
q4 = q4.merge(v, left_on=["ticker", "periodo_fin"],
              right_on=["ticker", "fin_trimestre"])
error = (q4["q4_musd"] - q4["ventas"]).abs()
print(len(q4), "fourth quarters; maximum error",
      round(error.max(), 2), "million")

342 fourth quarters; maximum error 0.0 million

The 342 computed fourth quarters match the published sales exactly: the facts are rounded to 100,000 USD (decimales = −5), the same precision as the sales table in millions with one decimal. With real data you have to add the concept changes between years and fiscal years that do not end in December (lesson 1.5.1). The 6 companies that changed ticker require joining by the ticker in force on each date, or better by a permanent identifier (id_empresa, the CIK).

Conference call transcripts

A transcript is a list of turns with speaker, role and text. The structure matters more than the full text: the presentation is written by management and is almost always optimistic; the analysts' questions reflect what worries the market. The corpus has 840 calls from 30 companies:

llam = pd.read_parquet("llamadas6_transcripciones.parquet")
una = llam[llam["id_llamada"] == llam["id_llamada"].iloc[0]]
print(una[["turno", "seccion", "rol", "hablante"]]
      .head(7).to_string(index=False))
print(llam.groupby(["seccion", "rol"]).size())

 turno      seccion                   rol       hablante
     1 presentacion              operador      Operadora
     2 presentacion relaciones_inversores Alba Salbasgon
     3 presentacion             directivo   Alba Varlazu
     4 presentacion             directivo    Nils Bastel
     5    preguntas              operador      Operadora
     6    preguntas              analista Rocco Basbasmi
     7    preguntas             directivo    Nils Bastel
seccion       rol
preguntas     analista                 3790
              directivo                3790
              operador                 1680
presentacion  directivo                1680
              operador                  840
              relaciones_inversores     840
dtype: int64

The data values are in Spanish: the sections are presentacion (presentation) and preguntas (questions), and the roles operador (operator), relaciones_inversores (investor relations), directivo (executive) and analista (analyst). With the turns separated, the tone of management and that of the analysts are measured separately with the lexicon of lesson 6.1.1:

lex = pd.read_csv("texto6_lexico.csv").query("lista == 'financiera'")
listas = {i: (set(g.loc[g["categoria"] == "positiva", "palabra"]),
              set(g.loc[g["categoria"] == "negativa", "palabra"]))
          for i, g in lex.groupby("idioma")}

def tono(texto, idioma):
    pos, neg = listas[idioma]
    w = re.findall(r"[a-záéíóúüñ]+", texto.lower())
    p, n = sum(x in pos for x in w), sum(x in neg for x in w)
    return (p - n) / (p + n + 1)

grupo = np.where(llam["rol"] == "analista", "analistas",
                 np.where(llam["rol"] == "directivo", "direccion",
                          "otros"))
textos_ll = (llam.assign(grupo=grupo)
             .groupby(["id_llamada", "grupo", "idioma"])["texto"]
             .agg(" ".join).reset_index())
textos_ll["tono"] = [tono(t, i) for t, i in
                     zip(textos_ll["texto"], textos_ll["idioma"])]
t_ll = textos_ll.pivot(index="id_llamada", columns="grupo",
                       values="tono")
ver_ll = pd.read_csv("llamadas6_verdad.csv").set_index("id_llamada")
print(t_ll[["direccion", "analistas"]].mean().round(2))
print("correlation with the analysts' true tone:",
      round(t_ll["analistas"].corr(ver_ll["tono_analistas"]), 2))

grupo
direccion    0.75
analistas    0.01
dtype: float64
correlation with the analysts' true tone: 0.89

Management (direccion) sounds positive on average (0.75) and the analysts (analistas), almost neutral (0.01). The dictionary tone of the questions correlates highly with the true one (0.89) because the generator's questions reuse template phrases, most of them with lexicon words: with real transcripts you cannot expect that much and a model is advisable (lessons 6.1.1 and 6.2.3, where guidance and tone are extracted with an LLM).

Social media: cashtags, spam and bots

Social media posts are short, informal and noisy. The corpus imitates the typical problems: cashtags ($TICKER) to know which company they talk about, spam with several cashtags and a link, automated accounts that copy a text minutes later and pump campaigns (pushing up the price of a small company with coordinated posts). The truth (social6_verdad.parquet) says what each post is:

posts = pd.read_parquet("social6_posts.parquet")
clase = pd.read_parquet("social6_verdad.parquet")[["id_post", "clase"]]
posts = posts.merge(clase, on="id_post")
posts["cashtags"] = posts["texto"].str.findall(r"\$([A-Z]{3,5})\b")
posts["n_tags"] = posts["cashtags"].str.len()
print(posts["clase"].value_counts().to_dict())
print(f"no recognizable cashtag: {(posts['n_tags'] == 0).mean():.1%}")
posts.groupby("clase")[["n_tags", "seguidores_autor",
                        "antiguedad_cuenta_dias"]].median()

{'organica': 47720, 'bot_copia': 9956, 'spam': 2090, 'pump': 1931}
no recognizable cashtag: 0.0%
           n_tags  seguidores_autor  antiguedad_cuenta_dias
clase
bot_copia     1.0              29.0                   150.0
organica      1.0              97.0                  1763.0
pump          1.0              30.0                   483.0
spam          3.0              29.0                   188.5

All posts in the corpus carry at least one recognizable cashtag; in real data, those that do not cannot be assigned to a company without resolving names and nicknames (lesson 6.7.5) and are discarded for the per-company signal. The medians already separate the classes (organica is organic, bot_copia a bot copy): spam carries several cashtags, and the bots and the pump accounts have few followers and much younger accounts than the organic ones. Three simple rules, without looking at the truth:

posts = posts.sort_values("ts")
spam = (posts["n_tags"] >= 2) & posts["texto"].str.contains(
    r"\.example|link|canal|bot|VIP|gratis|free", case=False)
# copy: the same text was already published in the previous 2 hours
anterior = posts.groupby("texto")["ts"].shift()
copia = (posts["ts"] - anterior) < pd.Timedelta(hours=2)
posts["regla"] = np.select([spam, copia], ["spam", "copy"], "ok")
pd.crosstab(posts["clase"], posts["regla"])

regla      copy     ok  spam
clase
bot_copia  9956      0     0
organica      8  47712     0
pump         95   1836     0
spam          0      0  2090

The spam rule finds the 2,090 spam posts without touching any organic one. The copy rule finds the 9,956 bot copies, at a small cost: 8 organic posts that, by chance, repeat another person's sentence within two hours (on real social media, set phrases and the template texts of some apps do the same). Pump campaigns are barely detected by these rules (95 of 1,931 posts): their texts vary and carry no links. They are recognized as a whole (many new accounts that suddenly talk about a small company and abnormal volume), and they are better treated as a risk to the signal than as a signal: in the corpus, the four campaigns push the price up 48-58% and give it back within a few sessions (level 6 README).

Agency copies: one story, one event

In the news corpus, 12,266 of the 39,570 items are syndicated copies of another: the same story at another agency, from 2 minutes to 6 hours later, with the headline touched up (“ACTUALIZA:”, “Última hora:”, the Spanish for “UPDATE:” and “Breaking:”), the body trimmed or translated into the other language. Lesson 6.1.3 measures the damage of counting them as events. To group them without the truth, two keys: the normalized headline (without prefixes or upper case) and a figure fingerprint (the numbers of the text, normalized between the Spanish and the English convention), both with the company and a 6-hour window:

noticias = pd.read_parquet("noticias6_corpus.parquet").sort_values("ts")
PREFIJO = re.compile(r"^(ACTUALIZA|Última hora|UPDATE|Breaking):\s*")

def cifras(texto, idioma):
    out = set()
    for x in re.findall(r"\d[\d.,]*\d|\d", texto):
        if idioma == "es":
            x = x.replace(".", "").replace(",", ".")
        else:
            x = x.replace(",", "")
        out.add(round(float(x), 1))
    return tuple(sorted(out))

def agrupar(df, usar_cifras=True, ventana=pd.Timedelta(hours=6)):
    grupo, visto = np.arange(len(df)), {}
    for i, r in enumerate(df.itertuples()):
        claves = [("t", r.ticker, PREFIJO.sub("", r.titular).lower())]
        c = cifras(r.titular + " " + r.cuerpo, r.idioma)
        if usar_cifras and c:
            claves.append(("c", r.ticker, c))
        for k in claves:
            if k in visto and r.ts - visto[k][1] <= ventana:
                grupo[i] = visto[k][0]
                break
        for k in claves:
            visto[k] = (grupo[i], r.ts)
    return grupo

noticias["g_titular"] = agrupar(noticias, usar_cifras=False)
noticias["g_cifras"] = agrupar(noticias)
print(noticias[["g_titular", "g_cifras"]].nunique().to_dict())

{'g_titular': 30675, 'g_cifras': 27898}

How closely do those groups resemble the true stories? Two pairwise measures over news items: precision (of the pairs the method puts together, how many really are the same story) and recall (of the pairs that are the same story, how many the method puts together):

from collections import Counter

historia = pd.read_csv("noticias6_verdad.csv")[["id_noticia",
                                                 "id_historia"]]
noticias = noticias.merge(historia, on="id_noticia")

def pares(etiquetas):
    return sum(n * (n - 1) // 2 for n in Counter(etiquetas).values())

def medir(col):
    juntos = pares(zip(noticias[col], noticias["id_historia"]))
    return pd.Series({
        "grupos": noticias[col].nunique(),
        "precision": juntos / pares(noticias[col]),
        "exhaustividad": juntos / pares(noticias["id_historia"])})

print("true stories:", noticias["id_historia"].nunique())
metodos = ["g_titular", "g_cifras"]
pd.DataFrame({c: medir(c) for c in metodos}).T.round(3)

true stories: 27162
            grupos  precision  exhaustividad
g_titular  30675.0      0.999          0.653
g_cifras   27898.0      0.997          0.929

The normalized headline almost never joins different stories (precision of 99.9%), but it only finds two out of three pairs of copies (exhaustividad is the recall column): translations and rewritten headlines escape it. The figure fingerprint raises recall to 92.9% with a minimal loss of precision (99.7%) and leaves 27,898 groups against 27,162 true stories: those left out are translations without any figure in the text (an appointment, a shareholder meeting), which only share the company and the time. The header figure summarizes the result. For those, the tool is a multilingual embedding: two texts with the same meaning in two languages end up close together (which MiniLM, an English model, does not achieve; lesson 6.1.1).

Exercises

1. The social media copy rule also flags organic posts. Add a condition on the account (for example, age under one year) and measure again against the truth. What do you gain and what do you lose?

joven = posts["antiguedad_cuenta_dias"] < 365
posts["regla2"] = np.select([spam, copia & joven],
                            ["spam", "copy"], "ok")
pd.crosstab(posts["clase"], posts["regla2"])

regla2     copy     ok  spam
clase
bot_copia  5711   4245     0
organica      1  47719     0
pump         47   1884     0
spam          0      0  2090

2. Restatement 8-Ks (Item 4.02) announce that financial statements already published are not reliable. List those in the corpus with their New York time and say in which session the price could react.

r402 = docs[docs["items"] == "4.02"].copy()
r402["hora_ny"] = r402["ts"].dt.tz_convert("America/New_York")
print(len(r402), "restatements")
print(r402["hora_ny"].dt.strftime("%H:%M").agg(["min", "max"]))
r402[["ticker", "hora_ny"]].head(3)

120 restatements
min    06:30
max    07:53
Name: hora_ny, dtype: str
  ticker                   hora_ny
0   MURI 2019-01-02 07:43:04-05:00
2   ZEZU 2019-01-08 06:32:00-05:00
4   DITI 2019-01-24 07:18:08-05:00

3. Design a daily process that collects the 8-Ks of 500 companies from EDGAR. How many requests does it need, how long does it take while respecting the access policy and what would you store from each filing?