--- title: "0.4.4 Files, pathlib and time zones" description: "Read and write CSV, JSON, YAML and TOML with pathlib, and convert times between markets with zoneinfo: why New York opens at 14:30 Madrid time some weeks." url: "https://pythonparatrading.com/en/python-trading-course/level-0/python-files-pathlib-time-zones/" lang: "en" image: "https://pythonparatrading.com/og/en/python-trading-course/level-0/python-files-pathlib-time-zones.png" --- # 0.4.4 · Files (CSV, JSON, YAML, TOML) and dates with time zones > A trading system's data lives in files, and its times live in time zones. This lesson covers the four usual text formats with `pathlib` and the `zoneinfo` module, with which the New York open stops being "at 15:30" and becomes what it is: a local time that moves. - Course: [Level 0: Python from scratch and a data mindset](https://pythonparatrading.com/en/python-trading-course/level-0/) - Module: 0.4 · Intermediate Python - Duration: 50 min - Prerequisites: [0.4.3 · Iterators, generators, decorators and context managers](https://pythonparatrading.com/en/python-trading-course/level-0/python-iterators-generators-decorators/) ## By the end of this lesson - Build paths with `pathlib` and read and write CSV, JSON, YAML and TOML with the standard library and PyYAML. - Recognize the traps of each format: types that arrive as text, dates that JSON does not support and symbols that YAML turns into booleans. - Create time-zone-aware dates with `zoneinfo`, convert them between markets and store them in UTC. - Compute the weeks in which the time difference between New York and Madrid changes, and explain the times that do not exist or repeat. ## Paths with pathlib A trading system reads prices, configurations and trades, and writes reports, logs and backtest results. Before opening a file you need to know where it is, and paths written as text (`"datos/" + simbolo + ".csv"`) break easily: too many or too few slashes, different separators on Windows and Linux, duplicated extensions. `pathlib.Path` represents a path as an object. The `/` operator joins parts, and its attributes give the name, the extension or the folder without manipulating text: ```python from pathlib import Path for p in sorted(Path(".").glob("*.csv")): print(f"{p.name:<22} {p.stem:<16} {p.stat().st_size:>8,} B") informes = Path("informes") / "2025" / "sint" informes.mkdir(parents=True, exist_ok=True) destino = informes / "resumen.csv" print(destino, destino.suffix, destino.parent.exists()) ``` ```output cartera_sint.csv cartera_sint 49,207 B operaciones_sint.csv operaciones_sint 2,440 B sint_diario.csv sint_diario 60,757 B informes/2025/sint/resumen.csv .csv True ``` `glob("*.csv")` lists the files that match the pattern (`rglob` also searches subfolders). `mkdir` with `parents=True` creates the intermediate folders, and with `exist_ok=True` it does not fail if they already exist: a reporting script can run every day without checking anything first. Relative paths, such as `Path(".")` or `"sint_diario.csv"`, are resolved from the **working directory**, the place the program was launched from (lesson 0.2.4). A script that works in the terminal and fails in a task scheduler almost always has that problem. The usual solution is to build paths from the location of the script itself: `Path(__file__).parent / "datos"`. ## CSV: everything arrives as text CSV is the most common format for prices and trades: one row per record, columns separated by commas and a header. The standard library's `csv` module takes care of quotes and separators; `DictReader` delivers each row as a dictionary with the header's keys. What it does not do is convert types: **everything arrives as text**. ```python import csv with open("operaciones_sint.csv", newline="", encoding="utf-8") as f: filas = list(csv.DictReader(f)) print(len(filas), list(filas[0])) print(filas[0]["cantidad"], type(filas[0]["cantidad"]).__name__) def pnl(fila: dict[str, str]) -> float: signo = 1 if fila["lado"] == "largo" else -1 bruto = signo * int(fila["cantidad"]) * ( float(fila["precio_salida"]) - float(fila["precio_entrada"])) return bruto - float(fila["comision"]) por_simbolo: dict[str, float] = {} for fila in filas: s = fila["simbolo"] por_simbolo[s] = por_simbolo.get(s, 0.0) + pnl(fila) for s, total in sorted(por_simbolo.items()): print(f"{s:<5} {total:>10,.2f}") ``` ```output 40 ['id', 'fecha_entrada', 'fecha_salida', 'simbolo', 'lado', 'cantidad', 'precio_entrada', 'precio_salida', 'comision'] 200 str ORO -3,237.96 RVAR 17.09 SINT 4,405.08 TECH -35.02 ``` `"200"` is text: adding to it would raise an error and multiplying it would repeat the string (lesson 0.4.2). That is why every numeric field goes through `int()` or `float()` before any arithmetic. pandas (lesson 0.5.2) infers the types on its own, and even so it is worth checking them. Writing is symmetrical. `csv.writer` takes rows as lists and handles separators and quotes. The `newline=""` argument to `open` is not decorative: without it, on Windows each row would end with an extra line break. ```python with destino.open("w", newline="", encoding="utf-8") as f: w = csv.writer(f) w.writerow(["simbolo", "pnl"]) for s, total in sorted(por_simbolo.items()): w.writerow([s, round(total, 2)]) print(destino.read_text(encoding="utf-8")) ``` ```output simbolo,pnl ORO,-3237.96 RVAR,17.09 SINT,4405.08 TECH,-35.02 ``` A typical trap in Spain, and in many other European locales: CSV files exported from a spreadsheet set to Spanish use `;` as the separator and a decimal comma (`142,22`). `float("142,22")` raises `ValueError`. It is read with `csv.reader(f, delimiter=";")` and the comma is replaced by a period before converting, or with `pd.read_csv(sep=";", decimal=",")`. What you must never do is remove the comma: `14222` is a different price. ## JSON: no dates or decimals JSON is the format of APIs: almost all brokers and data providers respond in JSON. It supports objects (dictionaries), lists, text, numbers, `true`, `false` and `null`. Nothing else: no dates, no `Decimal`, no sets. ```python import json from datetime import date resumen = {"estrategia": "cruce-50-200", "fecha": date(2025, 12, 31), "operaciones": len(filas), "pnl": {s: round(v, 2) for s, v in por_simbolo.items()}} try: json.dumps(resumen) except TypeError as e: print("TypeError:", e) resumen["fecha"] = resumen["fecha"].isoformat() texto = json.dumps(resumen, ensure_ascii=False, indent=2) Path("resumen.json").write_text(texto, encoding="utf-8") leido = json.loads(Path("resumen.json").read_text(encoding="utf-8")) print(leido["fecha"], type(leido["fecha"]).__name__) print(leido == resumen) ``` ```output TypeError: Object of type date is not JSON serializable 2025-12-31 str True ``` The date is saved as ISO 8601 text (`2025-12-31`) and comes back as text: whoever reads the JSON has to know that the field is a date and convert it with `date.fromisoformat`. That is the work Pydantic does (lesson 0.4.2) when it validates an API response: the model says the field is a date and Pydantic converts it. Two more details. `ensure_ascii=False` keeps accented characters as they are instead of escapes such as `\u00f3`. And `json.dumps` writes `NaN` for a not-a-number float, which is not standard JSON: many readers, browsers included, reject it. With `allow_nan=False` the error is raised when writing, not when reading. ## YAML and TOML: configurations Configurations are written by hand, and JSON is awkward for that: it does not allow comments and requires quotes around everything. The two usual formats are YAML and TOML. YAML is read with PyYAML (lesson 0.4.2); TOML, with `tomllib`, part of the standard library since Python 3.11. YAML guesses types from the text, and sometimes it guesses wrong. PyYAML follows version 1.1 of the standard, where `yes`, `no`, `on` and `off` are booleans. There are companies listed under those symbols: ```python import yaml Path("config.yaml").write_text("""\ simbolos: [SPY, ON, NO, "ON"] inicio: 2021-01-04 version_modelo: 1.10 """, encoding="utf-8") print(yaml.safe_load(Path("config.yaml").read_text(encoding="utf-8"))) ``` ```output {'simbolos': ['SPY', True, False, 'ON'], 'inicio': datetime.date(2021, 1, 4), 'version_modelo': 1.1} ``` `ON` (onsemi, on the Nasdaq) has become `True` and `NO` `False`; only the quoted `"ON"` is still text. The unquoted date has become a `date`, and version `1.10` the number `1.1`. In a list of symbols the error raises no exception: the strategy simply does not trade those assets, or fails much later when requesting the price of `True`. That is why configurations are validated with a model (lesson 0.4.2) that requires `list[str]`. TOML requires quotes around all text, so it does not have that problem. It is the format of `pyproject.toml` (lesson 0.2.2). `tomllib.load` needs the file opened in binary mode (`"rb"`): ```python import tomllib Path("estrategia.toml").write_text("""\ [estrategia] nombre = "cruce" simbolos = ["SPY", "ON", "NO"] inicio = 2021-01-04 [costes] comision_pb = 1.5 deslizamiento_pb = 3.0 """, encoding="utf-8") with open("estrategia.toml", "rb") as f: cfg = tomllib.load(f) print(cfg["estrategia"]) print(sum(cfg["costes"].values())) ``` ```output {'nombre': 'cruce', 'simbolos': ['SPY', 'ON', 'NO'], 'inicio': datetime.date(2021, 1, 4)} 4.5 ``` The symbols are still text and the date is still a date. `tomllib` only reads: writing TOML requires an external package (`tomli-w`). That fits how it is used: a configuration is written by hand and the program reads it. | Format | Reading in Python | Comments | Types | Typical use | |---|---|---|---|---| | CSV | `csv` (stdlib), pandas | No | All text | Prices, trades | | JSON | `json` (stdlib) | No | Basic, no dates | APIs, results | | YAML | PyYAML (`safe_load`) | Yes | Guesses them (careful) | Configurations | | TOML | `tomllib` (stdlib, read-only) | Yes | Explicit, with dates | `pyproject.toml`, configurations | ## Dates with time zones: zoneinfo A `datetime` without a time zone (*naive*) is a clock time without a place: "March 9, 2026, 09:30" does not say whether it is in New York, Madrid or Tokyo. One with a zone (*aware*) does represent a specific instant. `zoneinfo`, part of the standard library since Python 3.9, provides the zones of the IANA database, with their full history of clock changes, by name: `America/New_York`, `Europe/Madrid`, `Asia/Tokyo`. The NYSE core session runs from 09:30 to 16:00, New York time.[^1] Converting the open to other zones is a call to `astimezone`: ```python from datetime import datetime, time, timedelta from zoneinfo import ZoneInfo NY = ZoneInfo("America/New_York") MAD = ZoneInfo("Europe/Madrid") UTC = ZoneInfo("UTC") for dia in (date(2026, 3, 2), date(2026, 3, 9), date(2026, 3, 30)): apertura = datetime.combine(dia, time(9, 30), NY) print(apertura.isoformat(), "→", apertura.astimezone(MAD).strftime("%H:%M Madrid"), "|", apertura.astimezone(UTC).strftime("%H:%M UTC")) ``` ```output 2026-03-02T09:30:00-05:00 → 15:30 Madrid | 14:30 UTC 2026-03-09T09:30:00-04:00 → 14:30 Madrid | 13:30 UTC 2026-03-30T09:30:00-04:00 → 15:30 Madrid | 13:30 UTC ``` The same open, at the same New York time, falls at 15:30 Madrid time on March 2, at 14:30 on the 9th and again at 15:30 on the 30th. The shift comes from the clock-change rules: the United States moves its clocks forward on the **second** Sunday of March and back on the first Sunday of November;[^2] the European Union does it on the **last** Sunday of March and the last Sunday of October.[^3] Between those dates, the difference between the two cities is 5 hours instead of 6. In 2026, on business days: ```python def diferencia(dia: date) -> timedelta: """Hours between Madrid and New York at the NYSE open.""" ap = datetime.combine(dia, time(9, 30), NY) return ap.astimezone(MAD).utcoffset() - ap.utcoffset() dias = [date(2026, 1, 1) + timedelta(days=i) for i in range(365)] habiles = [d for d in dias if d.weekday() < 5] raros = [d for d in habiles if diferencia(d) != timedelta(hours=6)] print(len(raros), "business days with", diferencia(raros[0])) tramos = [[raros[0]]] for ayer, hoy in zip(raros, raros[1:]): if (hoy - ayer).days > 3: tramos.append([]) tramos[-1].append(hoy) for t in tramos: print(f" from {t[0]} to {t[-1]}: {len(t)} days") ``` ```output 20 business days with 5:00:00 from 2026-03-09 to 2026-03-27: 15 days from 2026-10-26 to 2026-10-30: 5 days ``` Twenty business days in 2026 on which New York opens an hour earlier for someone trading from Madrid: from March 9 to 27 and from October 26 to 30. The header figure plots every Monday of 2026 with the Madrid time of the open; the four highlighted weeks are the shifted ones. The consequences are practical. A program that sends orders "at 15:25 Madrid time, five minutes before the open" arrives an hour late in those weeks. A filter that discards "the first half hour of New York" using fixed Madrid times is applied at the wrong time. And a backtest with intraday bars in Madrid local time mixes two session schedules without anyone noticing. The solution is always the same: express each market's hours in its own zone and convert. ## Times that do not exist and times that repeat The clock change creates two anomalies in local time. In Madrid, on March 29, 2026 the clock jumps from 02:00 to 03:00: 02:30 on that day **does not exist**. On October 25, it goes from 03:00 back to 02:00: 02:30 happens **twice**. `zoneinfo` raises no errors; the `fold` attribute (0 or 1) chooses between the two occurrences of a repeated time: ```python inexistente = datetime(2026, 3, 29, 2, 30, tzinfo=MAD) print(inexistente.astimezone(UTC).astimezone(MAD)) for fold in (0, 1): repetida = datetime(2026, 10, 25, 2, 30, tzinfo=MAD, fold=fold) print(fold, repetida.astimezone(UTC).strftime("%H:%M UTC"), repetida.tzname()) try: print(datetime(2026, 3, 9, 15, 0) < datetime.now(UTC)) except TypeError as e: print("TypeError:", e) ``` ```output 2026-03-29 03:30:00+02:00 0 00:30 UTC CEST 1 01:30 UTC CET TypeError: can't compare offset-naive and offset-aware datetimes ``` The nonexistent time, after a round trip through UTC, becomes 03:30 summer time. The repeated one has two different instants, one hour apart: 00:30 and 01:30 UTC. And Python refuses to compare a date without a zone with one that has a zone, rather than assume anything. Markets almost never trade at those hours, but data does get generated: cryptocurrency files, which trade 24 hours a day, or a server's logs in local time, have a duplicated hour every fall and a gap every spring. The rules that avoid almost all problems: - **Store in UTC.** Timestamps of data, orders and logs go in UTC, which has no clock changes. - **Convert for display.** Local time is for people: reports, charts, alerts. - **Define market hours in their own zone.** "09:30 `America/New_York`", not "15:30 Madrid time". - **Never mix naive and aware.** If a file comes with dates without a zone, decide when reading it which zone they are in and convert them. The gap between closes is computed as before, each one in its own zone: ```python dia = date(2025, 12, 31) cierre_bme = datetime.combine(dia, time(17, 30), MAD) cierre_nyse = datetime.combine(dia, time(16, 0), NY) print(cierre_bme.astimezone(UTC).strftime("BME %H:%M UTC")) print(cierre_nyse.astimezone(UTC).strftime("NYSE %H:%M UTC")) print("NYSE closes", cierre_nyse - cierre_bme, "later") ``` ```output BME 16:30 UTC NYSE 21:00 UTC NYSE closes 4:30:00 later ``` Subtracting aware dates gives the real duration, because Python converts them to UTC before subtracting. Subtracting two naive local times on clock-change days would give an hour too many or too few. [^1]: NYSE, "Holidays & Trading Hours": *Core Trading Session: 9:30 a.m. to 4:00 p.m. ET*. Link in "Resources". [^2]: NIST, "Daylight Saving Time Rules": daylight saving time begins on the second Sunday of March at 2:00 a.m. and ends on the first Sunday of November at 2:00 a.m., local time. Link in "Resources". [^3]: Directive 2000/84/EC, transposed into Spanish law by Royal Decree 236/2002: the last Sunday of March and the last Sunday of October, at 1:00 UTC. Link in "Resources". [^4]: BME Exchange, "Horario Bolsa Española" (Spanish stock exchange trading hours): open market from 9:00 to 17:30 and closing auction from 17:30 to 17:35 (https://www.bolsasymercados.es/es/bme-exchange/horario-del-mercado.html). ## Exercises 1. Compute at what Madrid time the Tokyo Stock Exchange opens (09:00, `Asia/Tokyo`) on March 2 and March 30, 2026. Why does it change, if Japan has no daylight saving time? **Solution.** ```python TOK = ZoneInfo("Asia/Tokyo") for dia in (date(2026, 3, 2), date(2026, 3, 30)): ap = datetime.combine(dia, time(9, 0), TOK) print(dia, ap.astimezone(MAD).strftime("%H:%M Madrid (%Z)")) ``` ```output 2026-03-02 01:00 Madrid (CET) 2026-03-30 02:00 Madrid (CEST) ``` Japan does not change its clocks; Madrid does. In winter (CET, UTC+1) Tokyo is 8 hours ahead; in summer (CEST, UTC+2), 7. A market's schedule moves as seen from outside even though the market changes nothing. 2. Read `operaciones_sint.csv` and write to `largas.json` only the long SINT trades, with `cantidad` as an integer, the prices as floats and the dates as ISO text. Check how many there are by reading the JSON. **Solution.** ```python largas = [ {"entrada": f["fecha_entrada"], "salida": f["fecha_salida"], "cantidad": int(f["cantidad"]), "precio_entrada": float(f["precio_entrada"]), "precio_salida": float(f["precio_salida"])} for f in filas if f["simbolo"] == "SINT" and f["lado"] == "largo" ] Path("largas.json").write_text(json.dumps(largas, indent=2), encoding="utf-8") datos = json.loads(Path("largas.json").read_text(encoding="utf-8")) print(len(datos), datos[0]) ``` ```output 24 {'entrada': '2021-09-02', 'salida': '2021-10-12', 'cantidad': 180, 'precio_entrada': 108.64, 'precio_salida': 108.79} ``` The dates already arrive from the CSV as ISO text, so they are copied as they are. Quantities and prices are converted before writing: the JSON stores numbers, and whoever reads it will not have to convert them. 3. Write a function `en_sesion(instante, zona, abre, cierra)` that says whether an aware `datetime` falls within a market's hours expressed in its own zone. Check 15:00 Madrid time on March 16 and March 30, 2026 against the NYSE. **Solution.** ```python def en_sesion(instante: datetime, zona: ZoneInfo, abre: time, cierra: time) -> bool: local = instante.astimezone(zona) return local.weekday() < 5 and abre <= local.time() < cierra for dia in (16, 30): t = datetime(2026, 3, dia, 15, 0, tzinfo=MAD) print(t.date(), en_sesion(t, NY, time(9, 30), time(16, 0))) ``` ```output 2026-03-16 True 2026-03-30 False ``` At 15:00 Madrid time on March 16 the NYSE has already been open for half an hour (it is 10:00 in New York); on March 30 it has not opened yet (it is 09:00). The function knows nothing about holidays: a complete market calendar (level 1) adds the list of days without a session. ## Self-assessment 1. A YAML file contains `simbolos: [SPY, ON, NO]`. What does `yaml.safe_load` return? - A. `['SPY', 'ON', 'NO']` - B. A syntax error - C. `['SPY', None, None]` - D. `['SPY', True, False]` *Answer D: `['SPY', True, False]`.* PyYAML follows YAML 1.1, where `on`, `off`, `yes` and `no` (in any combination of upper and lower case) are booleans. Symbols that match those words must be put in quotes. 2. At what Madrid time does the NYSE open on Monday, March 16, 2026? - A. 15:30, as always - B. 14:30, because the United States has already moved its clocks forward and Europe has not yet - C. 16:30 - D. 09:30 *Answer B: 14:30, because the United States has already moved its clocks forward and Europe has not yet.* The United States changes its clocks on the second Sunday of March (March 8, 2026) and the EU on the last one (March 29). Between the two dates, the difference is 5 hours instead of 6. 3. Why is it a good idea to store timestamps in UTC? - A. Because it takes up less space - B. Because markets open in UTC - C. Because UTC has no clock changes: each instant has a single representation and differences between timestamps are always correct - D. Because Python does not support other zones *Answer C: Because UTC has no clock changes: each instant has a single representation and differences between timestamps are always correct.* In local time there are times that do not exist (the spring-forward change) and times that repeat (the fall-back change). In UTC that does not happen: store in UTC and convert to local time only for display. 4. You compare `datetime(2026, 3, 9, 15, 0)` (no zone) with an open created with `tzinfo=ZoneInfo("America/New_York")`. What happens? - A. `TypeError`: dates with and without a time zone cannot be compared - B. Python assumes the first one is in UTC - C. Python assumes the first one is in the computer's local time - D. It always returns `False` *Answer A: `TypeError`: dates with and without a time zone cannot be compared.* A date without a zone (naive) does not represent a specific instant. Python refuses to compare it with one that has a zone (aware) rather than guess. ## Key points - `pathlib.Path` builds paths with `/`, lists files with `glob` and reads or writes text in one line. Relative paths depend on the working directory. - CSV delivers text: every column has to be converted. JSON does not support dates or `Decimal`: they are serialized as ISO text. - YAML is convenient and treacherous: `ON` and `NO` are booleans and `1.10` is the number 1.1. TOML requires quotes around text and does not have that problem; `tomllib` reads it but does not write it. - The United States and the EU change their clocks on different dates: in 2026, the NYSE opens at 14:30 Madrid time from March 9 to 27 and from October 26 to 30. - Store in UTC, convert for display and never mix dates with and without a zone. The closes of two markets on the same date do not happen at the same time. ## Resources ### Documentation - [Python 3.14 documentation: zoneinfo](https://docs.python.org/3.14/library/zoneinfo.html) · docs.python.org - [Python 3.14 documentation: pathlib](https://docs.python.org/3.14/library/pathlib.html) · docs.python.org - [NYSE: trading hours and holidays](https://www.nyse.com/markets/hours-calendars) · nyse.com - [Royal Decree 236/2002 establishing summer time in Spain (BOE)](https://www.boe.es/eli/es/rd/2002/03/01/236) · boe.es - [NIST: daylight saving time rules in the United States](https://www.nist.gov/pml/time-and-frequency-division/popular-links/daylight-saving-time-dst) · nist.gov ### Data - [Level 0 data: trades of the fictitious ETF SINT (CSV)](https://pythonparatrading.com/curso/datos/nivel-0/operaciones_sint.csv) · Download · operaciones_sint.csv Previous lesson: [0.4.3 · Iterators, generators, decorators and context managers](https://pythonparatrading.com/en/python-trading-course/level-0/python-iterators-generators-decorators/) Next lesson: [0.4.5 · Professional logging and testing with pytest](https://pythonparatrading.com/en/python-trading-course/level-0/python-logging-and-testing-with-pytest/)