Overview
Italian athletics results are public, and unusable. They live on a federation portal built to answer one question at a time: this event, this year, this category. Twenty years of them, across every event and every age group, exist only as thousands of separate pages — and a question like “what happens to fast sixteen-year-olds?” has nowhere to be asked.
This project builds the missing database. A two-level scraper indexes the ranking tables first, then visits each athlete's profile for their full competition history: every year from 2005 to 2026, outdoor, indoor and road, from U14 to senior. The result is 451,164 athletes, 3.3 million ranking entries and 9.25 million individual results, deduplicated and normalised — as far as I know the most complete historical archive of Italian competitive athletics assembled for research.
It started as the final project for the Web Scraping Laboratory exam. It is now the database behind the rest of my research: the cohort study with Foro Italico, the career clustering and the predictive models all read from it.
What I built
- A scraper in two steps: first the ranking tables, to find out who exists; then each athlete's profile page, for their full competition history. Profiles are fetched several at a time to keep it quick.
- A resumable pipeline over every combination of year × activity type × gender × category × event — 11,132 runs, each recorded, so an interruption costs minutes rather than days.
- Cleaning and normalisation of marks, wind, dates, categories and venues, plus the deduplication that turns repeated appearances into one athlete.
- A five-table SQLite schema for collection, exported to Parquet for analysis: athletes, rankings, results, runs, and the separate U14 regional pipeline.
- Ten analysis phases on top: demographics, the 1989–1995 cohort study, k-means career archetypes, a co-participation network (σ = 9.16, modularity 0.576), time-series motifs with the matrix profile, and a linear model that predicted the 2025 Italian 100 m final.
The data is public and used for research only, following the terms of the FIDAL portal. Rate limiting and retries are part of the scraper, not an afterthought: the point was to collect an archive without being a nuisance to the server that holds it.
Look someone up
50,652 athletes · 3,398,269 results
Type a surname and pick from the list — the archive is full of namesakes, which is exactly the problem the pipeline had to solve. Everyone who placed in the top 100 of a FIDAL ranking since 2016 is in here, with their whole career: the index arrives two letters at a time, the results one bucket at a time, so nothing heavy loads until you search.
Inside the pipeline
Read-only · outputs from main.ipynb and utils/*.py
Six cells from the scraper: the session that survives twenty years of requests, the two levels of scraping, what the database ended up holding, and what it looks like once you ask it something — including a question about my own results.
import time, requests
from bs4 import BeautifulSoup
BASE = "https://www.fidal.it"
GRADUATORIE_URL = f"{BASE}/graduatorie.php"
def make_session() -> requests.Session:
s = requests.Session()
s.headers.update({"User-Agent": "Mozilla/5.0"})
return s
def get_with_retry(session, url, *, timeout=30, retries=3, backoff=1.4):
"""GET with exponential backoff — the portal is slow before it is unfriendly."""
delay = 1.0
for attempt in range(retries):
try:
r = session.get(url, timeout=timeout)
r.raise_for_status()
return r.text
except Exception:
if attempt == retries - 1:
raise
time.sleep(delay)
delay *= backoff
session = make_session()
print("session ready ·", GRADUATORIE_URL)# Step 1 — one ranking page: men's U23 100 m, outdoor, 2025.
# FIDAL answers a query like this with the season's best mark per athlete.
params = {"anno": "2025", "tipo_attivita": "P", "sesso": "M",
"categoria": "PM", "gara": "03", "limite": "100"}
html = get_with_retry(session, f"{GRADUATORIE_URL}?{urlencode(params)}")
soup = BeautifulSoup(html, "html.parser")
ranking = []
for tr in soup.select("table tbody tr"):
td = [clean_text(c.get_text()) for c in tr.select("td")]
ranking.append({
"Rank": td[0], "Prest.": td[1], "Vento": td[2], "Nominativo": td[3],
"Anno": td[4], "Società": td[5], "Città": td[6], "Data": td[7],
"Nominativo_url": absolutize_url(tr.select_one("a")["href"]),
})
pd.DataFrame(ranking).to_csv("2025_P_M_PM_03_100_metri_ranking.csv", index=False)
print(pd.DataFrame(ranking).head(5).to_string(index=False))# Step 2 — follow every profile link and take the whole history.
# Eight profiles at a time; each thread keeps its own session.
def scrape_profile(url: str) -> list[dict]:
soup = BeautifulSoup(get_with_retry(get_thread_session(), url), "html.parser")
return parse_athlete_results(soup, athlete_url=url)
with ThreadPoolExecutor(max_workers=8) as ex:
races = [row for rows in ex.map(scrape_profile, ranking_urls) for row in rows]
df = pd.DataFrame(races)
df.to_csv("all_100_metri_races_athletes_2025_P_M_PM_03.csv", index=False)
print(f"{len(df)} races from {df.athlete.nunique()} athletes")
print(df.head(3)[["athlete", "date_iso", "città", "prestazione", "vento"]].to_string(index=False))# Step 3 — one athlete, everything. This is my own profile page.
url = "https://www.fidal.it/atleta/GALLI-Lorenzo/faAAARkpOfa2o%3D"
me = pd.DataFrame(scrape_profile(url))
print(f"results : {len(me)}")
print(f"events : {me.gara.nunique()}")
print(f"seasons : {me.date_iso.str[:4].min()}–{me.date_iso.str[:4].max()}")
print(me.gara.value_counts().head(5).to_string())# Season best per year, per event. Wind is left as scraped.
me["p"] = pd.to_numeric(me.prestazione, errors="coerce")
# FIDAL mislabels the odd row: a 25.20 "100 m" is a 200 m in disguise
LIMITS = {"100 metri": (9.5, 16), "60 piani": (6.3, 12), "Salto in lungo/LJ": (2.0, 9.0)}
me = drop_outside(me, LIMITS) # 1 row dropped
for gara in LIMITS:
best = me[me.gara == gara].groupby("year").p.agg("max" if "lungo" in gara else "min")
ax.plot(best.index, best.values, marker="o")
plt.show()# Step 4 — the same two steps, for every year × activity × gender ×
# category × event. Each combination is a "run", recorded once it is done,
# so an interrupted night resumes instead of restarting.
runs = pd.read_parquet("fidal_db_parquet/runs.parquet")
print(runs.status.value_counts().to_string())
athletes = pd.read_parquet("fidal_db_parquet/athletes.parquet")
rankings = pd.read_parquet("fidal_db_parquet/rankings.parquet")
results = pd.read_parquet("fidal_db_parquet/athlete_results.parquet")
for name, df in [("athletes", athletes), ("rankings", rankings), ("results", results)]:
print(f"{name:<10} {len(df):>12,} rows")per_year = (rankings.assign(year=rankings.date_iso.str[:4].astype(int))
.groupby("year")
.agg(rows=("athlete_id", "size"),
athletes=("athlete_id", "nunique")))
ax.bar(per_year.index, per_year.athletes / 1000, label="Unique athletes")
ax.plot(per_year.index, per_year.rows / 1000, marker="o", label="Ranking entries")
plt.show()