Fix missing women’s world records by removing brittle table indices (WorldRecords.ipynb - Chapter 9)

[AAdewunmi]

Hi Paul,

I ran into this: men’s records were fine, but women’s records were missing, while working on WorldRecords.ipynb - Chapter 9.

Root cause
The notebook uses hard-coded table indices:
RECORDS = (0, 2, 4, 5)

Wikipedia’s table order has changed, so those indices now point at the wrong tables (often relay-only). This caused women’s records to be incomplete/missing and also caused downstream KeyError/IndexError issues.

Fix (robust for table reordering in WorldRecords.ipynb - Chapter 9)
Instead of fixed indices, pick tables based on the nearby headings (“Long course / Short course” + “Men / Women”) and merge them (Code drop-in replacement for the records-building cell):

import re, html as html_lib

# Find headings like <h2>...</h2>, <h3>...</h3>, etc.
head_re = re.compile(r"<h([2-6])[^>]*>(.*?)</h\1>", re.DOTALL | re.IGNORECASE)

# Find Wikipedia "wikitable" tables.
table_re = re.compile(
    r'<table[^>]*class="[^"]*wikitable[^"]*"[^>]*>.*?</table>',
    re.DOTALL | re.IGNORECASE,
)

def strip_tags(text):
    """Remove HTML tags/entities and clean up whitespace."""
    text = re.sub(r"<[^>]+>", " ", text)
    text = html_lib.unescape(text)
    return " ".join(text.split())

# Store each heading with its position so we can find the nearest one before a table.
headings = [
    (m.start(), int(m.group(1)), strip_tags(m.group(2)))
    for m in head_re.finditer(html)
]

def last_heading_before(pos, level):
    """Return the most recent heading of a given level before a position."""
    prior = [h for h in headings if h[0] < pos and h[1] == level]
    return prior[-1][2] if prior else ""

COURSES = ("LC Men", "LC Women", "SC Men", "SC Women")
records = {course: {} for course in COURSES}

# For each table, decide whether it belongs to LC/SC and Men/Women
# based on the nearest h2 and h3 headings.
for match in table_re.finditer(html):
    pos = match.start()
    h2 = last_heading_before(pos, 2)
    h3 = last_heading_before(pos, 3)

    if "Long course" in h2:
        prefix = "LC"
    elif "Short course" in h2:
        prefix = "SC"
    else:
        continue

    if "Men" in h3:
        sex = "Men"
    elif "Women" in h3:
        sex = "Women"
    else:
        continue

    course_key = f"{prefix} {sex}"
    table = gazpacho.Soup(match.group(0))

    # Extract the first two columns: Event and Time.
    for row in table.find("tr", mode="all")[1:]:
        cols = row.find("td", mode="all")
        if len(cols) >= 2:
            event = cols[0].text.strip()
            time = cols[1].text.strip()
            records[course_key][event] = time

Then re-run the json.dump(records, jf) cell to refresh records.json

Validate
After re-running the notebook:

len(records["LC Women"]), len(records["SC Women"])
records["LC Women"].get("100m butterfly")

Defensive app fix (prevents crashes)
In swimclub.py (def produce_bar_chart()), replace with conditional append:

"""Load world records and collect times for each course, using 'N/A' if missing."""

with open(JSONDATA) as jf:
    records = json.load(jf)

# The four record categories we want to check.
COURSES = ("LC Men", "LC Women", "SC Men", "SC Women")

# Convert the filename into a records lookup key (e.g., "100m butterfly").
lookup = event_lookup(fn)

times = []
for course in COURSES:
    # Always append something so times has exactly 4 items.
    if lookup in records[course]:
        times.append(records[course][lookup])
    else:
        times.append("N/A")

Why This Works
Headings like “Long course (50m pool)” and “Women” are more stable than table positions, so scraping survives table reordering. It fixes a real breakage (women’s records missing) and replaces brittle table indices with heading-based selection, which should be much more resilient to Wikipedia table reordering over time.

Hope you find this useful.

Regards,

Adrian

[headfirstpython]

See post #18 for my thoughts on this and the next post. Happy to further discuss the issues raised in that thread.