Refactor update_records.py to Avoid Missing Women’s Records (update_records.py - Chapter 9)

[AAdewunmi]

Hi Paul,

Based on my previous post #17, I updated my version of update_records.py (Chapter 9) by:

1. Removing brittle fixed table indices (they can change on Wikipedia).
2. Selecting tables using nearby headings (“Long course / Short course” + “Men / Women”).
3. Writing records.json using a local, repo-relative path (file) instead of a hard-coded Linux path.
4. Keeping the “ignore relay events” filter.

This fixed the “missing women’s world records” problem for me and made the script more robust.

See the commented refactored update_records.py below:

"""
Update `records.json` by scraping world records from Wikipedia.

This version avoids brittle fixed table indices by selecting tables
based on nearby headings: Long/Short course + Men/Women.
"""

import os
import re
import json
import gazpacho
import html as html_lib

URL = "https://en.wikipedia.org/wiki/List_of_world_records_in_swimming"
COURSES = ("LC Men", "LC Women", "SC Men", "SC Women")
JSONDATA = "records.json"
BASE_DIR = os.path.dirname(__file__)

# Download the page HTML once.
html = gazpacho.get(URL)

# Regex to find headings (h2/h3/etc) and wikitable blocks.
head_re = re.compile(r"<h([2-6])[^>]*>(.*?)</h\1>", re.DOTALL | re.IGNORECASE)
table_re = re.compile(
    r'<table[^>]*class="[^"]*wikitable[^"]*"[^>]*>.*?</table>',
    re.DOTALL | re.IGNORECASE,
)


def strip_tags(text: str) -> str:
    """Remove HTML tags/entities and normalize whitespace."""
    text = re.sub(r"<[^>]+>", " ", text)
    text = html_lib.unescape(text)
    return " ".join(text.split())


# Capture heading positions so we can find the nearest heading before each table.
headings = [
    (m.start(), int(m.group(1)), strip_tags(m.group(2)))
    for m in head_re.finditer(html)
]


def last_heading_before(pos: int, level: int) -> str:
    """Return the most recent heading of a given level before a position."""
    prior = [h for h in headings if h[0] < pos and h[1] == level]
    return prior[-1][2] if prior else ""


# Initialize the records dictionary.
records = {course: {} for course in COURSES}

# Walk every wikitable and decide which course/sex it belongs to.
for match in table_re.finditer(html):
    pos = match.start()
    table_html = match.group(0)

    h2 = last_heading_before(pos, level=2)
    h3 = last_heading_before(pos, level=3)

    # Determine LC vs SC from the nearest h2.
    if "Long course" in h2:
        prefix = "LC"
    elif "Short course" in h2:
        prefix = "SC"
    else:
        continue

    # Determine Men vs Women from the nearest h3.
    if "Men" in h3:
        sex = "Men"
    elif "Women" in h3:
        sex = "Women"
    else:
        continue

    course_key = f"{prefix} {sex}"
    table = gazpacho.Soup(table_html)

    # Extract Event + Time columns, skipping relays.
    for row in table.find("tr", mode="all")[1:]:
        cols = row.find("td", mode="all")
        if len(cols) < 2:
            continue
        event = cols[0].text.strip()
        time = cols[1].text.strip()
        if event and "relay" not in event.lower():
            records[course_key][event] = time

# Save next to this script (portable across OSes).
with open(os.path.join(BASE_DIR, JSONDATA), "w") as jf:
    json.dump(records, jf)

Validation

python - <<'PY'
import json
with open("9_Chapter/webapp/records.json") as f:
    records = json.load(f)
print(len(records["LC Women"]), len(records["SC Women"]))
PY

0 0

python - <<'PY'
import json
with open("9_Chapter/webapp/records.json") as f:
    records = json.load(f)
print(len(records["LC Women"]), len(records["SC Women"]))
PY

17 18

Hope you find useful.

Regards,

Adrian

[headfirstpython]

Hi Adrian.

Before I get going, let me begin by saying that I really appreciate the time you are putting into all of this, and for sharing your code with me (and the world).

There is an old joke which goes something like this: "You have a problem, and you've decided to fix it with a regular expression... You now have two problems!" 😉

To be clear... as an old Perl coder, I love regular expressions. But they do not make much of an appearance in Head First Python as the goal of my book is to teach Python, not to teach regular expressions (although I can appreciate what you are doing here with yours).

The goal of the web scraping code in the book is to demonstrate what's possible at the expense of producing code with a level of robustness which rivals that as written by a professional coder. With any webpage (but especially with Wikipedia), the content and structure of the target webpage can change often and sometimes quite dramatically. Rather than try to cover every eventuality, the goal was always to get the concept across with code which should work most of the time. Obviously, if those index values change due to Wikipedia edits, my code breaks, but it's a simple fix to resolve the issue and move on: simply change the index values.

I would think your code breaks just as quickly if someone messes with the headings being looked for. I'd also question whether the complexity you've added to the code is worth the perceived robustness that is provided (when a simple edit to the index values was all that's needed here for the original code to work again). Pages 372 through 378 in Chapter 8 somewhat discuss this issue by concentrating on the size of the tables on the page (rather than a table's relative position to anything else).

My goal is always to keep things as simple as possible with an eye on future maintenance of my code. As I said, I can appreciate what you are doing, but I'd caution that the added complexity of your code may not be worth it.

I'd love to hear your thoughts on this.

Regards.

--Paul.

[AAdewunmi]

Hi Paul,

Thanks for the thoughtful reply, and for the context on what you’re optimising for in the book. The “two problems” regex joke definitely landed 😎

I agree with the core goal you stated: teach the concept, keep the code approachable, and accept that Wikipedia is a moving target. I also agree that “change the index values and move on” is often the right call for a learner who just wants to keep progressing.

Where I was coming from is slightly different, and it’s mostly about the learner experience when it breaks.

• When the indices drift, the failure mode isn’t always obvious. In my case it looked like “men works, women doesn’t”, and then the app falls over later with KeyError/IndexError. For someone new, it’s hard to see that the root cause is “wrong tables selected”.
• So I tried to make the table selection fail in a way that’s more semantically tied to the page structure, even if it’s a bit more code.

On headings vs indices, I think you’re right in principle: both are brittle. I don’t think headings are magically stable. My (practical) bet was just that Wikipedia editors are more likely to reorder or insert tables than they are to rename “Long course” / “Short course” or swap the Men/Women section headings. If headings change, my approach breaks too, but it should break “loudly” in a way that points at what to update.

On complexity, I hear you. Dropping regex into Head First Python is a lot, and it’s not aligned with the teaching objective. I wasn’t suggesting it needs to go into the book verbatim. I mainly wanted to (a) document the root cause for anyone hitting the same issue, and (b) offer an alternative for people who want a slightly more resilient version, even if it sits outside the main learning path.

If it helps, there’s probably a middle ground that stays “Head First simple” and avoids regex entirely:

• keep the idea of selecting tables based on a lightweight heuristic (like you mention with table sizes), or
• keep indices but add one small guardrail so the notebook/app fails with a clearer message when the expected events aren’t present (e.g., assert that “100m freestyle” exists in each target dict, otherwise print a hint that Wikipedia table order changed and the indices need updating).

That would preserve the teaching intent and still reduce the “mysterious missing women’s records” problem.

Happy to align with whatever approach you prefer in the repo. If you want, I can propose a minimal “defensive check” patch that avoids regex and keeps the narrative of the chapter intact, while making the failure case easier for readers to diagnose.

Regards,

Adrian

[headfirstpython]

We are on the same page here. 😎 I'm planning to tweak some of the code this afternoon, then push the updates to GitHUB. I'll come up with a defensive check to warn when the indices need to change. Thanks, again, for your work here and your considered discussion. It is all very much appreciated.

--Paul.

[AAdewunmi]

Hi Paul,

Sounds good, and that approach makes a lot of sense for the book.
Thanks again for taking it on so quickly, and for being open to the discussion.

Regards,

Adrian