Chapter 9½ pd.read_html(URL) gives HTTPError

[Jelzap0]

Hi,
In chapter 9½ page 430, when doing

import pandas as pd

URL = "https://en.wikipedia.org/wiki/List_of_world_records_in_swimming"

tables = pd.read_html(URL)

It gives HTTPError: HTTP Error 403: Forbidden.

Here the complete traceback:

---------------------------------------------------------------------------
HTTPError                                 Traceback (most recent call last)
Cell In[2], line 1
----> 1 tables = pd.read_html(URL)

File ~/Learning/head-first-python/webapp/.venv/lib/python3.13/site-packages/pandas/io/html.py:1226, in read_html(io, match, flavor, header, index_col, skiprows, attrs, parse_dates, thousands, encoding, decimal, converters, na_values, keep_default_na, displayed_only, extract_links, dtype_backend, storage_options)
   1222 check_dtype_backend(dtype_backend)
   1224 io = stringify_path(io)
-> 1226 return _parse(
   1227     flavor=flavor,
   1228     io=io,
   1229     match=match,
   1230     header=header,
   1231     index_col=index_col,
   1232     skiprows=skiprows,
   1233     parse_dates=parse_dates,
   1234     thousands=thousands,
   1235     attrs=attrs,
   1236     encoding=encoding,
   1237     decimal=decimal,
   1238     converters=converters,
   1239     na_values=na_values,
   1240     keep_default_na=keep_default_na,
   1241     displayed_only=displayed_only,
   1242     extract_links=extract_links,
   1243     dtype_backend=dtype_backend,
   1244     storage_options=storage_options,
   1245 )

File ~/Learning/head-first-python/webapp/.venv/lib/python3.13/site-packages/pandas/io/html.py:979, in _parse(flavor, io, match, attrs, encoding, displayed_only, extract_links, storage_options, **kwargs)
    968 p = parser(
    969     io,
    970     compiled_match,
   (...)    975     storage_options,
    976 )
    978 try:
--> 979     tables = p.parse_tables()
    980 except ValueError as caught:
    981     # if `io` is an io-like object, check if it's seekable
    982     # and try to rewind it before trying the next parser
    983     if hasattr(io, "seekable") and io.seekable():

File ~/Learning/head-first-python/webapp/.venv/lib/python3.13/site-packages/pandas/io/html.py:237, in _HtmlFrameParser.parse_tables(self)
    229 def parse_tables(self):
    230     """
    231     Parse and return all tables from the DOM.
    232 
   (...)    235     list of parsed (header, body, footer) tuples from tables.
    236     """
--> 237     tables = self._parse_tables(self._build_doc(), self.match, self.attrs)
    238     return (self._parse_thead_tbody_tfoot(table) for table in tables)

File ~/Learning/head-first-python/webapp/.venv/lib/python3.13/site-packages/pandas/io/html.py:789, in _LxmlFrameParser._build_doc(self)
    786 parser = HTMLParser(recover=True, encoding=self.encoding)
    788 if is_url(self.io):
--> 789     with get_handle(self.io, "r", storage_options=self.storage_options) as f:
    790         r = parse(f.handle, parser=parser)
    791 else:
    792     # try to parse the input in the simplest way

File ~/Learning/head-first-python/webapp/.venv/lib/python3.13/site-packages/pandas/io/common.py:772, in get_handle(path_or_buf, mode, encoding, compression, memory_map, is_text, errors, storage_options)
    769     codecs.lookup_error(errors)
    771 # open URLs
--> 772 ioargs = _get_filepath_or_buffer(
    773     path_or_buf,
    774     encoding=encoding,
    775     compression=compression,
    776     mode=mode,
    777     storage_options=storage_options,
    778 )
    780 handle = ioargs.filepath_or_buffer
    781 handles: list[BaseBuffer]

File ~/Learning/head-first-python/webapp/.venv/lib/python3.13/site-packages/pandas/io/common.py:404, in _get_filepath_or_buffer(filepath_or_buffer, encoding, compression, mode, storage_options)
    402 # assuming storage_options is to be interpreted as headers
    403 req_info = urllib.request.Request(filepath_or_buffer, headers=storage_options)
--> 404 with urlopen(req_info) as req:
    405     content_encoding = req.headers.get("Content-Encoding", None)
    406     if content_encoding == "gzip":
    407         # Override compression based on Content-Encoding header

File ~/Learning/head-first-python/webapp/.venv/lib/python3.13/site-packages/pandas/io/common.py:281, in urlopen(*args, **kwargs)
    275 """
    276 Lazy-import wrapper for stdlib urlopen, as that imports a big chunk of
    277 the stdlib.
    278 """
    279 import urllib.request
--> 281 return urllib.request.urlopen(*args, **kwargs)

File ~/.local/share/uv/python/cpython-3.13.11-linux-x86_64-gnu/lib/python3.13/urllib/request.py:189, in urlopen(url, data, timeout, context)
    187 else:
    188     opener = _opener
--> 189 return opener.open(url, data, timeout)

File ~/.local/share/uv/python/cpython-3.13.11-linux-x86_64-gnu/lib/python3.13/urllib/request.py:495, in OpenerDirector.open(self, fullurl, data, timeout)
    493 for processor in self.process_response.get(protocol, []):
    494     meth = getattr(processor, meth_name)
--> 495     response = meth(req, response)
    497 return response

File ~/.local/share/uv/python/cpython-3.13.11-linux-x86_64-gnu/lib/python3.13/urllib/request.py:604, in HTTPErrorProcessor.http_response(self, request, response)
    601 # According to RFC 2616, "2xx" code indicates that the client's
    602 # request was successfully received, understood, and accepted.
    603 if not (200 <= code < 300):
--> 604     response = self.parent.error(
    605         'http', request, response, code, msg, hdrs)
    607 return response

File ~/.local/share/uv/python/cpython-3.13.11-linux-x86_64-gnu/lib/python3.13/urllib/request.py:533, in OpenerDirector.error(self, proto, *args)
    531 if http_err:
    532     args = (dict, 'default', 'http_error_default') + orig_args
--> 533     return self._call_chain(*args)

File ~/.local/share/uv/python/cpython-3.13.11-linux-x86_64-gnu/lib/python3.13/urllib/request.py:466, in OpenerDirector._call_chain(self, chain, kind, meth_name, *args)
    464 for handler in handlers:
    465     func = getattr(handler, meth_name)
--> 466     result = func(*args)
    467     if result is not None:
    468         return result

File ~/.local/share/uv/python/cpython-3.13.11-linux-x86_64-gnu/lib/python3.13/urllib/request.py:613, in HTTPDefaultErrorHandler.http_error_default(self, req, fp, code, msg, hdrs)
    612 def http_error_default(self, req, fp, code, msg, hdrs):
--> 613     raise HTTPError(req.full_url, code, msg, hdrs, fp)

HTTPError: HTTP Error 403: Forbidden

[AAdewunmi]

Hi @Jelzap0,

I ran into the same HTTPError: HTTP Error 403: Forbidden with pd.read_html(URL) (Wikipedia blocks pandas’ default urllib user-agent).

Fix: fetch the page with requests using a browser-like header, then pass the HTML to read_html:

import io
import requests
import certifi
import pandas as pd

URL = "https://en.wikipedia.org/wiki/List_of_world_records_in_swimming"

headers = {
    "User-Agent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/124.0.0.0 Safari/537.36",
    "Accept-Language": "en-US,en;q=0.9",
}

response = requests.get(URL, headers=headers, verify=certifi.where(), timeout=30)
response.raise_for_status()

tables = pd.read_html(io.StringIO(response.text))

Optional (only if you see SSL certificate errors):

import os, ssl, certifi
cafile = certifi.where()
os.environ["SSL_CERT_FILE"] = cafile
ssl._create_default_https_context = lambda: ssl.create_default_context(cafile=cafile)

Tip: if you’re in a notebook, restart the kernel and run from the top after changes.

Happy coding 😎

[headfirstpython]

There's maybe a bit too much code in that solution, and I feel using requests (although fine) is not something I would use here especially when we have gazpacho at our disposal.

I'd replace this line in the notebook:

tables = pd.read_html(URL)

with these lines of code (which effectively tricks Wikipedia into responding as Wikipedia believes it is talking to a recent Chrome browser):

import gazpacho
import io

AGENT = {
    "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/144.0.0.0 Safari/537.36",
}

html = gazpacho.get(URL, headers=AGENT)
tables = pd.read_html(io.StringIO(html))

As stated above, Wikipedia is trying to discourage web scraping with Python (pandas), which is a pity as I feel the usage discussed in the book is valid and reasonable. That said, I guess it's their site and Wikipedia can run it how they like.

Note that pandas currently has no way to specify the ID of the User-Agent (the browser) which is why we're doing things as shown in the above code. The use of the IO standard library converts the returned HTML into a format read_html can use, otherwise read_html thinks the string is a filename, can't find it when it goes looking for it, then crashes with an IOError. Using the conversion shown above fixes this issue.

Regards, and thanks.

--Paul.

[Jelzap0]

@headfirstpython @AAdewunmi
Thank you so much for your time🙏
Now everything works perfectly.

[headfirstpython]

You are very welcome, but kudos to Adrian who did the heavy lifting (I just tweaked).

[AAdewunmi]

No worries @Jelzap0 😎 ... Just building on the work of Paul (@headfirstpython)