A regex link checker breaks on the exact HTML it was meant to check

python - 2026-09-05

I noticed one of my own TIL entries had a dead link: the second entry linked to a sibling post using [text](static-site-instead-of-datasette.html), a relative link to the markdown source file. That resolves fine when GitHub renders the .md file directly, but my static site build script only ever writes .html files, so the link 404'd on the actual published site.

The fix for that part was straightforward: rewrite relative *.md links to *.html before handing the markdown off for conversion.

RELATIVE_MD_LINK_RE = re.compile(r"(?<=\]\()(?!https?://)([^)\s]+?)\.md(#[^)]*)?(?=\))")

def rewrite_relative_md_links(text):
    return RELATIVE_MD_LINK_RE.sub(lambda m: f"{m.group(1)}.html{m.group(2) or ''}", text)

Then the fix for the bug broke the build

To stop this happening again, I added a check_links.py step to CI: build the site, then walk every _site/**/*.html file, extract href attributes, and fail if any local one points at a file that doesn't exist. First version used a plain regex:

HREF_RE = re.compile(r'href="([^"]+)"')

It caught the bug I'd just fixed. Then, on the very next commit, it failed the build again — on a link that didn't exist:

BROKEN: github-pages/atom-feed-and-syntax-highlighting.html: broken link -> </span><span class=

That "link" is HTML markup, not a URL. The TIL post about the link bug shows the fix as a Python code sample, which itself contains the literal text href="..." (as an example, inside a fenced code block). Two things compounded:

My regex has no concept of "inside a <code> block" vs "inside a real <a> tag" — it just scans raw text for anything shaped like href="...", so the example text matched too.
Pygments syntax-highlights that code sample by wrapping tokens in  tags, which splits the literal string href="../feed.atom" across multiple spans: href=".... My regex's [^"]+ then greedily matched everything between the first " it found and the next one — which happened to be several spans away, capturing raw HTML markup as if it were a URL.

The actual fix was to stop treating HTML as text and parse it properly:

from html.parser import HTMLParser

class HrefCollector(HTMLParser):
    def __init__(self):
        super().__init__()
        self.hrefs = []

    def handle_starttag(self, tag, attrs):
        if tag == "a":
            for name, value in attrs:
                if name == "href" and value:
                    self.hrefs.append(value)

HTMLParser only calls handle_starttag for genuine tags — text content inside a <code> block that merely looks like href="..." is just character data to it, never mistaken for a real attribute. No more false positives, and it still catches a real broken link just as well (I verified by manually reintroducing one and confirming the checker still exits non-zero).

The lesson

Regexes over HTML break down exactly when the HTML gets interesting — and a link checker for a programming blog, where code samples routinely contain HTML-shaped strings, is about the worst place to find that out. If the content you're parsing might contain more of itself as text, use a real parser.

Created 2026-09-05T03:18:28+02:00 · Edit