#258: Fix the Internal Links in the RSS-Feed of MkDocs

The RSS feed plug-in for MkDocs is a bit of a troublemaker. While it works, there is a lot of room for improvement. One big disappointment is the handling of internal links. Unfortunately, the MkDocs pipeline is nothing you want to dive into and so I had to work around the problem.

What is the problem?

To create an internal link to another post in my blog, I use the relative path from the post I am writing to the post file I want to link to. In the build step MkDocs replaces that with the correct absolute link and everything works. The RSS feed uses the relative link, puts it into the description field that then shows up in the RSS reader. If someone clicks on the link, they end up on a site that does not exist.

The link as part of the RSS feed:

./../../2024/248-mkdocs-for-a-blog/248-mkdocs-for-a-blog.md

How it should look instead:

https://pythonfriday.dev/2024/10/248-mkdocs-for-a-blog/

As we can see, the slug is part of both links and that is the common part that we can use.

My approach to fix the links

Since the relative link does not contain the month, I cannot simply work with the feed. I need a different source to find the correct link and then replace the relative link with the absolute one. It helps that I want to fix the problem when MkDocs generated the whole site. That allows me to access the sitemap.xml file with all the pages in the blog. The entries there look like this:

    <url>
         <loc>https://PythonFriday.dev/2024/10/248-mkdocs-for-a-blog/</loc>
         <lastmod>2024-11-22</lastmod>
    </url>

My approach to fixing the links involves the following steps:

Read the sitemap.xml and extract the URLs of the posts
Turn the links from the sitemap into a dictionary with the slug as the key and the absolute link as the value
For each RSS feed file, we can now do this:
- find the relative links and create a dictionary with the folder name as a key and the relative link as its value
- Run through the relative link dictionary and replace every relative link with the absolute link from the sitemap link dictionary

Read the sitemap

The sitemap.xml is an XML file inside the generated site. It does not only contain the posts, but all pages, including the archive, the categories and the tags. We can ignore all those pages and only work with the posts where we extract the slug to build our dictionary for the absolute links:

def find_posts_in_sitemap(sitemap):
    posts = {}

    # Parse the XML file
    root = ET.parse(sitemap)

    # Define the namespace
    namespace = {'ns': 'http://www.sitemaps.org/schemas/sitemap/0.9'}

    # Extract all <loc> and skip what is not a post
    for loc in root.findall('ns:url/ns:loc', namespace):
        if "category" in loc.text:
            continue
        if "archive" in loc.text:
            continue
        if "page" in loc.text:
            continue
        if "tags" in loc.text:
            continue
        if "2" not in loc.text:
            continue

        slug = extract_slug_from_sitemap(loc.text)

        posts[slug] = loc.text

    print(f"{len(posts)} posts found")
    return posts


def extract_slug_from_sitemap(url):
    # capture the last part of the URL path before the trailing slash
    pattern = r'/([^/]+)/$'

    # Search for the pattern in the provided URL
    match = re.search(pattern, url)

    # If a match is found, return the slug
    if match:
        return match.group(1)
    return None

Fix the feed files

We can handle the XML and the JSON files for the RSS feed the same way. We use the list of valid posts from the sitemap and the file as a parameter into our fix_feed() function. There we read the file into a variable, search for links to Markdown files and find the slugs in them. After all these setup steps we can replace the relative link for a specific slug with the absolute one:

def fix_feed(feed, valid_posts):
    print(f"Work on file {feed}")
    with open(feed, 'r') as file:
        data = file.read()

    links_to_md_file = find_links_to_md_file(data)
    substitutes = find_replacement(valid_posts, links_to_md_file)

    for key, value in substitutes.items():
        data = data.replace(key, value)

    with open(feed, 'w') as file:
        file.write(data)


def find_links_to_md_file(text):
    links = []
    pattern = r'(\.\/[^"]+?\.md)'
    matches = re.findall(pattern, text)

    for match in matches:
        links.append(match)

    print(f"{len(links)} links found")
    unique_links = set(links)
    print(f"{len(unique_links)} unique links found")
    return unique_links


def extract_slug_from_link_to_md_file(text):
    # find slug, that here is the directory name
    pattern = r'/([^/]+)\.md$'

    # Search for the pattern in the provided text
    match = re.search(pattern, text)

    # If a match is found, return the slug
    if match:
        return match.group(1)
    return None


def find_replacement(valid_posts, links_to_md_file):
    substitutes = {}

    for link in links_to_md_file:
        slug = extract_slug_from_link_to_md_file(link)
        substitutes[link] = valid_posts[slug]

    return substitutes

Glue everything together

To run all the above methods in the right order, I created this glue code:

if __name__ == "__main__":
    sitemap = r"..\site\sitemap.xml"
    feed_rss_created = r"..\site\feed_rss_created.xml"
    feed_rss_updated = r"..\site\feed_rss_updated.xml"
    feed_json_created = r"..\site\feed_json_created.json"
    feed_json_updated = r"..\site\feed_json_updated.json"

    print(f"Sitemap {sitemap}")
    valid_posts = find_posts_in_sitemap(sitemap)

    fix_feed(feed_rss_created, valid_posts)
    fix_feed(feed_rss_updated, valid_posts)
    fix_feed(feed_json_created, valid_posts)
    fix_feed(feed_json_updated, valid_posts)

The whole code goes into the file fix_feed.py in the folder /scripts. If I now run the script, it turns the relative links into absolute ones and with that fixes this annoying problem in the RSS feed.