Skip to content

#251: Create HTTP Redirects to the New Blog

Now that the posts for Python Friday are on the new PythonFriday.dev domain, I need a way to redirect the visitors from the Improve & Repeat blog to the new location. Let us write a script to create the necessary HTTP redirects.

Apache mod_rewrite

Since Improve & Repeat runs on Apache, we can use the mod_rewrite module. This is a feature that let us use regular expressions to match requests and redirect them with a pattern. I want to keep things simple and skip most of the dynamic part. Instead, we create a rewrite rule for each post and explicitly set the new URL. This is a tiny bit more work at the beginning, but it will be simpler to understand for everyone who has to maintain it.

Status code 302 or 301?

HTTP status code 302 is for a temporary redirect, while 301 is a durable solution. We first should check if everything works with the 302 and only after a few weeks switch to 301. That way the browser keeps checking the original location, what would help us to fix a wrong redirect.

Find the old posts

To get a list of all the old posts, we can leverage the tools we used in #139: Parse Sitemaps With Python). With the ultimate sitemap parser we can read all URLs from the sitemap, and filter the ones containing Python Friday posts:

1
2
3
4
5
6
7
8
9
from usp.tree import sitemap_tree_for_homepage

tree = sitemap_tree_for_homepage('https://improveandrepeat.com/')

pages = []

for page in tree.all_pages():
    if "/python-friday-" in page.url and page.url not in pages:
        pages.append(page.url)

Create the rewrite rule with a transformation

The rewrite rule follows this pattern: 

1
RewriteRule ^path/old/post/?$ https://domain/path/new/ [QSD,R=302,L]

It takes two substitutions to create a new URL out of an old one. We need to remove the python-friday- part and change the domain from improveandrepeat.com to pythonfriday.dev. With our old and new URL, we can create a rewrite rule and write it to a file:

1
2
3
4
5
6
7
8
with open("redirects.txt", "w") as f:
    for page in pages:
        target = str(page)
        target = target.replace("improveandrepeat.com", "pythonfriday.dev")
        target = target.replace("/python-friday-", "/")
        source = str(page)
        source = source.replace("https://improveandrepeat.com/", "")
        f.write(f"RewriteRule ^{source}?$ {target} [QSD,R=302,L]\n")

Handle exceptions

For 248 out of the 249 posts we want to redirect, we can follow the rule above. Only post 186 needs a special treatment, because I missed that the Yoast plug-in changed the URL behind my back. We can put that special case into its own line at the end of our script:

1
2
3
4
5
pf186_old =  "2023/08/filter-data-in-your-pandas-dataframe/"
pf186_new = "https://pythonfriday.dev/2023/08/186-filter-data-in-pandas/"
...

    f.write(f"RewriteRule ^{pf186_old}?$ {pf186_new} [QSD,R=302,L]\n")

Run it

The full script should now look like this:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
from usp.tree import sitemap_tree_for_homepage

pf186_old =  "2023/08/filter-data-in-your-pandas-dataframe/"
pf186_new = "https://pythonfriday.dev/2023/08/186-filter-data-in-pandas/"

tree = sitemap_tree_for_homepage('https://improveandrepeat.com/')

pages = []

for page in tree.all_pages():
    if "/python-friday-" in page.url and page.url not in pages:
        pages.append(page.url)


with open("redirects.txt", "w") as f:
    for page in pages:
        target = str(page)
        target = target.replace("improveandrepeat.com", "pythonfriday.dev")
        target = target.replace("/python-friday-", "/")
        source = str(page)
        source = source.replace("https://improveandrepeat.com/", "")
        f.write(f"RewriteRule ^{source}?$ {target} [QSD,R=302,L]\n")

    f.write(f"RewriteRule ^{pf186_old}?$ {pf186_new} [QSD,R=302,L]\n")

print("redirects written to redirects.txt")

When we run it, we get a file named redirects.txt that has all the rewrite rules in it:

1
2
3
...
RewriteRule ^2021/12/python-friday-100/?$ https://pythonfriday.dev/2021/12/100/ [QSD,R=302,L]
...

Apply it in Apache

We need to log-in to the old site, open the .htaccess file for the page in an editor and paste our rules in the mod-rewrite section:

1
2
3
4
5
6
7
8
<IfModule mod_rewrite.c>
RewriteEngine On

# ENTER RULES HERE

RewriteCond %{SERVER_PORT}   !^443$
RewriteRule  (.*)  https://%{HTTP_HOST}/$1   [L]
</IfModule>

It may take a few minutes until the setting is active. You best check with a single post if the redirect works.

Verify it

With requests we can check if we get the correct redirect back from our old blog. Since the redirect happens automatically, we need to check the request history to see if it was a redirect or not:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
from usp.tree import sitemap_tree_for_homepage
import requests
import time

pages_original = []

pf186 =  "https://improveandrepeat.com/2023/08/filter-data-in-your-pandas-dataframe/"
tree = sitemap_tree_for_homepage('https://improveandrepeat.com/')

for page in tree.all_pages():
    if "/python-friday-" in page.url and page.url not in pages_original:
        pages_original.append(page.url)

pages_original.append(pf186)

print("\n" * 10)
print(f"pages to check: {len(pages_original)}")

for page in pages_original:
    time.sleep(1)
    r = requests.get(page)
    if r.history:
        old = r.history[0]
        print(f"{r.status_code} - {old.is_redirect} - {old.url} => {r.url}")
    else:
        print(f"{r.status_code} - False - {r.url}")

For my server I had to slow it down a bit to prevent errors from Cloudflare. If you run the script, you should get a line like this for all posts:

1
2
3
4
pages to check: 250
200 - True - https://improveandrepeat.com/2020/01/python-friday-1-lets-learn-python/ => https://pythonfriday.dev/2020/01/1-lets-learn-python/
200 - True - https://improveandrepeat.com/2020/01/python-friday-2-resources-to-learn-python/ => https://pythonfriday.dev/2020/01/2-resources-to-learn-python/
200 - True - https://improveandrepeat.com/2020/01/python-friday-3-numbers-booleans-none/ => https://pythonfriday.dev/2020/01/3-numbers-booleans-none/

Fix the URL

For some pages we may get a redirect but end up on a non-existing page. That happens if the title falls into the part of the slug generator in which MkDocs and WordPress differ. We could change the slug function or only fix the slug for the few posts that end up with the wrong URL. I prefer to add a slug meta tag in the header section of the Markdown file:

1
2
3
4
5
---
title: "#3: Numbers, Booleans & None"
slug: "3-numbers-booleans-none"
date: 2020-01-17 20:00:00
categories:

What about Cloudflare?

Should you use Cloudflare on your old site, you need to purge the cache so that Cloudflare no longer sends out the old posts. Everything else should work right away.

Next

With the redirects in place, we can look at some common mistakes with MkDocs and how we can prevent them.