Last week we covered the basic parts to automate our browser with Selenium. If you try that with real web pages, you quickly notice that they are often not enough. In this post we look at few helpful tricks to make your Selenium automation more robust.
With Selenium ready and a solution in place to download drivers with ease, we now can start with the interesting part and automate our browser.
Downloading drivers for Selenium is annoying. We need to find the right driver for the specific browser version that runs on our computer. Luckily for us, there is a project that can do this work for us.
The more JavaScript a web site has, the more makes it sense to access it with a web browser. Over the next weeks we explore how Selenium can help us to automate a web browser to get the data from a web page we are interested in.
We must renew TLS/SSL certificates before they reach their end of life. Otherwise, our users get an unfriendly message about an unsecure web page. Why not write a Python script that checks our sites and shows how many days are remaining for our certificates?
With requests, Beautiful Soup, and the ultimate sitemap parser we have everything together to create our own basic link checker. Let us combine the three parts to something useful and check if our web site has broken links.
Sitemaps are great starting points if you need to find a list of all pages in a web site. Let us look at a little helper package that allows us to read the different sitemap formats with ease.
Getting HTML pages into our Python application is a straightforward task, thanks to requests. But what do we do now? Let us look how Beautiful Soup can help us getting the data out of the often messed up HTML code.
Web crawlers are still a hot topic. More and more data is on the web; unfortunately, often not in the form we need it. Today we look at requests, the elegant HTTP library for Python that gives us a good starting point to interact with web applications.
With the basics of date and time covered, we can look at how the dateutil module can help us with relative dates.