For our experiment with Bert and Goodreads it did not matter if the review was written in Englisch or German. But not all tools are that flexible when it comes to the language. Often, we need to load a language-specific model and for that we need a reliable way to detect the language. Let us explore two libraries that can help us with this task in Python.
The star review we got back with Bert in last week’s post gave me an idea: how well does my written review of a book I read match the stars I gave that book? Let us use some Python code to answer this question.
Sentiment analysis is a powerful tool that allows us to understand the emotions and opinions behind written text. Be it reviews, social media posts, or customer feedback – if we know the emotions and how strong they are, we can flag important texts and prioritise them to address problems before they blow up.
In this post we use pre-trained models from Hugging Face and the Natural Language Toolkit (NLTK). That way we can run everything locally and jump directly to the analysis part to get fast feedback.
While we now have a script to access a local LLM, we have a usability problem: We have to wait until the LLM has found an answer before we see that something is going on. Until then our script looks dead, and we may think it failed while it just waits on the LLM. A word-by-word output as we get with the web interfaces or at least a paragraph-by-paragraph update may soften this problem. Let us see how we can shorten the feedback time.
Machine learning is the hot topic of the day, especially Large Language Models (LLM). Despite the hype, my experience with them has been mixed. Sometimes they provide great help on the first try, but often the result needs significant rework or is completely wrong. In this post we do the groundwork to quickly try a lot of local LLMs to find one that offers a net benefit for us. It will not necessarily be a durable solution, but we can hit the ground running and get feedback before we spend all the time to run the wrong model on our local machine.
The Social plug-in of MkDocs Material gives us the nice images with the titles of our blog posts when we share them on social media.
Those images are also part of the RSS feed, but once more not in a correct format. Let us see how we can fix that.
When we work with Pandas, we keep our data in a DataFrame. This format works well with the whole workflow, but it is a bit challenging when we dump it to the console so that we then can use it in a document. Luckily for us, there is a nice helper to convert the DataFrame to Markdown.
Last week we saw how we could work seamlessly with Excel in DuckDB. If you have Pandas and want to get a similar experience, this post is for you.
Excel is the backbone of many companies. While it comes with a nice format to share tabular data between departments, it is not ideal to work with it in an automated way. Even with the integrated Python and the development functionality, Excel feels something limited compared to what we can do with regular Python. In this post we use Excel only as a storage format and do the work in plain Python and DuckDB.
DuckDB offers us a fast access to analyse our data. A lot of the flexibility in DuckDB comes from extensions, that we can load when we need them. Let us explore how we can work with those extensions to solve common problems.