DuckDB is a fast database, that allows us to use SQL to interact with our data. But that is not where the story ends. While we can use a lot of common SQL syntax, we get a few helpful features that help us to work more effectively with our data. Let us explore them.
Last week we made our first steps with DuckDB and its in-memory capabilities. In this post we persist data with DuckDB so that we can reuse it between our scripts.
DuckDB is an in-process SQL OLAP (Online Analytical Processing) database designed for efficient analytical queries. Known as the "SQLite for analytics", it integrates seamlessly with Python, making it ideal for data analysis. Let us explore the benefits it can bring to our tool box. We focus on the Python integration, but you can use DuckDB with many more programming languages.
Last week we explored the uv tool and saw how much of pip we can reuse while profiting from the massive speed boost. In this post we explore the final part of uv and see how it can help us to manage our projects and prepare our package for pypi.org.
If we dive into machine learning, we quickly run into a problem: Incompatible package versions. We can solve that problem with virtual environments, but that brings us right to the next one. Pip is a great tool, but when we need to install the same package multiple times, pip will download the package again and again. That is slow and breaks the flow. Let us explore an alternative solution.
In last week’s post we explored the Parquet format and how we can work with it using pyarrow and fastparquet. Now it is time to find out how we can use Parquet files with Pandas so that we can profit from this storage efficient format in our daily work.
Now that we know how to create a large amount of test data with Faker, we should find an efficient way to store the data. Most developers know CSV files, but is there a more efficient format we can use? On my search to find an answer to this question, the Parquet format showed up and it sounds like the tool for this task. Let us find out if this is the case and how we can use it.
In this post we use pyarrow and fastparquet to work with Parquet files, while Pandas will be the topic of the next post.
Faker is a test data generator we explored in the last post. Despite the large list of options to generate data, sometimes we need data in a format that Faker cannot offer. Should that happen, we can create our own data provider. Let us figure out how to do that.
If you have ever worked on a project that needed realistic test data, you know how tedious it can be to create it manually. Hardcoding names, addresses, or even phone numbers into your test suite not only clutters your code but also makes it inflexible. Thankfully, the Python Faker library comes to the rescue.
Have you ever run into unexpected bugs when checking conditions in your Python code? If so, you might have stumbled upon a subtle, yet common mistake: using a method reference without parentheses when evaluating conditions.