In last week’s post we explored the Parquet format and how we can work with it using pyarrow and fastparquet. Now it is time to find out how we can use Parquet files with Pandas so that we can profit from this storage efficient format in our daily work.
Now that we know how to create a large amount of test data with Faker, we should find an efficient way to store the data. Most developers know CSV files, but is there a more efficient format we can use? On my search to find an answer to this question, the Parquet format showed up and it sounds like the tool for this task. Let us find out if this is the case and how we can use it.
In this post we use pyarrow and fastparquet to work with Parquet files, while Pandas will be the topic of the next post.
Last week we got pytest to run asynchronous test methods. That was the preparation step for this post where we switch to asynchronous SQLAlchemy for our to-do application. As it turns out, switching to asynchronous methods for SQLAlchemy takes a lot of work. Let us get through the different changes we need to make.
Our current way to create the tables in the database when we run the application works fine until we need to extend an existing table. Then SQLAlchemy will not do that for us, and we must make the change manually. With Alembic we have a solution for that problem that works great with SQLAlchemy. Let us add it to our to-do application.
Last week we moved from an in-memory data store to SQLAlchemy and persist our tasks inside an SQLite database. We have two things we need to optimize, or else we end up with problems along the way.
Until now we kept our data in a variable. While that worked with an example application, the data vanishes as soon as we restart our API. To get a more realistic application, we need to persist data for a longer time. Let us explore how we can integrate SQLAlchemy with FastAPI.
Before we can move ahead and add a database to our to-do API, we take a little detour and update SQLAlchemy to version 2. Much changed since I wrote about SQLAlchemy three years ago, but thanks to the early published guidelines, the largest part of the examples I used in my post can stay the same. Nevertheless, there are changes we need to know about, and we must update minor details to run the examples with the current version of SQLAlchemy.
If you work with web technologies these days, there is no way around JSON. Today we look at the basic operations to serialize our objects to JSON and turn JSON back into objects.
Sometimes the collection of data takes a lot longer than processing it. Wouldn't it be great if you could store whole Python objects? If something goes wrong, we can restart the analytics part and don't need to recreate the data.
SQLAlchemy is a massive database toolkit that you can use in various ways. Let us do a recap on the options you have and look for a good starting point for your situation.