Skip to content

#266: Using Parquet Files in Pandas

In last week’s post we explored the Parquet format and how we can work with it using pyarrow and fastparquet. Now it is time to find out how we can use Parquet files with Pandas so that we can profit from this storage efficient format in our daily work.

Why use Parquet files in Pandas?

Pandas integrates seamlessly with Parquet through the DataFrame - also a column-oriented technique. If we use both together, we can leverage the powerful data manipulation capabilities of Pandas and benefit from Parquet's efficient storage and retrieval.

Reading Parquet files with Pandas

To read a Parquet file into a DataFrame, we use the read_parquet function of Pandas:

1
2
3
4
5
6
7
import pandas as pd

# Read the Parquet file
df = pd.read_parquet('people.parquet')

# Display the DataFrame
print(df)

This code reads the people.parquet file and loads its contents into a DataFrame.

If our Parquet file contains specific columns of interest, we can read only those columns to optimize performance:

1
df = pd.read_parquet('people.parquet', columns=['name', 'age'])

Writing DataFrames to Parquet Files

To write a DataFrame to a Parquet file, we use the to_parquet method:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
import pandas as pd

# Sample DataFrame
data = {
    'id': [1, 2, 3],
    'name': ['Alice', 'Bob', 'Charlie'],
    'age': [25, 30, 35]
}
df = pd.DataFrame(data)

# Write to Parquet file
df.to_parquet('people.parquet')

By default, Pandas includes the index of the DataFrame in the Parquet file. If the index is not necessary, we can exclude it by setting the index parameter to False:

1
df.to_parquet('people.parquet', index=False)

Choosing the Engine

Pandas supports both pyarrow and fastparquet as engines for handling Parquet files. By default, Pandas attempts to use the pyarrow engine for working with Parquet files but will fall back to fastparquet if pyarrow is unavailable. We can specify the engine when we read from or write to a Parquet file:

1
2
3
4
5
# Reading with a specific engine
df = pd.read_parquet('people.parquet', engine='fastparquet')

# Writing with a specific engine
df.to_parquet('people.parquet', engine='pyarrow')

Limitations

When we work with Parquet files and Pandas, it is important to be aware of the limitations around the data types. Not all data types that work with Pandas can be stored in Parquet. For instance, complex data types like Interval or actual Python object types are not supported and will raise errors during serialization (see pandas.pydata.org)

Conclusion

Integrating Pandas with Parquet files streamlines the process of reading and writing data, combining the analytical strengths of Pandas with the storage efficiency of Parquet. By understanding the available options and parameters, we can optimize our data workflows to gain a performance boost and use less resources.