A comparison between fastparquet and pyarrow?

I used both fastparquet and pyarrow for converting protobuf data to parquet and to query the same in S3 using Athena. Both worked, however, in my use-case, which is a lambda function, package zip file has to be lightweight, so went ahead with fastparquet. (fastparquet library was only about 1.1mb, while pyarrow library was 176mb, and Lambda package limit is 250mb).

I used the following to store a dataframe as parquet file:

from fastparquet import writeparquet_file = path.join(filename + '.parq')write(parquet_file, df_data)

I would point out that the author of the speed comparison is also the author of pyarrow :) I can speak about the fastparquet case.

From your point of view, the most important thing to know is compatibility. Athena is not one of the test targets for fastparquet (or pyarrow), so you should test thoroughly before making your choice. There are a number of options that you may want to envoke (docs) for datetime representation, nulls, types, that may be important to you.

Writing to s3 using dask is certainly a test case for fastparquet, and I believe pyarrow should have no problem with that either.

However, since the question lacks concrete criteria, and I came here for a good "default choice", I want to state that pandas default engine for DataFrame objects is pyarrow (see pandas docs).