How to save file in Feather format\storage from Spark?

pandas apache-spark dataframe spark-dataframe feather

Not sure, you can do it directly, but you can transform first the Spark Dataframe (on pyspark) to a pandas and store it the to Feather:

pandas_df = spark_df.toPandas()

feather.write_feather(pandas_df, 'example_feather')

But I afraid, this will have an impact on the performance.

CodeHunter