PySpark: Take average of a column after using filter function

python apache-spark apache-spark-sql pyspark pyspark-sql

Aggregation function should be a value and a column name a key:

dataframe.filter(df['salary'] > 100000).agg({"age": "avg"})

Alternatively you can use pyspark.sql.functions:

from pyspark.sql.functions import col, avgdataframe.filter(df['salary'] > 100000).agg(avg(col("age")))

It is also possible to use CASE .. WHEN

from pyspark.sql.functions import whendataframe.select(avg(when(df['salary'] > 100000, df['age'])))

python apache-spark apache-spark-sql pyspark pyspark-sql

You can try this too:

dataframe.filter(df['salary'] > 100000).groupBy().avg('age')

CodeHunter