Convert PySpark dataframe column from list to string
While you can use a UserDefinedFunction
it is very inefficient. Instead it is better to use concat_ws
function:
from pyspark.sql.functions import concat_wsdf.withColumn("test_123", concat_ws(",", "test_123")).show()
+----+----------------+|uuid| test_123|+----+----------------+| 1|test,test2,test3|| 2|test4,test,test6|| 3|test6,test9,t55o|+----+----------------+
You can create a udf
that joins array/list and then apply it to the test column:
from pyspark.sql.functions import udf, coljoin_udf = udf(lambda x: ",".join(x))df.withColumn("test_123", join_udf(col("test_123"))).show()+----+----------------+|uuid| test_123|+----+----------------+| 1|test,test2,test3|| 2|test4,test,test6|| 3|test6,test9,t55o|+----+----------------+
The initial data frame is created from:
from pyspark.sql.types import StructType, StructFieldschema = StructType([StructField("uuid",IntegerType(),True),StructField("test_123",ArrayType(StringType(),True),True)])rdd = sc.parallelize([[1, ["test","test2","test3"]], [2, ["test4","test","test6"]],[3,["test6","test9","t55o"]]])df = spark.createDataFrame(rdd, schema)df.show()+----+--------------------+|uuid| test_123|+----+--------------------+| 1|[test, test2, test3]|| 2|[test4, test, test6]|| 3|[test6, test9, t55o]|+----+--------------------+
As of version 2.4.0, you can use array_join
.Spark docs
from pyspark.sql.functions import array_joindf.withColumn("test_123", array_join("test_123", ",")).show()