Spark Row to JSON
Spark 2.1 should have native support for this use case (see #15354).
import org.apache.spark.sql.functions.to_jsondf.select(to_json(struct($"c1", $"c2", $"c3")))
I use this command to solve the to_json problem:
output_df = (df.select(to_json(struct(col("*"))).alias("content")))
Here, no JSON parser, and it adapts to your schema:
import org.apache.spark.sql.functions.{col, concat, concat_ws, lit}df.select( col(df.columns(0)), col(df.columns(1)), concat( lit("{"), concat_ws(",",df.dtypes.slice(2, df.dtypes.length).map(dt => { val c = dt._1; val t = dt._2; concat( lit("\"" + c + "\":" + (if (t == "StringType") "\""; else "") ), col(c), lit(if(t=="StringType") "\""; else "") ) }):_*), lit("}") ) as "C").collect()