Filter by array value in Spark DataFrame

scala apache-spark dataframe elasticsearch apache-spark-sql

In Spark 1.5+ you can use array_contains function:

df.where(array_contains($"people.artist.id", "153"))

If you use an earlier version you can try an UDF like this:

val containsId = udf(  (rs: Seq[Row], v: String) => rs.map(_.getAs[String]("id")).exists(_ == v))df.where(containsId($"people.artist", lit("153")))

CodeHunter