Spark: optimized join with ElasticSearch index
This is an expected behavior, yes elaticsearch-hadoop connector supports pushdown predicate but there is no push when you join.
This is because the join operation does not know anything about how the keys are partitioned in your dataframes.
By default, this operation will hash all the keys of both dataframes, sending all the elements with the same key hash across the network to the same machine, and then join together the elements with the same key on that machine.
And that's why you get that execution plan without the predicate being pushed down.
EDIT : It seems like the connector supports since the version 2.1 the IN clause. You ought using that if your DataFrame a isn't big.