Spark - How to count number of records by key

You're nearly there! All you need is a countByValue:

val countOfFemalesByCountry = femaleOnly.map(_(13)).countByValue()// Prints (Australia, 230), (America, 23242), etc.

(In your example, I assume you meant x(10) rather than x._10)

All together:

sc.textFile("/home/cloudera/desktop/file.txt")    .map(_.split(","))    .filter(x => x(10) == "Female")    .map(_(13))    .countByValue()

Have you considered manipulating your RDD using the Dataframes API ?

It looks like you're loading a CSV file, which you can do with spark-csv.

Then it's a simple matter (if your CSV is titled with the obvious column names) of:

import com.databricks.spark.csv._val countryGender = sqlContext.csvFile("/home/cloudera/desktop/file.txt") // already splits by field  .filter($"gender" === "Female")  .groupBy("country").count().show()

If you want to go deeper in this kind of manipulation, here's the guide:https://spark.apache.org/docs/latest/sql-programming-guide.html

You can easily create a key, it doesn't have to be in the file/database. For example:

val countryGender = sc.textFile("/home/cloudera/desktop/file.txt")                .map(_.split(","))                .filter(x => x._10 == "Female")                .map(x => (x._13, x._10))    // <<<< here you generate a new key                .groupByKey();