Spark RDD - Mapping with extra arguments
You can use an anonymous function either directly in a
flatMapjson_data_rdd.flatMap(lambda j: processDataLine(j, arg1, arg2))or to curry
processDataLinef = lambda j: processDataLine(dataline, arg1, arg2)json_data_rdd.flatMap(f)You can generate
processDataLinelike this:def processDataLine(arg1, arg2): def _processDataLine(dataline): return ... # Do something with dataline, arg1, arg2 return _processDataLinejson_data_rdd.flatMap(processDataLine(arg1, arg2))toolzlibrary provides usefulcurrydecorator:from toolz.functoolz import curry@currydef processDataLine(arg1, arg2, dataline): return ... # Do something with dataline, arg1, arg2json_data_rdd.flatMap(processDataLine(arg1, arg2))Note that I've pushed
datalineargument to the last position. It is not required but this way we don't have to use keyword args.Finally there is
functools.partialalready mentioned by Avihoo Mamka in the comments.