Access element of a vector in a Spark DataFrame (Logistic Regression probability vector) [duplicate]
Update:
It seems like there is a bug in spark that prevents you from accessing individual elements in a dense vector during a select statement. Normally you should would be able to access them just like you would a numpy array, but when trying to run the code previously posted, you may get the error pyspark.sql.utils.AnalysisException: "Can't extract value from probability#12;"
So, one way to handle this to avoid this silly bug is to use a udf. Similar to the other question, you can define a udf in the following way:
from pyspark.sql.functions import udffrom pyspark.sql.types import FloatTypefirstelement=udf(lambda v:float(v[0]),FloatType())cv_predictions_prod.select(firstelement('probability')).show()Behind the scenes this still accesses the elements of the DenseVector like a numpy array, but it doesn't throw the same bug as before.
Since this is getting a lot of upvotes, I figured I should strike through the incorrect portion of this answer.
Original answer:A dense vector is just a wrapper for a numpy array. So you can access the elements in the same way that you would access the elements of a numpy array.
There are several ways to access individual elements of an array in a dataframe. One is to explicitly call the column cv_predictions_prod['probability'] in your select statement. By explicitly calling the column, you can perform operations on that column, like selecting the first element in the array. For example:
cv_predictions_prod.select(cv_predictions_prod['probability'][0]).show()should solve the problem.