Duplicate columns in Spark Dataframe

The best way would be to change the column name upstream ;)

However, it seems that is not possible, so there are a couple of options:

1. If the case of the columns are different("email" vs "Email") you can turn on case sensitivity:

        sql(sqlContext, "set spark.sql.caseSensitive=true")

2. If the column names are exactly the same, you will need to manually specify the schema and skip the first row to avoid the headers:

   customSchema <- structType(structField("year", "integer"), structField("make", "string"),structField("model", "string"),structField("comment", "string"),structField("blank", "string"))df <- read.df(sqlContext, "cars.csv", source = "com.databricks.spark.csv", header="true", schema = customSchema)

Try renaming the column.

You can select it by position instead of the select call.

colnames(df)[column number of interest] <- 'deleteme'

Alternatively you could just drop the column directly

 newdf <- df[,-x]

Where x is the column number you don't want.

Update:

If the above don't work, you could set header to false and then use the first row to rename columns:

  df <- read.df(    sqlContext,    FILE_PATH,    source = "com.databricks.spark.csv",    header = "FALSE",    mode = "DROPMALFORMED"  )#get first row to use as column namesmycolnames <- df[1,]#edit the dup column *in situ*mycolnames[x] <- 'IamNotADup'colnames(df) <- df[1,]# drop the first row:df <- df[-1,]

You can also create a new dataframe using toDF.

Here's the same thing, for pyspark: Selecting or removing duplicate columns from spark dataframe