spark [dataframe].write.option(“mode”,“overwrite”).saveAsTable(“foo”) fails with ‘already exists’ if foo exists

I think I am seeing a bug in spark where mode ‘overwrite’ is not respected, rather an exception is thrown on an attempt to do saveAsTable into a table that already exists (using mode ‘overwrite’).

Below is a little scriptlet that reproduces the issue. The last statement results in a stack trace reading:

 org.apache.spark.sql.AnalysisException: Table `example` already exists.;

Any advice much appreciated.

spark.sql("drop table if exists example ").show()
case class Person(first: String, last: String, age: Integer)
val df = List(
    Person("joe", "x", 9),
    Person("fred", "z", 9)).toDF()
df.write.option("mode","overwrite").saveAsTable("example")

val recover1 = spark.read.table("example")
recover1.show()


val df3 = List(
    Person("mouse", "x", 9),
    Person("golf", "z", 9)).toDF()

 df3.write.
    option("mode","overwrite").saveAsTable("example")      

val recover4 = spark.read.table("example")
recover4.show()

Answer

saveAsTable doesn’t check extra options, use mode directly

df3.write.mode(SaveMode.Overwrite).saveAsTable("example")

df3.write.mode("overwrite").saveAsTable("example")

Advertisement

Answer