`SaveMode.Overwrite` on a non-existent path throws `NoSuchTableException`

[hujiayao]

Summary

df.write.format("lance").mode("overwrite").save("<new_path>") fails with org.apache.spark.sql.catalyst.analysis.NoSuchTableException when <new_path> does not yet exist. This is the most idiomatic Spark write call, and it diverges from how parquet/delta/iceberg handle the same pattern.

The same call succeeds with SaveMode.ErrorIfExists or SaveMode.Ignore.

Reproduction

import org.apache.spark.sql.SaveMode
import spark.implicits._

val newPath = "s3://bucket/some/brand/new/path"   // does not exist yet
val df = Seq((1L, "Alice", 25), (2L, "Bob", 30)).toDF("id", "name", "age")

df.write.mode(SaveMode.Overwrite).format("lance").save(newPath)

Stack trace:

org.apache.spark.sql.catalyst.analysis.NoSuchTableException:
  [TABLE_OR_VIEW_NOT_FOUND] The table or view ... cannot be found.  SQLSTATE: 42P01
    at org.lance.spark.BaseLanceNamespaceSparkCatalog.loadTableFromPath(BaseLanceNamespaceSparkCatalog.java:1509)
    at org.lance.spark.BaseLanceNamespaceSparkCatalog.loadTableInternal(BaseLanceNamespaceSparkCatalog.java:1388)
    at org.lance.spark.BaseLanceNamespaceSparkCatalog.loadTable(BaseLanceNamespaceSparkCatalog.java:557)
    at org.apache.spark.sql.DataFrameWriter.saveInternal(DataFrameWriter.scala:376)
    at org.apache.spark.sql.DataFrameWriter.save(DataFrameWriter.scala:302)

Environment

• lance-spark: 0.5.0-beta.1 (also confirmed on v0.4.0)
• Spark: 3.5.2
• Tested on Databricks Runtime 15.4 LTS and 16.4 LTS (Java 17); not Databricks-specific — the failing code path is purely in BaseLanceNamespaceSparkCatalog + Spark's DataFrameWriter.

Root cause

In Spark 3.5's DataFrameWriter.saveInternal, the two save-mode branches behave differently when the data source implements SupportsCatalogOptions:

Mode	loadTable probe	Next step	Result
ErrorIfExists	not called	CreateTableAsSelect	✓
Ignore	not called	CreateTableAsSelect (ignoreIfExists=true)	✓
Append	called, unguarded	n/a	✗ propagates (arguably correct)
Overwrite	called, unguarded	n/a	✗ propagates ← this bug

The Append | Overwrite branch in DataFrameWriter.scala calls catalog.loadTable(ident) directly without catching NoSuchTableException, so lance-spark's loadTableFromPath throwing for a missing path propagates straight to the user.

Parquet/Delta don't hit this because they only implement TableProvider, not SupportsCatalogOptions, and route through saveToV1Source.

Workaround

For first-time writes of a new path, use SaveMode.ErrorIfExists or SaveMode.Ignore:

df.write.mode(SaveMode.ErrorIfExists).format("lance").save(newPath)

Subsequent Append/Overwrite against the now-existing path work normally.

[LuciferYang]

I'm trying to fix this issue in upstream Spark

• [SPARK-57068][SQL] Make SaveMode.Overwrite create the table when missing for SupportsCatalogOptions sources apache/spark#56111

[LuciferYang]

Update on the upstream fix: I'm closing apache/spark#56111 rather than merging it.

The PR would have made mode("overwrite") create the table on a missing identifier, but the approach was rejected in review and I agree with the reasoning. For a source that implements SupportsCatalogOptions (which lance-spark does), save() resolves a real catalog table identity, so it's catalog DML, not path-based data writing:

• mode("overwrite") builds OverwriteByExpression(Literal(true)) — SQL INSERT OVERWRITE (truncate + write).
• mode("append") builds AppendData — SQL INSERT INTO.

Both SQL verbs require the target to already exist — there's no auto-create on insert — so Overwrite/Append throwing on a missing table is the intended, internally consistent behavior for a catalog-backed source. The PR special-cased only Overwrite-missing → create, which breaks the Append/Overwrite symmetry and doesn't actually match the V1 file-source model either (there, Append-on-missing creates too). The create-or-replace verb already exists separately as saveAsTable (ReplaceTableAsSelect(orCreate = true) = SQL CREATE OR REPLACE TABLE AS SELECT).

So this won't be changed in Spark. Recommended usage for lance-spark:

• First write to a new path: SaveMode.ErrorIfExists or SaveMode.Ignore — both route to CreateTableAsSelect and create it (as already noted in the issue).
• Create-or-replace by table identifier: df.write.format("lance").mode("overwrite").saveAsTable(<table>).
• Overwrite/Append on an already-existing path work normally.