[Spark] Unity Catalog integration foundation and repository setup for Delta Lake #5523

tdas · 2025-11-19T00:15:09Z

🥞 Stacked PR

Use this link to review incremental changes.

stack/spark-uc-pr1-foundation [Files changed]
- stack/spark-uc-pr2-dml-framework [Files changed]
  - stack/spark-uc-pr3-ddl [Files changed]
    - stack/spark-uc-pr4-utility [Files changed]
      - stack/spark-uc-pr5-read [Files changed]

Which Delta project/connector is this regarding?

Description

Establishes foundational Unity Catalog integration for Delta Lake with embedded UC server lifecycle management and comprehensive test framework.

Adds new sparkUnityCatalog SBT module with Unity Catalog 0.3.0 dependencies, UnityCatalogSupport trait for managing UC server lifecycle in tests, and UnityCatalogSupportSuite with 4 integration tests validating UC-Delta connectivity and table operations. Uses shaded UC server JAR to avoid dependency conflicts. Compatible with Spark 4.0 and Delta Lake. Includes repository setup (.gitignore updates).

How was this patch tested?

Added UnityCatalogSupportSuite with 4 comprehensive integration tests
Tests validate UC server startup, catalog creation, and basic table operations
All tests pass with Unity Catalog 0.3.0 and Spark 4.0
Validates end-to-end UC server connectivity and Delta table registration

Does this PR introduce any user-facing changes?

No. This PR only adds test infrastructure for Unity Catalog integration. No user-facing functionality is changed.

- Moved UnityCatalogSupport.scala and UnityCatalogSupportSuite.scala to com.sparkuctest package - Updated package declarations from org.apache.spark.sql.delta to com.sparkuctest - All 4 tests pass successfully with Spark 4.0

tdas · 2025-11-19T03:05:51Z

spark/unitycatalog/src/test/scala/com/sparkuctest/UnityCatalogSupport.scala

+    // scalastyle:off println
+    println(s"Unity Catalog server started and ready at $unityCatalogUri")
+    println(s"Created catalog '$unityCatalogName' with schema 'default'")
+    // scalastyle:on println


make all prints into log lines

tdas · 2025-11-19T03:06:48Z

spark/unitycatalog/src/test/scala/com/sparkuctest/UnityCatalogSupport.scala

+    Thread.sleep(5000)
+


instead of sleeping a fixed out can we poll to check when the server is ready

tdas · 2025-11-19T03:07:53Z

spark/unitycatalog/src/test/scala/com/sparkuctest/UnityCatalogSupportSuite.scala

+  private def createUCClient(): ApiClient = {
+    val client = new ApiClient()
+    // Extract port from unityCatalogUri
+    val port = unityCatalogUri.split(":")(2).toInt
+    client.setScheme("http")
+    client.setHost("localhost")
+    client.setPort(port)
+    client
+  }


can the UnityCatalogSupport trait provide the client

spark/unitycatalog/src/test/scala/com/sparkuctest/UnityCatalogSupportSuite.scala

build.sbt

…, and dependency management

…ed JAR

tdas · 2025-11-19T05:39:14Z

build.sbt

+      // Standard test dependencies
+      "org.scalatest" %% "scalatest" % scalaTestVersion % "test",
+
+      // Unity Catalog dependencies - exclude Jackson to use Spark's Jackson 2.15.x


remove jackson version

Maybe we need to change the unitycatalog's jackson version, so that we can align the spark's.

huan233usc

Looking good! just some questions on server setup and formatting.

huan233usc · 2025-11-19T06:58:23Z

spark/unitycatalog/src/test/scala/com/sparkuctest/UnityCatalogSupport.scala

@@ -0,0 +1,258 @@
+/*
+ * Copyright (2021) The Delta Lake Project Authors.


huan233usc · 2025-11-19T18:12:05Z

spark/unitycatalog/src/test/scala/com/sparkuctest/UnityCatalogSupport.scala

+ * limitations under the License.
+ */
+
+package com.sparkuctest


Is org.apache.spark.sql.delta.test.unty a better naming?

i am intentionally keeping it out of the package so that we dont accidentally depend or anything internal to delta or uc. closer to how any user / app would run

But it's not a com package right ? maybe we can use io.unitycatalog.ittest ?

btw, should we write the test in scala or java ? will it be better to use Java to maintain those tests ?

huan233usc · 2025-11-19T18:42:54Z

spark/unitycatalog/src/test/scala/com/sparkuctest/UnityCatalogSupport.scala

+    ucTempDir.deleteOnExit()
+
+    // Find an available port
+    ucPort = findAvailablePort()


nit: There is a small potential for the racing of port if the port is taken between here and Line 147. Should we include these logic in the retry? (I think might be ok to keep things as is as it is a test that is not parallelzable)

huan233usc · 2025-11-19T19:23:09Z

spark/unitycatalog/src/test/scala/com/sparkuctest/UnityCatalogSupportSuite.scala

+    val response: ListTablesResponse = tablesApi.listTables(catalogName, schemaName, null, null)
+
+    import scala.jdk.CollectionConverters._
+    if (response.getTables != null) {


The null check here seems to be redundant. If there is no tables, we will get empty list?

this is to handle the case if any server returns the corresponding field in the response json as null (that is field is missing) instead of an empty [].

build.sbt

huan233usc · 2025-11-19T19:28:57Z

spark/unitycatalog/src/test/scala/com/sparkuctest/UnityCatalogSupport.scala

+   * Creates a Unity Catalog API client configured for this server.
+   */
+  def createUnityCatalogClient(): ApiClient = {
+    val port = unityCatalogUri.split(":")(2).toInt


Should we directly us ucPort

haaah. stupid claude.

huan233usc · 2025-11-19T19:30:54Z

spark/unitycatalog/src/test/scala/com/sparkuctest/UnityCatalogSupportSuite.scala

+    // 3. Verify we can query UC server directly via SDK
+    val ucTables = listTables(unityCatalogName, "default")
+    // Should succeed even if empty - this confirms UC server is responding
+    assert(ucTables != null, "Should be able to query UC server via SDK")


This assertion should always pass as listTables never returns null, should we remove?

huan233usc · 2025-11-19T19:38:50Z

spark/unitycatalog/src/test/scala/com/sparkuctest/UnityCatalogSupportSuite.scala

+      """)
+
+      // If we got here, the catalog is working
+      val tables = spark.sql(s"SHOW TABLES IN $unityCatalogName.default")


Does checkAnswer works?https://github.com/apache/spark/blob/61668ad02671af0a80ec6d91ddaf02e6b3f042e6/sql/core/src/test/scala/org/apache/spark/sql/QueryTest.scala#L137

checkAnswer( spark.sql(s"SHOW TABLES IN $unityCatalogName.default") .select("tableName") .filter($"tableName" === "test_verify_catalog"), Row("test_verify_catalog") :: Nil )

seems like more code than this version :)

tdas · 2025-11-19T21:27:41Z

spark/unitycatalog/src/test/scala/com/sparkuctest/UnityCatalogSupportSuite.scala

+      val result = spark.sql(s"SELECT * FROM $testTable ORDER BY id").collect()
+      assert(result.length == 3, s"Should have 3 rows, got ${result.length}")


this should use checkAnswer

openinx

Thanks for the work, just left several comments. It's a great work to get start with the unitycatalog integration testing work.

openinx · 2025-11-20T23:11:03Z

.gitignore

 kernel/kernel-benchmarks/benchmark_report.json

+# Unity Catalog test artifacts
+spark/unitycatalog/etc/


Which tests will generate those artifacts ?

openinx · 2025-11-20T23:18:59Z

build.sbt


+
+val unityCatalogVersion = "0.3.0"
+val sparkUnityCatalogJacksonVersion = "2.15.4" // We are using Spark 4.0's Jackson version 2.15.x, to override Unity Catalog 0.3.0's version 2.18.x


Unrelated: I think we may need to relocate the Jackson in the oss-unitycatalog spark connector, since it's such a general client-side jar, which will run under different runtime environments. And for those common jars, it can be easily conflicted with the jars introduced from other projects.

I had an issue for oss-unitycatalog before: unitycatalog/unitycatalog#1141

i agree.. jackson was a massive pain. we need to fix it.

openinx · 2025-11-20T23:20:12Z

build.sbt

+
+lazy val sparkUnityCatalog = (project in file("spark/unitycatalog"))
+  .dependsOn(spark % "compile->compile;test->test;provided->provided")
+  .disablePlugins(JavaFormatterPlugin, ScalafmtPlugin)


Why disable the java and scala formatter plugin ? Seems weird ?

openinx · 2025-11-20T23:23:00Z

build.sbt

+    // This is a test-only module - no production sources
+    Compile / sources := Seq.empty,
+
+    Test / javaOptions ++= Seq("-ea"),


It seems like uncommon to use -ea (or -enableassertions) for testing ? If we all use the org.assertj, then maybe we can remove this option, do I understand correctly ?

openinx · 2025-11-20T23:29:32Z

build.sbt

+      // Standard test dependencies
+      "org.scalatest" %% "scalatest" % scalaTestVersion % "test",
+
+      // Unity Catalog dependencies - exclude Jackson to use Spark's Jackson 2.15.x


Maybe we need to change the unitycatalog's jackson version, so that we can align the spark's.

openinx · 2025-11-20T23:30:26Z

build.sbt

    ),
    assembly / assemblyMergeStrategy := {
      // Discard `module-info.class` to fix the `different file contents found` error.
-      // TODO Upgrade SBT to 1.5 which will do this automatically


nit: Is this related to this PR ?

openinx · 2025-11-20T23:40:44Z

spark/unitycatalog/src/test/scala/com/sparkuctest/UnityCatalogSupport.scala

+        val testClient = new ApiClient()
+        testClient.setScheme("http")
+        testClient.setHost("localhost")
+        testClient.setPort(ucPort)


Why not use the createUnityCatalogClient directly ?

openinx · 2025-11-20T23:52:05Z

spark/unitycatalog/src/test/scala/com/sparkuctest/UnityCatalogSupportSuite.scala

+
+  test("UnityCatalogSupport trait starts UC server and configures Spark correctly") {
+    // 1. Verify UC server is accessible via URI
+    assert(unityCatalogUri.startsWith("http://localhost:"),


How will it work if we want to run this same tests on different UC server setup ?

we can make this a env variable. if this env variable is set, then we use that URL instead of starting a new UC OSS server.

tdas changed the title ~~spark-uc module~~ Unity Catalog integration foundation and repository setup for Delta Lake Nov 19, 2025

tdas changed the title ~~Unity Catalog integration foundation and repository setup for Delta Lake~~ [Spark] Unity Catalog integration foundation and repository setup for Delta Lake Nov 19, 2025

tdas added 3 commits November 18, 2025 20:47

spark-uc module

56a74b1

Move Unity Catalog test support to com.sparkuctest package

5784bfe

- Moved UnityCatalogSupport.scala and UnityCatalogSupportSuite.scala to com.sparkuctest package - Updated package declarations from org.apache.spark.sql.delta to com.sparkuctest - All 4 tests pass successfully with Spark 4.0

Add Unity Catalog PR strategy and repository setup

1b7fe04

tdas force-pushed the stack/spark-uc-pr1-foundation branch from 8d685a6 to 1b7fe04 Compare November 19, 2025 01:48

tdas added 3 commits November 18, 2025 21:19

Fix sparkUnityCatalog to use CrossSparkVersions.sparkDependentSettings

892c1d0

Make sparkUnityCatalog tests run only on Spark 4.0

8fc5ac2

Sync: conditional test execution for Spark 4.0

e1b8157

tdas commented Nov 19, 2025

View reviewed changes

spark/unitycatalog/src/test/scala/com/sparkuctest/UnityCatalogSupportSuite.scala Outdated Show resolved Hide resolved

tdas commented Nov 19, 2025

View reviewed changes

spark/unitycatalog/src/test/scala/com/sparkuctest/UnityCatalogSupportSuite.scala Outdated Show resolved Hide resolved

tdas commented Nov 19, 2025

View reviewed changes

spark/unitycatalog/src/test/scala/com/sparkuctest/UnityCatalogSupportSuite.scala Show resolved Hide resolved

tdas commented Nov 19, 2025

View reviewed changes

build.sbt Outdated Show resolved Hide resolved

tdas commented Nov 19, 2025

View reviewed changes

build.sbt Outdated Show resolved Hide resolved

tdas added 4 commits November 18, 2025 22:29

Address PR comments: improve logging, server readiness, test coverage…

407f726

…, and dependency management

Address PR comments: improve logging, server readiness, test coverage…

0778546

…, and dependency management

Use dependency overrides to resolve Jackson conflicts instead of shad…

9017d89

…ed JAR

Use dependency overrides to resolve Jackson conflicts instead of shad…

3ec5f93

…ed JAR

tdas commented Nov 19, 2025

View reviewed changes

tdas added 2 commits November 19, 2025 00:46

Revert sbt-assembly upgrade and cleanup module-info merge strategy

2f6adec

Revert sbt-assembly upgrade and cleanup module-info merge strategy

c0fa580

huan233usc reviewed Nov 19, 2025

View reviewed changes

tdas commented Nov 19, 2025

View reviewed changes

openinx reviewed Nov 20, 2025

View reviewed changes

		@@ -0,0 +1,258 @@
		/*
		* Copyright (2021) The Delta Lake Project Authors.

		val result = spark.sql(s"SELECT * FROM $testTable ORDER BY id").collect()
		assert(result.length == 3, s"Should have 3 rows, got ${result.length}")



		val unityCatalogVersion = "0.3.0"
		val sparkUnityCatalogJacksonVersion = "2.15.4" // We are using Spark 4.0's Jackson version 2.15.x, to override Unity Catalog 0.3.0's version 2.18.x

[Spark] Unity Catalog integration foundation and repository setup for Delta Lake #5523

Are you sure you want to change the base?

[Spark] Unity Catalog integration foundation and repository setup for Delta Lake #5523

Conversation

tdas commented Nov 19, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

🥞 Stacked PR

Which Delta project/connector is this regarding?

Description

How was this patch tested?

Does this PR introduce any user-facing changes?

Uh oh!

tdas Nov 19, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

tdas Nov 19, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

huan233usc left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

openinx left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

tdas commented Nov 19, 2025 •

edited

Loading

tdas Nov 19, 2025 •

edited

Loading

tdas Nov 19, 2025 •

edited

Loading