Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Class TestHiveMetastore use getSystemClassLoader instead of getClass.getClassLoader in setupMetastoreDB #12131

Open
1 of 3 tasks
2MD opened this issue Jan 29, 2025 · 0 comments · May be fixed by #12140
Open
1 of 3 tasks
Labels
bug Something isn't working

Comments

@2MD
Copy link

2MD commented Jan 29, 2025

Apache Iceberg version

1.7.1 (latest release)

Query engine

Spark

Please describe the bug 🐞

val icebergVersion = "1.7.1"
val sparkVersion = "3.3.2"

lazy val icebergDependencies = Seq(
"org.apache.iceberg" %% "iceberg-spark-runtime-3.3" % icebergVersion,
"org.scala-lang.modules" % "scala-collection-compat_2.12" % "2.12.0",
"org.apache.iceberg" % "iceberg-aws-bundle" % icebergVersion,
"org.apache.iceberg" % "iceberg-hive-metastore" % icebergVersion
)
"org.apache.hive" % "hive-metastore" % "3.1.3" % Provided,
"org.apache.spark" %% "spark-hive" % sparkVersion % Provided,
("org.apache.hive" % "hive-exec" % "3.1.3").classifier("core") % Provided,
("org.apache.iceberg" % "iceberg-hive-metastore" % icebergVersion % IntegrationTest).classifier("tests")

I put hive-schema-3.1.0.derby.sql in my resources folder
project/
├── src/
│ ├── it/
│ │ ├── resources/
│ │ │ └── hive-schema-3.1.0.derby.sql

I have some test

var metastore: TestHiveMetastore = _
  var uris: String = _
  implicit var spark: SparkSession = _

  override def beforeAll(): Unit = {
    super.beforeAll()
    metastore = new TestHiveMetastore()
    metastore.start()
    uris = metastore.hiveConf().get("hive.metastore.uris")
    spark = SparkSession.builder
      .master("local[*]")
      .config("spark.hadoop.hive.metastore.uris", uris)
      .config("spark.sql.legacy.parquet.nanosAsLong", "false")
      .config("spark.sql.extensions", "org.apache.iceberg.spark.extensions.IcebergSparkSessionExtensions")
      .config(s"spark.sql.catalog.$icebergCatalog", "org.apache.iceberg.spark.SparkCatalog")
      .config(s"spark.sql.catalog.$icebergCatalog.type", "hive")
      .config("spark.hadoop.fs.s3a.endpoint", minioContainer.getHostAddress)
      .config("spark.hadoop.fs.s3a.access.key", minioContainer.getMinioAccessValue)
      .config("spark.hadoop.fs.s3a.secret.key", minioContainer.getMinioAccessValue)
      .config("spark.hadoop.fs.s3a.path.style.access", "true")
      .config("spark.hadoop.fs.s3a.impl", "org.apache.hadoop.fs.s3a.S3AFileSystem")
      .config("spark.sql.iceberg.check-ordering", "false")
      .config("write.metadata.delete-after-commit.enabled", "false")
      .config("spark.sql.iceberg.handle-timestamp-without-timezone", "true")
      .enableHiveSupport()
      .getOrCreate()
    s3.createBucket(bucket)
  }

I run my test > sbt "IntegrationTest/testOnly"

Caused by: java.lang.NullPointerException
[error] at java.base/java.io.Reader.(Reader.java:167)
[error] at java.base/java.io.InputStreamReader.(InputStreamReader.java:72)
[error] at org.apache.iceberg.hive.TestHiveMetastore.setupMetastoreDB(TestHiveMetastore.java:281)
[error] at org.apache.iceberg.hive.TestHiveMetastore.(TestHiveMetastore.java:106)
[error] at ru.samokat.datalake.loader.BaseLoadSuite.beforeAll(BaseLoadSuite.scala:49)
[error] at ru.samokat.datalake.loader.BaseLoadSuite.beforeAll$(BaseLoadSuite.scala:46)

He can't find hive-schema-3.1.0.derby.sql because file loaded from SystemClassLoader in this place:

private static void setupMetastoreDB(String dbURL) throws SQLException, IOException {
    Connection connection = DriverManager.getConnection(dbURL);
    ScriptRunner scriptRunner = new ScriptRunner(connection, true, true);

    ClassLoader classLoader = ClassLoader.getSystemClassLoader();
    InputStream inputStream = classLoader.getResourceAsStream("hive-schema-3.1.0.derby.sql");
    try (Reader reader = new InputStreamReader(inputStream)) {
      scriptRunner.runScript(reader);
    }
  }

Willingness to contribute

  • I can contribute a fix for this bug independently
  • I would be willing to contribute a fix for this bug with guidance from the Iceberg community
  • I cannot contribute a fix for this bug at this time
@2MD 2MD added the bug Something isn't working label Jan 29, 2025
@2MD 2MD linked a pull request Jan 31, 2025 that will close this issue
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
Development

Successfully merging a pull request may close this issue.

1 participant