Skip to content

feat(sqllogictest): Add support for iceberg datafusion sqllogictest integration #1764

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Open
wants to merge 10 commits into
base: main
Choose a base branch
from
Open

feat(sqllogictest): Add support for iceberg datafusion sqllogictest integration #1764

wants to merge 10 commits into from

Conversation

lliangyu-lin
Copy link
Contributor

@lliangyu-lin lliangyu-lin commented Oct 20, 2025 •
edited
Loading

Which issue does this PR close?

What changes are included in this PR?

  • Add sqllogictests entry point sqllogictests.rs
  • Update Schedule to track EngineRunner instead of Engine
  • Update datafusion engine to run slt file with sqllogictest

Are these changes tested?

  • cargo test --test sqllogictests
    Finished `test` profile [unoptimized + debuginfo] target(s) in 0.46s
     Running tests/sqllogictests.rs (target/debug/deps/sqllogictests-8ee6c1e0fdedebe4)

running 1 test
test schedule: df_test.toml ... ok

test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.01s

lliangyu-lin
lliangyu-lin commented Oct 20, 2025
View reviewed changes
crates/sqllogictest/src/engine/datafusion.rs Outdated

async fn create_catalog(_: &TomlTable) -> anyhow::Result<Arc<dyn CatalogProvider>> {
todo!()
let temp_dir = TempDir::new()?;
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Planning to make catalog configurable based on schedule file in separate PR

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please create an issue to track this and add comment here to refer to that issue.

@lliangyu-lin
Copy link
Contributor Author

Getting No space left on device during compilation in CI. Will check if we could increase the memory or if any extra dependencies were added.

liurenjie1024
liurenjie1024 reviewed Oct 21, 2025
View reviewed changes
Copy link
Contributor

@liurenjie1024 liurenjie1024 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @lliangyu-lin for this pr!

crates/sqllogictest/src/engine/datafusion.rs Outdated

async fn create_catalog(_: &TomlTable) -> anyhow::Result<Arc<dyn CatalogProvider>> {
todo!()
let temp_dir = TempDir::new()?;
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please create an issue to track this and add comment here to refer to that issue.

@lliangyu-lin lliangyu-lin mentioned this pull request Oct 23, 2025
Open
lliangyu-lin
lliangyu-lin commented Oct 23, 2025
View reviewed changes
.github/workflows/ci.yml
- macos-latest
- windows-latest
steps:
- name: Maximize build space (Ubuntu)
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is same used in unit test workflow: https://github.com/apache/iceberg-rust/blob/main/.github/workflows/ci.yml#L128-L131

liurenjie1024
liurenjie1024 reviewed Oct 23, 2025
View reviewed changes
Copy link
Contributor

@liurenjie1024 liurenjie1024 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @lliangyu-lin for this pr, generally LGTM! Just one minor nit.

crates/sqllogictest/testdata/schedules/df_test.toml
[engines]
df = { type = "datafusion" }

[catalog]
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We don't need this any more? If so, it would be confusing to put it here.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@liurenjie1024 liurenjie1024 liurenjie1024 left review comments

At least 1 approving review is required to merge this pull request.

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@lliangyu-lin @liurenjie1024