-
Notifications
You must be signed in to change notification settings - Fork 1.5k
Failed optimizations with Int64 type #15291
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Comments
Thanks @aectaan -- what is the error message that you get? |
@alamb it's related to common subexpr eliminate:
|
Thanks @aectaan |
I wonder if you can get a pure SQL ( |
Thank you @alamb! No, it doesn't require anything custom. Unfortunately There are two optimisation rules, that involved into bug:
Order and set of rules is default. Disabling other rules doesn't change anything. If we swap order of first |
Maybe you can use set datafusion.sql_parser.dialect = 'postgres'; > set datafusion.sql_parser.dialect = 'postgres';
0 row(s) fetched.
Elapsed 0.001 seconds.
> PREPARE req(BIGINT) AS
WITH aggregations_group AS (
SELECT
count(col_utf8) FILTER (WHERE $1 - 1 <= col_int64) as foo,
count(col_utf8) FILTER (WHERE $1 - 2 <= col_int64 AND col_uint32 >= 0) as bar,
count(col_utf8) FILTER (WHERE $1 - 2 <= col_int64 AND col_uint32 >= 0) as baz
FROM test
)
SELECT * FROM aggregations_group;
Error during planning: table 'datafusion.public.test' not found |
@alamb On head commit.
|
That's interesting, same request performed via datafusion-cli returns without error that test have. Behaviour of test and df-cli differs starting from 34d9d3a |
Looks like datafusion-cli doesn't apply optimisations at all. After applying attached patch behaviour is the same as with test. |
Ok, probably it's related to |
@alamb What's the reason to cast numeric columns to i64/u64 and not to smallest compatible type? Why not to try something like this: macro_rules! try_parse_num {
($num:expr, $ty:ident) => {{
if let Ok(n) = $num.parse::<$ty>() {
return Ok(lit(n));
}
}};
}
try_parse_num!(signed_number, i8);
if !negative {
try_parse_num!(unsigned_number, u8);
}
try_parse_num!(signed_number, i16);
if !negative {
try_parse_num!(unsigned_number, u16);
}
try_parse_num!(signed_number, i32);
if !negative {
try_parse_num!(unsigned_number, u32);
}
try_parse_num!(signed_number, i64);
if !negative {
try_parse_num!(unsigned_number, u64);
} I replaced code at
|
I think it makes some sense from a False Sharing perspective but I'm not entirely sure about that. |
Describe the bug
Datafusion optimizer produce different behaviour with different types of arguments in request. Also behaviour is dependent on positions of arguments.
To Reproduce
Add this test to
optimiser_integrations.rs
Also add a
Field::new("col_int64", DataType::Int64, true)
column to test table.Failed request will execute succesfuly with any of this changes:
Expected behavior
Expected to get successful result not dependent on types or expression order
Additional context
No response
The text was updated successfully, but these errors were encountered: