Incorrect Behavior of Collecting a filtered iterator to a BooleanArray

[tobixdev]

Describe the bug

Collecting to a BooleanArray produces unintuitive results if the upper bound of the iterator is an over estimation. At least I think after looking at the code.

Is this intended behavior? If not, I could try to come up with a fix.

To Reproduce

Tested with Arrow v56.2.0 (via DataFusion 50)

The following test reproduces this:

    #[test]
    fn test_boolean_array_from() {
        let values = vec![Some(true), None, Some(true), Some(false)]
            .into_iter()
            .filter(Option::is_some)
            .collect::<BooleanArray>();
        assert_debug_snapshot!(values, @r"
        BooleanArray
        [
          true,
          true,
          false,
          null,
        ]
        ")
    }

Expected behavior

I'd have expected the following Array (without the null):

        BooleanArray
        [
          true,
          true,
          false,
        ]

Additional context

The result of the "same" operation on an Int64Array:

    #[test]
    fn test_int64_array_from() {
        let values = vec![Some(1), None, Some(2), Some(3)]
            .into_iter()
            .filter(Option::is_some)
            .collect::<Int64Array>();
        assert_debug_snapshot!(values, @r"
        PrimitiveArray<Int64>
        [
          1,
          2,
          3,
        ]
        ")
    }

[jhorstmann]

Agree, this looks like a bug. The FromIterator impl uses the upper bound of the input as the resulting length:

arrow-rs/arrow-array/src/array/boolean_array.rs

Line 443 in b9c2bf7

let data_len = data_len.expect("Iterator must be sized"); // panic if no upper bound.

[tobixdev]

I think a sustainable fix would be to separate the implementation into from_iter and from_trusted_len_iter similar to PrimtiveArray.

The former would use a (probably slower) builder for creating the boolean array, while the latter would do a similar thing to the current implementation.

[alamb]

The former would use a (probably slower) builder for creating the boolean array, while the latter would do a similar thing to the current implementation.

Makes sense to me -- thank you @jhorstmann and @tobixdev

[alamb]

label_issue.py automatically added labels {'arrow'} from #8525