Remove record_batch! macro once upstream updates

[timsaucer]

Is your feature request related to a problem or challenge?

Follow on to apache/arrow-rs#6588

Once arrow-rs releases and we update datafusion, we can remove the record_batch! macro.

Describe the solution you'd like

Remove and use upstream method instead.

Describe alternatives you've considered

No response

Additional context

No response

[buraksenn]

take

[buraksenn]

I will keep an eye on the update and do the required change if it is okay.

[timsaucer]

Sounds great! It will probably take a while as we have to wait for the upstream to make a release, and then DF to update to that release version.

[buraksenn]

Should this be done? I saw this #12846. I can delete create_array and record_batch macros to make it use the same as in arrow. Also they are also not identical:

• Datafusion: ($(($name: expr, $type: ident, $values: expr)),*)
• Arrow: ($(($name: expr, $type: ident, [$($values: expr),])),)

@timsaucer @alamb ?

[alamb]

Should this be done? I saw this #12846. I can delete create_array and record_batch macros to make it use the same as in arrow. Also they are also not identical:

• Datafusion: ($(($name: expr, $type: ident, $values: expr)),*)
• Arrow: ($(($name: expr,
  t
  y
  p
  e
  :
  i
  d
  e
  n
  t
  ,
  [
  ($values: expr),])),)

@timsaucer @alamb ?

I recommend:

1. Switch to use the upstream arrow one
2. Deprecate (don't remove) the API in datafusion, per https://datafusion.apache.org/library-user-guide/api-health.html

[buraksenn]

Should this be done? I saw this #12846. I can delete create_array and record_batch macros to make it use the same as in arrow. Also they are also not identical:

• Datafusion: ($(($name: expr, $type: ident, $values: expr)),*)
• Arrow: ($(($name: expr,
  t
  y
  p
  e
  :
  i
  d
  e
  n
  t
  ,
  [
  ($values: expr),])),)

@timsaucer @alamb ?

I recommend:

1. Switch to use the upstream arrow one
2. Deprecate (don't remove) the API in datafusion, per https://datafusion.apache.org/library-user-guide/api-health.html

Will do as recommended. Thanks @alamb

[buraksenn]

I've took another look into this and tried to change it but the issue is that macro in arrow only accepts array literals such as:
[1,2,3]. However, in the datafusion macro it accepts vectors. One reference of this macro is in this function:

    let batches = vec![
        create_record_batch(1, 5),
        create_record_batch(6, 1),
        create_record_batch(7, 5),
    ];

fn create_record_batch(start_value: i32, num_values: usize) -> RecordBatch {
    let end_value = start_value + num_values as i32;
    let a_vals: Vec<i32> = (start_value..end_value).collect();
    let b_vals: Vec<f64> = a_vals.iter().map(|v| *v as f64).collect();

    record_batch!(("a", Int32, a_vals), ("b", Float64, b_vals)).unwrap()
}

As far as I've tried, achieving the same is not possible with macro in arrow crate in the same way. That's why I think we can update the macro in arrow to the same in datafusion and then deprecate it here. But since I'm not Rust expert not sure on this.

Arrow: https://github.com/apache/arrow-rs/blob/c4dbf0d8af6ca5a19b8b2ea777da3c276807fc5e/arrow-array/src/record_batch.rs#L153-L174
Datafusion:

datafusion/datafusion/common/src/test_util.rs

Lines 338 to 357 in bfabd48

	macro_rules! record_batch {
	($(($name: expr, $type: ident, $values: expr)),*) => {
	{
	let schema = std::sync::Arc::new(arrow_schema::Schema::new(vec![
	$(
	arrow_schema::Field::new($name, arrow_schema::DataType::$type, true),
	)*
	]));
	let batch = arrow_array::RecordBatch::try_new(
	schema,
	vec![$(
	$crate::create_array!($type, $values),
	)*]
	);
	batch
	}
	}
	}

cc @timsaucer @alamb

[alamb]

Thanks for looking at this @buraksenn

I thought we more or less ported the macro from DataFusion to Arrow so the different is surprising, but I haven't had a chance to look into it in detail.

[ByteBaker]

@alamb @buraksenn I think that would be me, since I made the call to go with array literals than vec![] since the former avoids allocation. But now that I see it, should I make a patch in the upstream and change it there, or shall I make the change here directly? The syntax also felt a little more elegant this way so I stayed. Need your advice.

[alamb]

Thanks @ByteBaker -- I don't have any strong opinion / advice here unfortunately

[ByteBaker]

@alamb @buraksenn is this issue still on? Sorry I had been caught up last several months but now I'm able to contribute again.

If everyone can agree on one of the two paths

• Change arrow-rs to follow the same pattern as in datafusion
• Change datafusion to sync with arrow-rs' macro

I'll code them. If it's gonna be DF, I'll make necessary code sync and call changes as well.

[alamb]

I think you should change DataFusion to sync with arrow-rs macro (basically remove the datafusion macro and use the arrow one)

The rationale is that will reduce the downstream impact on other libraries