GH-40062: [C++][Python] Conversion of Table to Arrow Tensor by AlenkaF · Pull Request #41870 · apache/arrow

AlenkaF · 2024-05-29T09:25:25Z

Rationale for this change

There is currently no method to convert Arrow Table to Arrow Tensor (conversion from columnar format to a contiguous block of memory). This work is a continuation of RecordBatch::ToTensor work, see #40058.

What changes are included in this PR?

This PR:

implements Table::ToTensor conversion
adds bindings to Python
adds benchmarks in C++
removes the code in RecordBatch::ToTensor and uses the Table implementation (RecordBatch::ToTensor benchmarks checked)

Are these changes tested?

Yes, in C++ and Python.

Are there any user-facing changes?

No, it is a new feature.

GitHub Issue: [C++][Python] Conversion of Table to Arrow Tensor #40062

AlenkaF · 2024-05-29T15:55:11Z

Benchmarks for RecordBatch::ToTensor after the changeing the implementation to use Table::ToTensor:

(pyarrow-dev) alenkafrim@alenka-mac arrow % archery --quiet benchmark diff --benchmark-filter=BatchToTensorSimple
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Non-regressions: (7)
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
                                                  benchmark      baseline     contender  change %                                                                                                                                                                                                  counters
  BatchToTensorSimple<Int64Type>/size:4194304/num_columns:3 8.540 GiB/sec 8.826 GiB/sec     3.351   {'family_index': 3, 'per_family_instance_index': 3, 'run_name': 'BatchToTensorSimple<Int64Type>/size:4194304/num_columns:3', 'repetitions': 1, 'repetition_index': 0, 'threads': 1, 'iterations': 1545}
  BatchToTensorSimple<Int16Type>/size:4194304/num_columns:3 4.515 GiB/sec 4.583 GiB/sec     1.516    {'family_index': 1, 'per_family_instance_index': 3, 'run_name': 'BatchToTensorSimple<Int16Type>/size:4194304/num_columns:3', 'repetitions': 1, 'repetition_index': 0, 'threads': 1, 'iterations': 787}
 BatchToTensorSimple<Int64Type>/size:4194304/num_columns:30 5.355 GiB/sec 5.426 GiB/sec     1.320   {'family_index': 3, 'per_family_instance_index': 4, 'run_name': 'BatchToTensorSimple<Int64Type>/size:4194304/num_columns:30', 'repetitions': 1, 'repetition_index': 0, 'threads': 1, 'iterations': 971}
 BatchToTensorSimple<Int16Type>/size:4194304/num_columns:30 2.113 GiB/sec 2.120 GiB/sec     0.331   {'family_index': 1, 'per_family_instance_index': 4, 'run_name': 'BatchToTensorSimple<Int16Type>/size:4194304/num_columns:30', 'repetitions': 1, 'repetition_index': 0, 'threads': 1, 'iterations': 380}
BatchToTensorSimple<Int16Type>/size:4194304/num_columns:300 2.009 GiB/sec 1.976 GiB/sec    -1.620  {'family_index': 1, 'per_family_instance_index': 5, 'run_name': 'BatchToTensorSimple<Int16Type>/size:4194304/num_columns:300', 'repetitions': 1, 'repetition_index': 0, 'threads': 1, 'iterations': 363}
    BatchToTensorSimple<Int16Type>/size:65536/num_columns:3 5.391 GiB/sec 5.141 GiB/sec    -4.645    {'family_index': 1, 'per_family_instance_index': 0, 'run_name': 'BatchToTensorSimple<Int16Type>/size:65536/num_columns:3', 'repetitions': 1, 'repetition_index': 0, 'threads': 1, 'iterations': 61484}
BatchToTensorSimple<Int64Type>/size:4194304/num_columns:300 7.797 GiB/sec 7.429 GiB/sec    -4.716 {'family_index': 3, 'per_family_instance_index': 5, 'run_name': 'BatchToTensorSimple<Int64Type>/size:4194304/num_columns:300', 'repetitions': 1, 'repetition_index': 0, 'threads': 1, 'iterations': 1374}

--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Regressions: (17)
--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
                                                  benchmark        baseline       contender  change %                                                                                                                                                                                                 counters
 BatchToTensorSimple<Int8Type>/size:4194304/num_columns:300 698.025 MiB/sec 642.690 MiB/sec    -7.927  {'family_index': 0, 'per_family_instance_index': 5, 'run_name': 'BatchToTensorSimple<Int8Type>/size:4194304/num_columns:300', 'repetitions': 1, 'repetition_index': 0, 'threads': 1, 'iterations': 117}
  BatchToTensorSimple<Int8Type>/size:4194304/num_columns:30 936.761 MiB/sec 849.504 MiB/sec    -9.315   {'family_index': 0, 'per_family_instance_index': 4, 'run_name': 'BatchToTensorSimple<Int8Type>/size:4194304/num_columns:30', 'repetitions': 1, 'repetition_index': 0, 'threads': 1, 'iterations': 164}
 BatchToTensorSimple<Int32Type>/size:4194304/num_columns:30   2.943 GiB/sec   2.664 GiB/sec    -9.484  {'family_index': 2, 'per_family_instance_index': 4, 'run_name': 'BatchToTensorSimple<Int32Type>/size:4194304/num_columns:30', 'repetitions': 1, 'repetition_index': 0, 'threads': 1, 'iterations': 530}
   BatchToTensorSimple<Int8Type>/size:4194304/num_columns:3   1.220 GiB/sec   1.103 GiB/sec    -9.540    {'family_index': 0, 'per_family_instance_index': 3, 'run_name': 'BatchToTensorSimple<Int8Type>/size:4194304/num_columns:3', 'repetitions': 1, 'repetition_index': 0, 'threads': 1, 'iterations': 226}
BatchToTensorSimple<Int32Type>/size:4194304/num_columns:300   3.350 GiB/sec   3.004 GiB/sec   -10.308 {'family_index': 2, 'per_family_instance_index': 5, 'run_name': 'BatchToTensorSimple<Int32Type>/size:4194304/num_columns:300', 'repetitions': 1, 'repetition_index': 0, 'threads': 1, 'iterations': 603}
     BatchToTensorSimple<Int8Type>/size:65536/num_columns:3   1.343 GiB/sec   1.193 GiB/sec   -11.189    {'family_index': 0, 'per_family_instance_index': 0, 'run_name': 'BatchToTensorSimple<Int8Type>/size:65536/num_columns:3', 'repetitions': 1, 'repetition_index': 0, 'threads': 1, 'iterations': 15407}
  BatchToTensorSimple<Int32Type>/size:4194304/num_columns:3   6.492 GiB/sec   5.679 GiB/sec   -12.518  {'family_index': 2, 'per_family_instance_index': 3, 'run_name': 'BatchToTensorSimple<Int32Type>/size:4194304/num_columns:3', 'repetitions': 1, 'repetition_index': 0, 'threads': 1, 'iterations': 1170}
    BatchToTensorSimple<Int32Type>/size:65536/num_columns:3   8.703 GiB/sec   7.530 GiB/sec   -13.478   {'family_index': 2, 'per_family_instance_index': 0, 'run_name': 'BatchToTensorSimple<Int32Type>/size:65536/num_columns:3', 'repetitions': 1, 'repetition_index': 0, 'threads': 1, 'iterations': 99016}
    BatchToTensorSimple<Int64Type>/size:65536/num_columns:3  17.419 GiB/sec  14.934 GiB/sec   -14.269  {'family_index': 3, 'per_family_instance_index': 0, 'run_name': 'BatchToTensorSimple<Int64Type>/size:65536/num_columns:3', 'repetitions': 1, 'repetition_index': 0, 'threads': 1, 'iterations': 198847}
    BatchToTensorSimple<Int8Type>/size:65536/num_columns:30   1.246 GiB/sec   1.013 GiB/sec   -18.692   {'family_index': 0, 'per_family_instance_index': 1, 'run_name': 'BatchToTensorSimple<Int8Type>/size:65536/num_columns:30', 'repetitions': 1, 'repetition_index': 0, 'threads': 1, 'iterations': 14331}
   BatchToTensorSimple<Int16Type>/size:65536/num_columns:30   3.813 GiB/sec   3.045 GiB/sec   -20.148  {'family_index': 1, 'per_family_instance_index': 1, 'run_name': 'BatchToTensorSimple<Int16Type>/size:65536/num_columns:30', 'repetitions': 1, 'repetition_index': 0, 'threads': 1, 'iterations': 43240}
   BatchToTensorSimple<Int32Type>/size:65536/num_columns:30   5.497 GiB/sec   3.822 GiB/sec   -30.460  {'family_index': 2, 'per_family_instance_index': 1, 'run_name': 'BatchToTensorSimple<Int32Type>/size:65536/num_columns:30', 'repetitions': 1, 'repetition_index': 0, 'threads': 1, 'iterations': 63621}
   BatchToTensorSimple<Int8Type>/size:65536/num_columns:300 665.489 MiB/sec 452.284 MiB/sec   -32.037   {'family_index': 0, 'per_family_instance_index': 2, 'run_name': 'BatchToTensorSimple<Int8Type>/size:65536/num_columns:300', 'repetitions': 1, 'repetition_index': 0, 'threads': 1, 'iterations': 7122}
   BatchToTensorSimple<Int64Type>/size:65536/num_columns:30   7.306 GiB/sec   4.883 GiB/sec   -33.166  {'family_index': 3, 'per_family_instance_index': 1, 'run_name': 'BatchToTensorSimple<Int64Type>/size:65536/num_columns:30', 'repetitions': 1, 'repetition_index': 0, 'threads': 1, 'iterations': 83661}
  BatchToTensorSimple<Int16Type>/size:65536/num_columns:300   1.024 GiB/sec 646.927 MiB/sec   -38.317 {'family_index': 1, 'per_family_instance_index': 2, 'run_name': 'BatchToTensorSimple<Int16Type>/size:65536/num_columns:300', 'repetitions': 1, 'repetition_index': 0, 'threads': 1, 'iterations': 11642}
  BatchToTensorSimple<Int64Type>/size:65536/num_columns:300   1.208 GiB/sec 711.915 MiB/sec   -42.439 {'family_index': 3, 'per_family_instance_index': 2, 'run_name': 'BatchToTensorSimple<Int64Type>/size:65536/num_columns:300', 'repetitions': 1, 'repetition_index': 0, 'threads': 1, 'iterations': 13994}
  BatchToTensorSimple<Int32Type>/size:65536/num_columns:300   1.158 GiB/sec 678.147 MiB/sec   -42.812 {'family_index': 2, 'per_family_instance_index': 2, 'run_name': 'BatchToTensorSimple<Int32Type>/size:65536/num_columns:300', 'repetitions': 1, 'repetition_index': 0, 'threads': 1, 'iterations': 13406}

…to tensor.cc (#41932) ### Rationale for this change This is a precursor PR to #41870 with the purpose to make the review of #41870 easier (the diff of the code will be visible as it currently isn't because the code was moved to table.cc. I should also live in tensor.cc). ### What changes are included in this PR? The code from `RecordBatch::ToTensor` in record_batch.cc is moved to `RecordBatchToTensor` in tensor.cc. ### Are these changes tested? Existing tests should pass. ### Are there any user-facing changes? No. **This PR does not close the linked issue yet, it is just a precursor!** * GitHub Issue: #40062 Authored-by: AlenkaF <frim.alenka@gmail.com> Signed-off-by: AlenkaF <frim.alenka@gmail.com>

jorisvandenbossche

Generally looks good! Some minor comments, and wondering if we can reduce the duplication in testing a bit

AlenkaF · 2024-06-10T16:01:42Z

I have researched the benchmark regression a bit and found that:

running the benchmarks for RecordBatch::ToTensor shows up to 40% of change in time (regressions)
removing Table creation but keeping the code as is, hardcoding the for loop over the chunks to one iteration, makes the regression fall to maximum of 20%

benchmark diff output

(pyarrow-dev) alenkafrim@alenka-mac build % archery --quiet benchmark diff --benchmark-filter=ToTensorSimple
----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Non-regressions: (7)
----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
                                                benchmark       baseline      contender  change %                                                                                                                                                                                                 counters
 BatchToTensorSimple<Int64Type>/size:65536/num_columns:30  7.321 GiB/sec  7.341 GiB/sec     0.275  {'family_index': 3, 'per_family_instance_index': 1, 'run_name': 'BatchToTensorSimple<Int64Type>/size:65536/num_columns:30', 'repetitions': 1, 'repetition_index': 0, 'threads': 1, 'iterations': 84665}
  BatchToTensorSimple<Int64Type>/size:65536/num_columns:3 17.341 GiB/sec 17.385 GiB/sec     0.256  {'family_index': 3, 'per_family_instance_index': 0, 'run_name': 'BatchToTensorSimple<Int64Type>/size:65536/num_columns:3', 'repetitions': 1, 'repetition_index': 0, 'threads': 1, 'iterations': 197830}
BatchToTensorSimple<Int32Type>/size:65536/num_columns:300  1.153 GiB/sec  1.136 GiB/sec    -1.413 {'family_index': 2, 'per_family_instance_index': 2, 'run_name': 'BatchToTensorSimple<Int32Type>/size:65536/num_columns:300', 'repetitions': 1, 'repetition_index': 0, 'threads': 1, 'iterations': 13151}
BatchToTensorSimple<Int64Type>/size:65536/num_columns:300  1.221 GiB/sec  1.198 GiB/sec    -1.838 {'family_index': 3, 'per_family_instance_index': 2, 'run_name': 'BatchToTensorSimple<Int64Type>/size:65536/num_columns:300', 'repetitions': 1, 'repetition_index': 0, 'threads': 1, 'iterations': 13997}
BatchToTensorSimple<Int16Type>/size:65536/num_columns:300  1.027 GiB/sec  1.005 GiB/sec    -2.092 {'family_index': 1, 'per_family_instance_index': 2, 'run_name': 'BatchToTensorSimple<Int16Type>/size:65536/num_columns:300', 'repetitions': 1, 'repetition_index': 0, 'threads': 1, 'iterations': 11502}
 BatchToTensorSimple<Int16Type>/size:65536/num_columns:30  3.824 GiB/sec  3.728 GiB/sec    -2.521  {'family_index': 1, 'per_family_instance_index': 1, 'run_name': 'BatchToTensorSimple<Int16Type>/size:65536/num_columns:30', 'repetitions': 1, 'repetition_index': 0, 'threads': 1, 'iterations': 43449}
BatchToTensorSimple<Int16Type>/size:4194304/num_columns:3  4.435 GiB/sec  4.322 GiB/sec    -2.550   {'family_index': 1, 'per_family_instance_index': 3, 'run_name': 'BatchToTensorSimple<Int16Type>/size:4194304/num_columns:3', 'repetitions': 1, 'repetition_index': 0, 'threads': 1, 'iterations': 792}

----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Regressions: (17)
----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
                                                  benchmark        baseline        contender  change %                                                                                                                                                                                                  counters
 BatchToTensorSimple<Int64Type>/size:4194304/num_columns:30   5.354 GiB/sec    5.078 GiB/sec    -5.159   {'family_index': 3, 'per_family_instance_index': 4, 'run_name': 'BatchToTensorSimple<Int64Type>/size:4194304/num_columns:30', 'repetitions': 1, 'repetition_index': 0, 'threads': 1, 'iterations': 959}
    BatchToTensorSimple<Int32Type>/size:65536/num_columns:3   8.656 GiB/sec    8.107 GiB/sec    -6.348    {'family_index': 2, 'per_family_instance_index': 0, 'run_name': 'BatchToTensorSimple<Int32Type>/size:65536/num_columns:3', 'repetitions': 1, 'repetition_index': 0, 'threads': 1, 'iterations': 96401}
BatchToTensorSimple<Int64Type>/size:4194304/num_columns:300   7.884 GiB/sec    7.371 GiB/sec    -6.506 {'family_index': 3, 'per_family_instance_index': 5, 'run_name': 'BatchToTensorSimple<Int64Type>/size:4194304/num_columns:300', 'repetitions': 1, 'repetition_index': 0, 'threads': 1, 'iterations': 1140}
 BatchToTensorSimple<Int16Type>/size:4194304/num_columns:30   2.109 GiB/sec    1.969 GiB/sec    -6.655   {'family_index': 1, 'per_family_instance_index': 4, 'run_name': 'BatchToTensorSimple<Int16Type>/size:4194304/num_columns:30', 'repetitions': 1, 'repetition_index': 0, 'threads': 1, 'iterations': 378}
BatchToTensorSimple<Int16Type>/size:4194304/num_columns:300   2.007 GiB/sec    1.869 GiB/sec    -6.878  {'family_index': 1, 'per_family_instance_index': 5, 'run_name': 'BatchToTensorSimple<Int16Type>/size:4194304/num_columns:300', 'repetitions': 1, 'repetition_index': 0, 'threads': 1, 'iterations': 360}
   BatchToTensorSimple<Int32Type>/size:65536/num_columns:30   5.514 GiB/sec    5.116 GiB/sec    -7.218   {'family_index': 2, 'per_family_instance_index': 1, 'run_name': 'BatchToTensorSimple<Int32Type>/size:65536/num_columns:30', 'repetitions': 1, 'repetition_index': 0, 'threads': 1, 'iterations': 62798}
BatchToTensorSimple<Int32Type>/size:4194304/num_columns:300   3.346 GiB/sec    3.066 GiB/sec    -8.379  {'family_index': 2, 'per_family_instance_index': 5, 'run_name': 'BatchToTensorSimple<Int32Type>/size:4194304/num_columns:300', 'repetitions': 1, 'repetition_index': 0, 'threads': 1, 'iterations': 601}
   BatchToTensorSimple<Int8Type>/size:65536/num_columns:300 669.230 MiB/sec  598.420 MiB/sec   -10.581    {'family_index': 0, 'per_family_instance_index': 2, 'run_name': 'BatchToTensorSimple<Int8Type>/size:65536/num_columns:300', 'repetitions': 1, 'repetition_index': 0, 'threads': 1, 'iterations': 7493}
    BatchToTensorSimple<Int16Type>/size:65536/num_columns:3   5.393 GiB/sec    4.745 GiB/sec   -12.015    {'family_index': 1, 'per_family_instance_index': 0, 'run_name': 'BatchToTensorSimple<Int16Type>/size:65536/num_columns:3', 'repetitions': 1, 'repetition_index': 0, 'threads': 1, 'iterations': 61699}
 BatchToTensorSimple<Int8Type>/size:4194304/num_columns:300 700.642 MiB/sec  611.987 MiB/sec   -12.653   {'family_index': 0, 'per_family_instance_index': 5, 'run_name': 'BatchToTensorSimple<Int8Type>/size:4194304/num_columns:300', 'repetitions': 1, 'repetition_index': 0, 'threads': 1, 'iterations': 123}
    BatchToTensorSimple<Int8Type>/size:65536/num_columns:30   1.247 GiB/sec    1.075 GiB/sec   -13.836    {'family_index': 0, 'per_family_instance_index': 1, 'run_name': 'BatchToTensorSimple<Int8Type>/size:65536/num_columns:30', 'repetitions': 1, 'repetition_index': 0, 'threads': 1, 'iterations': 14200}
  BatchToTensorSimple<Int32Type>/size:4194304/num_columns:3   6.465 GiB/sec    5.567 GiB/sec   -13.879   {'family_index': 2, 'per_family_instance_index': 3, 'run_name': 'BatchToTensorSimple<Int32Type>/size:4194304/num_columns:3', 'repetitions': 1, 'repetition_index': 0, 'threads': 1, 'iterations': 1156}
  BatchToTensorSimple<Int8Type>/size:4194304/num_columns:30 938.704 MiB/sec  792.766 MiB/sec   -15.547    {'family_index': 0, 'per_family_instance_index': 4, 'run_name': 'BatchToTensorSimple<Int8Type>/size:4194304/num_columns:30', 'repetitions': 1, 'repetition_index': 0, 'threads': 1, 'iterations': 164}
 BatchToTensorSimple<Int32Type>/size:4194304/num_columns:30   2.944 GiB/sec    2.453 GiB/sec   -16.660   {'family_index': 2, 'per_family_instance_index': 4, 'run_name': 'BatchToTensorSimple<Int32Type>/size:4194304/num_columns:30', 'repetitions': 1, 'repetition_index': 0, 'threads': 1, 'iterations': 529}
  BatchToTensorSimple<Int64Type>/size:4194304/num_columns:3   8.618 GiB/sec    7.157 GiB/sec   -16.959   {'family_index': 3, 'per_family_instance_index': 3, 'run_name': 'BatchToTensorSimple<Int64Type>/size:4194304/num_columns:3', 'repetitions': 1, 'repetition_index': 0, 'threads': 1, 'iterations': 1521}
   BatchToTensorSimple<Int8Type>/size:4194304/num_columns:3   1.197 GiB/sec 1008.475 MiB/sec   -17.748     {'family_index': 0, 'per_family_instance_index': 3, 'run_name': 'BatchToTensorSimple<Int8Type>/size:4194304/num_columns:3', 'repetitions': 1, 'repetition_index': 0, 'threads': 1, 'iterations': 227}
     BatchToTensorSimple<Int8Type>/size:65536/num_columns:3   1.314 GiB/sec    1.057 GiB/sec   -19.601     {'family_index': 0, 'per_family_instance_index': 0, 'run_name': 'BatchToTensorSimple<Int8Type>/size:65536/num_columns:3', 'repetitions': 1, 'repetition_index': 0, 'threads': 1, 'iterations': 15032}

Plan to also try profiling in python (py-spy doesn't work on MacOS, any other suggestions maybe?). Update: installed py-spy with brew and it works, looking at the .svg at the moment.

jorisvandenbossche · 2024-06-11T07:06:54Z

I have researched the benchmark regression a bit and found that:

Do you see those regressions of up to 40% for both row major and column major conversions? And both for uniform vs mixed type with casting?

AlenkaF · 2024-06-11T07:15:14Z

Do you see those regressions of up to 40% for both row major and column major conversions? And both for uniform vs mixed type with casting?

Benchmarks for RecordBatch only test row-major conversion. The newly added Table benchmarks test both. I think that was due to the fact we were adding features for RecordBatch::ToTensor step by step and we needed one simple benchmark that we could check while adding the features. Row-major conversion was the last to be added.

As for the types, we only test uniform types in C++ benchmarks at the moment.

ps: haven't been able to find extract any information with neither py-spy nor cProfile.

thisisnic · 2025-11-18T12:30:39Z

Thank you for your contribution. Unfortunately, this
pull request has been marked as stale because it has had no activity in the past 365 days. Please remove the stale label
or comment below, or this PR will be closed in 14 days. Feel free to re-open this if it has been closed in error. If you
do not have repository permissions to reopen the PR, please tag a maintainer.

Co-authored-by: Rok Mihevc <rok@mihevc.org>

Copilot

Pull request overview

Copilot reviewed 12 out of 12 changed files in this pull request and generated 5 comments.

Copilot

Pull request overview

Copilot reviewed 12 out of 12 changed files in this pull request and generated 1 comment.

raulcd · 2026-07-01T07:26:26Z

@github-actions crossbow submit wheel-cp314

github-actions · 2026-07-01T07:29:04Z

Revision: c821847

Submitted crossbow builds: ursacomputing/crossbow @ actions-436d7bdb5c

Task	Status
wheel-macos-monterey-cp314-cp314-amd64
wheel-macos-monterey-cp314-cp314-arm64
wheel-macos-monterey-cp314-cp314t-amd64
wheel-macos-monterey-cp314-cp314t-arm64
wheel-manylinux-2-28-cp314-cp314-amd64
wheel-manylinux-2-28-cp314-cp314-arm64
wheel-manylinux-2-28-cp314-cp314t-amd64
wheel-manylinux-2-28-cp314-cp314t-arm64
wheel-musllinux-1-2-cp314-cp314-amd64
wheel-musllinux-1-2-cp314-cp314-arm64
wheel-musllinux-1-2-cp314-cp314t-amd64
wheel-musllinux-1-2-cp314-cp314t-arm64
wheel-windows-cp314-cp314-amd64
wheel-windows-cp314-cp314t-amd64

raulcd · 2026-07-01T07:35:13Z

@ursabot please benchmark

rok · 2026-07-01T07:35:19Z

Benchmark runs are scheduled for commit 7d2e68f. Watch https://buildkite.com/apache-arrow and https://conbench.arrow-dev.org for updates. A comment will be posted here when the runs are complete.

Copilot

Pull request overview

Copilot reviewed 12 out of 12 changed files in this pull request and generated no new comments.

AlenkaF · 2026-07-01T09:10:50Z

I have applied all code suggestions, thank you @tadeja and @rok for the reviews, much much appreciated!
As for the Copilot review, I have included a change from one comment that dealt with possible overflow on buffer allocation step.

Job failures do not seem connected.

rok

This looks good now, let's merge!

rok · 2026-07-01T12:59:26Z

Great to see this land!

conbench-apache-arrow · 2026-07-01T14:05:55Z

Thanks for your patience. Conbench analyzed the 4 benchmarking runs that have been run so far on PR commit 7d2e68f.

There were 19 benchmark results indicating a performance regression:

Pull Request Run on amd64-c6a-4xlarge-linux at 2026-07-01 08:47:27Z
- BatchToTensorSimple (C++) with params=/size:524288/num_columns:3, source=cpp-micro, suite=arrow-tensor-benchmark
- BatchToTensorSimple (C++) with params=/size:32768/num_columns:30, source=cpp-micro, suite=arrow-tensor-benchmark
and 17 more (see the report linked below)

The full Conbench report has more details.

conbench-apache-arrow · 2026-07-02T15:20:34Z

After merging your PR, Conbench analyzed the 2 benchmarking runs that have been run so far on merge-commit ca37093.

There were no benchmark performance regressions. 🎉

The full Conbench report has more details.

rok · 2026-07-02T16:56:22Z

Yaaay no regressions on the final commit! I've restarted benchmarks for the other two machines just in case, let's see what happens.

AlenkaF · 2026-07-03T13:03:33Z

Oh, that is great! I wasn't really sure what to do with the first result (#41870 (comment))! 🤞

rok · 2026-07-03T13:11:56Z

Huh, this is interesting. I'm not sure how to interpret but there are changes. Some data types appear to have better results on others worse results. why would this be.

AlenkaF · 2026-07-03T15:55:38Z

Hm, there has been a change visible for me locally also. But on average the result should be similar. I did the research then but don't remember exactly what was the reason and whatever I write now will be nonsense =) Will do the research again and paste here the likely reason for this.

The only connected comment from the PR I found is this one: #41870 (comment)

github-actions Bot added Component: C++ Component: Python awaiting review Awaiting review labels May 29, 2024

AlenkaF mentioned this pull request Jun 3, 2024

GH-40062: [C++] Move RecordBatch::ToTensor code from record_batch.cc to tensor.cc #41932

Merged

AlenkaF force-pushed the gh-40062-table-to-tensor branch from 13c49a7 to 15574f8 Compare June 5, 2024 12:51

jorisvandenbossche reviewed Jun 6, 2024

View reviewed changes

Comment thread cpp/src/arrow/table.h Outdated

Comment thread cpp/src/arrow/tensor.cc Outdated

Comment thread cpp/src/arrow/tensor.cc Outdated

Comment thread python/pyarrow/table.pxi Outdated

Comment thread python/pyarrow/tests/test_table.py Outdated

github-actions Bot added awaiting changes Awaiting changes awaiting review Awaiting review and removed awaiting review Awaiting review labels Jun 6, 2024

AlenkaF force-pushed the gh-40062-table-to-tensor branch from 15574f8 to 4decf7f Compare June 10, 2024 15:56

github-actions Bot added awaiting review Awaiting review and removed awaiting review Awaiting review awaiting changes Awaiting changes labels Jun 10, 2024

github-actions Bot added awaiting committer review Awaiting committer review and removed awaiting review Awaiting review labels Jun 11, 2024

jorisvandenbossche reviewed Jun 11, 2024

View reviewed changes

Comment thread cpp/src/arrow/table.h Outdated

Comment thread python/pyarrow/table.pxi Outdated

github-actions Bot added awaiting changes Awaiting changes and removed awaiting committer review Awaiting committer review labels Jun 11, 2024

github-actions Bot added awaiting change review Awaiting change review and removed awaiting changes Awaiting changes labels Jun 11, 2024

github-actions Bot added the Status: stale-warning Issues and PRs flagged as stale which are due to be closed if no indication otherwise label Nov 18, 2025

AlenkaF removed the Status: stale-warning Issues and PRs flagged as stale which are due to be closed if no indication otherwise label Nov 19, 2025

Add Table::ToTensor and bindings to Python with Python tests

c8691ab

Copilot started reviewing on behalf of AlenkaF July 1, 2026 06:53 View session

AlenkaF and others added 2 commits July 1, 2026 09:01

Apply suggestions from code review

8f1d593

Co-authored-by: Rok Mihevc <rok@mihevc.org>

Apply suggestion from @rok

c821847

Co-authored-by: Rok Mihevc <rok@mihevc.org>

Copilot AI reviewed Jul 1, 2026

View reviewed changes

Comment thread cpp/src/arrow/tensor.cc

Comment thread python/pyarrow/tests/test_table.py

Comment thread cpp/src/arrow/tensor.cc

Comment thread cpp/src/arrow/tensor.h

Comment thread cpp/src/arrow/tensor.cc

Copilot AI review requested due to automatic review settings July 1, 2026 07:10

Copilot started reviewing on behalf of AlenkaF July 1, 2026 07:11 View session

Copilot AI reviewed Jul 1, 2026

View reviewed changes

Comment thread cpp/src/arrow/tensor.cc Outdated

Run linter

7d2e68f

Use internal::MultiplyWithOverflow when allocating buffer

81bb077

Copilot AI review requested due to automatic review settings July 1, 2026 08:11

Copilot started reviewing on behalf of AlenkaF July 1, 2026 08:12 View session

Copilot AI reviewed Jul 1, 2026

View reviewed changes

rok approved these changes Jul 1, 2026

View reviewed changes

AlenkaF merged commit ca37093 into apache:main Jul 1, 2026
67 of 69 checks passed

AlenkaF removed the awaiting merge Awaiting merge label Jul 1, 2026

AlenkaF deleted the gh-40062-table-to-tensor branch July 1, 2026 12:23

AlenkaF mentioned this pull request Jul 1, 2026

[C++][Python] Conversion of Table to Arrow Tensor #40062

Closed

AlenkaF mentioned this pull request Jul 1, 2026

[C++][Python] Row-major conversion of Table/RecordBatch to Arrow Tensor #40063

Closed

Uh oh!

Conversation

AlenkaF commented May 29, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Rationale for this change

What changes are included in this PR?

Are these changes tested?

Are there any user-facing changes?

Uh oh!

AlenkaF commented May 29, 2024

Uh oh!

jorisvandenbossche left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

AlenkaF commented Jun 10, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Uh oh!

jorisvandenbossche commented Jun 11, 2024

Uh oh!

AlenkaF commented Jun 11, 2024

Uh oh!

thisisnic commented Nov 18, 2025

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

raulcd commented Jul 1, 2026

Uh oh!

github-actions Bot commented Jul 1, 2026

Uh oh!

raulcd commented Jul 1, 2026

Uh oh!

rok commented Jul 1, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

AlenkaF commented Jul 1, 2026

Uh oh!

rok left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

rok commented Jul 1, 2026

Uh oh!

conbench-apache-arrow Bot commented Jul 1, 2026

Uh oh!

conbench-apache-arrow Bot commented Jul 2, 2026

Uh oh!

rok commented Jul 2, 2026

Uh oh!

AlenkaF commented Jul 3, 2026

Uh oh!

rok commented Jul 3, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

AlenkaF commented Jul 3, 2026

Uh oh!

Reviewers

Assignees

Labels

AlenkaF commented May 29, 2024 •

edited

Loading

AlenkaF commented Jun 10, 2024 •

edited

Loading

rok commented Jul 3, 2026 •

edited

Loading