GH-45523: [R] Implement Utf8View type bindings by thisisnic · Pull Request #49712 · apache/arrow

thisisnic · 2026-04-11T18:15:28Z

Rationale for this change

No bindings for Utf8View type in the R package

What changes are included in this PR?

Implement bindings

Are these changes tested?

Yep

Are there any user-facing changes?

Yep, adding functionality.

AI Usage

Heavily used Codex/Claude here. I'm not confident of every line of code. I read things over, and iterated on it making sure that tests pass and nothing seemed wildly incorrect.

GitHub Issue: [R] An "cannot handle Array of type <utf8_view>" error occurs when reading ipc file include strings output by r-polars #45523

thisisnic · 2026-04-11T18:17:20Z

We should rebase once #49710 is merged as changes from that PR are in this branch as they were needed to make it work.

thisisnic · 2026-05-06T20:49:12Z

+      return this->value_builder_->Append(
+          std::string_view(view_.bytes, static_cast<size_t>(view_.size)));


The old code passed (const char*, int32_t) which matches StringBuilder::Append but not StringViewBuilder::Append (which takes int64_t). Switching to std::string_view works for both builder types.

thisisnic · 2026-05-06T20:52:43Z

A lot of .Rd updates as I used a newer roxygen2 version.

Copilot

Pull request overview

This PR adds R bindings and conversion support for Arrow utf8_view / string_view, addressing IPC/table conversion failures for data containing StringView columns.

Changes:

Adds string_view() R data type binding, export registration, and type tests.
Implements R↔Arrow conversion paths for StringView arrays and dictionary values.
Updates generated documentation and related converter support in C++/Python.

Reviewed changes

Copilot reviewed 13 out of 23 changed files in this pull request and generated 2 comments.

Show a summary per file

File	Description
r/R/type.R	Adds `StringView` R6 type, constructor, and canonical type aliases.
r/R/arrowExports.R	Registers R wrapper for `StringView__initialize`.
r/NAMESPACE	Exports `string_view`.
r/src/datatype.cpp	Maps Arrow `STRING_VIEW` to R `StringView` and initializes `utf8_view`.
r/src/r_to_arrow.cpp	Adds R-to-Arrow StringView conversion and dictionary StringView value handling.
r/src/array_to_vector.cpp	Adds Arrow-to-R StringView and wider dictionary index conversion support.
r/src/arrowExports.cpp	Registers native StringView initialization entry point.
cpp/src/arrow/util/converter.h	Enables dictionary converters for `StringViewType`.
python/pyarrow/src/arrow/python/python_to_arrow.cc	Adjusts Python dictionary StringView append call.
r/tests/testthat/test-Array.R	Adds StringView array round-trip tests.
r/tests/testthat/test-Table.R	Adds table/dictionary StringView and wider index tests.
r/tests/testthat/test-data-type.R	Adds StringView data type and code round-trip tests.
r/DESCRIPTION	Updates roxygen metadata.
r/man/data-type.Rd	Adds `string_view()` documentation entry.
r/man/acero.Rd	Regenerated Acero documentation.
r/man/arrow-package.Rd	Regenerated package author documentation.
r/man/csv_convert_options.Rd	Regenerated CSV conversion docs.
r/man/csv_read_options.Rd	Regenerated CSV read docs.
r/man/CsvReadOptions.Rd	Regenerated CSV read options docs.
r/man/enums.Rd	Regenerated enum documentation.
r/man/JsonFileFormat.Rd	Regenerated JSON file format docs.
r/man/reexports.Rd	Regenerated reexports docs.
r/man/vctrs_extension_array.Rd	Regenerated vctrs extension docs.

Files not reviewed (10)

r/man/CsvReadOptions.Rd: Language not supported
r/man/JsonFileFormat.Rd: Language not supported
r/man/acero.Rd: Language not supported
r/man/arrow-package.Rd: Language not supported
r/man/csv_convert_options.Rd: Language not supported
r/man/csv_read_options.Rd: Language not supported
r/man/data-type.Rd: Language not supported
r/man/enums.Rd: Language not supported
r/man/reexports.Rd: Language not supported
r/man/vctrs_extension_array.Rd: Language not supported

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Copilot

Pull request overview

Copilot reviewed 13 out of 23 changed files in this pull request and generated 1 comment.

Files not reviewed (10)

r/man/CsvReadOptions.Rd: Language not supported
r/man/JsonFileFormat.Rd: Language not supported
r/man/acero.Rd: Language not supported
r/man/arrow-package.Rd: Language not supported
r/man/csv_convert_options.Rd: Language not supported
r/man/csv_read_options.Rd: Language not supported
r/man/data-type.Rd: Language not supported
r/man/enums.Rd: Language not supported
r/man/reexports.Rd: Language not supported
r/man/vctrs_extension_array.Rd: Language not supported

…ings

Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>

Copilot

Pull request overview

Copilot reviewed 12 out of 13 changed files in this pull request and generated no new comments.

Files not reviewed (1)

r/man/data-type.Rd: Language not supported

thisisnic · 2026-06-18T15:00:31Z

Current CI failure is unrelated; #50219

thisisnic · 2026-06-18T15:02:24Z

Lots of rounds with Claude, seems to make sense! I tested the code from the original issue and it works with this PR.

Copilot

Pull request overview

Copilot reviewed 12 out of 13 changed files in this pull request and generated 2 comments.

Files not reviewed (1)

r/man/data-type.Rd: Generated file

Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>

Copilot

Pull request overview

Copilot reviewed 12 out of 13 changed files in this pull request and generated 1 comment.

Files not reviewed (1)

r/man/data-type.Rd: Generated file

Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>

Copilot

Pull request overview

Copilot reviewed 12 out of 13 changed files in this pull request and generated no new comments.

Files not reviewed (1)

r/man/data-type.Rd: Generated file

pitrou

Thanks for doing this @thisisnic . Some comments below, but the main question is: can we also add support for BinaryView? It should be relatively easy given all the work already done here.

pitrou · 2026-06-22T08:38:53Z

      DICTIONARY_CASE(LargeBinaryType);
      DICTIONARY_CASE(StringType);
      DICTIONARY_CASE(LargeStringType);
+      DICTIONARY_CASE(StringViewType);


Why not also BinaryViewType?

pitrou · 2026-06-22T08:41:05Z

  )
 )
+StringView <- R6Class(
+  "StringView",


Is there a reason for calling it StringView instead of Utf8View (like the existing Utf8 and LargeUtf8)?

pitrou · 2026-06-22T08:41:25Z

    code = function(namespace = FALSE) call2("binary", .ns = if (namespace) "arrow")
  )
 )
 LargeBinary <- R6Class(


Why not also add BinaryView here?

pitrou · 2026-06-22T08:42:37Z

+    // StringViewArray uses a different memory layout (views + data buffers) rather
+    // than offsets, so skip the offset-based fast path and fall through to the
+    // GetView()-based element loop below.
+    if (array->type_id() != Type::STRING_VIEW) {


Can use is_binary_view_like(array->type_id()) to also match BinaryView.

pitrou · 2026-06-22T08:43:05Z

+    // than offsets, so skip the offset-based fast path and fall through to the
+    // GetView()-based element loop below.
+    if (array->type_id() != Type::STRING_VIEW) {
+      auto p_offset = array->data()->GetValues<int32_t>(1);


So this doesn't work for LargeBinary and LargeString, which have 64-bit offsets, right?

pitrou · 2026-06-22T08:51:28Z

+template <typename T>
+struct RConverterTrait<T, enable_if_t<is_binary_view_like_type<T>::value &&
+                                      !is_string_view_type<T>::value>> {
  // not implemented


Why not implement it too? It should be reasonably easy given you already the implementations for Binary and StringView.

pitrou · 2026-06-22T08:52:34Z


+template <typename T>
+class RPrimitiveConverter<T, enable_if_string_view<T>>
+    : public PrimitiveConverter<T, RConverter> {


It seems there's a lot in common between this and the regular String converter (only UnsafeAppendUtf8Strings differs AFAICT), perhaps it's worth factoring things out in a common base class?

pitrou · 2026-06-22T08:54:00Z

  expect_array_roundtrip(c("itsy", NA, "spider"), utf8())
  expect_array_roundtrip(c("itsy", NA, "spider"), large_utf8(), as = large_utf8())
+
+  expect_array_roundtrip(c("itsy", NA, "", "spider"), string_view(), as = string_view())


String view arrays have a different representation for "short" and "long" strings (i.e. > 12 characters), I suggest exercising that:

Suggested change

expect_array_roundtrip(c("itsy", NA, "", "spider"), string_view(), as = string_view())

expect_array_roundtrip(c("itsy", NA, "", "spider"), string_view(), as = string_view())

expect_array_roundtrip(c("itsy", NA, "", "a long non-inlined string", "another long string"), string_view(), as = string_view())

pitrou · 2026-06-22T08:55:20Z

  expect_equal_data_frame(tab_large, fact)
 })

+test_that("Table converts dictionary arrays with wider index types back to R", {


This test does not seem related, is it just adding a test that was overlooked before?

pitrou · 2026-06-22T08:55:41Z

+})
+
+test_that("Table converts dictionary arrays with string_view values", {
+  expected <- data.frame(foo = factor(c("x", NA, "x")))


Would also suggest testing with longer strings here.

github-actions Bot added Component: R Component: C++ awaiting committer review Awaiting committer review labels Apr 11, 2026

thisisnic mentioned this pull request Apr 30, 2026

GH-39603: [R] Error: Cannot convert Dictionary Array of type dictionary<values=large_string, indices=uint32, ordered=0> to R #49710

Merged

thisisnic force-pushed the GH-45523-ipc-polars branch 2 times, most recently from e8768d0 to bb711a7 Compare May 5, 2026 16:41

github-actions Bot added Component: Python awaiting changes Awaiting changes and removed awaiting committer review Awaiting committer review labels May 5, 2026

thisisnic commented May 6, 2026

View reviewed changes

Comment thread r/src/r_to_arrow.cpp

Copilot AI review requested due to automatic review settings May 28, 2026 09:37

thisisnic force-pushed the GH-45523-ipc-polars branch from de59365 to cf9ad10 Compare May 28, 2026 09:37

Copilot started reviewing on behalf of thisisnic May 28, 2026 09:37 View session

github-actions Bot added awaiting change review Awaiting change review and removed awaiting changes Awaiting changes labels May 28, 2026

Copilot AI reviewed May 28, 2026

View reviewed changes

Comment thread r/src/r_to_arrow.cpp Outdated

Comment thread r/man/acero.Rd Outdated

Copilot AI review requested due to automatic review settings May 28, 2026 12:11

Copilot started reviewing on behalf of thisisnic May 28, 2026 12:11 View session

Copilot AI reviewed May 28, 2026

View reviewed changes

Comment thread r/src/r_to_arrow.cpp Outdated

thisisnic added 8 commits June 10, 2026 15:59

Allow dictionaries from a wider range of types for indices

94275d7

Implement string view

dfdad81

revert unwanted docs change

52fc2b2

add context

3af2dd0

expand tests

7f6b26a

Fix offset bug

887beda

Fix DictionaryBuilder<StringViewType> Append signature in Python bind…

3c6820e

…ings

Fix DictionaryBuilder::Append for StringViewType in Python bindings

fc6dec6

thisisnic and others added 2 commits June 10, 2026 16:07

Fix docs build issue

21e7014

Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>

Run make doc

1842e7e

thisisnic force-pushed the GH-45523-ipc-polars branch from a74c9c1 to 1842e7e Compare June 10, 2026 15:10

Simplify tests

2b56912

Copilot AI review requested due to automatic review settings June 10, 2026 15:18

Copilot started reviewing on behalf of thisisnic June 10, 2026 15:18 View session

Simplify tests further

74c32cc

Copilot AI reviewed Jun 10, 2026

View reviewed changes

thisisnic added 2 commits June 18, 2026 14:33

appease linter

c3b63c7

Simplify code as per Codex feedback

ee58f8e

thisisnic marked this pull request as ready for review June 18, 2026 15:01

thisisnic requested a review from jonkeane as a code owner June 18, 2026 15:01

Copilot AI review requested due to automatic review settings June 18, 2026 15:01

thisisnic requested a review from pitrou as a code owner June 18, 2026 15:01

Copilot started reviewing on behalf of thisisnic June 18, 2026 15:01 View session

Copilot AI reviewed Jun 18, 2026

View reviewed changes

Comment thread python/pyarrow/src/arrow/python/python_to_arrow.cc

Comment thread r/src/array_to_vector.cpp Outdated

Potential fix for pull request finding

809f578

Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>

Copilot AI review requested due to automatic review settings June 18, 2026 16:30

Copilot started reviewing on behalf of thisisnic June 18, 2026 16:31 View session

Copilot AI reviewed Jun 18, 2026

View reviewed changes

Comment thread r/src/array_to_vector.cpp

Potential fix for pull request finding

d095d1b

Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>

Copilot AI review requested due to automatic review settings June 18, 2026 16:43

Copilot started reviewing on behalf of thisisnic June 18, 2026 16:43 View session

Copilot AI reviewed Jun 18, 2026

View reviewed changes

pitrou reviewed Jun 22, 2026

View reviewed changes

		return this->value_builder_->Append(
		std::string_view(view_.bytes, static_cast<size_t>(view_.size)));

	expect_array_roundtrip(c("itsy", NA, "", "spider"), string_view(), as = string_view())
	expect_array_roundtrip(c("itsy", NA, "", "spider"), string_view(), as = string_view())
	expect_array_roundtrip(c("itsy", NA, "", "a long non-inlined string", "another long string"), string_view(), as = string_view())

Conversation

thisisnic commented Apr 11, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Rationale for this change

What changes are included in this PR?

Are these changes tested?

Are there any user-facing changes?

AI Usage

Uh oh!

thisisnic commented Apr 11, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

thisisnic commented May 6, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

thisisnic commented Jun 18, 2026

Uh oh!

thisisnic commented Jun 18, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

pitrou left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

thisisnic commented Apr 11, 2026 •

edited

Loading