Skip to content
Merged
Show file tree
Hide file tree
Changes from 4 commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 3 additions & 2 deletions parquet-variant-compute/benches/variant_kernels.rs
Original file line number Diff line number Diff line change
Expand Up @@ -19,8 +19,9 @@ use arrow::array::{Array, ArrayRef, StringArray};
use arrow::util::test_util::seedable_rng;
use criterion::{Criterion, criterion_group, criterion_main};
use parquet_variant::{Variant, VariantBuilder};
use parquet_variant_compute::variant_get::{GetOptions, variant_get};
use parquet_variant_compute::{VariantArray, VariantArrayBuilder, json_to_variant};
use parquet_variant_compute::{
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I moved all the functions into the main parquet_variant_compute module

GetOptions, VariantArray, VariantArrayBuilder, json_to_variant, variant_get,
};
use rand::Rng;
use rand::SeedableRng;
use rand::distr::Alphanumeric;
Expand Down
2 changes: 1 addition & 1 deletion parquet-variant-compute/src/cast_to_variant.rs
Original file line number Diff line number Diff line change
Expand Up @@ -34,7 +34,7 @@ use arrow_schema::ArrowError;
/// ```
/// # use arrow::array::{Array, ArrayRef, Int64Array};
/// # use parquet_variant::Variant;
/// # use parquet_variant_compute::cast_to_variant::cast_to_variant;
/// # use parquet_variant_compute::cast_to_variant;
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

likewise, I removed this extra level of namespacing

/// // input is an Int64Array, which will be cast to a VariantArray
/// let input = Int64Array::from(vec![Some(1), None, Some(3)]);
/// let result = cast_to_variant(&input).unwrap();
Expand Down
17 changes: 11 additions & 6 deletions parquet-variant-compute/src/lib.rs
Original file line number Diff line number Diff line change
Expand Up @@ -20,10 +20,14 @@
//! ## Main APIs
//! - [`VariantArray`] : Represents an array of `Variant` values.
//! - [`VariantArrayBuilder`]: For building [`VariantArray`]
//! - [`json_to_variant`]: Function to convert a batch of JSON strings to a `VariantArray`.
//! - [`variant_to_json`]: Function to convert a `VariantArray` to a batch of JSON strings.
//! - [`mod@cast_to_variant`]: Module to cast other Arrow arrays to `VariantArray`.
//! - [`variant_get`]: Module to get values from a `VariantArray` using a specified [`VariantPath`]
//!
//! # Compute Kernels
//! - [`json_to_variant()`]: Function to convert Arrays of JSON strings to a `VariantArray`.
//! - [`variant_to_json()`]: Function to convert a `VariantArray` to arrays of JSON strings.
//! - [`cast_to_variant()`]: Cast Arrow arrays to `VariantArray`.
//! - [`variant_get()`]: Convert `VariantArray` (or an inner path) to ArrowArrays type
//! - [`shred_variant()`]: Shred a `VariantArray`
//! - [`unshred_variant()`]: Unshred a `VariantArray`.
//!
//! ## 🚧 Work In Progress
//!
Expand All @@ -36,15 +40,15 @@
//! [Variant issue]: https://github.com/apache/arrow-rs/issues/6736

mod arrow_to_variant;
pub mod cast_to_variant;
mod cast_to_variant;
mod from_json;
mod shred_variant;
mod to_json;
mod type_conversion;
mod unshred_variant;
mod variant_array;
mod variant_array_builder;
pub mod variant_get;
mod variant_get;
mod variant_to_arrow;

pub use variant_array::{BorrowedShreddingState, ShreddingState, VariantArray, VariantType};
Expand All @@ -56,3 +60,4 @@ pub use shred_variant::shred_variant;
pub use to_json::variant_to_json;
pub use type_conversion::CastOptions;
pub use unshred_variant::unshred_variant;
pub use variant_get::{GetOptions, variant_get};
13 changes: 5 additions & 8 deletions parquet/src/variant.rs
Original file line number Diff line number Diff line change
Expand Up @@ -22,14 +22,11 @@
//! Note: Requires the `variant_experimental` feature of the `parquet` crate to be enabled.
//!
//! # Features
//! * [`Variant`] represents variant value, which can be an object, list, or primitive.
//! * [`VariantBuilder`] for building `Variant` values.
//! * [`VariantArray`] for representing a column of Variant values.
//! * [`json_to_variant`] and [`variant_to_json`] for converting to/from JSON.
//! * [`cast_to_variant()`] for casting other Arrow arrays to `VariantArray`.
//! * [`VariantType`] Arrow ExtensionType for Parquet Variant logical type.
//! [`variant_get`] to extracting a value by path and functions to convert
//! between `Variant` and JSON.
//! * Representation of [`Variant`], and [`VariantArray`] for working with
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this was largely a copy of what was in parquet_variant and parquet_variant_compute so I changed the docs to highlight the most important functions and structures and link to the rest of the docs

//! Variant values (see [`parquet_variant`] for more details)
//! * Kernels for working with arrays of Variant values
//! such as conversion between `Variant` and JSON, and shredding/unshredding
//! (see [`parquet_variant_compute`] for more details)
//!
//! # Example: Writing a Parquet file with Variant column
//! ```rust
Expand Down
Loading