Skip to content

ARCHITECTURE.md

Latest commit

 

History

History
244 lines (144 loc) · 7.88 KB

ARCHITECTURE.md

File metadata and controls

244 lines (144 loc) · 7.88 KB
Raw

Arnio Architecture

Arnio is designed to provide high-performance, memory-efficient data ingestion and cleaning by leveraging C++ while maintaining a seamless, declarative Python API.

This document outlines the core architecture and the boundary between Python and C++.

1. The Core Philosophy

Data preprocessing often involves operations that are inherently slow in Python (e.g., string manipulation, repeated passes over data). Arnio solves this by:

  1. Loading data directly into C++ memory structures.

  2. Prioritizing native C++ execution for core cleaning operations to avoid Python GIL contention, while seamlessly supporting Python-backed and custom steps.

  3. Translating the final dataset to a pandas.DataFrame via a boundary that aims to minimize unnecessary copies.

2. Python ↔ C++ Boundary

The boundary is managed using pybind11.

The C++ core is compiled into a Python extension module (_arnio_cpp). The Python API (in arnio/) serves as a lightweight, type-hinted wrapper around this compiled extension.

graph TD
    A[Python User Code] --> B[Arnio Python API]
    B -->|pybind11| C[C++ Core _arnio_cpp]

    subgraph C++ Runtime
        C --> D[CsvReader]
        C --> E[Frame / Column]
        C --> F[Cleaning Primitives]
    end

    F -->|Return| E
    E -->|to_pandas| A
Loading

3. Data Model

Arnio's data model is columnar, strongly resembling Apache Arrow or modern Pandas internals.

Column

A Column represents a single 1D array of homogeneous data.

  • Variant Storage: Data is stored using std::variant over strongly-typed std::vectors (e.g., std::vector<int64_t>, std::vector<std::string>).

4. Pandas Dtype Compatibility

Arnio supports a focused set of pandas dtypes directly through its native C++ columnar model. This section reflects the current implementation behavior and indicates which dtype workflows are supported versus rejected during validation.

Fully Supported

The following dtypes are natively supported and map efficiently to strongly typed C++ vectors:

  • int64
  • float64
  • bool
  • string

These allow efficient parsing, cleaning operations, and zero-copy or near zero-copy conversion back to pandas where possible.

Currently Unsupported

The following dtypes are currently rejected by from_pandas() validation and will raise a user-facing TypeError with guidance on how to preprocess the column:

  • datetime64[ns]
  • category
  • mixed object columns
  • timedelta64[ns]
  • complex64
  • complex128

These dtypes require conversion to supported representations before processing.

Nullable Dtype Behavior

When converting Arnio data back to pandas, null-mask information is preserved where supported and may be represented using pandas nullable extension dtypes such as:

  • Int64
  • BooleanDtype
  • StringDtype

This outbound conversion behavior should not be interpreted as full inbound support for nullable pandas extension dtypes in from_pandas().

User-facing Behavior

When unsupported dtypes are encountered, Arnio provides clear user-facing errors instead of silent failures.

For best performance and compatibility, users are encouraged to prefer strongly typed columns such as int64, float64, bool, and string.

5. Pipeline Execution & Dispatch Flow

  • Null Handling: Nulls are tracked via a separate boolean mask (std::vector<bool>), allowing the underlying data vectors to remain dense and cache-friendly.

The pipeline() function orchestrates data flow by prioritizing C++ efficiency while allowing Python extensibility.

Arnio supports a mix of C++-backed steps, Python-backed built-ins, and custom pipeline steps.

When a step is invoked, the system follows a priority-based dispatch model:

Built-in Step Path

The system first queries _STEP_REGISTRY.

This built-in registry routes operations to highly optimized C++-backed steps (executing directly within the Frame / C++ core), while also housing Python-backed built-ins.

Custom Pipeline Steps

If the name is absent from the built-in registries, it searches _PYTHON_STEP_REGISTRY for custom, user-defined Python fallbacks.

Frame

A Frame is an ordered collection of Column objects, representing a 2D dataset.

6. Converting to Pandas

The Frame maintains an index mapping column names to their respective Column objects for O(1) access.

The Conversion Penalty

Because Python-based steps expect a pandas.DataFrame, the system performs a roundtrip:

Frame → to_pandas() → from_pandas() → Frame

Performance Note

This roundtrip involves memory re-allocation.

  • to_pandas() creates a DataFrame representation.

  • from_pandas() re-infers types to re-populate the internal data structures.

Core cleaning primitives should ideally be implemented as C++ built-ins to bypass this overhead.

6. Converting to Pandas

The to_pandas() function is a critical boundary.

It uses the NumPy C-API (via pybind11's buffer protocol) to expose the underlying C++ std::vector memory to pandas, aiming to avoid element-by-element copies for numerics and booleans where supported.

String columns currently require instantiation of Python str objects.

7. Data Quality and Schema Validation

Arnio is split into two layers:

  • The C++ layer handles parsing CSVs, storing data in memory, and cleaning operations such as drop_nulls() and strip_whitespace(). These execute through pybind11 directly in C++.

  • The Python/pandas layer handles data quality: profiling, validation, and schema checks.

Each function in the quality layer behaves as follows:

profile()

  • Converts an ArFrame to a pandas DataFrame internally.
  • Computes per-column statistics including null counts, duplicate rows, data types, and unique value ratios.
  • Returns a DataQualityReport.

suggest_cleaning()

  • Accepts an ArFrame or an existing DataQualityReport.
  • If given an ArFrame, it calls profile() first.
  • Returns a list of cleaning steps that can be passed to pipeline().

auto_clean()

  • Accepts an ArFrame.
  • Calls profile() internally.
  • Applies suggested cleaning steps directly to the original ArFrame.
  • Returns a cleaned ArFrame.

validate()

  • Converts ArFrame to a pandas DataFrame internally.
  • Evaluates each column against rules defined in a Schema including nullability, dtype, range, pattern, and semantic constraints.
  • Returns a ValidationResult.

Flow Diagram

graph TD
    A[ArFrame] -->|or| D[suggest_cleaning]
    A --> B[profile]
    A --> C[auto_clean]
    A --> H[validate]
    B --> E[DataQualityReport]
    E -->|or| D
    D --> F[step list]
    C --> G[cleaned ArFrame]
    H --> I[ValidationResult]
Loading

8. Cleaning Module Architecture

The cleaning module is designed around immutable semantics to ensure data integrity across pipeline steps.

To ensure consistency, the C++ core utilizes internal helper functions:

  • resolve_subset: Translates user-provided column names into integer indices.

  • select_rows: A utility that takes a list of row indices and constructs a new Frame.

  • row_key: Generates a deterministic string serialization key for a row to facilitate duplicate detection.

Copy-Always Semantics

Arnio follows an immutable design pattern.

Most cleaning operations do not modify the existing Frame in-place; instead, they produce a new std::vector<Column> and return a brand-new Frame instance, often utilizing std::move to transfer ownership efficiently.

9. Error Handling & Translation

Arnio uses a unified exception hierarchy to bridge the C++/Python boundary.

Base Exception

  • ArnioError: The base exception class.

Specialized Exceptions

  • CsvReadError

  • UnknownStepError

  • TypeCastError

When supported by the specific implementation, exceptions raised within the core are translated into standard Python exceptions to maintain interpreter stability.