Allow CSR-Style Offsets as an Alternative Encoding for Unstructured Topologies

[jlee-kitware]

We’ve run into a recurring issue in Catalyst/VTK when consuming Conduit Mesh Blueprint data for unstructured topologies. The Blueprint currently uses two arrays, sizes and offsets, both of length num_cells. Many downstream consumers (VTK, Viskores, GPU kernels, sparse/irregular codes) instead expect a CSR-style prefix-sum array of length num_cells + 1, where the last entry equals the total connectivity length.

Because Conduit does not currently allow a CSR-style representation, Catalyst has to allocate a new array and compute these offsets internally. That defeats zero-copy handling of connectivity, especially for GPU-resident meshes where the original data already exists in the required form.

We would like to ask whether the Mesh Blueprint could optionally accept a CSR-style encoding instead of sizes + offsets. Something like:

topologies:
  topo:
    type: unstructured
    coordset: coords
    elements:
      shape: poly
      connectivity: [...]
      csr_offsets: [...]

This would not add redundant data. It would simply allow the producer to supply the format many consumers already use, and avoid the extra copy.

Would the Blueprint team be open to discussing this? We can provide examples, reasoning, or a prototype PR if helpful.

[cyrush]

@jlee-kitware we are aware of the issue with the choice of VTK / Viskores connectivity, but have not encountered simulation codes that store this natively. Are there examples of this style in data producers?

When using conduit we always know the exact length of the connectivity array, so having that length placed at the end of the offsets array is redundant (and opens up the possibly for it to be inconsistent)

If anything we could create a special case for:

sizes (array of number of cells)
offsets (array of number of cells + 1)

Consumer can use sizes to get infer the number of cells.

That said, offsets are not assumed to conform to a prefix sum -- they can be arbitrary offsets.

Can VTK/Viskores generally accept only a csr_offset without more info (known size or shape of each entry)?

Note: The conduit one-to-many relation is used for more things beyond mesh connectivity, so we want to keep it as simple as possible.

[jlee-kitware]

Thanks for the reply Cyrus, I'll try to answer each of your points:

--

We do see CSR-style connectivity coming directly from GPU-enabled simulation codes, but it’s actually broader than that:

• Several commercial and in-house CFD codes (including STAR-CCM+ from my personal experience) store unstructured connectivity in CSR.
• Many GPU-targeted solvers naturally produce num_cells+1 offsets because it maps directly to adjacency lists and prefix-sum kernels.
• When ParaView/VTK filters operate on polygonal/polyhedral meshes, VTK itself often ends up producing CSR-form connectivity internally.

So CSR isn’t just a simulation-producer convention, it also shows up mid-pipeline when Catalyst uses ParaView to extract or transform mesh data. In those cases, Catalyst/Viskores already receives CSR-form offsets from VTK.

--

Agreed that mathematically the final offset value=connectivity_length is redundant if you also have sizes. The issue for us isn’t semantic — it’s the extra allocation.

When Catalyst receives Blueprint connectivity with:

• sizes of length num_cells
• offsets of length num_cells

VTK/Viskores requires:

• csr_offsets of length num_cells + 1

Since Catalyst cannot modify application-owned memory, we always have to allocate a new GPU buffer, copy the offsets, compute and append the last entry. This forces device memory allocation + full copy every timestep, even though the simulation or VTK often already has a correct CSR array.

--

Totally understood conduit has general one-to-many relation and should not assume prefix sums. We are not proposing changing that.

We are simply asking for an optional path for the special case where the producer already does use prefix sums (or when the data comes from VTK, which guarantees it).

--

Can VTK/Viskores consume CSR without sizes?

Yes, absolutely.

VTK/Viskores only needs:

• connectivity
• a CSR-style offset array (num_cells + 1)

Sizes are always computed implicitly as offset[i+1] - offset[i].

This is already what Catalyst does today — we just want to avoid the extra buffer copy that occurs before that point.

--

If Blueprint allowed:

elements:
  connectivity: [...]
  csr_offsets: [...]   # length = num_cells + 1

then:

• Catalyst could accept CSR connectivity directly from simulations,
• or reuse CSR connectivity generated by VTK during extraction,
• without performing buffer copies or materializing temporary arrays.

A consumer that still wants sizes could trivially reconstruct them.

So we are looking for a non-breaking addition to the existing schema, and it does not remove support for arbitrary offsets or sizes. It just gives a fast path where the producer (or a previous consumer like ParaView) already has the exact structure required downstream.

Hopefully that provides more context and answers your questions. LMK if you have more questions/ideas.
BR
Jeff

[cyrush]

Thanks for these details.

I think Blueprint predates the VTK 9.0 and Viskores connectivity choices.

Sizes are always computed implicitly as offset[i+1] - offset[i].

That is interesting and I wasn't aware of this.

That means that VTK / Viskores cannot actually fully support Blueprint data w/o potentially repacking the connectivity array. In practice is must be rare, but it does surprise me.

In addition to connectivity and csr_offsets - all cases I am aware of creating explicit connectivity also needs an array of shape types. So those two aren't sufficient alone, or at least the size information could also be inferred from the explicit list of shape types that is required. I guess that's is why I am puzzled by this constraint.

I get that CSR is more efficient, but seems to be more constrained vs prior VTK connectivity.

The choice makes less data admissible for zero copy, since arbitrary offsets into connectivity aren't supported.

Adding csr_offsets with full support for the blueprint would take a good amount of effort.

Adding a new case to blueprint verify is easy, but there are many transforms and consumers that would require expanded logic.

It's possible that the allowing offsets with num cells +1 is less disruptive, but the subtlety of that difference is also scary.

For Blueprint, have a rule of thumb to find several instances of a common usage pattern before solidifying it.

Concretely knowing which codes this would help could help support a change.

I will say -- we are also exposed to this & pay a cost in Ascent with Viskores.

However we expect this cost because sim codes codes we are supporting don't have connectivity zero-copy compatible with that specific style.

[spyridon97]

The number of cells can either be extracted by the size of shapes array is simply csr_offsets' size minus 1.

[jlee-kitware]

“I think Blueprint predates the VTK 9.0 and Viskores connectivity choices.”

Yes, that matches our understanding. Since Catalyst, VTK, and Viskores all use CSR, Blueprint is the one remaining place where the representations diverge.

“Sizes are always computed implicitly as offset[i+1] - offset[i]. That is interesting and I wasn't aware of this.”

Right — this is exactly why CSR works well for us: the prefix-sum structure encodes sizes implicitly and avoids keeping multiple redundant views of the same information. When Blueprint requires both sizes and offsets, VTK/Viskores can't use the offsets array directly because it isn’t prefix-sum and doesn’t have the extra sentinel entry.

“That means that VTK / Viskores cannot actually fully support Blueprint data w/o potentially repacking the connectivity array.”

Correct. Today Catalyst must always repack to supply a CSR offsets array to VTK/Viskores. This is the only remaining copy preventing full zero-copy connectivity. For GPU-resident workflows that cost is noticeable.

“All cases I am aware of creating explicit connectivity also need an array of shape types… so those two aren't sufficient alone.”

Agree that types are still required. CSR offsets doesn’t replace types — it only replaces the need for sizes. For mixed-element unstructured grids or face/edge connectivity, CSR + types works the same way it does in VTK/Viskores today.

“I get that CSR is more efficient, but seems to be more constrained vs prior VTK connectivity. The choice makes less data admissible for zero copy, since arbitrary offsets aren't supported.”

It’s true that Blueprint allows arbitrary offsets. But in practice, all consumers downstream of Catalyst — VTK, Viskores, Ascent, ParaView filters, rendering backends — already require CSR offsets. That is, the admitted set of “zero-copy-compatible” data is already constrained by VTK/Viskores, not by Catalyst. The Blueprint representation simply forces a repack step because its offsets do not match the CSR structure those consumers use.

“Adding csr_offsets with full support… would take a good amount of effort.”

Fully understood — this is why we are suggesting optional CSR offsets, not a replacement. Blueprint would remain fully backward compatible:

• If csr_offsets is present, VTK/Viskores/Catalyst can zero-copy.
• If not, consumers fall back to the existing logic.

This avoids altering any existing data producers or consumers unless they opt in.

“Offsets with numCells+1 may be less disruptive, but the subtlety is scary.”

Totally fair — and this is exactly why an explicit csr_offsets array may be cleaner. It avoids overloading the meaning of the existing offsets. But maybe its an easier compromise, not sure yet.

“Concretely knowing which codes this would help could support a change.”

Understood, a bespoke solution for Catalyst is undesirable. For reference, the CSR-style offsets appear naturally in:

• GPU CFD solvers that store cell connectivity (cell-to-point lists) in prefix-sum form for efficient device kernels (FV/FE/AMR codes, unstructured solvers on GPUs).
• DOE and DOD simulation codes whose mesh data structures use prefix-sum connectivity on device for parallel execution. I could be wrong but Kestrel, Helios, LAAVA, FUN3d, Mint...
• Catalyst/VTK/Viskores themselves, which require CSR-style connectivity (offsets of length numCells+1) for zero-copy construction of vtkCellArray / Viskores ArrayHandle.
• Commercial solvers (e.g., Star-CCM+, fluent), which export cell connectivity in CSR/prefix-sum form for interoperability and performance.

So even if few producers expose CSR today, nearly all downstream consumers already require it.

“Ascent also pays this cost with Viskores.”

Yes — which is why we’re collectively interested in reducing unnecessary repacking when the sim already provides prefix-sum offsets.
LMK if I misinterpreted your concerns.

[cyrush]

@jlee-kitware sorry for the pause in the discussion.

We can consider this, this could be come a specific type of the one-to-many relation blueprint protocol.

We need to work out a few details:

• What is the proper unique name and node structure for this case.
  For example, how does the name impact how we detect this case vs others, are there gotchas for non connectivity cases.

• What transforms do we need to/from the prefix sum representation to the general case?

Can we extend concrete helpers for o2m (iterators, etc) to support this case