Skip to content

Support creating assets from PLY files larger than 2GB#228

Open
comoc wants to merge 3 commits into
aras-p:mainfrom
comoc:fix-2gb
Open

Support creating assets from PLY files larger than 2GB#228
comoc wants to merge 3 commits into
aras-p:mainfrom
comoc:fix-2gb

Conversation

@comoc

@comoc comoc commented Jun 29, 2026

Copy link
Copy Markdown

Problem

Creating a GaussianSplatAsset from a large PLY file (2.38 GB, ~10.3M splats, full SH) failed — first with a files larger than 2GB are not supported error, and after lifting that check, with a hard Editor crash inside memcpy.

Both issues come from size calculations done in int that overflow once a per-splat buffer exceeds 2GB (InputSplatData is a fixed 248 bytes/splat, so 8.6M splats ≈ 2GB).

Fix

  • PLYFileReader: previously loaded the entire binary body into one NativeArray<byte> (capped at ~2GB) and rejected larger files outright. Now opens the stream positioned at the body via OpenAndReadHeader and reads it in 1M-splat batches straight into the destination InputSplatData array (whose element count still fits in an int, so >2GB total is fine).
  • ReorderMorton: duplicated the splat array via the NativeArray copy constructor, whose internal size is length * sizeof computed in int — this overflows past 2GB and crashes inside memcpy. Replaced with an explicit long-sized UnsafeUtility.MemCpy and a pointer-based gather.

Notes / scope

  • This enables import and rendering of >8.6M-splat clouds. Editing (Edit/Resize) and Export PLY remain bounded by the existing kMaxSplats = 8.6M limit and are out of scope here.
  • Tested manually in the Editor with a 2.38 GB / 10.3M-splat PLY at High quality.

Large Gaussian Splat PLY files (>8.6M splats) crashed asset creation in
two places, both due to int-sized size calculations overflowing past 2GB:

- PLYFileReader loaded the entire binary body into a single
  NativeArray<byte> (capped at ~2GB) and rejected larger files outright.
  Read the body in batches straight into the InputSplatData array instead.
- ReorderMorton duplicated the splat array via the NativeArray copy
  constructor, whose internal size is computed as length*sizeof in int and
  overflows once the array exceeds 2GB (e.g. 10.3M splats * 248 bytes),
  crashing inside memcpy. Use an explicit long-sized MemCpy and a
  pointer-based gather.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@comoc comoc marked this pull request as ready for review June 29, 2026 13:26
Copilot AI review requested due to automatic review settings June 29, 2026 13:26

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR enables importing very large (>2GB) binary little-endian PLY Gaussian splat files by avoiding int-overflowing size calculations and by streaming PLY body data in batches instead of loading it all at once.

Changes:

  • Updated PLYFileReader to expose OpenAndReadHeader(...) so callers can stream the binary body starting from the correct file position (removing the previous “>2GB not supported” behavior).
  • Updated GaussianFileReader to read PLY vertex data in 1M-splat batches into a reusable buffer and convert directly into the destination NativeArray<InputSplatData>.
  • Updated GaussianSplatAssetCreator.ReorderMorton to avoid NativeArray copy paths that compute byte sizes in int, replacing them with an explicit UnsafeUtility.MemCpy using long.

Reviewed changes

Copilot reviewed 3 out of 3 changed files in this pull request and generated 1 comment.

File Description
package/Editor/Utils/PLYFileReader.cs Adds a streaming-friendly header reader that returns a positioned FileStream; removes whole-file body loading.
package/Editor/Utils/GaussianFileReader.cs Streams PLY body data in batches into the destination splat array to support files exceeding ~2GB.
package/Editor/GaussianSplatAssetCreator.cs Fixes >2GB splat-array reordering by replacing overflow-prone copy logic with long-sized MemCpy and pointer-based gather.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment thread package/Editor/Utils/GaussianFileReader.cs
comoc and others added 2 commits June 29, 2026 22:37
ReorderPLYData was a [BurstCompile] static method invoked as a direct
call, which Burst compiles through the function-pointer path. That path
can fail to compile in time on first use, surfacing as "Burst failed to
compile the function pointer ReorderPLYData$BurstManaged(...)" and
aborting asset creation on the first attempt.

Convert it to a standard IJobParallelFor Burst job: this uses the regular
job compilation path (no flaky function pointer), and parallelizes the
per-splat scatter across worker threads as a bonus.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Per PR review feedback on the large-PLY read path:
- Wrap the batched read + conversion in try/catch and dispose the
  destination NativeArray (potentially multi-GB) if reading or converting
  throws partway (e.g. a truncated file), instead of leaking it.
- Use typed InputSplatData* pointer arithmetic for the per-batch
  destination offset to make the element intent clear.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants