Support creating assets from PLY files larger than 2GB#228
Open
comoc wants to merge 3 commits into
Open
Conversation
Large Gaussian Splat PLY files (>8.6M splats) crashed asset creation in two places, both due to int-sized size calculations overflowing past 2GB: - PLYFileReader loaded the entire binary body into a single NativeArray<byte> (capped at ~2GB) and rejected larger files outright. Read the body in batches straight into the InputSplatData array instead. - ReorderMorton duplicated the splat array via the NativeArray copy constructor, whose internal size is computed as length*sizeof in int and overflows once the array exceeds 2GB (e.g. 10.3M splats * 248 bytes), crashing inside memcpy. Use an explicit long-sized MemCpy and a pointer-based gather. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
There was a problem hiding this comment.
Pull request overview
This PR enables importing very large (>2GB) binary little-endian PLY Gaussian splat files by avoiding int-overflowing size calculations and by streaming PLY body data in batches instead of loading it all at once.
Changes:
- Updated
PLYFileReaderto exposeOpenAndReadHeader(...)so callers can stream the binary body starting from the correct file position (removing the previous “>2GB not supported” behavior). - Updated
GaussianFileReaderto read PLY vertex data in 1M-splat batches into a reusable buffer and convert directly into the destinationNativeArray<InputSplatData>. - Updated
GaussianSplatAssetCreator.ReorderMortonto avoidNativeArraycopy paths that compute byte sizes inint, replacing them with an explicitUnsafeUtility.MemCpyusinglong.
Reviewed changes
Copilot reviewed 3 out of 3 changed files in this pull request and generated 1 comment.
| File | Description |
|---|---|
| package/Editor/Utils/PLYFileReader.cs | Adds a streaming-friendly header reader that returns a positioned FileStream; removes whole-file body loading. |
| package/Editor/Utils/GaussianFileReader.cs | Streams PLY body data in batches into the destination splat array to support files exceeding ~2GB. |
| package/Editor/GaussianSplatAssetCreator.cs | Fixes >2GB splat-array reordering by replacing overflow-prone copy logic with long-sized MemCpy and pointer-based gather. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
ReorderPLYData was a [BurstCompile] static method invoked as a direct call, which Burst compiles through the function-pointer path. That path can fail to compile in time on first use, surfacing as "Burst failed to compile the function pointer ReorderPLYData$BurstManaged(...)" and aborting asset creation on the first attempt. Convert it to a standard IJobParallelFor Burst job: this uses the regular job compilation path (no flaky function pointer), and parallelizes the per-splat scatter across worker threads as a bonus. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Per PR review feedback on the large-PLY read path: - Wrap the batched read + conversion in try/catch and dispose the destination NativeArray (potentially multi-GB) if reading or converting throws partway (e.g. a truncated file), instead of leaking it. - Use typed InputSplatData* pointer arithmetic for the per-batch destination offset to make the element intent clear. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
Creating a
GaussianSplatAssetfrom a large PLY file (2.38 GB, ~10.3M splats, full SH) failed — first with afiles larger than 2GB are not supportederror, and after lifting that check, with a hard Editor crash insidememcpy.Both issues come from size calculations done in
intthat overflow once a per-splat buffer exceeds 2GB (InputSplatDatais a fixed 248 bytes/splat, so 8.6M splats ≈ 2GB).Fix
PLYFileReader: previously loaded the entire binary body into oneNativeArray<byte>(capped at ~2GB) and rejected larger files outright. Now opens the stream positioned at the body viaOpenAndReadHeaderand reads it in 1M-splat batches straight into the destinationInputSplatDataarray (whose element count still fits in anint, so >2GB total is fine).ReorderMorton: duplicated the splat array via theNativeArraycopy constructor, whose internal size islength * sizeofcomputed inint— this overflows past 2GB and crashes insidememcpy. Replaced with an explicitlong-sizedUnsafeUtility.MemCpyand a pointer-based gather.Notes / scope
Export PLYremain bounded by the existingkMaxSplats = 8.6Mlimit and are out of scope here.