Skip to content

[Xet] Basic shard creation #1633

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Open
wants to merge 44 commits into
base: main
Choose a base branch
from
Open

[Xet] Basic shard creation #1633

wants to merge 44 commits into from
+5,570 −1,905

Conversation

coyotte508
Copy link
Member

@coyotte508 coyotte508 commented Jul 16, 2025 •
edited
Loading

cc @Kakulukian @assafvayner for viz, follow up to #1616

Based on https://github.com/huggingface/xet-core/blob/7e41fb0dd7cfb276222b9668d0b97a984647721e/spec/shard.md

Need to handle:

  • split into multiple shards when xorb or file info grows too big
  • uploading xorbs & shards (and we need to upload xorbs before shards referencing them)

coyotte508
coyotte508 commented Jul 16, 2025
View reviewed changes
coyotte508 added 25 commits July 16, 2025 21:38
@coyotte508
remove shard key expiry
6676431
@coyotte508
actually make API calls to xet backend to upload shards/xorbs
3fc715b
@coyotte508
fix prefix for shard upload
7a38fc7
@coyotte508
fix verificaiton data
947a926
@coyotte508
update wasm bindings
9cd2e66
@coyotte508
fixup! update wasm bindings
bf8ae1c
@coyotte508
no need to compute shard hash client-side
70f0a0e
@coyotte508
no need for shard hash
431949b
@coyotte508
Merge remote-tracking branch 'origin/main' into shard-creation
515cae6
@coyotte508
add va prefix to xorb too
adbe363
@coyotte508
progress events for uploading xorbs
2f9d4a0
@coyotte508
commit leftover xorb after all chunks have been processed
a13da79
@coyotte508
integrate xet upload in commit function
3cbde5e
@coyotte508
add local dedup for xet uploads
651f6c0
@coyotte508
move chunk caching to its own class/file
f3e190f
@coyotte508
Make sure to not OOB when writing shards
06dcd6d
@coyotte508
dedup boolean when loading chunks from wasm
c093c64
@coyotte508
global dedup! (just need hmac algorithm)
2e27699
@coyotte508
delay file events until matching xorb is emitted
7c397dd
@coyotte508
add dedup ratio to information
5f3a61b
@coyotte508
fixup! add dedup ratio to information
66299a3
@coyotte508
use hmac function from wasm
988d85b
@coyotte508
add bench script
433284e
@coyotte508
fix wasm instantiation
74dfb71
@coyotte508
fix api calls
b19209f
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@assafvayner assafvayner assafvayner left review comments

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@coyotte508 @assafvayner