Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[DNM] optimization for shuffle read stage #1

Open
wants to merge 6 commits into
base: topic/3rd/sparkucx/implement-writer-and-reader
Choose a base branch
from
Open

[DNM] optimization for shuffle read stage #1

wants to merge 6 commits into from

Conversation

jeynmann
Copy link

@jeynmann jeynmann commented Aug 8, 2023

Merge the optimization PR#27 of sparkucx into the sparkdpu branch.

id executor data time/s stage0/s stage1/s stage2/s
tcp 4 38GB 97.6 24.0 48.2 12.2
sparkdpu - PR# 4 38GB 111.5 24.5 61.9 12.1
sparkdpu - ofir 4 38GB 115.4 25.5 62.9 14.3

brianplus and others added 6 commits April 25, 2023 10:58
@brianplus
Load UCX libs explicitly (#26)
2d06a7e 
* Load UCX libs explicitly once ucx shuffle manager is created

Signed-off-by: Brian Sheng <[email protected]>

* Add comments for the code change

Signed-off-by: Brian Sheng <[email protected]>

---------

Signed-off-by: Brian Sheng <[email protected]>
@JeynmannZ
optimize progress
3729e10
@JeynmannZ
Merge branch 'topic/optimize_progress' into dpu_dev
8e460a8
@JeynmannZ
add inline
606590d
@JeynmannZ
Merge branch 'topic/3rd/sparkucx/implement-writer-and-reader' into dp…
7f10bd0 
…u_dev
@JeynmannZ
rm no use changes
8fbda9e
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@jeynmann @brianplus @JeynmannZ