Skip to content

Conversation

@TianyiZhao1437
Copy link
Collaborator

@TianyiZhao1437 TianyiZhao1437 commented Jan 8, 2026

📋 PR Title Format

The PR title should follow the format:

type(scope): concise message (max 50 chars)

Where:

  • type is one of: feat, fix, docs, refactor, perf, test, chore.
  • scope is optional and describes the part of the codebase affected (e.g., auth, ui, api).
  • concise message is a short description of the change (max 50 chars).

📝 Change Type

Please select the type of change this PR introduces (choose one or more):

  • feat: New feature.
  • fix: Bug fix.
  • docs: Documentation only changes.
  • refactor: A code change that neither fixes a bug nor adds a feature.
  • perf: Performance improvement.
  • test: Adding missing tests or correcting existing tests.
  • chore: Maintenance tasks (e.g., updating dependencies).

💡 Description

MLX rope now supports batch offset ml-explore/mlx#2564. Improve engine performance when handling batch.

Qwen/Qwen3-8B on M4 Max bs=16 per decode layer:
before: 2.5ms
after: 0.3ms

🔗 Related Issues

List any issues this PR closes or relates to:

  • Closes #IssueNumber (e.g., Closes #123)

✅ Checklist

Please ensure the following points are addressed before merging:

  • I have performed a self-review of my own code.
  • I have added/updated tests that prove my fix or feature works (if applicable).
  • I have updated the documentation (if necessary).
  • My code follows the project's style guidelines.

@TianyiZhao1437 TianyiZhao1437 marked this pull request as ready for review January 8, 2026 03:47
@TianyiZhao1437 TianyiZhao1437 requested a review from a team January 8, 2026 03:47
@TianyiZhao1437 TianyiZhao1437 changed the title feat(mlx): rope supports batch perf(mlx): rope supports batch offset Jan 8, 2026
@TianyiZhao1437 TianyiZhao1437 requested a review from gufengc January 8, 2026 03:56
@TianyiZhao1437 TianyiZhao1437 merged commit 780fb09 into main Jan 8, 2026
8 checks passed
@TianyiZhao1437 TianyiZhao1437 deleted the batch-rope branch January 8, 2026 04:46
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants