Skip to content

[Attention] R3 Attention Transform #485

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 1 commit into from
Oct 30, 2025
Merged

[Attention] R3 Attention Transform #485

merged 1 commit into from
Oct 30, 2025
+117 −33

Conversation

@kylesayrs
Copy link
Collaborator

@kylesayrs kylesayrs commented Oct 8, 2025 •
edited
Loading

Purpose

  • Support R3 transforms on attention

Postrequisites

Changes

  • Add hooks for Q_ATTN and K_CACHE targets
  • Update get_transform_size to support modules which do not have weights
  • Remove unnecessary checks for "weight attribute of module (attention modules do not have weights)
  • Add misc util TransformLocation. is_online for checking if a transform location is applied in an online way

Testing

  • Added R3 test applied to module
  • Able to create R3 rotated models

@kylesayrs kylesayrs force-pushed the kylesayrs/r3-only branch 4 times, most recently from 4cc5ace to 9ead292 Compare October 14, 2025 04:21
@kylesayrs kylesayrs force-pushed the kylesayrs/r3-on-attn branch 2 times, most recently from 17258ed to 98734bf Compare October 15, 2025 17:15
@kylesayrs kylesayrs marked this pull request as ready for review October 15, 2025 17:23
@kylesayrs kylesayrs mentioned this pull request Oct 15, 2025
Open
Base automatically changed from kylesayrs/r3-only to main October 23, 2025 14:35
@kylesayrs
squash
10d6037 
Signed-off-by: Kyle Sayers <[email protected]>
@kylesayrs kylesayrs force-pushed the kylesayrs/r3-on-attn branch from 2977f82 to 10d6037 Compare October 29, 2025 14:11
brian-dellabetta
brian-dellabetta approved these changes Oct 29, 2025
View reviewed changes
Copy link
Collaborator

@brian-dellabetta brian-dellabetta left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

very nice! one nit:

shanjiaz
shanjiaz approved these changes Oct 30, 2025
View reviewed changes
Copy link
Collaborator

@shanjiaz shanjiaz left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good to me!

@kylesayrs kylesayrs merged commit 52792be into main Oct 30, 2025
3 checks passed
@kylesayrs kylesayrs deleted the kylesayrs/r3-on-attn branch October 30, 2025 19:20
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@brian-dellabetta brian-dellabetta brian-dellabetta approved these changes

@shanjiaz shanjiaz shanjiaz approved these changes

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

4 participants

@kylesayrs @brian-dellabetta @shanjiaz