-
-
Notifications
You must be signed in to change notification settings - Fork 1.4k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Implementing exaone3.5 #1480
base: main
Are you sure you want to change the base?
Implementing exaone3.5 #1480
Conversation
…ementing-exaone3
…ementing-exaone3
@KareemMusleh Thanks for the effort! I was just curious of the general process you go through to implement something like this. Is it different for every model, and how so? |
@qingy1337 I was thinking about creating a writeup for this (assuming this gets accepted), as this is my first pull request to a large public repo. But the gist of it is this:
This is different than previous model integrations because exaone has the exact same architecture as llama |
For this PR to be ready the all attention refactor fix PR should be merged first. Because it uses the latest transformers |
This is my first attempt at an implementation of exaone into unsloth, it was requested in this issue.
I didn't want to implement exaone into a separate model class because exaone follows the llama architecture as was discussed in this issue. As of now I am having problems with the from_pretrained function when using config and state_dict. I've already opened an issue in transformers about it. I'll try to solve that problem soon.