Does the OLMoE model and its code support customizing the MoE routing mechanism for fine-tuning?

[fendi001]

Does the open-sourced OLMoE model and its code support modifying the MoE routing mechanism for fine-tuning? If yes, could you please show me where is the source code located to enable modify the routing mechanism?

[Muennighoff]

Sure you can e.g. modify it here https://github.com/huggingface/transformers/blob/0ef339ff1b63bb03a388c79bfbebec9085e10564/src/transformers/models/olmoe/modeling_olmoe.py#L589 & fine-tune via transformers if you want