Kubeflow Trainer is a Kubernetes-native project designed for large language models (LLMs) fine-tuning and enabling scalable, distributed training of machine learning (ML) models across various frameworks, including PyTorch, JAX, TensorFlow, and others.
You can integrate other ML libraries such as HuggingFace, DeepSpeed, or Megatron-LM with Kubeflow Training to orchestrate their ML training on Kubernetes.
Kubeflow Trainer allows you effortlessly develop your LLMs with the Kubeflow Python SDK and build Kubernetes-native Training Runtimes with Kubernetes Custom Resources APIs.
The following KubeCon + CloudNativeCon 2024 talk provides an overview of Kubeflow Trainer capabilities:
Please check the official Kubeflow documentation to install and get started with Kubeflow Trainer.
The following links provide information on how to get involved in the community:
- Join our
#kubeflow-trainingSlack channel. - Attend the bi-weekly AutoML and Training Working Group community meeting.
- Check out who is using Kubeflow Trainer.
Please refer to the CONTRIBUTING guide.
Please refer to the CHANGELOG.
Kubeflow Trainer project is currently in alpha status, and APIs may change. If you are using Kubeflow Training Operator V1, please refer to this migration document.
Kubeflow Community will maintain the Training Operator V1 source code at
the release-1.9 branch.
You can find the documentation for Kubeflow Training Operator V1 in these guides.
This project was originally started as a distributed training operator for TensorFlow and later we merged efforts from other Kubeflow Training Operators to provide a unified and simplified experience for both users and developers. We are very grateful to all who filed issues or helped resolve them, asked and answered questions, and were part of inspiring discussions. We'd also like to thank everyone who's contributed to and maintained the original operators.
- PyTorch Operator: list of contributors and maintainers.
- MPI Operator: list of contributors and maintainers.
- XGBoost Operator: list of contributors and maintainers.
- Common library: list of contributors and maintainers.