Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: Add feature gate #798

Open
wants to merge 1 commit into
base: main
Choose a base branch
from
Open

Conversation

yeahdongcn
Copy link
Contributor

This PR introduces a feature gate mechanism in ktransformers, providing centralized control over specific features. This helps streamline feature management and reduces the frequent invocation of get_compute_capability.
Vendors can extend this file to implement more fine-grained controls.

Testing Done

$ python ./ktransformers/local_chat.py --force_think true --cpu_infer 64 --model_path deepseek-ai/DeepSeek-R1 --gguf_path /home/gpuserver/models/DeepSeek-R1-Q4_K_M --port 10002
flashinfer not found, use triton for linux
Feature gate initialized: KTRANSFORMERS_USE_TORCH_NATIVE=False, KTRANSFORMERS_USE_FLASHINFER=False
using custom modeling_xxx.py.
using default_optimize_rule for DeepseekV3ForCausalLM
Injecting model as ktransformers.operators.models . KDeepseekV2Model
...

@yeahdongcn
Copy link
Contributor Author

#787 is no longer necessary.

Signed-off-by: Xiaodong Ye <[email protected]>
@yeahdongcn
Copy link
Contributor Author

Rebased from upstream/main.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant