Skip to content

[WIP][AQUA] GPU Shape Recommendation #1221

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Open
wants to merge 9 commits into
base: main
Choose a base branch
from
Open

[WIP][AQUA] GPU Shape Recommendation #1221

wants to merge 9 commits into from
+3,200 −51

Conversation

elizjo
Copy link
Member

@elizjo elizjo commented Jul 7, 2025 •
edited
Loading

Wrote an additional POST API and aqua command for recommending GPU shapes for a particular model

ads aqua recommend shapes --model_ocid 'ocid1.datasciencemodel.oc1.<ocid>' ""

Returns
Screenshot 2025-08-01 at 2 19 10 PM

{
  "model_ocid" : "ocid1.datasciencemodel.oc1.<ocid>"
}

Returns

{                                                                                                                                                                                                                                            
    "display_name": "Almawave/Velvet-14B",
    "recommendations": [
        {
            "shape_details": {
                "available": false,
                "core_count": null,
                "memory_in_gbs": null,
                "name": "BM.GPU.MI300X.8",
                "shape_series": "GPU",
                "gpu_specs": {
                    "gpu_memory_in_gbs": 1536,
                    "gpu_count": 8,
                    "gpu_type": "MI300X",
                    "quantization": [
                        "fp8",
                        "gguf"
                    ],
                    "ranking": {
                        "cost": 90,
                        "performance": 90
                    }
                }
            },
            "configurations": [
                {
                    "model_details": {
                        "model_size_gb": 28.16,
                        "kv_cache_size_gb": 26.84,
                        "total_model_gb": 55.0
                    },
                    "deployment_params": {
                        "quantization": "bfloat16",
                        "max_model_len": 131072,
                        "params": ""
                    },
                    "recommendation": "No override PARAMS needed. \n\nModel fits well within the allowed compute shape (55.0GB used / 1536.0GB allowed)."
                }
            ]
        }
    ],
     "troubleshoot": ""
}

Status: business logic works, API works, unit tests finished, rich diff CLI table finished
Screenshot 2025-07-29 at 12 04 57 PM

@oracle-contributor-agreement oracle-contributor-agreement bot added the OCA Verified All contributors have signed the Oracle Contributor Agreement. label Jul 7, 2025
@github-actions GitHub Actions
Copy link

github-actions bot commented Jul 7, 2025

📌 Cov diff with main:

Coverage-0%

📌 Overall coverage:

Coverage-18.62%

@mrDzurb mrDzurb changed the title GPU Shape Recommendation [AQUA] GPU Shape Recommendation Jul 7, 2025
@github-actions GitHub Actions
Copy link

github-actions bot commented Jul 8, 2025

📌 Cov diff with main:

Coverage-0%

📌 Overall coverage:

Coverage-18.62%

@mrDzurb mrDzurb changed the title [AQUA] GPU Shape Recommendation [WIP][AQUA] GPU Shape Recommendation Jul 15, 2025
@elizjo elizjo force-pushed the ODSC-74228/GPU-Shape-Recommendation branch from 2f54f8b to 26e08a2 Compare July 25, 2025 17:29
@github-actions GitHub Actions
Copy link

📌 Cov diff with main:

Coverage-32%

📌 Overall coverage:

Coverage-58.28%

@github-actions GitHub Actions
Copy link

📌 Cov diff with main:

Coverage-54%

📌 Overall coverage:

Coverage-58.46%

@github-actions GitHub Actions
Copy link

📌 Cov diff with main:

Coverage-0%

📌 Overall coverage:

Coverage-18.43%

@github-actions GitHub Actions
Copy link

📌 Cov diff with main:

Coverage-70%

📌 Overall coverage:

Coverage-58.58%

@github-actions GitHub Actions
Copy link

github-actions bot commented Aug 1, 2025

📌 Cov diff with main:

Coverage-70%

📌 Overall coverage:

Coverage-58.58%

mrDzurb
mrDzurb reviewed Aug 1, 2025
View reviewed changes
ads/aqua/resources/gpu_shapes_index.json
},
"VM.GPU.A10.1": {
"gpu_count": 1,
"gpu_memory_in_gbs": 24,
"gpu_type": "A10"
"gpu_type": "A10",
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Let's add FP8 for the A10 shapes as well.

ads/aqua/common/utils.py Outdated
@@ -1287,6 +1287,7 @@ def load_gpu_shapes_index(

# Merge: remote shapes override local
local_shapes = local_data.get("shapes", {})
remote_data = {}
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why do we need this?

ads/aqua/extension/__init__.py
@@ -13,6 +13,7 @@
from ads.aqua.extension.evaluation_handler import __handlers__ as __eval_handlers__
from ads.aqua.extension.finetune_handler import __handlers__ as __finetune_handlers__
from ads.aqua.extension.model_handler import __handlers__ as __model_handlers__
from ads.aqua.extension.recommend_handler import __handlers__ as __gpu_handlers__
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe we can name it as __shape_handler?

ads/aqua/shaperecommend/llm_config.py Outdated
Detects quantization bit-width as a string (e.g., '4bit', '8bit') from Hugging Face config dict.
"""
if raw.get("load_in_8bit"):
return "8bit"
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It would be better to move this into constants.

ads/aqua/shaperecommend/llm_config.py Outdated
If model is un-quantized, uses the weight size.
If model is pre-quantized, uses the quantization level.
"""
key = (self.quantization or self.weight_dtype or "float32").lower()
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Let's move "float32" to constants

ads/aqua/shaperecommend/llm_config.py Outdated
"""
vals = []
curr = min_len
max_seq_len = 16384 if not self.max_seq_len else self.max_seq_len
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Let's move the numbers like 16384 to constants and add some description there

@github-actions GitHub Actions
Copy link

github-actions bot commented Aug 2, 2025

📌 Cov diff with main:

Coverage-69%

📌 Overall coverage:

Coverage-58.57%

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@mrDzurb mrDzurb mrDzurb left review comments

@darenr darenr Awaiting requested review from darenr darenr is a code owner

@mayoor mayoor Awaiting requested review from mayoor mayoor is a code owner

@VipulMascarenhas VipulMascarenhas Awaiting requested review from VipulMascarenhas VipulMascarenhas is a code owner

@qiuosier qiuosier Awaiting requested review from qiuosier qiuosier is a code owner

@ahosler ahosler Awaiting requested review from ahosler ahosler is a code owner

At least 2 approving reviews are required to merge this pull request.

Assignees
No one assigned
Labels
OCA Verified All contributors have signed the Oracle Contributor Agreement.
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@elizjo @mrDzurb