more skills

.claude/skills/lmdeploy-service-launch/SKILL.md

@@ -0,0 +1,146 @@
+---
+name: lmdeploy-service-launch
+description: Start, monitor, and stop lmdeploy REST API services for any supported device/backend. Use when Codex needs to run `lmdeploy serve api_server` for a model path, choose model names from the model being served, set backend/device/max-batch/cache and user-provided parallel strategy flags, run outside the sandbox when devices are unavailable inside it, preserve server logs, check readiness, handle port conflicts, and cleanly stop the service before another run.
+---
+
+# LMDeploy Service Launch
+
+## Purpose
+
+Use this skill only for the server side of a RESTful lmdeploy run. Keep it
+generic: do not assume Ascend, CUDA, qwen, a fixed model name, or a fixed model
+path unless the user or referenced script specifies them.
+
+## Inputs To Resolve
+
+Before launching, resolve these from the user request, referenced scripts, or
+local context:
+
+- `MODEL_PATH`: required.
+- `MODEL_NAME`: use the user-provided name if present. Otherwise choose a short
+  name that matches the model being served, such as `qwen` for Qwen-family
+  models or `interns` for InternS/InternLM-family models. If unsure, derive a
+  stable lowercase name from the model path basename, then query `/v1/models`
+  after startup and use the served id for clients.
+- `BACKEND`: default to the referenced script or existing project convention.
+- `DEVICE`: use the user-provided device if present. If the user did not specify
+  one, infer it from the host with the device-selection checks below. Do not
+  hard-code a device in this skill.
+- `PORT`: default to the referenced script or an available local port.
+- `MAX_BATCH_SIZE`, `CACHE_BLOCK_SEQ_LEN`, `CACHE_MAX_ENTRY_COUNT`, and other
+  server capacity flags from the user's comparison setup.
+- `PARALLEL_FLAGS`: user-provided parallel strategy flags. Do not assume this is
+  `--tp`; it may be tensor parallel, expert parallel, data parallel, or another
+  lmdeploy-supported strategy.
+- `LOG_FILE`: always write server stdout/stderr to a preserved log file.
+
+If the user asks to compare models, keep all comparable server parameters
+aligned unless the user explicitly asks to vary one.
+
+## Device Selection
+
+Respect explicit user input first. If the user specifies `ascend`, `cuda`,
+`maca`, `camb`, CPU, or another lmdeploy-supported device, use that value and do
+not auto-detect a different one.
+
+If the user does not specify a device, probe the host before launching:
+
+```bash
+command -v npu-smi >/dev/null 2>&1 && npu-smi info
+command -v nvidia-smi >/dev/null 2>&1 && nvidia-smi
+```
+
+Use the probe result to choose the lmdeploy `--device` value:
+
+- `npu-smi info` succeeds and shows available devices: use `ascend`.
+- `nvidia-smi` succeeds and shows available GPUs: use `cuda`.
+- both succeed: prefer the device implied by the referenced script, current
+  environment variables, or user context; otherwise ask before choosing.
+- neither succeeds: check the referenced script or local docs. If still
+  unknown, ask the user for `DEVICE` instead of guessing.
+
+Record the probe command and outcome in the run log/report when auto-detection
+is used.
+
+## Launch Template
+
+Adapt the command to the current model and device. The template intentionally
+uses placeholders instead of qwen- or Ascend-specific defaults:
+
+```bash
+cd <WORKDIR>
+
+export PYTHONPATH=<LMDEPLOY_ROOT>:<PROJECT_ROOT>:${PYTHONPATH:-}
+# Export device/vendor environment variables only when required by the selected
+# stack or referenced script.
+
+lmdeploy serve api_server <MODEL_PATH> \
+  --model-name <MODEL_NAME> \
+  --backend <BACKEND> \
+  --device <DEVICE> \
+  --server-port <PORT> \
+  --max-batch-size <MAX_BATCH_SIZE> \
+  --cache-block-seq-len <CACHE_BLOCK_SEQ_LEN> \
+  --cache-max-entry-count <CACHE_MAX_ENTRY_COUNT> \
+  <PARALLEL_FLAGS> \
+  <EXTRA_SERVER_FLAGS>
+```
+
+Omit flags that the local lmdeploy version does not support. If a referenced
+script uses a different option name for port or device, follow the script and
+local `lmdeploy serve api_server --help`.
+
+## Device And Sandbox Handling
+
+If the server needs accelerator devices and the sandbox cannot see them, run the
+service command outside the sandbox with the required approval. Do not try to
+measure performance from an environment that cannot enumerate the requested
+device.
+
+Treat device visibility failures separately from OOM:
+
+- device not visible: stop and report the environment issue
+- port already bound: stop the old service or choose another port
+- OOM: apply the benchmark's cache/memory fallback policy if one was given
+
+## Readiness And Health
+
+Poll the server log until one of these happens:
+
+- ready: the log contains `Uvicorn running`
+- failure: the process exits or the log contains a traceback/fatal runtime error
+- timeout: startup exceeds the expected model load window
+
+After readiness, optionally confirm the served model id:
+
+```bash
+curl -s http://127.0.0.1:<PORT>/v1/models
+```
+
+Use that id as the client `--model` value unless the user explicitly requires a
+different model name.
+
+## Stop And Cleanup
+
+After each benchmark case:
+
+1. Stop the client first.
+2. Stop the server cleanly with interrupt/terminate.
+3. Wait for the service process to exit.
+4. Confirm the port is free before starting another server.
+5. Wait for device runtime cleanup if the backend leaves worker processes or
+   communication ports alive briefly.
+
+Never leave a service session running at the end of the task.
+
+## Server Report Items
+
+Record these in the run report:
+
+- exact server command or script path
+- model path and served model name
+- backend, device, port, and parallel strategy flags
+- max batch size, cache block length, cache max entry count
+- extra flags, including speculative/MTP flags if used
+- startup result and log path
+- any retries, cache changes, OOMs, port conflicts, or device errors

.claude/skills/lmdeploy-service-launch/agents/openai.yaml

@@ -0,0 +1,4 @@
+interface:
+  display_name: "LMDeploy Service Launch"
+  short_description: "Start lmdeploy API services"
+  default_prompt: "Start an lmdeploy api_server for the requested model and device, preserving logs, checking readiness, and cleaning up the service afterward."

.claude/skills/restful-pressure-benchmark/SKILL.md

@@ -0,0 +1,180 @@
+---
+name: restful-pressure-benchmark
+description: Run RESTful pressure benchmarks with lmdeploy's profile_restful_api.py after starting an lmdeploy API service. Use when Codex needs to choose ShareGPT native-length or random-length workloads, set tokenizer/model/client parameters without hard-coding qwen, compare models or max-batch settings, preserve logs, inspect running batch and KV cache behavior, and report throughput and cache/OOM handling.
+---
+
+# RESTful Pressure Benchmark
+
+## Required Setup
+
+Before running any client workload, use the `lmdeploy-service-launch` skill to
+start the server and obtain:
+
+- `BASE_URL` or `HOST`/`PORT`
+- served `MODEL_NAME`
+- `MODEL_PATH`
+- `LMDEPLOY_ROOT`, supplied by the user or referenced script
+- server log path
+- server settings such as backend, device, max batch size, and cache values
+
+Do not assume the model is named `qwen`. The client `--model` must match the
+served model id or the user-provided model name. If the launch skill chose
+`qwen` for a Qwen model or `interns` for an InternS/InternLM-style model, use
+that same value here.
+
+By default, set `TOKENIZER_PATH` equal to `MODEL_PATH`. Override it only when
+the user or local benchmark script explicitly gives a separate tokenizer.
+
+Locate the client script from the user-provided lmdeploy checkout. Prefer:
+
+```text
+<LMDEPLOY_ROOT>/benchmark/profile_restful_api.py
+```
+
+If the file is not there, search under `<LMDEPLOY_ROOT>` and record the actual
+path used in the report. Do not hard-code an environment-specific absolute path
+in this skill or in reusable instructions.
+
+## Workload Selection
+
+Support both workload families. Choose from the user request:
+
+### ShareGPT Native Length
+
+Use this when the user asks for ShareGPT/default/native lengths. Do not set
+fixed random lengths.
+
+```bash
+timeout <LIMIT_SECONDS> python3 <PROFILE_RESTFUL_API> \
+  --backend lmdeploy \
+  --dataset-name sharegpt \
+  --dataset-path <SHAREGPT_JSON> \
+  --tokenizer <TOKENIZER_PATH> \
+  --num-prompts <NUM_PROMPTS> \
+  --host <HOST> \
+  --port <PORT> \
+  --model <MODEL_NAME> \
+  --disable-warmup
+```
+
+If the user asks to force ShareGPT output length, add
+`--sharegpt-output-len <TOKENS>`. Otherwise leave the native output lengths.
+
+### Random Lengths
+
+Use this when the user asks for synthetic/random token lengths, including fixed
+length cases such as 8k input / 8k output.
+
+```bash
+timeout <LIMIT_SECONDS> python3 <PROFILE_RESTFUL_API> \
+  --backend lmdeploy \
+  --dataset-name random \
+  --dataset-path <SHAREGPT_JSON> \
+  --random-input-len <INPUT_LEN> \
+  --random-output-len <OUTPUT_LEN> \
+  --random-range-ratio <RATIO> \
+  --tokenizer <TOKENIZER_PATH> \
+  --num-prompts <NUM_PROMPTS> \
+  --host <HOST> \
+  --port <PORT> \
+  --model <MODEL_NAME> \
+  --disable-warmup
+```
+
+Always pass a local `--dataset-path` for random workloads if the script uses
+ShareGPT text as prompt material; otherwise it may try to download a dataset.
+
+Choose `--random-range-ratio` intentionally:
+
+- `1`: fixed lengths exactly equal to `--random-input-len` and
+  `--random-output-len`
+- `0 < ratio < 1`: random lengths in a bounded range
+- `0`: lengths may range down to zero; do not use this for fixed-length tests
+
+Confirm the client log prints the expected aggregate `#Input tokens` and
+`#Output tokens` before treating a run as valid.
+
+## Time, Prompt Count, And Fairness
+
+Respect the user's time limit. If none is given, use a conservative timeout and
+adjust `NUM_PROMPTS` so each run finishes in the requested window.
+
+For comparisons, keep these aligned unless the user asks otherwise:
+
+- workload family and workload parameters
+- `NUM_PROMPTS`
+- server max batch size and cache settings
+- tokenizer rule
+- timeout policy
+- client flags such as streaming, warmup, and request rate
+
+If a run is interrupted, times out, or used wrong length parameters, mark it
+invalid and rerun rather than mixing it into the comparison table.
+
+## Monitoring And Statistics
+
+Preserve both client and server logs. From the client log, report:
+
+- successful request count
+- benchmark duration
+- total input tokens
+- total generated tokens
+- request throughput
+- input token throughput
+- output token throughput
+
+From the server log, extract `Engine (running/waiting)` and `KV cache` samples.
+Report:
+
+- running batch min, max, average
+- count of samples near the configured max batch size
+- KV cache maximum
+- counts of samples at or above 99%, 99.9%, and 100%
+- whether there was sustained KV saturation
+
+Do not treat sustained high KV cache as OOM if the server keeps progressing.
+
+## Cache And OOM Policy
+
+Use the cache policy from the user request or service setup. If the user says to
+start from `CACHE_MAX_ENTRY_COUNT=0.8`, do so for every run. Reduce cache only
+after a confirmed OOM, using the requested decrement step. Record the value used
+for every test.
+
+Do not change cache for:
+
+- wrong tokenizer/model name
+- dataset/proxy/download failure
+- port conflict
+- device visibility failure
+- ordinary KV cache saturation with ongoing progress
+
+## Failure Handling
+
+- If the server is not ready, return to `lmdeploy-service-launch`.
+- If the client cannot find the served model, query `/v1/models` and rerun with
+  the served id.
+- If token counts do not match the intended workload, fix client parameters and
+  rerun.
+- If MTP/speculative decoding is requested and fails, stop that series and
+  report the exact failing log path and error snippet.
+- Always stop the server after the final client run.
+
+## Report Checklist
+
+Write a run report with:
+
+- server setup from `lmdeploy-service-launch`
+- lmdeploy root and `profile_restful_api.py` path
+- client command and workload type
+- model path, served model name, tokenizer path
+- dataset path, prompt count, requested lengths, and observed token totals
+- timeout and whether the run completed
+- throughput table
+- speed ratios and percent gaps for comparisons
+- running batch and KV cache table
+- cache values, OOM status, retries, and failures
+- raw log paths
+
+Keep benchmark results run-specific. Do not hard-code previous model numbers in
+the skill.

.claude/skills/restful-pressure-benchmark/agents/openai.yaml

@@ -0,0 +1,4 @@
+interface:
+  display_name: "RESTful Pressure Benchmark"
+  short_description: "Run RESTful pressure benchmarks"
+  default_prompt: "Run a RESTful pressure benchmark through profile_restful_api.py, using the LMDeploy service launch skill first, then report throughput, batch behavior, cache settings, and logs."