[docs] Refresh inference benchmarks and promote efficiency guides

[tomaarsen]

Hello!

Heads up, this PR was AI-generated and human-reviewed.

Pull Request overview

• Refresh CPU benchmarks and add native llama.cpp comparisons
• Clarify backend recommendations across hardware and model sizes
• Promote Speeding up Inference in the documentation navigation

Details

I've refreshed the CPU benchmarks and added a comparison on Hugging Face Jobs cpu-upgrade instances. OpenVINO INT8 leads locally, while llama.cpp leads on the cloud instance, so the recommendations now emphasize testing on deployment hardware. The GPU comparison also extends through Qwen3-Embedding-8B, where llama.cpp becomes competitive with Sentence Transformers.

I've rewritten the benchmark discussion around the main findings, with GPU and CPU tabs and expandable sections for methodology, cloud results and the detailed llama.cpp comparison. The efficiency guides now appear directly after Pretrained Models in the navigation for all four model families, with links in their Usage documentation sidebars. Existing page URLs are preserved.

Validation covered benchmark RST parsing, expandable-section structure, image references, SVG XML and navigation links. A full Sphinx build and browser layout check remain unverified in this environment.

• Tom Aarsen

[Copilot]

🔵 Needs a closer look

Two moderate benchmark documentation inconsistencies remain, along with a stale breadcrumb.

Pull request overview

Refreshes inference benchmarks and promotes efficiency guides across the documentation.

Changes:

• Adds GPU, CPU, cloud CPU, and llama.cpp comparisons.
• Clarifies backend recommendations and methodology.
• Promotes efficiency guides in navigation and usage sidebars.

File summaries

File	Summary and review notes
index.rst	Updates navigation; stale generated API breadcrumbs remain (nit, 1 vote).
docs/sparse_encoder/usage/usage.rst	Adds an efficiency-guide sidebar link.
docs/sentence_transformer/usage/usage.rst	Adds an efficiency-guide sidebar link.
docs/sentence_transformer/usage/efficiency.rst	Refreshes benchmarks and recommendations; llama.cpp methodology coverage and the GPU recommendation flowchart need updates (moderate, 1 vote each).
docs/sentence_transformer/pretrained_models.md	Updates the efficiency-guide reference.
docs/quickstart.rst	Updates efficiency-guide links.
docs/multi_vector_encoder/usage/usage.rst	Adds an efficiency-guide sidebar link.
docs/cross_encoder/usage/usage.rst	Adds an efficiency-guide sidebar link.

Review details

Suppressed comments (3)

docs/sentence_transformer/usage/efficiency.rst:678

• The new Qwen3-Embedding-8B results report llama.cpp Q8_0 as faster than the three Sentence Transformers configurations compared here, but the Recommendations flowchart below still sends every GPU model to float16 + Flash Attention and has no llama.cpp/model-size branch. That makes the page's actionable recommendation inconsistent with the new 8B benchmark; add a size/backend decision or explicitly scope the flowchart to Sentence Transformers backends.

On Qwen3-Embedding-8B, the comparison shifts in favor of llama.cpp: Q8_0 with default embedding-table placement is about 12%, 2% and 12% faster than Sentence Transformers with BF16, FA2 and unpadding on short sentences, NQ answers and long reviews, respectively. More aggressive quantization does not help here, as Q4 is slower than Q8 on all three workloads. These results make llama.cpp worth considering for larger models, particularly when quantization helps them fit in memory, although models above 8B were not tested.

docs/sentence_transformer/usage/efficiency.rst:621

• The new benchmark figures explicitly include llama.cpp (including the GPU and cloud CPU results), but this backend list still ends at OpenVINO and never defines the llama.cpp configurations or their baseline. Readers therefore cannot map the new llama.cpp bars to the stated methodology; add the relevant llama.cpp entries here or explicitly scope this list to the Sentence Transformers/ONNX/OpenVINO backends and link to the detailed comparison below.

- ``openvino``: OpenVINO, via ``backend="openvino"``.

- ``openvino-qint8``: OpenVINO quantized to int8 via ``export_static_quantized_openvino_model(..., quantization_config=OVQuantizationConfig(), ...)`` and ``backend="openvino"``.

index.rst:273

• The promoted breadcrumb is still stale in the generated API documentation: sentence_transformers/backend/quantize.py:42,131 and sentence_transformers/backend/optimize.py:43 still render Sentence Transformer > Usage > Speeding up Inference. Users reaching this page through those docstrings will therefore see the old hierarchy; update those labels along with the navigation rename.

   * How do I make Sentence Transformer models **faster**? `Sentence Transformers > Speeding up Inference <docs/sentence_transformer/usage/efficiency.html>`_

• Files reviewed: 8/14 changed files
• Comments generated: 0
• Review effort level: Lite

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.