Docling-serve UI hangs indefinitely when selecting pipeline Vlm

[pierdomenicobatzu]

Linked to the issue #399 (which still seems to be happening to me).

When selecting pipeline type = Vlm from the docling-serve UI, inference seems to run indefinitely (100% usage of local GPU). The auto-selected VLM model is Granite-docling.

I've reproduced this behavior using the following docling-serve API invocation:

curl -X 'POST' \
  'http://localhost:5001/v1/convert/source' \
  -H 'accept: application/json' \
  -H 'Content-Type: application/json' \
  -d '{
    "sources": [{"kind": "http", "url": "https://arxiv.org/pdf/2501.17887"}],
    "options": {
      "ocr_engine": "easyocr",
      "pdf_backend": "dlparse_v4",
      "from_formats": ["pdf", "docx"],
      "force_ocr": false,
      "image_export_mode": "placeholder",
      "do_ocr": true,
      "ocr_lang": ["en", "it"],
      "table_mode": "accurate",
      "to_formats": ["md", "json", "html", "text", "doctags"],
      "abort_on_error": false,
      "pipeline": "vlm",
      "vlm_pipeline_model": "granite_docling"
    }
  }'

However. I've recently found out that this problem doesn't reproduce if you invoke the docling-serve API the following way (HuggingFace parameters documented at ):

curl -X 'POST' \
  'http://localhost:5001/v1/convert/source' \
  -H 'accept: application/json' \
  -H 'Content-Type: application/json' \
  -d '{
    "sources": [{"kind": "http", "url": "https://arxiv.org/pdf/2501.17887"}],
    "options": {
      "ocr_engine": "easyocr",
      "pdf_backend": "dlparse_v4",
      "from_formats": ["pdf", "docx"],
      "force_ocr": false,
      "image_export_mode": "placeholder",
      "do_ocr": true,
      "ocr_lang": ["en", "it"],
      "table_mode": "accurate",
      "to_formats": ["md", "json", "html", "text", "doctags"],
      "abort_on_error": false,
      "pipeline": "vlm",
      "vlm_pipeline_model_local": {
        "repo_id": "ibm-granite/granite-docling-258M",
        "inference_framework": "transformers",
        "transformers_model_type": "automodel-imagetexttotext",
        "temperature": 0.0,
        "prompt": "Convert this page to docling.",
        "response_format": "doctags"
      }
    }
  }'

PLEASE NOTE: The hanging problem doesn't reproduce by running the underlying docling library the following way:

docling --pipeline vlm --vlm-model granite_docling "https://arxiv.org/pdf/2501.17887"

So, it seems that the underlying docling library is executed in a way more similar to the second docling-serve API invocation example, while the docling-serve UI seems to be more similar to the first example above.

docling-serve version = 1.14.3
docling version = 2.80.0
GPU: NVIDIA RTX 4000 SFF Ada Generation

[dosubot]

The vlm_pipeline_model parameter you're using is deprecated and has known issues. This is likely the root cause of your hanging problem.

Recommended fix: Switch to the newer vlm_pipeline_preset parameter:

{
  "options": {
    "pipeline": "vlm",
    "vlm_pipeline_preset": "granite_docling"
  }
}

The deprecated fields (vlm_pipeline_model, vlm_pipeline_model_local, vlm_pipeline_model_api) are still supported for backward compatibility but will be removed in a future release. Your workaround using vlm_pipeline_model_local with explicit HuggingFace parameters works because it bypasses whatever resolution logic is causing the hang in vlm_pipeline_model.

If you need custom model configuration beyond the preset, you can use vlm_pipeline_custom_config instead (requires admin to set DOCLING_SERVE_ALLOW_CUSTOM_VLM_CONFIG=true).

To reply, just mention @dosu.

---

^{How did I do? Good | Irrelevant | Incorrect | Verbose | Hallucination | Report 🐛 | Other}

   

[pierdomenicobatzu]

The vlm_pipeline_model parameter you're using is deprecated and has known issues. This is likely the root cause of your hanging problem.

Recommended fix: Switch to the newer vlm_pipeline_preset parameter:

{
"options": {
"pipeline": "vlm",
"vlm_pipeline_preset": "granite_docling"
}
}
The deprecated fields (vlm_pipeline_model, vlm_pipeline_model_local, vlm_pipeline_model_api) are still supported for backward compatibility but will be removed in a future release. Your workaround using vlm_pipeline_model_local with explicit HuggingFace parameters works because it bypasses whatever resolution logic is causing the hang in vlm_pipeline_model.

If you need custom model configuration beyond the preset, you can use vlm_pipeline_custom_config instead (requires admin to set DOCLING_SERVE_ALLOW_CUSTOM_VLM_CONFIG=true).

To reply, just mention @dosu.

How did I do? Good | Irrelevant | Incorrect | Verbose | Hallucination | Report 🐛 | Other

   

@dosu tried also with the newer vlm_pipeline_preset parameter suggested, same result: hangs indefinitely.

[dosubot]

Thanks for testing that—I apologize for the incomplete suggestion. The real issue is a known bug with the Transformers backend hanging on NVIDIA GPUs.

The granite_docling preset uses an AUTO_INLINE engine that defaults to Transformers on CUDA systems (because prefer_vllm=False by default), which triggers this hang.

Working solutions:

1. Use vLLM explicitly with vlm_pipeline_model_local:

{
  "options": {
    "pipeline": "vlm",
    "vlm_pipeline_model_local": {
      "repo_id": "ibm-granite/granite-docling-258M",
      "inference_framework": "vllm",
      "prompt": "Convert this page to docling.",
      "response_format": "doctags",
      "scale": 2.0
    }
  }
}

2. Or use the vLLM-specific preset (if available in your version):

{
  "options": {
    "pipeline": "vlm",
    "vlm_pipeline_model": "GRANITEDOCLING_VLLM"
  }
}

Also ensure you have UVICORN_WORKERS=1 set—LocalOrchestrator doesn't support multiple workers and can cause additional hanging issues.

The CLI likely works because it may have vLLM installed and prioritized, or has different environment configuration.

To reply, just mention @dosu.

---

^{How did I do? Good | Irrelevant | Incorrect | Verbose | Hallucination | Report 🐛 | Other}

   

[pierdomenicobatzu]

@dosu thanks for your reply.

I've verified that the usage of the "vllm" inference framework inside docling-serve works. I'm using your proposed solution:

{
  "options": {
    "pipeline": "vlm",
    "vlm_pipeline_model_local": {
      "repo_id": "ibm-granite/granite-docling-258M",
      "inference_framework": "vllm",
      "prompt": "Convert this page to docling.",
      "response_format": "doctags",
      "scale": 2.0
    }
  }
}

PLEASE NOTE (1): This configuration needs to be fixed inside the docling-serve UI as well (because it defaults to the hanging implementation when "Vlm" is selected as the pipeline).

PLEASE NOTE (2): Please consider to add the "vllm" dependency when installing with pip, because "vllm" seems to be missing both from https://github.com/docling-project/docling-serve/blob/main/pyproject.toml and from https://github.com/docling-project/docling/blob/main/pyproject.toml#L100 .

[dosubot]

Glad the vLLM configuration worked for you!

Regarding your notes:

On vLLM dependency: vLLM is actually commented out in docling's optional dependencies intentionally due to:

1. Platform restrictions — vLLM only supports Linux x86_64 and caused installation failures on Linux arm64
2. Dependency conflicts — vLLM has strict torch version pinning that causes lockfile conflicts

Users on supported platforms need to install vLLM separately (pip install vllm).

On the UI issue: You're right that the Gradio UI only exposes pipeline type selection but doesn't expose VLM-specific options like vlm_pipeline_preset or inference framework selection. When "VLM" is selected, it defaults to the AUTO_INLINE engine which picks Transformers on NVIDIA GPUs—leading to the hang.

These are valid feature requests for the maintainers:

1. Add VLM backend/preset configuration to the Gradio UI
2. Consider changing the default backend selection to vLLM on Linux x86_64 systems where it's available

To reply, just mention @dosu.

---

^{How did I do? Good | Irrelevant | Incorrect | Verbose | Hallucination | Report 🐛 | Other}