Run docling-serve VLM through external API

[bestonephoenix]

I am trying hard to get docling-serve to run using a VLM per default, using an external API endpoint, unsuccessfully.
The system continues to look for internal and seems to ignore all variables set.

using docling-serve 1.15.1

ENV Variables:
DOCLING_SERVE_ENABLE_REMOTE_SERVICES=true
DOCLING_SERVE_ALLOW_CUSTOM_VLM_CONFIG=true
DOCLING_SERVE_VLM_API_URL= "https://xxx/v1/chat/completions"
DOCLING_SERVE_VLM_API_HEADERS_JSON='{"Authorization":"Bearer xxx"}'
DOCLING_SERVE_VLM_API_PARAMS_JSON='{"model": "granite_docling","max_completion_tokens:"1500}'
DOCLING_SERVE_VLM_RESPONSE_FORMAT="<enum_value>"
DOCLING_SERVE_VLM_TEMPERATURE="0.0"
DOCLING_SERVE_DEFAULT_PIPELINE=vlm
DOCLING_SERVE_MAX_SYNC_WAIT=600

Error shown is
ERROR:docling_jobkit.orchestrators.local.worker:Worker 1 failed to process job 420d1082-ed82-4003-bdb6-06a843786198: Model 'ibm-granite/granite-docling-258M' not found in artifacts_path.
Expected location: /opt/app-root/src/.cache/docling/models/ibm-granite--granite-docling-258M

Trying from ui and external.

[dosubot]

Hi @bestonephoenix! I'm Dosu and I’m helping the docling-serve team.

The issue is that DOCLING_SERVE_DEFAULT_PIPELINE=vlm sets the pipeline to the local VLM pipeline, which tries to load the ibm-granite/granite-docling-258M model from disk — the DOCLING_SERVE_VLM_API_URL and related env vars don't redirect that pipeline to an external API.

To use an external API endpoint, you should use presets or custom configs instead. There are two approaches:

Option A: Server-side custom VLM preset (recommended)

Define a custom preset via environment variable and set it as the default:

DOCLING_SERVE_ENABLE_REMOTE_SERVICES=true
DOCLING_SERVE_ALLOW_CUSTOM_VLM_CONFIG=true
DOCLING_SERVE_CUSTOM_VLM_PRESETS='{"my_external_vlm": {"engine_options": {"engine_type": "api"}, "api_overrides": {"api": {"url": "https://xxx/v1/chat/completions", "headers": {"Authorization": "Bearer xxx"}, "params": {"model": "granite_docling", "max_completion_tokens": 1500}}}}}'
DOCLING_SERVE_DEFAULT_VLM_PRESET=my_external_vlm

[1] [2]

Option B: Per-request custom config

Pass the external API config in each request body:

{
  "options": {
    "pipeline": "vlm",
    "vlm_custom_config": {
      "engine_options": {"engine_type": "api"},
      "api_overrides": {
        "api": {
          "url": "https://xxx/v1/chat/completions",
          "headers": {"Authorization": "Bearer xxx"},
          "params": {"model": "granite_docling", "max_completion_tokens": 1500}
        }
      }
    }
  }
}

Also note: there's a minor JSON syntax error in your env var — "max_completion_tokens:"1500} is missing the closing quote after max_completion_tokens. It should be "max_completion_tokens": 1500.

The key takeaway is that engine_type: "api" is what tells docling-serve to call an external endpoint rather than loading a local model [2]. Without that, it defaults to local inference and looks for the model files on disk [3].

To reply, just mention @dosu.

---

^{How did I do? Good | Irrelevant | Incorrect | Verbose | Hallucination | Report 🐛 | Other}

   

[bestonephoenix]

I went for the server-side config, but it still tries to find it locally - I am trying first using the UI to eliminate possible glitches: standard pipeline (CPU) works, but VLM returns a failure and no API call is being issued.

xxx@instance-xxx-1627:~$ ERROR:docling_jobkit.orchestrators.local.worker:Worker 0 failed to process job 565d54e2-89cc-4e9b-9664-42eec25450d9: Model 'ibm-granite/granite-docling-258M' not found in artifacts_path.
Expected location: /opt/app-root/src/.cache/docling/models/ibm-granite--granite-docling-258M
Available models in /opt/app-root/src/.cache/docling/models: docling-project--docling-models, RapidOcr, docling-project--docling-layout-heron, docling-project--DocumentFigureClassifier-v2.0, EasyOcr

To fix this issue:
  1. Download the model: docling-tools models download-hf-repo ibm-granite/granite-docling-258M
  2. Or remove --artifacts-path to enable auto-download
  3. Or use a different model that exists in your artifacts_path
INFO:     170.85.2.245:24318 - "POST /ui/gradio_api/queue/join HTTP/1.1" 200 OK
INFO:     170.85.2.245:24318 - "GET /ui/gradio_api/queue/data?session_hash=fo3z2yov9bj HTTP/1.1" 200 OK
INFO:     170.85.2.245:24318 - "POST /ui/gradio_api/queue/join HTTP/1.1" 200 OK
INFO:     170.85.2.245:24318 - "GET /ui/gradio_api/queue/data?session_hash=fo3z2yov9bj HTTP/1.1" 200 OK
INFO:     127.0.0.1:37372 - "GET /v1/status/poll/565d54e2-89cc-4e9b-9664-42eec25450d9?wait=5 HTTP/1.1" 200 OK
ERROR:docling_serve.gradio_ui:Error processing file(s): Task failed with status 'failure'

I have removed as much as possible from the config leaving only the VLM preset

DOCLING_SERVE_ENABLE_UI=true
DOCLING_SERVE_ENABLE_REMOTE_SERVICES=true
DOCLING_SERVE_MAX_SYNC_WAIT=600
DOCLING_SERVE_ALLOW_CUSTOM_VLM_CONFIG=true
DOCLING_SERVE_CUSTOM_VLM_PRESETS='{"llama-vlm": {"engine_options": {"engine_type": "api"}, "api_overrides": {"api": {"url": "https://xxx/v1/chat/completions", "headers": {"Authorization": "Bearer xxx"}, "params": {"model": "granite_docling", "max_completion_tokens": 1500}}}}}'
DOCLING_SERVE_DEFAULT_VLM_PRESET=llama-vlm

[dosubot]

This looks like a gap in how the Gradio UI interacts with the preset system. The UI only sends pipeline="vlm" in the request but never includes the vlm_pipeline_preset parameter [1]. When that field is omitted, the server's fallback logic doesn't consult DOCLING_SERVE_DEFAULT_VLM_PRESET — instead, it falls back to hardcoded legacy model specs that use AUTO_INLINE (local model loading) [2] [3]. So your custom preset configuration is correct but simply never gets used.

Workaround: Use the REST API directly and explicitly pass the preset:

curl -X POST "http://localhost:5000/v1/convert/source/async" \
  -H "Content-Type: application/json" \
  -d '{
    "sources": [{"url": "https://example.com/document.pdf"}],
    "options": {
      "pipeline": "vlm",
      "vlm_pipeline_preset": "llama-vlm"
    }
  }'

Or use "vlm_pipeline_preset": "default" which should resolve to your DOCLING_SERVE_DEFAULT_VLM_PRESET=llama-vlm [4].

The underlying issue is that the Gradio UI has no controls for VLM preset selection, and the server doesn't automatically inject the configured default preset when the field is absent. This seems like a bug/feature gap worth tracking.

To reply, just mention @dosu.

---

^{How did I do? Good | Irrelevant | Incorrect | Verbose | Hallucination | Report 🐛 | Other}
   

[bestonephoenix]

@dosu I tried using "vlm_pipeline_preset": "default" which should resolve to my DOCLING_SERVE_DEFAULT_VLM_PRESET=llama-vlm. But it continues to try using the internal VLM - never makes an attempt to call out to my external API :(

[dosubot]

The likely problem is that your custom preset JSON structure is incorrect — specifically, it's missing the required model_spec field and the API configuration should go directly into engine_options, not under api_overrides. When the preset fails validation at request time, the system silently falls back to the built-in local model. Also, invalid JSON in the env var is silently swallowed (returns empty dict with no error) [1].

Try this corrected configuration:

DOCLING_SERVE_ENABLE_REMOTE_SERVICES=true
DOCLING_SERVE_ALLOW_CUSTOM_VLM_CONFIG=true
DOCLING_SERVE_CUSTOM_VLM_PRESETS='{"llama-vlm": {"model_spec": {"name": "Custom API VLM", "default_repo_id": "ibm-granite/granite-docling-258M", "prompt": "Convert this page to docling.", "response_format": "doctags"}, "engine_options": {"engine_type": "api", "url": "https://xxx/v1/chat/completions", "headers": {"Authorization": "Bearer xxx"}, "params": {"model": "granite_docling", "max_completion_tokens": 1500}, "timeout": 60.0, "concurrency": 1}, "scale": 2.0, "batch_size": 1}}'
DOCLING_SERVE_DEFAULT_VLM_PRESET=llama-vlm

Key differences from your current config:

1. Added model_spec — this is required and includes name, default_repo_id, prompt, and response_format [2] [3]
2. Moved API config into engine_options directly — url, headers, params go here, not under api_overrides.api [4]
3. Added scale and batch_size for completeness

You can verify your JSON parses correctly before starting the server:

echo "$DOCLING_SERVE_CUSTOM_VLM_PRESETS" | jq .

Also enable debug logging with DOCLING_SERVE_LOG_LEVEL=DEBUG to get more visibility into what's happening.

When making requests, either use vlm_pipeline_preset: "llama-vlm" directly, or vlm_pipeline_preset: "default" which should resolve to llama-vlm via your DOCLING_SERVE_DEFAULT_VLM_PRESET setting [5]. Note that the Gradio UI still won't pass the preset — you'll need to use the REST API to test this.

To reply, just mention @dosu.

---

^{How did I do? Good | Irrelevant | Incorrect | Verbose | Hallucination | Report 🐛 | Other}
   

[bestonephoenix]

I have set the environment variable
DOCLING_SERVE_DEFAULT_VLM_PRESET=llama-vlm

I am not able to see docling making a call to external API.

Having extended the Logs to INFO, I see the below lines: could it be that somehow my env only activates picture_description using VLM, but not processing of the entire page through VLM?

4/09/2026, 07:30:20 PM
STDERR
INFO:docling.models.factories.base_factory:Loading plugin 'docling_defaults'
04/09/2026, 07:30:20 PM
STDERR
INFO:docling.models.factories:Registered table structure engines: ['docling_tableformer', 'docling_tableformer_v2']
04/09/2026, 07:30:20 PM
STDERR
INFO:docling.models.factories.base_factory:Loading plugin 'docling_defaults'
04/09/2026, 07:30:20 PM
STDERR
INFO:docling.models.factories:Registered layout engines: ['layout_object_detection', 'docling_layout_default', 'docling_experimental_table_crops_layout']
04/09/2026, 07:30:20 PM
STDERR
INFO:docling_jobkit.convert.manager:artifacts_path is set to a valid directory. No model weights will be downloaded at runtime.
04/09/2026, 07:30:20 PM
STDERR
INFO:docling.document_converter:Initializing pipeline for StandardPdfPipeline with options hash e0074757da0fd7ce3e0e8012f4d4d8f2
04/09/2026, 07:30:20 PM
STDERR
INFO:docling.models.factories.base_factory:Loading plugin 'docling_defaults'
04/09/2026, 07:30:20 PM
STDERR
INFO:docling.models.factories:Registered picture descriptions: ['picture_description_vlm_engine', 'vlm', 'api']
04/09/2026, 07:30:20 PM
STDERR
INFO:docling.utils.accelerator_utils:Accelerator device: 'cpu'
04/09/2026, 07:30:20 PM
STDERR
[INFO] 2026-04-09 17:30:20,826 [RapidOCR] base.py:22: Using engine_name: onnxruntime
04/09/2026, 07:30:20 PM
STDERR
[INFO] 2026-04-09 17:30:20,828 [RapidOCR] main.py:53: Using /opt/app-root/src/.cache/docling/models/RapidOcr/onnx/PP-OCRv4/det/ch_PP-OCRv4_det_infer.onnx
04/09/2026, 07:30:20 PM
STDERR
[INFO] 2026-04-09 17:30:20,922 [RapidOCR] base.py:22: Using engine_name: onnxruntime
04/09/2026, 07:30:20 PM
STDERR
[INFO] 2026-04-09 17:30:20,922 [RapidOCR] main.py:53: Using /opt/app-root/src/.cache/docling/models/RapidOcr/onnx/PP-OCRv4/cls/ch_ppocr_mobile_v2.0_cls_infer.onnx
04/09/2026, 07:30:20 PM
STDERR
[INFO] 2026-04-09 17:30:20,955 [RapidOCR] base.py:22: Using engine_name: onnxruntime
04/09/2026, 07:30:20 PM
STDERR
[INFO] 2026-04-09 17:30:20,955 [RapidOCR] main.py:53: Using /opt/app-root/src/.cache/docling/models/RapidOcr/onnx/PP-OCRv4/rec/ch_PP-OCRv4_rec_infer.onnx
04/09/2026, 07:30:21 PM
STDERR
INFO:docling.models.stages.ocr.auto_ocr_model:Auto OCR model selected rapidocr with onnxruntime.
04/09/2026, 07:30:21 PM
STDERR
INFO:docling.utils.accelerator_utils:Accelerator device: 'cpu'
04/09/2026, 07:30:21 PM
STDERR
Loading weights:   0%|          | 0/770 [00:00<?, ?it/s]
Loading weights:  60%|██████    | 463/770 [00:00<00:00, 4626.93it/s]
Loading weights: 100%|██████████| 770/770 [00:00<00:00, 4650.86it/s]
04/09/2026, 07:30:21 PM
STDERR
DEBUG:docling_ibm_models.layoutmodel.layout_predictor:LayoutPredictor settings: {'model_name': 'RTDetrV2ForObjectDetection', 'safe_tensors_file': '/opt/app-root/src/.cache/docling/models/docling-project--docling-layout-heron/model.safetensors', 'device': 'cpu', 'num_threads': 4, 'image_size': SizeDict(height=640, width=640, longest_edge=None, shortest_edge=None, max_height=None, max_width=None), 'threshold': 0.3}
04/09/2026, 07:30:21 PM
STDERR
INFO:docling.utils.accelerator_utils:Accelerator device: 'cpu'