Unclear about how to use `picture_description_custom_config` and ` vlm_pipeline_custom_config` together in new version

[hdnh2006]

Hello,

Thanks for this fantastic tool. The potential is incredible, however TBH, the documentation is terrible.

I appreciate the effort making an incredible code, but I was working on trying to make work together the parameters picture_description_api and vlm_pipeline_model_api and they were deprecated in favor of picture_description_custom_config and vlm_pipeline_custom_config. However, there's no a single example in the documentation that help us to understand how this new version is working.

In the past, I was able to configure only vlm_pipeline_model_api in the following way:

{
  "pipeline": "vlm",
  "do_ocr": false,
  "do_picture_description": true,
  "image_export_mode": "embedded",
  "include_images": "true",
  "to_formats": [
    "md"
  ],
  "vlm_pipeline_model_api": "{\"url\": \"https://my_openai_compatible_endpoint/api/chat/completions\", \"headers\": {\"Authorization\": \"Bearer sk-my-token\"}, \"params\": {\"model\": \"MY-VLM-MODEL\", \"max_tokens\": 8192}, \"prompt\": \"Convert this page to markdown. Do not miss any text and only output the bare markdown! Your response will be used as a RAG by another model, it is important you don't make any mistake. DO YOUR BEST\", \"response_format\": \"markdown\", \"timeout\": 180.0, \"concurrency\": 16}",
  "picture_description_api": "{\"url\": \"https://my_openai_compatible_endpoint/api/chat/completions\", \"headers\": {\"Authorization\": \"Bearer sk-my-token\"}, \"params\": {\"model\": \"MY-VLM-MODEL\"}, \"prompt\": \"Describe this image in detail for RAG purposes, your description must be absolutely detailed\", \"timeout\": 180.0, \"concurrency\": 16}"
}

How can I do it now? I want to integrate it to Open WebUI

[dosubot]

Here's how to migrate from vlm_pipeline_model_api and picture_description_api to the new custom config format:

1. Enable custom configs on the server (required environment variables):

export DOCLING_SERVE_ENABLE_REMOTE_SERVICES=true
export DOCLING_SERVE_ALLOW_CUSTOM_VLM_CONFIG=true
export DOCLING_SERVE_ALLOW_CUSTOM_PICTURE_DESCRIPTION_CONFIG=true

2. Migrated config example (using both vlm_pipeline_custom_config and picture_description_custom_config):

{
  "options": {
    "to_formats": ["md"],
    "do_picture_description": true,
    "image_export_mode": "embedded",
    "include_images": true,
    "picture_description_custom_config": {
      "engine_options": {"engine_type": "api"},
      "model_spec": {
        "prompt": "Describe this image in detail for RAG purposes, your description must be absolutely detailed",
        "response_format": "text"
      },
      "generation_config": {"max_tokens": 8192},
      "api_overrides": {
        "api": {
          "url": "https://my_openai_compatible_endpoint/api/chat/completions",
          "headers": {"Authorization": "Bearer sk-my-token"},
          "params": {"model": "MY-VLM-MODEL"},
          "timeout": 180.0
        }
      }
    },
    "vlm_pipeline_custom_config": {
      "engine_options": {"engine_type": "api"},
      "model_spec": {
        "prompt": "Convert this page to markdown. Do not miss any text and only output the bare markdown!",
        "response_format": "markdown"
      },
      "generation_config": {"max_tokens": 8192},
      "api_overrides": {
        "api": {
          "url": "https://my_openai_compatible_endpoint/api/chat/completions",
          "headers": {"Authorization": "Bearer sk-my-token"},
          "params": {"model": "MY-VLM-MODEL"},
          "timeout": 180.0
        }
      }
    }
  },
  "file_sources": [{"base64_string": "<base64-pdf>", "filename": "doc.pdf"}]
}

Key migration mappings:

• url → api_overrides.api.url
• headers → api_overrides.api.headers
• params.model → api_overrides.api.params.model
• max_tokens → generation_config.max_tokens
• prompt → model_spec.prompt
• response_format → model_spec.response_format
• timeout → api_overrides.api.timeout

Note: The concurrency parameter is not directly available in the new format.

Also worth noting: as discussed in issue #522, there's work planned to allow configuring custom presets via config files, which will simplify this further.

To reply, just mention @dosu.

---

^{How did I do? Good | Irrelevant | Incorrect | Verbose | Hallucination | Report 🐛 | Other}

   

[hdnh2006]

Ok @dosu, but I asume I have to set "pipeline": "vlm", any other parameter I am missing?

[dosubot]

Actually, no — you don't need to set "pipeline": "vlm" explicitly. When you provide vlm_pipeline_custom_config, it automatically triggers VLM pipeline behavior.

However, I did miss a required field in model_spec — you need default_repo_id. Here's the corrected minimal config:

{
  "options": {
    "to_formats": ["md"],
    "do_picture_description": true,
    "vlm_pipeline_custom_config": {
      "engine_options": {"engine_type": "api"},
      "model_spec": {
        "name": "My Custom Model",
        "default_repo_id": "your-org/your-model",
        "prompt": "Convert this page to markdown...",
        "response_format": "markdown"
      },
      "generation_config": {"max_tokens": 8192},
      "api_overrides": {
        "api": {
          "url": "https://my_openai_compatible_endpoint/api/chat/completions",
          "headers": {"Authorization": "Bearer sk-my-token"},
          "params": {"model": "MY-VLM-MODEL"},
          "timeout": 180.0
        }
      }
    }
  },
  "file_sources": [{"base64_string": "<base64-pdf>", "filename": "doc.pdf"}]
}

The required fields in vlm_pipeline_custom_config are:

• engine_options.engine_type (use "api" for remote endpoints)
• model_spec.name, model_spec.default_repo_id, model_spec.prompt, model_spec.response_format

For default_repo_id, you can use any reasonable identifier string since you're pointing to a custom API endpoint anyway.

To reply, just mention @dosu.

---

^{How did I do? Good | Irrelevant | Incorrect | Verbose | Hallucination | Report 🐛 | Other}

   

[hdnh2006]

Okay @dosu, but I am still getting no description for images, this is my current config file for the integration with Open WebUI:

{
  "pipeline": "vlm",
  "do_ocr": false,
  "image_export_mode": "embedded",
  "include_images": "true",
  "do_code_enrichment": true,
  "do_formula_enrichment": true,
  "abort_on_error": false,
  "return_as_file": false,
  "to_formats": [
    "md"
  ],
  "vlm_pipeline_model_api": "{\"url\": \"https://my_open_ai_compatible/api/chat/completions\", \"headers\": {\"Authorization\": \"Bearer sk-1234\"}, \"params\": {\"model\": \"MY-VLM-MODEL\", \"max_tokens\": 8192}, \"prompt\": \"Convert this page to markdown without using block codes. Do not miss any text and only output the bare markdown! Your response will be used as a RAG by another model, it is important you don't make any mistake, if the image is empty, just reply empty page. DO YOUR BEST and DO NOT MAKE ANY MISTAKE\", \"response_format\": \"markdown\", \"timeout\": 180.0, \"concurrency\": 16}",
  "picture_description_api": "{\"url\": \"https://my_open_ai_compatible/api/chat/completions\", \"headers\": {\"Authorization\": \"Bearer sk-1234\"}, \"params\": {\"model\": \"MY-VLM-MODEL\"}, \"prompt\": \"Describe this image in detail for RAG purposes, your description must be absolutely detailed\", \"timeout\": 180.0, \"concurrency\": 16}"
}

Am I missing something?

[dosubot]

I can see two things missing from your config:

1. do_picture_description is not set — it defaults to false, so even though you have picture_description_api configured, picture description is never triggered.

2. image_alt_mode is not set to "description" — it defaults to "static", which always renders "Image" as the alt text in markdown. Even if descriptions were generated, they wouldn't appear in the output without this.

Try adding these to your config:

{
  "do_picture_description": true,
  "image_alt_mode": "description",
  "picture_description_area_threshold": 0,
  ...rest of your config
}

Setting picture_description_area_threshold to 0 ensures all images get described regardless of size [1].

So the full corrected config would look like:

{
  "pipeline": "vlm",
  "do_ocr": false,
  "do_picture_description": true,
  "image_alt_mode": "description",
  "picture_description_area_threshold": 0,
  "image_export_mode": "embedded",
  "include_images": "true",
  "do_code_enrichment": true,
  "do_formula_enrichment": true,
  "abort_on_error": false,
  "return_as_file": false,
  "to_formats": ["md"],
  "vlm_pipeline_model_api": "{\"url\": \"https://my_open_ai_compatible/api/chat/completions\", \"headers\": {\"Authorization\": \"Bearer sk-1234\"}, \"params\": {\"model\": \"MY-VLM-MODEL\", \"max_tokens\": 8192}, \"prompt\": \"Convert this page to markdown without using block codes. Do not miss any text and only output the bare markdown! Your response will be used as a RAG by another model, it is important you don't make any mistake, if the image is empty, just reply empty page. DO YOUR BEST and DO NOT MAKE ANY MISTAKE\", \"response_format\": \"markdown\", \"timeout\": 180.0, \"concurrency\": 16}",
  "picture_description_api": "{\"url\": \"https://my_open_ai_compatible/api/chat/completions\", \"headers\": {\"Authorization\": \"Bearer sk-1234\"}, \"params\": {\"model\": \"MY-VLM-MODEL\"}, \"prompt\": \"Describe this image in detail for RAG purposes, your description must be absolutely detailed\", \"timeout\": 180.0, \"concurrency\": 16}"
}

Let me know if that fixes it!

To reply, just mention @dosu.

---

^{How did I do? Good | Irrelevant | Incorrect | Verbose | Hallucination | Report 🐛 | Other}