Configuring local models served by llama.cpp for paperqa

[rosegarden-coder]

Appreciate the enormous efforts to release the new version v2025.12.17 and possibly future release of paperQA v3.

After some tinkering, I recorded the working steps in the next comment.

[rosegarden-coder]

The steps are much easier than I thought to make llama.cpp llama-server work with paperqa.

1. Set environment variables for llama-server api address

The llama-server api runs on a LAN computer with ip 192.168.0.2 port 8080.

export OPENAI_API_BASE=http://192.168.0.2:8080/v1
export OPENAI_API_KEY=sk-no-key-needed

That's it. You don't need to complicated settings to make paperqa work with local models.

2. Configure the embedding model in a json file.

The embedding setting is separate. Here is the content of the ~/.pqa/settings/embedding.json file.

{
  "embedding": "openai/custom-embedding",
  "embedding_config": {
    "model_list": [
      {
        "model_name": "openai/custom-embedding",
        "litellm_params": {
          "model": "text-embedding-3-small",
          "api_base": "http://192.168.0.2:8081/v1",
          "api_key": "sk-no-key-needed",
          "drop_params": true
        }
      }
    ]
  },
  "verbosity": 3,
  "agent": {
    "agent_type": "ToolSelector",
    "index": {
      "paper_directory": "/home/x/papers",
      "index_directory": "/home/x/.pqa/indexes"
    }
  }
}

3. Run the interference model and embedding models

Note, you need a model with vision support to work properly with the new version paperqa v2025.12.17.

Running the gguf model with its mmproj file:

llama-server --port 8080 --host 0.0.0.0 --n-gpu-layers 99 --ctx-size 32768 --flash-attn on -m ./Ministral-3-14B-Instruct-2512-UD-Q8_K_XL.gguf --mmproj ./Ministral-3-14B-Instruct-2512-UD-Q8_K_XL.mmproj-BF16.gguf

Running the embedding model:

llama-server -m ./Qwen3-Embedding-8B-Q4_K_M.gguf -c 32768 --port 8081 --host 0.0.0.0 -ngl 99 --embedding

4. Ask a question in command line mode:

pqa -s ~/.pqa/settings/embedding ask "When is Christmas?"

Voilà, you are good to go.

[rosegarden-coder]

Some follow-up caveats:

1. You don't need a model with vision support to use older version PaperQA version: 5.28.0, the steps are the same to work with llama-server, you just do no need to load the vision model with --mmproj
2. I would recommend switching from ollama to llama.cpp (llama-server) because ollama had problems of tool calling with models such as qwen3:30b-a3b served on its website.
3. [Ministral-3-14B-Instruct](https://huggingface.co/unsloth/Ministral-3-14B-Instruct-2512-GGUF) despite its size works surprisingly well. Please note, some other models even with vision support will connect timeout. Another model I have tested to be working is Qwen3-VL-30B-A3B. Mistral and Qwen produce very solid models and you can just experiment with them.
4. I never used corporate apis with paperqa and I see no need to do so: because we do not need its vast internal knowledge to answer the questions. As long as local models can reason and understand languages well, it will be able to provide good quality answers with the benefit of not sending your data to them. Maybe I made the wrong assumption without actually using the corporate apis.

[jamesbraza]

Hi @rosegarden-coder thanks for looking into all of this, it's great and I agree with what all you've detailed 👍

Would you be open to contributing this to our docs/tutorials/ as an IPython notebook?

[rosegarden-coder]

@jamesbraza Yes. I record the steps in my notes anyway. Sharing the steps and learn from each other would be a great experience for all.

[rosegarden-coder]

Update: to use vllm to serve the embedding model:

vllm serve ~/Qwen3-Embedding-8B \
  --runner pooling \
  --trust-remote-code \
  --dtype auto \
  --port 8081 \
  --served-model-name Qwen3-Embedding-8B

For paperqa Version: 2026.2.16 use this embedding json configuration (changed into openai_like):

{
    "embedding": "openai_like/custom-embedding",
    "embedding_config": {
      "model_list": [
        {
          "model_name": "openai_like/custom-embedding",
          "litellm_params": {
            "model": "openai_like/Qwen3-Embedding-8B",
            "api_base": "http://192.168.0.2:8081/v1",
            "api_key": "sk-no-key-needed",
            "drop_params": true
          }
        }
      ]
    },
    "verbosity": 3,
    "agent": {
      "agent_type": "ToolSelector",
      "index": {
        "paper_directory": "/home/x/papers",
        "index_directory": "/home/x/.pqa/indexes"
      }
    }
  }

for more information, refer to this post