Help/possible bug: custom local endpoint shows ~37K max context tokens and triggers summarisation earlier than backend window

[robert-stevenson-1]

Hi, I’m running LibreChat against a self‑hosted local LLM stack (proxy + llama.cpp server) and I’m trying to align LibreChat’s effective context window that is shown in the UI with the larger window configured in my backend.

It’s very possible I’m missing a configuration step or using modelSpecs incorrectly, so I’d appreciate guidance rather than assuming this is a bug straight away.

Environment

• LibreChat version: 0.8.7
• Deployment: Docker (LibreChat) on Linux
• Backend: local OpenAI‑compatible proxy in front of llama.cpp (llama-server)
• Endpoint type: custom (OpenAI‑style /v1/*)
• Hardware: multi‑GPU setup (e.g. RTX‑class cards)

Goal

I have several large‑context models hosted via llama.cpp (Qwen‑family 27B/35B) with context sizes around 200k–262k configured via --ctx-size. I’d like LibreChat’s “Maximum context tokens” and summarisation logic to reflect those larger effective windows, rather than a smaller generic default.

Right now, it appears that conversations with these models are being summarised earlier than necessary relative to the backend’s actual window.

Backend configuration (summary)

One representative model:

macros:
  cluster_args: "--rpc <gpu-cluster-ip>:50052 -ngl all --flash-attn on --port ${PORT} -fitc 64000 -t 5 --no-ui --fit on -fitt 512 --tensor-split 1,1"

models:
  Qwen3.6-27B-MTP-Code-NoQuant:
    path: "/path/to/Qwen3.6-27B-MTP-GGUF/Qwen3.6-27B-Q8_0.gguf"
    cmd: >
      llama-server -m /path/to/Qwen3.6-27B-MTP-GGUF/Qwen3.6-27B-Q8_0.gguf
      ${cluster_args}
      --ctx-size 200000
      --temp 0.6
      --top-p 0.95
      --top-k 20
      --min-p 0.0
      --presence-penalty 0.0
      --repeat-penalty 1.0
      --spec-type draft-mtp --spec-draft-n-max 2
      --parallel 1
      --chat-template-file /path/to/templates/qwen/chat_template.jinja
      --chat-template-kwargs "{\"preserve_thinking\":true}"
      --mmproj /path/to/Qwen3.6-27B-MTP-GGUF/mmproj-BF16.gguf
      --image-min-tokens 1024
    metadata:
      context_length: 200000
      description: "Qwen3.6-27B MTP Code, 200k ctx, cluster backend"
      tags:
        - qwen
        - code
        - mtp
        - 200k_ctx

The proxy is started with a config pointing at this file and listens on a public port (e.g. 0.0.0.0:11434).

The OpenAI‑style /v1/models endpoint for this id returns:

{
  "id": "Qwen3.6-27B-MTP-Code-NoQuant",
  "object": "model",
  "created": 1783895129,
  "owned_by": "llama-swap",
  "meta": {
    "llamaswap": {
      "context_length": 200000,
      "description": "Qwen3.6-27B MTP Code, 200k ctx, cluster backend",
      "tags": ["qwen","code","mtp","200k_ctx"]
    }
  },
  "status": {
    "value": "unloaded"
  }
}

So the backend is aware of a 200k window and exposes that in metadata (nested under meta.llamaswap.context_length).

LibreChat configuration

In librechat.yaml I have a custom endpoint and model specs like:

version: 1.1.0

endpoints:
  agents:
    recursionLimit: 50
    maxRecursionLimit: 100
  custom:
  - name: local-llama-cluster
    apiKey: dummy-key
    baseURL: http://host.docker.internal:11434/v1
    models:
      default:
        - Qwen3.6-27B-MTP-Code-NoQuant
      fetch: true
    titleConvo: true
    titleModel: current_model
    modelDisplayLabel: Llama.cpp

modelSpecs:
  enforce: false
  prioritize: true
  list:
    - name: Qwen3.6-27B-General
      label: Qwen3.6 27B General
      default: true
      preset:
        endpoint: local-llama-cluster
        model: Qwen3.6-27B-General
      capabilities:
        - vision
        - tools
      parameters:
        maxContextTokens: 262144

    - name: Qwen3.6-27B-MTP-Code
      label: Qwen3.6 27B MTP Code
      preset:
        endpoint: local-llama-cluster
        model: Qwen3.6-27B-MTP-Code
      capabilities:
        - vision
        - tools
      parameters:
        maxContextTokens: 262144

    - name: Qwen3.6-27B-MTP-Code-NoQuant
      label: Qwen3.6 27B MTP Code NoQuant
      preset:
        endpoint: local-llama-cluster
        model: Qwen3.6-27B-MTP-Code-NoQuant
      capabilities:
        - vision
        - tools
      parameters:
        maxContextTokens: 200000

    - name: Qwen3.6-35B-A3B-Q6-General
      label: Qwen3.6 35B A3B Q6 General
      preset:
        endpoint: local-llama-cluster
        model: Qwen3.6-35B-A3B-Q6-General
      capabilities:
        - vision
        - tools
      parameters:
        maxContextTokens: 262144

So there is:

• A generic custom endpoint pointing at the local OpenAI‑compatible API.
• Per‑model modelSpecs entries that set maxContextTokens to match the backend --ctx-size.

Observed behaviour

When I select Qwen3.6-27B-MTP-Code-NoQuant via the custom endpoint in LibreChat:

• The UI’s “Maximum context tokens” section still reports a limit of roughly 37K tokens.
• Conversations with this model appear to start summarising and collapsing history around that smaller limit, even though the backend window is much larger (200k).
• The other Qwen models on the same endpoint also seem to share this kind of cap, despite their modelSpecs.maxContextTokens being 262144.

In other words, summarisation and context trimming appear to be happening earlier than necessary for these large‑context models.

What I might be missing

I’m not sure whether:

• I’m misunderstanding how modelSpecs.parameters.maxContextTokens is supposed to work for custom endpoints.
• LibreChat currently ignores modelSpecs for generic custom/OpenAI endpoints and uses an internal default for context window / summarisation.
• I should be exposing context_length differently in /v1/models (e.g. as a top‑level field instead of nested under meta) for LibreChat to pick it up.
• There is another per‑endpoint or per‑provider setting I need to adjust to make LibreChat use the larger context values.

Questions

1. For custom OpenAI‑style endpoints like this, is modelSpecs.parameters.maxContextTokens expected to override the effective context window used by LibreChat’s token accounting and summarisation, or is it currently only wired for specific providers?
2. Is there a recommended way to tell LibreChat “this model id has a 200k context window” on a generic custom endpoint, without modifying the LibreChat source?
3. Should I be exposing a top‑level context_length in /v1/models for LibreChat to detect, or is that metadata ignored for custom endpoints?

If the current behaviour is expected and I’m just missing a configuration step, I’d be grateful for any pointers to the right docs or an example librechat.yaml pattern for large‑context custom models.

If there’s a known limitation with custom endpoints and large context windows (causing summarisation to kick in based on an internal cap), I’d be interested in any suggested workarounds or in helping test changes that make modelSpecs.maxContextTokens or /v1/models metadata feed into that logic.

Thanks very much in advance.
(issue based on a summarisation of a multi hour back and forth with Perplexity AI, hence it might sound LLM-y, but has been reviewed and corrected where approriate)

[robert-stevenson-1]

Got a bit confused with the change from an issue to a discussion, but got an update:

Resolved it (debatedly in a less ideal/temporarily way, as I think there is a more elegant/automatic way to get the model info better).
Between me and an LLM that has web search abilities, we ended up missing the documentation here: https://www.librechat.ai/docs/configuration/token_usage#custom-endpoint-token-config

Basically, I need to use tokenConfig on the custom endpoint in librechat.yaml

endpoints:
  custom:
    - name: local-llama-cluster
      apiKey: dummy-key
      baseURL: http://host.docker.internal:11434/v1
      models:
        default:
          - Qwen3.6-27B-MTP-Code-NoQuant
        fetch: true
      titleConvo: true
      titleModel: current_model
      modelDisplayLabel: Llama.cpp

      tokenConfig:
        Qwen3.6-27B-General:
          prompt: 1
          completion: 1
          context: 262144

        Qwen3.6-27B-MTP-Code:
          prompt: 1
          completion: 1
          context: 262144

        Qwen3.6-27B-MTP-Code-NoQuant:
          prompt: 1
          completion: 1
          context: 200000

        Qwen3.6-35B-A3B-Q6-General:
          prompt: 1
          completion: 1
          context: 262144

[robert-stevenson-1]

Maybe, since some self‑hosted backends already expose (or have the ability to enable this exposure) their effective context window via /v1/models or similar metadata, it might be helpful if LibreChat could automatically derive the context value for custom endpoints from that information (or from modelSpecs.maxContextTokens if it is appropriate) when available.

That would reduce the need for manual tokenConfig entries and make it easier for users to plug in large‑context local models without having to keep multiple places in sync.

Of course, the explicit tokenConfig override is still very useful for edge cases and fine‑tuning.

[danny-avila]

Thanks for the thorough writeup, and glad tokenConfig sorted it. That is the intended mechanism, so to close out your questions:

Where the ~37K comes from: when a conversation is not carrying an explicit maxContextTokens, LibreChat derives the window by matching the model id against a built-in token map, using substring matching and taking the longest matching key. Qwen3.6-27B-MTP-Code-NoQuant only matches the generic qwen3 key (40960), not the larger qwen3-coder or qwen3-vl-* entries (262144), because those exact substrings are not present in your id. After the reserved output headroom, that lands near the ~37K you saw, and that value is what drives summarisation.

Q1 (does modelSpecs.parameters.maxContextTokens override the effective window?): It is not provider-gated, and it does feed the budget, but only for conversations actually started from that spec. With enforce: false, the raw endpoint models stay directly selectable, and selecting one bypasses the spec and lands on the name-based lookup above. tokenConfig is endpoint-level and is checked first inside that same lookup, so it applies no matter how the model is chosen, which is why it fixed both the displayed limit and summarisation.

Q2 (recommended way to declare 200k on a generic custom endpoint): Yes, tokenConfig on the custom endpoint is the recommended and documented approach, exactly the page you linked. Only context affects the window. prompt and completion are for cost display, so 1/1 is fine.

Q3 (should you expose context_length in /v1/models?): It will not help today. The model fetch reads only each entry's id and ignores the rest, including both a top-level context_length and your meta.llamaswap.context_length. So there is no need to restructure that response.

On auto-deriving the value: reasonable idea. The main blocker is that the OpenAI /v1/models schema has no standard context-length field, so backends expose it inconsistently (top-level vs vendor-nested like your meta.llamaswap.*), and there is nothing portable to key off yet. A feature request for opt-in detection is welcome, with tokenConfig staying as the explicit override for edge cases.