Struggling with talking to self hosted ollama and open-webui (mistral) via API

[bodleytunes]

Hi

Im testing trying to connect to my llm instance hosted on a PC, which works fine if I access via the web UI, but any attempt at talking to it via API always fails as I think I have some parameters incorrect in my Python code.

in docker I see everything is running fine and its all reachable via ping and curl so connectivity is fine.

CONTAINER ID   IMAGE                                COMMAND               CREATED          STATUS                    PORTS                                             NAMES
93d553d2e862   ollama/ollama:latest                 "/bin/ollama serve"   34 minutes ago   Up 34 minutes             0.0.0.0:11434->11434/tcp, [::]:11434->11434/tcp   ollama
d49a542ae250   ghcr.io/open-webui/open-webui:main   "bash start.sh"       55 minutes ago   Up 34 minutes (healthy)   0.0.0.0:3000->8080/tcp, [::]:3000->8080/tcp       open-webui

on the crawl4ai front I have this setup for the llm extraction strategy:

def get_self_hosted_llm_extraction_strategy() -> LLMExtractionStrategy:
    return LLMExtractionStrategy(
        #provider="litellm",
        provider="ollama/mistral-7b-instruct-q4_0",
        base_url="http://10.12.99.108:3000",  # Ensure this matches your port mapping.
        api_token="sk-8604___de4b23",  # If your service requires an API token.
        extraction_type="schema",
        instruction=(
            """
            1. Extract all rows of trips from the journey planner.
            2. For each row, extract the departure and arrival times as well as the prices.
            3. Return the extracted information as a JSON object.
            """
        ),
        input_format="markdown",
        verbose=True
    )

And also this:

            extraction_strategy = get_self_hosted_llm_extraction_strategy()
            extraction_result = extraction_strategy.run(
                url=f"raw:{html_content}",
                sections=['trips']
            )

And then keep receiving this error :(

ock index: 0

Give Feedback / Get Help: https://github.com/BerriAI/litellm/issues/new
LiteLLM.Info: If you need to debug this error, use `litellm._turn_on_debug()'.


Provider List: https://docs.litellm.ai/docs/providers

Error in thread execution: litellm.APIConnectionError: OllamaException - {"detail":"Method Not Allowed"}

Any ideas where Im going wrong or what to try to get it working? I think its some argument that is wrong or possibly I should be hitting ollama directly but that doesn't work either.

Just to add, if I hit ollama container directly on 11434, I get a different error:

Give Feedback / Get Help: https://github.com/BerriAI/litellm/issues/new
LiteLLM.Info: If you need to debug this error, use `litellm._turn_on_debug()'.


Provider List: https://docs.litellm.ai/docs/providers

Error in thread execution: litellm.APIConnectionError: OllamaException - {"error":"model 'mistral-7b-instruct-q4_0' not found"}

Thanks,

Jon.

[bodleytunes]

OK I got it working it was a typo with the model

def get_self_hosted_llm_extraction_strategy() -> LLMExtractionStrategy:
    return LLMExtractionStrategy(
        provider="ollama/mistral:7b-instruct-q4_0",
        base_url="http://10.12.99.108:11434",  # Ensure this matches your port mapping.
        api_token="sk-86042ce___de4b23",  # If your service requires an API token.
        extraction_type="schema",
        instruction=(
            """
            1. Extract all rows of trips from the journey planner.
            2. For each row, extract the departure and arrival times as well as the prices.
            3. Return the extracted information as a JSON object.
            """
        ),
        input_format="markdown",
        verbose=True,
        apply_chunking=True,
        timeout=6000
    )

Its a bit slow but at least its not erroring any more!

Jon.

[bodleytunes]

Not sure why but all it returns is:

[
  {
    "tags": [
      "introduction"
    ],
    "error": false,
    "index": 0,
    "content": [
      "This is the first paragraph of the article, which provides an introduction and overview of the main topic."
    ]
  },
  {
    "tags": [
      "background"
    ],
    "error": false,
    "index": 1,
    "content": [
      "This is the second paragraph, which delves into the history and background of the topic.",
      "It provides context and sets the stage for the rest of the article."
    ]
  }
]

🤷🏻