Offline LLMs

[Hrishikesh007788]

@gventuri Is there a way to connect pandasai with LLM in our local machine completely offline? And is there is way to serve the local LLM in a server such that anyone can access it?

[philchass]

Same question, I can't find the way to do it

[richfly]

Are you trying with LM Studio?

[mspronesti]

@Hrishikesh007788 Just import the local LLM you want to use via langchain and pass it to a SmartDataframe. It will be internally converted by pandas-ai. Have a look at the docs for more (scroll down to the last section).

Example:

from langchain_community.llms.huggingface_pipeline import HuggingFacePipeline
from transformers import AutoModelForCausalLM, AutoTokenizer, pipeline

model_id = "gpt2" 
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(model_id)
pipe = pipeline("text-generation", model=model, tokenizer=tokenizer, max_new_tokens=100)
llm = HuggingFacePipeline(pipeline=pipe)

df = SmartDataframe(..., config={"llm": llm})

Hope this helps.

[lordpba]

use ollama, should work (as I know under Linux and Mac actually)

example (after installed ollama on local machine):

from langchain_community.llms import Ollama
llm= Ollama(model='mistral', temperature=0.1)

[nv-zhangineer]

I tried this, and always get empty LLM object....
the df from SmartDataframe simply stops working without any errors. I also confirmed that localhost:11434 works and I was also able to hit the Ollama GUI via another port.

from langchain_ollama.llms import OllamaLLM
import pandas as pd
from pandasai import SmartDataframe
llm= OllamaLLM(model='llama3:8b', temperature=0.1, base_url="http://localhost:11434")
print (llm)   <<<<<<<<< returns: `Params: {}`
data = pd.read_csv(uploader_file)
print ("Data read")
df = SmartDataframe(data, config={"llm": llm})

[nv-zhangineer]

I realized that I was using 3.0 version which doesn't seem to have support for Ollama, so I downgraded to 2.0, but still experiencing the same issue. In the v2.0 document, I used the exactly same code

from pandasai.llm.local_llm import LocalLLM
import pandas as pd
from pandasai import SmartDataframe
ollama_llm = LocalLLM(api_base="http://localhost:11434/v1", model="llama3:8b")
data = pd.read_csv("data.csv")
df = SmartDataframe(data, config={"llm": ollama_llm})  <<< This code basically not returning anything and not moving forward either.
print ("Data read")

[nv-zhangineer]

okay, what eventually worked for me is to disable caching. If I don't disable caching, then the smartdataframe basically hangs and quits

    config = {
        "llm": ollama_llm,
        "enable_cache": False,  # Disable caching to avoid cache-related hangs