Skip to content

Commit ab7f9da

Browse files
committed
Add OCI LangChain wrapper, docs, and dependency integration
Wire oci_langchain wrapper using langchain-oci with OCI SDK client construction, max_completion_tokens for OpenAI models, retry and timeout support. Add OCI integration docs, test fixtures, and declare uv extra conflicts for langchain-oci compatibility.
1 parent f42c9cd commit ab7f9da

15 files changed

Lines changed: 1446 additions & 860 deletions

File tree

docs/source/build-workflows/llms/index.md

Lines changed: 43 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -28,6 +28,7 @@ NVIDIA NeMo Agent Toolkit supports the following LLM providers:
2828
| [OpenAI](https://openai.com) | `openai` | OpenAI API |
2929
| [AWS Bedrock](https://aws.amazon.com/bedrock/) | `aws_bedrock` | AWS Bedrock API |
3030
| [Azure OpenAI](https://learn.microsoft.com/en-us/azure/ai-foundry/openai/quickstart) | `azure_openai` | Azure OpenAI API |
31+
| [OCI Generative AI](https://docs.oracle.com/en-us/iaas/Content/generative-ai/home.htm) | `oci` | OCI Generative AI via the OCI SDK-backed `langchain-oci` integration |
3132
| [LiteLLM](https://github.com/BerriAI/litellm) | `litellm` | LiteLLM API |
3233
| [Hugging Face](https://huggingface.co) | `huggingface` | Hugging Face API |
3334
| [Hugging Face Inference](https://huggingface.co/docs/api-inference) | `huggingface_inference` | Hugging Face Inference API, Endpoints, and TGI |
@@ -52,6 +53,15 @@ llms:
5253
azure_openai_llm:
5354
_type: azure_openai
5455
azure_deployment: gpt-4o-mini
56+
oci_llm:
57+
_type: oci
58+
model_name: nvidia/Llama-3.1-Nemotron-Nano-8B-v1
59+
region: us-chicago-1
60+
compartment_id: ocid1.compartment.oc1..example
61+
auth_type: API_KEY
62+
auth_profile: DEFAULT
63+
auth_file_location: ~/.oci/config
64+
provider: meta
5565
litellm_llm:
5666
_type: litellm
5767
model_name: gpt-4o
@@ -118,6 +128,39 @@ The AWS Bedrock LLM provider is defined by the {py:class}`~nat.llm.aws_bedrock_l
118128
* `credentials_profile_name` - The credentials profile name to use for the model
119129
* `max_retries` - The maximum number of retries for the request
120130

131+
### OCI Generative AI
132+
133+
You can use the following fields to configure the OCI Generative AI LLM provider:
134+
135+
* `region` - OCI region for the Generative AI service (defaults to `us-chicago-1`). The service endpoint is derived automatically.
136+
* `endpoint` - Optional explicit endpoint URL. Overrides the region-derived endpoint when set.
137+
* `compartment_id` - The OCI compartment OCID used for inference requests
138+
* `auth_type` - OCI SDK auth mode such as `API_KEY`, `SECURITY_TOKEN`, `INSTANCE_PRINCIPAL`, or `RESOURCE_PRINCIPAL`
139+
* `auth_profile` - OCI config profile name for file-backed auth
140+
* `auth_file_location` - Path to the OCI config file
141+
* `provider` - Optional provider override such as `meta`, `google`, `cohere`, or `openai`
142+
143+
The OCI Generative AI LLM provider is defined by the {py:class}`~nat.llm.oci_llm.OCIModelConfig` class.
144+
145+
* `model_name` - The name of the model to use
146+
* `region` - OCI region (defaults to `us-chicago-1`). The endpoint is derived from `https://inference.generativeai.{region}.oci.oraclecloud.com`.
147+
* `endpoint` - Optional explicit endpoint URL. Overrides the region-derived endpoint.
148+
* `compartment_id` - OCI compartment OCID
149+
* `auth_type` - OCI SDK auth type
150+
* `auth_profile` - OCI profile name for file-backed auth
151+
* `auth_file_location` - Path to the OCI config file
152+
* `provider` - Optional OCI provider override such as `meta`, `google`, `cohere`, or `openai`
153+
* `temperature` - The temperature to use for the model
154+
* `top_p` - The top-p value to use for the model
155+
* `max_tokens` - The maximum number of tokens to generate
156+
* `seed` - The seed to use for the model
157+
* `max_retries` - The maximum number of retries for the request
158+
* `request_timeout` - HTTP request timeout in seconds
159+
160+
:::{note}
161+
This provider targets OCI Generative AI through the OCI SDK-backed `langchain-oci` path and does not enable the Responses API.
162+
:::
163+
121164
### Azure OpenAI
122165

123166
You can use the following environment variables to configure the Azure OpenAI LLM provider:

docs/source/components/integrations/index.md

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -23,4 +23,5 @@ limitations under the License.
2323
./frameworks.md
2424
./a2a.md
2525
AWS Bedrock <./integrating-aws-bedrock-models.md>
26-
```
26+
OCI Generative AI <./integrating-oci-generative-ai-models.md>
27+
```
Lines changed: 99 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,99 @@
1+
<!--
2+
SPDX-FileCopyrightText: Copyright (c) 2025-2026, NVIDIA CORPORATION & AFFILIATES. All rights reserved.
3+
SPDX-License-Identifier: Apache-2.0
4+
5+
Licensed under the Apache License, Version 2.0 (the "License");
6+
you may not use this file except in compliance with the License.
7+
You may obtain a copy of the License at
8+
9+
http://www.apache.org/licenses/LICENSE-2.0
10+
11+
Unless required by applicable law or agreed to in writing, software
12+
distributed under the License is distributed on an "AS IS" BASIS,
13+
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
14+
See the License for the specific language governing permissions and
15+
limitations under the License.
16+
-->
17+
18+
# NVIDIA NeMo Agent Toolkit OCI Integration
19+
20+
The NeMo Agent Toolkit supports integration with multiple [LLM](../../build-workflows/llms/index.md) providers, including OCI Generative AI. The `oci` provider uses OCI SDK authentication and is designed for OCI Generative AI model and endpoint access. For workflow parity with the AWS Bedrock path, the toolkit also includes a LangChain wrapper built on `langchain-oci`.
21+
22+
To view the full list of supported LLM providers, run `nat info components -t llm_provider`.
23+
24+
## Configuration
25+
26+
### Prerequisites
27+
Before integrating OCI, ensure you have:
28+
29+
- access to OCI Generative AI in the target region
30+
- a valid OCI auth method such as `API_KEY`, `SECURITY_TOKEN`, `INSTANCE_PRINCIPAL`, or `RESOURCE_PRINCIPAL`
31+
- the target compartment OCID
32+
- the target OCI region (defaults to `us-chicago-1`) or a custom endpoint URL
33+
34+
Common deployment patterns include:
35+
36+
- OCI Generative AI regional endpoints
37+
- custom OCI Generative AI endpoints
38+
- OCI-hosted inference for NVIDIA Nemotron used as a live integration target
39+
40+
### Example Configuration
41+
Add the OCI LLM configuration to your workflow config file:
42+
43+
```yaml
44+
llms:
45+
oci_llm:
46+
_type: oci
47+
model_name: nvidia/Llama-3.1-Nemotron-Nano-8B-v1
48+
region: us-chicago-1
49+
compartment_id: ocid1.compartment.oc1..example
50+
auth_type: API_KEY
51+
auth_profile: DEFAULT
52+
temperature: 0.0
53+
max_tokens: 1024
54+
top_p: 1.0
55+
request_timeout: 60
56+
```
57+
58+
### Configurable Options
59+
* `model_name`: The name of the OCI-hosted model to use (required)
60+
* `region`: OCI region for the Generative AI service (defaults to `us-chicago-1`). The service endpoint is derived automatically.
61+
* `endpoint`: Optional explicit service endpoint URL. Overrides the region-derived endpoint when set.
62+
* `compartment_id`: OCI compartment OCID
63+
* `auth_type`: OCI SDK auth type
64+
* `auth_profile`: OCI profile name for file-backed auth
65+
* `auth_file_location`: Path to the OCI config file
66+
* `provider`: Optional OCI provider override such as `meta`, `google`, `cohere`, or `openai`
67+
* `temperature`: Controls randomness in the output (0.0 to 1.0)
68+
* `max_tokens`: Maximum number of tokens to generate
69+
* `top_p`: Top-p sampling parameter (0.0 to 1.0)
70+
* `seed`: Optional random seed
71+
* `max_retries`: Maximum number of retries for the request
72+
* `request_timeout`: HTTP request timeout in seconds
73+
74+
### Limitations
75+
* This provider targets OCI Generative AI through the OCI SDK-backed `langchain-oci` path.
76+
* The Responses API is not enabled for this provider in the current release.
77+
78+
## Nemotron On OCI
79+
80+
One strong OCI deployment pattern is NVIDIA Nemotron hosted on OCI and exposed through an OpenAI-compatible route. In that setup, the toolkit can validate live integration behavior against the OCI-hosted Nemotron endpoint while the official provider and LangChain wrapper cover the OCI Generative AI path.
81+
82+
## Usage
83+
Reference the OCI LLM in your configuration:
84+
85+
```yaml
86+
llms:
87+
oci_llm:
88+
_type: oci
89+
model_name: nvidia/Llama-3.1-Nemotron-Nano-8B-v1
90+
region: us-chicago-1
91+
compartment_id: ocid1.compartment.oc1..example
92+
auth_profile: DEFAULT
93+
```
94+
95+
## Troubleshooting
96+
* `401 Unauthorized`: verify the OCI profile, signer, and IAM permissions for Generative AI.
97+
* `404 Not Found`: confirm the regional endpoint or custom endpoint URL is correct.
98+
* `Connection errors`: verify OCI networking and regional endpoint reachability.
99+
* `Tool calling issues`: verify the served model supports tool calling and that the serving stack is configured for it.

docs/source/conf.py

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -379,6 +379,8 @@ def _build_api_tree() -> Path:
379379
'/extend/custom-components/gated-fields.html',
380380
'extend/integrating-aws-bedrock-models':
381381
'/components/integrations/integrating-aws-bedrock-models.html',
382+
'extend/integrating-oci-generative-ai-models':
383+
'/components/integrations/integrating-oci-generative-ai-models.html',
382384
'extend/memory':
383385
'/extend/custom-components/memory.html',
384386
'extend/object-store':

docs/source/get-started/installation.md

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -27,6 +27,7 @@ The following [LLM](../build-workflows/llms/index.md) API providers are supporte
2727
- OpenAI
2828
- AWS Bedrock
2929
- Azure OpenAI
30+
- OCI Generative AI
3031

3132
## Packages
3233

examples/frameworks/agno_personal_finance/pyproject.toml

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -34,7 +34,7 @@ classifiers = ["Programming Language :: Python"]
3434
[tool.setuptools_dynamic_dependencies]
3535
dependencies = [
3636
"nvidia-nat[agno,test] == {version}",
37-
"openai~=1.106",
37+
"openai>=1.106,<3.0.0",
3838
]
3939

4040
[tool.uv.sources]

examples/frameworks/multi_frameworks/pyproject.toml

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -38,7 +38,7 @@ dependencies = [
3838
"beautifulsoup4~=4.13",
3939
"markdown-it-py~=3.0",
4040
"nvidia-haystack~=0.3.0",
41-
"openai~=1.106",
41+
"openai>=1.106,<3.0.0",
4242
]
4343

4444
[tool.uv.sources]

packages/nvidia_nat_agno/pyproject.toml

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -57,7 +57,7 @@ dependencies = [
5757
"agno>=1.2.3,<2.0.0",
5858
"google-search-results>=2.4.2,<3.0.0",
5959
"litellm>=1.74,<1.82.7", # pin to known-good version
60-
"openai~=1.106",
60+
"openai>=1.106,<3.0.0",
6161
]
6262

6363
[tool.setuptools_dynamic_dependencies.optional-dependencies]

packages/nvidia_nat_langchain/pyproject.toml

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -67,6 +67,7 @@ dependencies = [
6767
"langchain-litellm>=0.3.5,<1.0.0",
6868
"langchain-milvus>=0.3.3,<1.0.0",
6969
"langchain-nvidia-ai-endpoints>=1.0.2,<2.0.0",
70+
"langchain-oci>=0.2.4,<1.0.0",
7071
"langchain-openai>=1.1.6,<2.0.0",
7172
"langchain-tavily>=0.2.16,<1.0.0",
7273
"langgraph>=1.0.5,<2.0.0",

packages/nvidia_nat_langchain/src/nat/plugins/langchain/llm.py

Lines changed: 58 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -37,6 +37,7 @@
3737
from nat.llm.huggingface_llm import HuggingFaceConfig
3838
from nat.llm.litellm_llm import LiteLlmModelConfig
3939
from nat.llm.nim_llm import NIMModelConfig
40+
from nat.llm.oci_llm import OCIModelConfig
4041
from nat.llm.openai_llm import OpenAIModelConfig
4142
from nat.llm.utils.hooks import _create_metadata_injection_client
4243
from nat.llm.utils.thinking import BaseThinkingInjector
@@ -54,6 +55,12 @@
5455
ModelType = TypeVar("ModelType")
5556

5657

58+
def _get_langchain_oci_chat_model():
59+
from langchain_oci import ChatOCIGenAI
60+
61+
return ChatOCIGenAI
62+
63+
5764
def _patch_llm_based_on_config(client: ModelType, llm_config: "LLMBaseConfig") -> ModelType:
5865

5966
from langchain_core.language_models import LanguageModelInput
@@ -222,6 +229,57 @@ async def openai_langchain(llm_config: OpenAIModelConfig, _builder: Builder):
222229
yield _patch_llm_based_on_config(client, llm_config)
223230

224231

232+
@register_llm_client(config_type=OCIModelConfig, wrapper_type=LLMFrameworkEnum.LANGCHAIN)
233+
async def oci_langchain(llm_config: OCIModelConfig, _builder: Builder):
234+
import oci
235+
from langchain_oci.common.auth import create_oci_client_kwargs
236+
237+
validate_no_responses_api(llm_config, LLMFrameworkEnum.LANGCHAIN)
238+
239+
ChatOCIGenAI = _get_langchain_oci_chat_model()
240+
241+
model_kwargs: dict[str, Any] = {}
242+
if llm_config.temperature is not None:
243+
model_kwargs["temperature"] = llm_config.temperature
244+
if llm_config.top_p is not None:
245+
model_kwargs["top_p"] = llm_config.top_p
246+
if llm_config.max_tokens is not None:
247+
if llm_config.provider and llm_config.provider.lower() == "openai":
248+
model_kwargs["max_completion_tokens"] = llm_config.max_tokens
249+
else:
250+
model_kwargs["max_tokens"] = llm_config.max_tokens
251+
if llm_config.seed is not None:
252+
model_kwargs["seed"] = llm_config.seed
253+
254+
client_kwargs = create_oci_client_kwargs(
255+
auth_type=llm_config.auth_type,
256+
service_endpoint=llm_config.endpoint,
257+
auth_file_location=llm_config.auth_file_location,
258+
auth_profile=llm_config.auth_profile,
259+
)
260+
client_kwargs["retry_strategy"] = oci.retry.RetryStrategyBuilder(
261+
max_attempts=llm_config.max_retries + 1 # OCI SDK counts total attempts (initial + retries)
262+
).get_retry_strategy()
263+
if llm_config.request_timeout is not None:
264+
client_kwargs["timeout"] = (10, llm_config.request_timeout)
265+
oci_client = oci.generative_ai_inference.GenerativeAiInferenceClient(**client_kwargs)
266+
267+
client = ChatOCIGenAI(
268+
client=oci_client,
269+
model_id=llm_config.model_name,
270+
service_endpoint=llm_config.endpoint,
271+
compartment_id=llm_config.compartment_id,
272+
auth_type=llm_config.auth_type,
273+
auth_profile=llm_config.auth_profile,
274+
auth_file_location=llm_config.auth_file_location,
275+
provider=llm_config.provider,
276+
is_stream=getattr(llm_config, "stream", False),
277+
model_kwargs=model_kwargs or None,
278+
)
279+
280+
yield _patch_llm_based_on_config(client, llm_config)
281+
282+
225283
@register_llm_client(config_type=DynamoModelConfig, wrapper_type=LLMFrameworkEnum.LANGCHAIN)
226284
async def dynamo_langchain(llm_config: DynamoModelConfig, _builder: Builder):
227285
"""

0 commit comments

Comments
 (0)