Remove Codex-as-a-tool (#90)

* Remove Codex-as-a-tool

* increase code coverage

* run hatch fmt

* update readme to use Validator API

* remove extra line

Author: LukeMainwaring

DEVELOPMENT.md

@@ -16,7 +16,7 @@ You can run the tests on your local machine with:
 hatch test
 ```
 
-The [`test` command][hatch-test] supports options such as `-c` for measuring test coverage, `-a` for testing with a matrix of Python versions, and appending an argument like `tests/test_codex_tool.py::test_to_llamaindex_tool` for running a single test.
+The [`test` command][hatch-test] supports options such as `-c` for measuring test coverage, `-a` for testing with a matrix of Python versions, and appending an argument like `tests/test_validator.py::test_validate_expert_answer` for running a single test.
 
 [hatch-test]: https://hatch.pypa.io/latest/tutorials/testing/overview/
 
@@ -79,8 +79,8 @@ Automated releases are handled by the [release workflow][release-workflow] which
 
 1. Bump the version in `src/cleanlab_codex/__about__.py`. You can use the [`hatch version`][hatch-version] command to do this.
 2. Ensure that the release notes are updated in [`CHANGELOG.md`][changelog]:
-    - You should update the `[Unreleased]` header to the new version and add a new `[Unreleased]` section at the top of the file.
-    - You should update the link for the `[Unreleased]` code and add a new link to the code diff for the new version.
+   - You should update the `[Unreleased]` header to the new version and add a new `[Unreleased]` section at the top of the file.
+   - You should update the link for the `[Unreleased]` code and add a new link to the code diff for the new version.
 3. Create a PR and merge these changes into the `main` branch.
 4. After the PR is merged into `main`, create a new release tag by running `git tag v<output of hatch version>` (i.e. `git tag v0.0.1`).
 5. Push the tag to the repository by running `git push origin <tag>`.
@@ -90,7 +90,6 @@ Automated releases are handled by the [release workflow][release-workflow] which
 [hatch-version]: https://hatch.pypa.io/latest/version/#updating
 [changelog]: CHANGELOG.md
 
-
 ### How to build and install the package locally
 
 You may want to build and install the package locally - for example, to see how your changes affect other Python code in a script or Jupyter notebook.
@@ -110,14 +109,3 @@ Testing, type checking, and formatting/linting is [checked in CI][ci].
 To target a different Codex backend environment (i.e. staging or local), set the `CODEX_BASE_URL` environment variable. Example: `export CODEX_BASE_URL=http://localhost:8080`.
 
 ## Style guide
-
-### Adding integrations with external libraries
-
-When adding integrations with external libraries, always use a lazy import. The external dependency should not be required to use the `cleanlab-codex` library. Wrap the lazy import in a `try`/`except` block to catch the `ImportError` and raise a `MissingDependencyError` with a helpful message. See [codex_tool.py](src/cleanlab_codex/codex_tool.py) file for examples one of which is shown below:
-
-```python
-try:
-    from cleanlab_codex.utils.smolagents import CodexTool as SmolagentsCodexTool
-except ImportError as e:
-    raise MissingDependencyError("smolagents", "https://github.com/huggingface/smolagents") from e
-```

README.md

@@ -4,7 +4,7 @@
 
 Codex enables you to seamlessly leverage knowledge from Subject Matter Experts (SMEs) to improve your RAG/Agentic applications.
 
-The `cleanlab-codex` library provides a simple interface to integrate Codex's capabilities into your RAG application. 
+The `cleanlab-codex` library provides a simple interface to integrate Codex's capabilities into your RAG application.
 See immediate impact with just a few lines of code!
 
 ## Demo
@@ -15,31 +15,30 @@ Install the package:
 pip install cleanlab-codex
 ```
 
-Integrating Codex into your RAG application as a tool is as simple as:
+Integrating Codex into your RAG application is as simple as:
 
 ```python
-from cleanlab_codex import CodexTool
-
-def rag(question, system_prompt, tools) -> str:
-    """Your RAG/Agentic code here"""
-    ...
-
-# Initialize the Codex tool
-codex_tool = CodexTool.from_access_key("your-access-key")
-
-# Update your system prompt to include information on how to use the Codex tool
-system_prompt = f"""Answer the user's Question based on the following Context. If the Context doesn't adequately address the Question, use the {codex_tool.tool_name} tool to ask an outside expert."""
-
-# Convert the Codex tool to a framework-specific tool
-framework_specific_codex_tool = codex_tool.to_<framework_name>_tool() # i.e. codex_tool.to_llamaindex_tool(), codex_tool.to_openai_tool(), etc.
-
-# Pass the Codex tool to your RAG/Agentic framework
-response = rag(question, system_prompt, [framework_specific_codex_tool])
+from cleanlab_codex import Validator
+validator = Validator(codex_access_key=...) # optional configurations can improve accuracy/latency
+
+# Your existing RAG code:
+context = rag_retrieve_context(user_query)
+prompt = rag_form_prompt(user_query, retrieved_context)
+response = rag_generate_response(prompt)
+
+# Detect bad responses and remediate with Cleanlab
+results = validator.validate(query=query, context=context, response=response,
+    form_prompt=rag_form_prompt)
+
+final_response = (
+    results["expert_answer"] # Codex's answer
+    if results["is_bad_response"] and results["expert_answer"]
+    else response # Your RAG system's initial response
+)
 ```
 
-(Note: Exact code will depend on the RAG/Agentic framework you are using. [Other integrations](https://help.cleanlab.ai/codex/concepts/integrations/) are available if you prefer to avoid Tool Calls.)
 <!-- TODO: add demo video -->
-<!-- Video should show Codex tool added to a RAG system, question asked that requires knowledge from an outside expert, Codex tool used to ask an outside expert, and expert response returned to the user -->
+<!-- Video should show Codex added to a RAG system, question asked that requires knowledge from an outside expert, Codex used to ask an outside expert, and expert response returned to the user -->
 
 ## Why Codex?
 - **Detect Knowledge Gaps and Hallucinations**: Codex identifies knowledge gaps and incorrect/untrustworthy responses in your AI application, to help you know which questions require expert input.

pyproject.toml

@@ -96,6 +96,7 @@ exclude_lines = [
   "if TYPE_CHECKING:",
   "if _TYPE_CHECKING:",
 ]
+show_missing = true
 
 [tool.hatch.envs.coverage]
 detached = true

src/cleanlab_codex/__init__.py

@@ -1,7 +1,6 @@
 # SPDX-License-Identifier: MIT
 from cleanlab_codex.client import Client
-from cleanlab_codex.codex_tool import CodexTool
 from cleanlab_codex.project import Project
 from cleanlab_codex.validator import Validator
 
-__all__ = ["Client", "CodexTool", "Project", "Validator"]
+__all__ = ["Client", "Project", "Validator"]

src/cleanlab_codex/codex_tool.py

@@ -9,7 +9,6 @@ class IntegrationType(str, Enum):
     """Supported methods for integrating Codex into a RAG system using this library."""
 
     BACKUP = "backup"
-    TOOL = "tool"
 
 
 class _AnalyticsMetadata:

src/cleanlab_codex/internal/analytics.py

@@ -37,7 +37,7 @@ def __str__(self) -> str:
 class Project:
     """Represents a Codex project.
 
-    To integrate a Codex project into your RAG/Agentic system, we recommend using one of our abstractions such as [`CodexTool`](/codex/api/python/codex_tool).
+    To integrate a Codex project into your RAG/Agentic system, we recommend using one of our abstractions such as [`Validator`](/codex/api/python/validator).
     """
 
     def __init__(self, sdk_client: _Codex, project_id: str, *, verify_existence: bool = True):

src/cleanlab_codex/project.py

@@ -1,17 +1,5 @@
-from cleanlab_codex.utils.aws import Tool as AWSConverseTool
-from cleanlab_codex.utils.aws import ToolSpec as AWSToolSpec
-from cleanlab_codex.utils.aws import format_as_aws_converse_tool
 from cleanlab_codex.utils.function import FunctionParameters
-from cleanlab_codex.utils.openai import Function as OpenAIFunction
-from cleanlab_codex.utils.openai import Tool as OpenAITool
-from cleanlab_codex.utils.openai import format_as_openai_tool
 
 __all__ = [
     "FunctionParameters",
-    "OpenAIFunction",
-    "OpenAITool",
-    "AWSConverseTool",
-    "AWSToolSpec",
-    "format_as_openai_tool",
-    "format_as_aws_converse_tool",
 ]