feat: Add support for loading context from TOML files including pyproject.toml

docs/how-tos/cli-reference.md

@@ -9,6 +9,8 @@ The dependencies for the Apache Hamilton CLI can be installed via
 pip install sf-hamilton[cli]
 ```
 
+The CLI includes support for TOML files via the `tomli` library. When using TOML configuration files, the extra dependencies will be automatically available.
+
 You can verify the installation with
 
 ```console
@@ -28,8 +30,9 @@ hamilton --help
 **Commands**:
 
 * `build`: Build a single Driver with MODULES
-* `diff`: Diff between the current MODULES and their...
-* `version`: Version NODES and DATAFLOW from dataflow...
+* `diff`: Diff between the current MODULES and their specified GIT_REFERENCE
+* `validate`: Validate DATAFLOW execution for the given CONTEXT
+* `version`: Version NODES and DATAFLOW from dataflow with MODULES
 * `view`: Build and visualize dataflow with MODULES
 
 ## `hamilton build`
@@ -48,6 +51,10 @@ $ hamilton build [OPTIONS] MODULES...
 
 **Options**:
 
+* `--name TEXT`: Name of the dataflow. Default: Derived from MODULES.
+* `--context FILE`: Path to Driver context file [.json, .py, .toml]. For TOML files, Hamilton looks for either:
+    - Top-level Hamilton headers: `HAMILTON_CONFIG`, `HAMILTON_FINAL_VARS`, `HAMILTON_INPUTS`, `HAMILTON_OVERRIDES`
+    - Tool-specific section: `[tool.hamilton]` with `config`, `final_vars`, `inputs`, `overrides` sub-keys
 * `--help`: Show this message and exit.
 
 ## `hamilton diff`
@@ -66,11 +73,78 @@ $ hamilton diff [OPTIONS] MODULES...
 
 **Options**:
 
+* `--name TEXT`: Name of the dataflow. Default: Derived from MODULES.
+* `--context FILE`: Path to Driver context file [.json, .py, .toml]. For TOML files, Hamilton looks for either:
+    - Top-level Hamilton headers: `HAMILTON_CONFIG`, `HAMILTON_FINAL_VARS`, `HAMILTON_INPUTS`, `HAMILTON_OVERRIDES`
+    - Tool-specific section: `[tool.hamilton]` with `config`, `final_vars`, `inputs`, `overrides` sub-keys
+* `--output-file-path PATH`: Output path of visualization. If path is a directory, use NAME for file name. [default: .]
 * `--git-reference TEXT`: [default: HEAD]
 * `--view / --no-view`: [default: no-view]
-* `--output-file-path PATH`: [default: diff.png]
 * `--help`: Show this message and exit.
 
+## `hamilton validate`
+
+Validate DATAFLOW execution for the given CONTEXT
+
+**Usage**:
+
+```console
+$ hamilton validate [OPTIONS] MODULES...
+```
+
+**Arguments**:
+
+* `MODULES...`: [required]
+
+**Options**:
+
+* `--context FILE`: [required] Path to Driver context file [.json, .py, .toml]. For TOML files, Hamilton looks for either:
+    - Top-level Hamilton headers: `HAMILTON_CONFIG`, `HAMILTON_FINAL_VARS`, `HAMILTON_INPUTS`, `HAMILTON_OVERRIDES`
+    - Tool-specific section: `[tool.hamilton]` with `config`, `final_vars`, `inputs`, `overrides` sub-keys
+* `--name TEXT`: Name of the dataflow. Default: Derived from MODULES.
+* `--help`: Show this message and exit.
+
+## Using TOML Files for Configuration
+
+Starting with version 1.90.0, the Hamilton CLI supports loading configuration from TOML files, including `pyproject.toml`. You can use either of these two formats:
+
+### Format 1: Top-level Hamilton headers
+
+In your TOML file, define the Hamilton configuration headers at the top level:
+
+```toml
+# example_context.toml
+HAMILTON_CONFIG = {param1 = "value1", param2 = 42}
+HAMILTON_FINAL_VARS = ["final_result", "output_value"]
+HAMILTON_INPUTS = {input_value = 100, string_input = "example"}
+HAMILTON_OVERRIDES = {override_param = "override_value"}
+```
+
+### Format 2: Tool-specific section (recommended for pyproject.toml)
+
+For projects using `pyproject.toml`, it's recommended to place Hamilton configuration in the `[tool.hamilton]` section:
+
+```toml
+# pyproject.toml
+[tool.hamilton]
+config = {param1 = "value1", param2 = 42}
+final_vars = ["final_result", "output_value"]
+inputs = {input_value = 100, string_input = "example"}
+overrides = {override_param = "override_value"}
+```
+
+### Usage
+
+You can use TOML configuration files with all Hamilton CLI commands that support the `--context` option:
+
+```console
+hamilton build --context config.toml my_module.py
+hamilton validate --context config.toml my_module.py
+hamilton view --context config.toml my_module.py
+hamilton diff --context config.toml my_module.py
+hamilton version --context config.toml my_module.py
+```
+
 ## `hamilton version`
 
 Version NODES and DATAFLOW from dataflow with MODULES
@@ -87,6 +161,10 @@ $ hamilton version [OPTIONS] MODULES...
 
 **Options**:
 
+* `--name TEXT`: Name of the dataflow. Default: Derived from MODULES.
+* `--context FILE`: Path to Driver context file [.json, .py, .toml]. For TOML files, Hamilton looks for either:
+    - Top-level Hamilton headers: `HAMILTON_CONFIG`, `HAMILTON_FINAL_VARS`, `HAMILTON_INPUTS`, `HAMILTON_OVERRIDES`
+    - Tool-specific section: `[tool.hamilton]` with `config`, `final_vars`, `inputs`, `overrides` sub-keys
 * `--help`: Show this message and exit.
 
 ## `hamilton view`
@@ -105,5 +183,9 @@ $ hamilton view [OPTIONS] MODULES...
 
 **Options**:
 
-* `--output-file-path PATH`: [default: ./dag.png]
+* `--name TEXT`: Name of the dataflow. Default: Derived from MODULES.
+* `--context FILE`: Path to Driver context file [.json, .py, .toml]. For TOML files, Hamilton looks for either:
+    - Top-level Hamilton headers: `HAMILTON_CONFIG`, `HAMILTON_FINAL_VARS`, `HAMILTON_INPUTS`, `HAMILTON_OVERRIDES`
+    - Tool-specific section: `[tool.hamilton]` with `config`, `final_vars`, `inputs`, `overrides` sub-keys
+* `--output-file-path PATH`: Output path of visualization. If path is a directory, use NAME for file name. [default: .]
 * `--help`: Show this message and exit.

docs/how-tos/pre-commit-hooks.md

@@ -106,6 +106,13 @@ In `v1`, the dataflow could be validated for `C` without any inputs. Now, a deve
 // will call .validate_execution(final_vars["C"])
 ```
 
+Or, using a TOML file:
+
+```toml
+# context.toml
+HAMILTON_FINAL_VARS = ["C"]  # will call .validate_execution(final_vars["C"])
+```
+
 ```{note}
 pre-commit hooks can prevent commits from breaking a core path, but you should use unit and integration tests for more robust checks.
 ```
@@ -125,6 +132,7 @@ To use them, add this snippet to your `.pre-commit-config.yaml` and adapt it to
       hamilton build my_module.py,
       hamilton build my_module2.py,
       hamilton validate --context context.json my_module.py my_module2.py,
+      hamilton validate --context context.toml my_module.py my_module2.py,  # example with TOML file
     ]
 ```
 

hamilton/cli/__main__.py

@@ -81,7 +81,7 @@ class CliState:
     typer.Option(
         "--context",
         "-ctx",
-        help="Path to Driver context file [.json, .py]",
+        help="Path to Driver context file [.json, .py, .toml]",
         exists=True,
         dir_okay=False,
         readable=True,

hamilton/cli/logic.py

@@ -269,7 +269,6 @@ def visualize_diff(
 
 
 # TODO refactor ContextLoader to a class
-# TODO support loading from pyproject.toml
 def load_context(file_path: Path) -> dict:
     if not file_path.exists():
         raise FileNotFoundError(f"`{file_path}` doesn't exist.")
@@ -279,6 +278,8 @@ def load_context(file_path: Path) -> dict:
         context = _read_json_context(file_path)
     elif extension == ".py":
         context = _read_py_context(file_path)
+    elif extension in [".toml", ".tml"]:
+        context = _read_toml_context(file_path)
     else:
         raise ValueError(f"Received extension `{extension}` is unsupported.")
 
@@ -337,3 +338,43 @@ def _read_py_context(file_path: Path) -> dict:
         context[k] = getattr(module, k, None)
 
     return context
+
+
+def _read_toml_context(file_path: Path) -> dict:
+    """Read context from a TOML file. For pyproject.toml, looks for Hamilton configuration in [tool.hamilton] section."""
+    try:
+        import tomli  # Using tomli for compatibility with older Python versions
+    except ImportError:
+        # Provide a helpful error message if tomli is not available
+        raise ImportError(
+            "tomli is required to read TOML files. "
+            "Install it with `pip install tomli` or `pip install sf-hamilton[cli]` which includes TOML support."
+        )
+
+    with open(file_path, 'rb') as f:
+        data = tomli.load(f)
+
+    # First check if there's a [tool.hamilton] section in pyproject.toml
+    # This is where Hamilton-specific configuration would typically go
+    hamilton_config = data.get('tool', {}).get('hamilton', {})
+
+    # If we find Hamilton-specific config, use it as the context
+    if hamilton_config:
+        context = {
+            CONFIG_HEADER: hamilton_config.get('config', {}),
+            FINAL_VARS_HEADER: hamilton_config.get('final_vars', []),
+            INPUTS_HEADER: hamilton_config.get('inputs', {}),
+            OVERRIDES_HEADER: hamilton_config.get('overrides', {}),
+        }
+    else:
+        # Otherwise, check for top-level Hamilton context headers
+        context = {}
+        for k in [
+            CONFIG_HEADER,
+            FINAL_VARS_HEADER,
+            INPUTS_HEADER,
+            OVERRIDES_HEADER,
+        ]:
+            context[k] = data.get(k, None)
+
+    return context

pyproject.toml

@@ -53,7 +53,10 @@ dependencies = [
 ]
 
 [project.optional-dependencies]
-cli = ["typer"]
+cli = [
+    "typer",
+    "tomli",
+]
 dask = ["dask[complete]"]  # commonly you'll want everything.
 dask-array = ["dask[array]"]
 dask-dataframe = ["dask[dataframe]"]

tests/cli/resources/test_context.toml

@@ -0,0 +1,7 @@
+# Test TOML file for Hamilton CLI context loading
+
+# Define Hamilton headers as top-level values
+HAMILTON_CONFIG = {test_param = "test_value"}
+HAMILTON_FINAL_VARS = ["final_var1", "final_var2"]
+HAMILTON_INPUTS = {input_value = 42}
+HAMILTON_OVERRIDES = {override_value = "override"}

tests/cli/resources/test_tool_hamilton.toml

@@ -0,0 +1,8 @@
+# Test TOML file for Hamilton CLI context loading using [tool.hamilton] section
+
+[tool.hamilton]
+# Hamilton-specific configuration
+config = {test_param = "test_value"}
+final_vars = ["final_var1", "final_var2"]
+inputs = {input_value = 42, string_input = "test_string"}
+overrides = {override_value = "override"}

tests/cli/test_logic.py

@@ -93,3 +93,42 @@ def test_diff_node_versions():
     assert diff["reference_only"] == ["orders_per_customer"]
     assert diff["current_only"] == ["orders_per_distributor"]
     assert diff["edit"] == ["average_order_by_customer", "customer_summary_table"]
+
+
+def test_load_context_from_toml(monkeypatch):
+    """Test loading context from a TOML file with top-level Hamilton headers."""
+    monkeypatch.setenv("HAMILTON_CONFIG", "HAMILTON_CONFIG")
+    monkeypatch.setenv("HAMILTON_FINAL_VARS", "HAMILTON_FINAL_VARS")
+    monkeypatch.setenv("HAMILTON_INPUTS", "HAMILTON_INPUTS")
+    monkeypatch.setenv("HAMILTON_OVERRIDES", "HAMILTON_OVERRIDES")
+
+    toml_path = Path(__file__).parent / "resources" / "test_context.toml"
+
+    # Load context from TOML file
+    context = logic.load_context(toml_path)
+
+    # Check that the expected values are loaded
+    assert context["HAMILTON_CONFIG"] == {"test_param": "test_value"}
+    assert context["HAMILTON_INPUTS"] == {"input_value": 42}
+    assert context["HAMILTON_OVERRIDES"] == {"override_value": "override"}
+    # The TOML file has an array of final variables
+    assert context["HAMILTON_FINAL_VARS"] == ["final_var1", "final_var2"]
+
+
+def test_load_context_from_toml_tool_hamilton(monkeypatch):
+    """Test loading context from a TOML file with [tool.hamilton] section."""
+    monkeypatch.setenv("HAMILTON_CONFIG", "HAMILTON_CONFIG")
+    monkeypatch.setenv("HAMILTON_FINAL_VARS", "HAMILTON_FINAL_VARS")
+    monkeypatch.setenv("HAMILTON_INPUTS", "HAMILTON_INPUTS")
+    monkeypatch.setenv("HAMILTON_OVERRIDES", "HAMILTON_OVERRIDES")
+
+    toml_path = Path(__file__).parent / "resources" / "test_tool_hamilton.toml"
+
+    # Load context from TOML file with tool.hamilton section
+    context = logic.load_context(toml_path)
+
+    # Check that the expected values from [tool.hamilton] section are loaded
+    assert context["HAMILTON_CONFIG"] == {"test_param": "test_value"}
+    assert context["HAMILTON_INPUTS"] == {"input_value": 42, "string_input": "test_string"}
+    assert context["HAMILTON_OVERRIDES"] == {"override_value": "override"}
+    assert context["HAMILTON_FINAL_VARS"] == ["final_var1", "final_var2"]