grass.tools: Add API and CLI to access tools without a session #5843
+1,788
−2
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This adds a Tools class which allows to access GRASS tools (modules) to be accessed using methods. Once an instance is created, calling a tool is calling a function (method) similarly to grass.jupyter.Map. Unlike grass.script, this does not require generic function name and unlike grass.pygrass module shortcuts, this does not require special objects to mimic the module families. Outputs are handled through a returned object which is result of automatic capture of outputs and can do conversions from known formats using properties. Usage example is in the _test() function in the file. The code is included under new grass.experimental package which allows merging the code even when further breaking changes are anticipated.
…ute with that stdin
…are different now)
… the max tries, focusing on timeout
…ting clearly ends the loop, the elapsed time may be significantly higher for some timeouts given that the lock process execution takes a second to execute.
Use of NumPy array IO with the standalone tools APIThe combination of NumPy array IO (from #5878) with the standalone tools API (from #5843 - this PR) allows to use tools with NumPy arrays without a project: from grass.experimental.standalone_tools import StandaloneTools
tools = StandaloneTools()
slope = tools.r_slope_aspect(elevation=np.ones((2, 3)), slope=np.ndarray)Complications with computational regionWith how the StandaloneTools are implemented now, the following will fail because the initially set region will be incompatible with the array size in the second call (see option 1 on the region comment above): from grass.experimental.standalone_tools import StandaloneTools
tools = StandaloneTools()
slope1 = tools.r_slope_aspect(elevation=np.ones((2, 3)), slope=np.ndarray)
slope2 = tools.r_slope_aspect(elevation=np.ones((5, 5)), slope=np.ndarray)One way how to avoid it is providing some parameter to StandaloneTools, like from grass.experimental.standalone_tools import StandaloneTools
slope1 = StandaloneTools().r_slope_aspect(elevation=np.ones((2, 3)), slope=np.ndarray)
slope2 = StandaloneTools().r_slope_aspect(elevation=np.ones((5, 5)), slope=np.ndarray)Having the multiple calls and having that instance immediately forgotten does not look that great. Evaluating length of user codeOne could also argue that, in case of NumPy arrays, really plain functions are preferable over calling a tool as a method of an object because even a single call still requires creation of an object beforehand or in the same statement as in these two examples: from grass.experimental.standalone_tools import StandaloneTools
tools = StandaloneTools()
slope = tools.r_slope_aspect(elevation=np.ones((2, 3)), slope=np.ndarray)from grass.experimental.standalone_tools import StandaloneTools
slope = StandaloneTools().r_slope_aspect(elevation=np.ones((5, 5)), slope=np.ndarray)Shortcut object in the libraryWe could create a StandaloneTools object on the Python module level, so that users can import it. This would be similar to grass.pygrass.modules.shortcuts (hence calling it shortcut here). In the library, we would have: # grass/experimental/standalone_tools.py
tools = StandaloneTools(refresh_region=True, keep_data=False, use_one_project=False)And then the user code would be: # myscript.py
from grass.experimental.standalone_tools import tools
slope = tools.r_slope_aspect(elevation=np.ones((5, 5)), slope=np.ndarray)This would exist alongside the option to create one or more StandaloneTools objects, and it would likely have different configuration (independent region, no data preserved, for truly standalone runs). Result would be possible confusion due to another option and some inconsistency, but it might be the best way how to provide such API because it creates simplest user code. |
wenzeslaus
changed the title
…lean up tests and doc-strings.
wenzeslaus
added a commit
that referenced
this pull request
Oct 7, 2025
This is adding r.pack files (aka native GRASS raster files) as input and output to tools when called through the Tools object. Tool calls such as r_grow can take r.pack files as input or output. The format is distinguished by the file extension. Notably, tool calls such as r_mapcalc don't pass input or output data as separate parameters (expressions or base names), so they can be used like that only when a wrapper exists (r_mapcalc_simple) or, in the future, when more information is included in the interface or passed between the tool and the Tools class Python code. Similarly, tools with multiple inputs or outputs in a single parameter are currently not supported. The code is using --json with the tool to get the information on what is input and what is output, because all are files which may or may not exists (this is different from NumPy arrays where the user-provided parameters clearly say what is input (object) and what is output (class)). Consequently, the whole import-export machinery is only started when there are files in the parameters as identified by the parameter converter class. Currently, the in-project raster names are driven by the file names. This will break for parallel usage and will not work for vector as is. While it is good for guessing the right (and nice) name, e.g., for r.mapcalc expression, ultimately, unique names retrieved with an API function are likely the way to go. When cashing is enabled (either through use go context manager or explicitly), import of inputs is skipped when they were already imported or when they are known outputs. Without cache, data is deleted after every tool (function) call. Cashing is keeping the in-project data in the project (as opposed to a hidden cache or deleting them). The parameter to explicitly drive this is called use_cache (originally keep_data). The objects track what is imported and also track import and cleaning tasks at function call versus object level. The data is cleaned even in case of exceptions. The interface was clarified by creating a private/protected version of run_cmd which has the internal-only parameters. This function uses a single try-finally block to trigger the cleaning in case of exceptions. While generally the code supports paths as both strings and Path objects, the actual decisions about import are made from the list of strings form of the command. From caller perspective, overwrite is supported in the same way as for in-project GRASS rasters. The tests use module scope to reduce fixture setup by couple seconds. Changes include a minor cleanup of comments in tests related to testing result without format=json and with, e.g., --json option. The class documentation discusses overhead and parallelization because the calls are more costly and there is a significant state of the object now with the cache and the rasters created in the background. This includes discussion of the NumPy arrays, too, and slightly improves the wording in part discussing arrays. This is building on top of #2923 (Tools API, and it is parallel with #5878 (NumPy array IO), although it runs at a different stage than NumPy array conversions and uses cache for the imported data (may be connected more with the arrays in the future). This can be used efficiently in Python with Tools (caching, assuming project) and in a limited way also with the experimental run subcommand in CLI (no caching, still needs an explicit project). There is more potential use of this with the standalone tools concept (#5843). The big picture is also discussed in #5830.
echoix
added
the
conflicts/needs rebase
|
Dependent #2923 was merged |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Building on top of #2923 (not merged), this adds functionality which allows accessing "packed" native GRASS rasters to be used as tool parameters in command line:
The above syntax is not actually implemented, but the code below works:
PYTHONPATH=$(grass --config python-path) python -m grass.app run r.slope.aspect elevation=.../elevation.pack slope=.../slope.packThe same functionality is also available from Python where it copies the syntax of plain Tools from #2923:
The above syntax does not fully work, but the following one does:
This PR is not meant for merging as is, but currently represents a final combination of all different features proposed. See discussion #5830 for details.