Support meta.yaml, pyproject.toml and versions.json

[vyasr]

In addition to conda requirements files, the rapid dependency generator needs to support a few additional use cases. In order of priority, these are:

• pyproject.toml: We need to be able to encode build-time dependencies into pyproject.toml files for Python packages. This will become much more important when we start officially supporting pip installation. At present most requirements information encoded in setup.py or pyproject.toml is essentially ignored by all our conda-only build processes, but that is going to change soon.
• meta.yaml: We need to be able to write out the appropriate sections of meta.yaml. Currently the generator will produce the env files that are helpful for developers or for use in different CI stages, but the conda package definition files are still manually maintained and could go out of sync.
• versions.json: We need to be able to write out versions to some sort of dependency file that can be handled in CMake. This necessity arises because there are some dependencies (treelite for cuML is the one that comes to mind) that are needed both at C++ compile time -- which means they are brought in via CPM.cmake -- and at Python run/test time -- which means they must be handled by setuptools. I propose a versions.json file because that is already essentially supported out of the box by rapids-cmake. I'll note that at this time treelite is the only obvious example of this that I see, so supporting this is not crucial. However, I think it would be minimal extra work on top of what we would already have to do to enable pyproject.toml, so I think it would be worth it. If we go this route, we should follow the rapids-cmake spec for versions.json.

Enabling the above will require making the following changes (in roughly this order based on which tasks depend on others as well as their relative priorities):

1. Generalizing the dependencies.yml file format. Currently, the file format hardcodes the conda_and_requirements key as the way to specify requirements that are common to both types of dependency files. Instead of this, we will need to modify the file to have something like a file_types key that accepts an array of values and writes those requirements to all the associated files. That would allow one set of entries to be shared across an arbitrary subset of different files.
2. Enabling read-modify-write behavior rather than a simple overwrite behavior in the generator. Currently the generator simply reads dependencies.yml, generates complete dependency lists, and then writes out those lists to the appropriate files, overwriting any preexisting files. While that is appropriate for requirements.txt and environment.yml files, that is not appropriate for pyproject.toml files or meta.yaml files. The latter types of files could contain additional information beyond a simple list of dependencies and that information must persist.
3. Implement writing of TOML files. The TOML package is the easiest choice here.
4. Write out the meta.yaml dependencies sections into new sub yml files, then use conda-build's jinja functions to read those files in for each section.
5. Implement writing of JSON files. We can just use the built-in json module.
6. Update rapids-cmake to support reading arbitrary JSON files. That should just require extracting some of the logic from rapids_cpm_load_preset_versions into a separate function.

Some additional notes and caveats:

• The version styles used by CPM.cmake do not support the sorts of complex version string used by setuptools/conda. Instead these version strings are hardcoded to a specific number. However, I do not think this is an issue because in cases where we hardcode a dependency for CMake we should force Python to use the same exact version (no fuzzy deps), so those dependencies can always be written as $PKG=$VERSION rather than $PKG>x,<y etc.
• CPM downloads require a git tag/branch. I'm not yet sure how we would want to support this. It may require adding some keys to the appropriate section. The best outcome here is likely that we will maintain separate sections for versions.json and Python requirements for a given dependency, and the versions.json section will support the additional keys like git tag/branch, but then we'll use a YAML anchor inside dependencies.yml containing the version so that there remains only a single source of truth.

[ajschmidt8]

The PR below was opened to adjust the dependencies.yaml schema in order to support future output types.

• Update dependencies.yaml Schema #11

[vyasr]

We now support writing out pyproject.toml files. I will open a separate thread to discuss issues with supporting meta.yaml files.

[vyasr]

We have started rolling out usage of rattler-build instead of conda-build for building our conda packages in rapidsai/build-planning#47. rattler-build follows the new format laid out in CEP 13 and CEP 14. The major relevant difference between this format and the legacy format is that the new recipe format is strict YAML, so we can parse the file using standard YAML parsers. The new format is also somewhat simpler than the old format, so wroting a standard tree-based YAML parser in dfg would be quite a bit easier to do.

We made a couple of attempts to support meta.yaml files in #28 and #45, but ran into a number of issues.

• feat: Support updating conda recipe meta.yaml file #28: This approach involved writing out the whole meta.yaml file. This was extremely challenging because meta.yaml files is really a superset of YAML due to the Jinja templating that it supports, so we couldn't use standard YAML parsers at all. This approach was quickly abandoned at the time.
• feat: Generate meta.yaml dependencies #45: In this PR we took a simpler approach to generate dependencies for each dependency group specified in meta.yaml and then reading that into the recipe. This approach worked relatively better, but still ran into some issues with non-YAML syntax such as the Jinja functions that conda-build supports for specifying dependencies.

Now that we are switching to the new recipe format, this problem should be substantially simplified. We may still think that the approach of #45 (writing out separate dependency lists for each group) is preferable; it is definitely simpler, but has the downside that adding new dependency sections (for example) requires modifying both dependencies.yaml and the recipe.yaml file, whereas if overwriting the file directly dfg could do this for us. Whichever approach we take, though, we should be in a position to experiment again here once the rattler-build transition is complete.

[bdice]

Surfacing this again. Should we look into generating conda pinnings with the new recipe.yaml format? We recently made a mistake in rapids-dask-dependency with different wheel/conda pinnings. I added a quick check in rapidsai/rapids-dask-dependency#148, but it'd be nice to rule out the problem more broadly by generating our rattler recipe dependencies. Could be a good 26.06 goal for the build team.

cc: @gforsyth @jameslamb for vis

[vyasr]

Maybe good to bring it up in the next sync to see if it can be prioritized. I'd love to see it done, I just don't know how it slots in with other items on the priority list.